Because of the ability of the human ear to detect latencies of a few milliseconds and jitter on an even shorter time scale, music synthesis provides one of the most demanding testbeds for real-time systems. Harmonicon is a MIDI synthesizer we have written entirely in Java running on top of our Metronome-based JVM.
Harmonicon implements synthesis of most of the Soundfont2 specification, and can produce high-quality piano sound. Harmonicon is able to provide response within 1 ms of that achievable with a dedicated hardware synthesizer. We are in the process of performing measurements of absolute latency, jitter, and fidelity.
Continuing experiments involve the use of our Expedited Real-time Threads (XRTs) to perform CD-quality waveform generation at 44.1 KHz. Currently we are able to achieve 22.05 KHz real-time waveform synthesis.
Presentations
Hear the Garbage Collector: a Software Synthesizer in Java, presentation by Florian Bömers at the University of Salzburg, June 2006.
Publications
Joshua Auerbach, David F. Bacon, Florian Bömers, and Perry Cheng,
Real-time Music Synthesis in Java using the Metronome Garbage Collector.
Submitted for publication.
PDF
An experimental analysis of the real-time capabilities of Metronome technology with a custom-built all-Java MIDI music synthesizer. We achieved 5.7ms worst-case keyboard-to-speaker latencies, comparable to a Kurzweil K2000R synthesizer The system is studied extensively and a comparison is made with the performance on other types of garbage collectors.
Daniel Spoonhower, Joshua Auerbach, David F. Bacon, Perry Cheng, and David Grove,
Eventrons: A Safe Programming Construct for High-Frequency Hard Real-Time Applications.
Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (Ottawa, Ontario, June 2006), ACM SIGPLAN Notices, volume 41, number 6, pp. 283-294.
PDF
This paper is primarily about the XRT (aka Eventron) technology but includes a use of the audio waveform output as a benchmark.
DEMONSTRATION
This is a demonstration of the real-time capabilties of IBM's WebSphere RealTime (WRT) Java virtual machine product, with its unique real-time garbage collection technology, code-named Metronome, developed at IBM's T.J. Watson Research Center.
The application being demonstrated is a MIDI music synthesizer we wrote entirely in Java, running with a 2 millisecond buffer size (making it competitive with hardware synthesizers and suitable for live performance). This is an extreme hard real-time system, since any interference by the hardware, operating system, virtual machine, or daemon processes would cause an audible "glitch" in the music.
Music synthesis is, in fact, more stringent in its real-time needs than many other hard real-time systems. For instance, avionics typically operate at a period of 20 milliseconds, or about 10 times longer than the synthesizer.
Visualization of the all-Java Synthesizer running on top of a non-real-time Java virtual machine.
Above is a screenshot of the video we'll be showing you. The synthesizer takes a stream of MIDI notes (shown in the top panel) and turns them into digital audio data (shown in the middle panel) while the Java virtual machine runs and garbage collects memory allocated by the program (shown in the bottom panel).
The screenshot and video were made with our TuningFork real-time visualization and trace analysis tool.
An incoming MIDI note comes from an instrument like a keyboard, and represents a command like "play middle-C on a piano" (but also includes information about how hard the key was pressed, for how long, etc). The synthesizer takes the note and looks it up in a soundbank, applies sound-shaping filters, mixes it together with any other currently playing notes, and then sends the resulting CD-quality audio to a sound card to be played on a speaker.
The synthesizer was written in completely standard Java and made extensive use of object-oriented features and dynamic memory allocation. In addition, to stress the system additional threads are running and allocating about 8 megabytes per second of object data.
In the screenshot above, you can clearly see gaps in the audio data (the green waveform in the middle panel). These gaps are caused by Java's automatic memory management or garbage collection. Garbage collection is a crucial component of Java's ease of programming, safety, security, and portability. However, up until now this has also meant unpredictable interruptions of the program, which are shown by the red bars. You can clearly see that every time there is a major garbage collection it takes about 100 milliseconds and there is a large gap in the sound.
There are also smaller garbage collections (the thin red lines) which collect a memory area devoted to newly allocated objects, also called the nursery. Nursery collections take about 5 milliseconds and their effects are not obviously visible in the audio waveform, but if you listen carefully you will hear them in the audio -- between the big gaps caused by full garbage collections, the audio has a kind of "static" caused by the nursery collections.
Now, play the video and see how standard Java garbage collectors interfere with real-time programs (16MB Quicktime, 4MB Windows Media Video).
Visualization of the all-Java Synthesizer running on top of IBM's WebSphere RealTime Java virtual machine with Metronome Garbage Collection technology.
In the screenshot above, you can see the same synthesizer code running with the exact same input on IBM's WebSphere RealTime Java virtual machine, on top of an open-source real-time Linux kernel co-developed by the IBM Linux Technology Center.
As you can see, there are no gaps in the audio. That's because the Metronome garbage collector does its work in tiny slices of about 500 microseconds each. That's why the garbage collections are "pink", because the visualizer is actually showing hundreds of tiny little pieces of garbage collection activity. The slices are so short they don't cause any perturbation of the synthesizer application, even with a 2 millisecond latency.
Now, play the video and hear for yourself how much better music sounds with Metronome (16MB Quicktime, 4MB Windows Media Video).
Zoomed-in View of 140 milliseconds of Metronome Garbage Collection
These slices are spaced and grouped regularly, with about 6 slices every 10 milliseconds. The regular spacing of the slices combined with their predictable size is a key part of why Metronome works so well for so many kinds of real-time applications.
WebSphere RealTime (WRT) with Metronome is being used by Raytheon to develop the software for the Navy's new DDG 1000 destroyer and is being integrated into telecommunication systems. Pilot projects are underway with customers in other industries, especially by financial institutions for real-time stock and derivatives trading.