CodeSounding library: from source code to computer generated music sounds (sonification)

SourceForge.net Logo

Demo

Wiki Home   Download   Documentation  

CodeSounding: computer generated music sounds from a source code structure [Serbo-Croatian translation]

CodeSounding is an open source sonification framework which makes possible to hear how any existing Java program "sounds like", by assigning instruments and pitches to code statements (if, for, etc) and playing them as they are executed at runtime. In this way the flowing of execution is played as a flow of music and its rhythm changes depending on user interaction.
The major challenges are 1) keeping realtime synchronization between instructions executed and sounds played, 2) how to "compress" the CPU speed into a meaningful rhythm, that is how to going from something like 2GHz to 44kHz.
We propose different approaches, with increasing degrees of realtime synchronization / "code fidelity":
  • every instruction can append one note into an ABC file, which is then converted in a MIDI and played in a separated thread. In this approach we tipically pick one note every thousandth instructions, nevertheless it allows interesting polyphonic pieces.
  • we can play MIDI notes directly, via jMusic. We tried a sort of dynamic pitch modulation, depending on the logarithm of the instructions triggered and on how quickly changes its slope. It sounds "dark" and has a poor realtime synchronization: see a live.
  • every MIDI-based approach does not allow a real sound mixing; it is possible to play two instruments on the same time, but you cannot change a timbre on realtime (unless by using pitch bend messages, although this affects just the frequency). Therefore a different approach could be to write directly into an audio buffer - in our case, we write into a JACK buffer. This allows a finer wave control and a good code synchronization, but requires to write a custom synthesis algorithm by hand.
  • finally, to allow a greater control on realtime sound synthesis we exploit some unit generator-based solutions:
    • with JSyn programming-oriented people can trigger sinusoids, filters, etc. without leaving the Java environment. Note that JSyn is not an open source software.
    • musicians and GUI-oriented people can use either Max/MSP (commercial) or Pure Data (open source) in order to make visual patches and exploiting the existing objects. The interactions with CodeSounding are based on UDP packets, sent to different ports, so the sonification can live on a different machine. In any case it stayes on a separated process.
It is also possible to programmatically extend the framework at any level, for example by subclassing the codesounding.SilentProcessor class.

Source code Musical score

Video examples

  • Trying CodeSounding with JSyn: every thousandth Java istruction triggers an enveloped oscillation (of 50 ms), with each instruction having a different frequency. They are finally added and reverberated. Faders enable you to change the oscillators's frequencies, how many instructions are skipped and the grains's envelope. Furthermore the skipped instructions scale the amplitude of each oscillator.

  • CodeSounding data can be pulled by a JACK audio server:


Notes
[] Also in Serbo-Croatian language, thanks to Jovana Milutinovich from Geeks Education
~ Copyright © 2006 2012 - Roberto Mannai (robermann@gmail.com) ~