Sonorous.js
An open-source audio library, built for the modern web.
This tracker mixer is a simple example that uses Sonorous.js to manipulate each track of the song Reborn by Tillian.
Background
I developed Sonorous.js during my time at eko, where I currently work as a software engineer. Eko is an interactive video company, and aims to deliver a seamless, dynamic audio/video stream to its users. Before Sonorous.js, we used an audio library called Howler. It was causing several bugs and hadn’t been updated to leverage modern JavaScript features. I set out to write a new audio library- one for the modern browser.
Goals
Intuitive API
WebAudio is ultimately how most sound gets played on the web. However, the WebAudio API is complex and low-level. I wanted to create a library that would allow developers to easily perform simple audio operations such as play, pause, etc.
Cross Browser Support
Each browser has implemented the WebAudio API differently, which means code that works on Chrome may not work on Safari, and vice versa. I wanted to build a library that reliably handled this behavior for developers.
Memory Management
Audio buffers can be huge. Just 20 minutes of audio decompressed at once would be more than enough to crash a mobile browser. I wanted this library to give developers some control over how audio buffer memory is handled.
Intuitive API
WebAudio is admittedly very powerful - you can craft an interactive musical journey, influence the ambience, or recreate an instrument built in the 1920s. WebAudio can also be a huge headache.
For starters, the API is low level and complex. For the simple background music or sound effect, WebAudio can feel like trying to take a sip of water out of a firehose. (For the record, it should stay that way- in order to build complex audio experiences, you need something that offers a lot of control.) While there are libraries out there that offer abstractions over WebAudio, I’ve found them old and outdated. A coworker of mine once said, “Whenever an audio plugin related bug is discovered, instead of quickly fixing it and moving on, we need to spend a whole day debugging internal implementation [of our 3rd party library].”
Sonorous abstracts away the complexity of WebAudio and allows you to play a sound in just 3 lines of code.
Playing a sound using WebAudio
Playing a sound using Sonorous.js.
Sonorous.js is comprised of a singleton manager and instances of a “sonor”. A sonor represents a specific audio file and you can perform basic operations like play, pause, set volume, etc. The manager handles all instances of sonors and handles global operations, like unlocking the audio context.
There have been a lot of changes to JavaScript/the browser, and I wanted to take advantage of that. For one, Web Audio is now supported in every modern browser, so Sonorous doesn’t deal with HTML5 Audio at all. Web Audio is superior in every way; HTML5 Audio is pretty old and outdated and doesn’t offer all the flexibility that Web Audio has. I also wanted to leverage the more modular ES6 class structure to create extensible architecture with clear interfaces. Debugging cross-browser issues could be tricky with Howler, but Sonorous’s architecture was designed to make it easy to pinpoint and fix issues. The current implementation could also easily be expanded in the future as WebAudio features grow.
Cross Browser Support
Sonorous provides many major advantages other than ease-of-use. If the WebAudio APIs weren’t tricky enough to understand on their own, there’s the added complexity of each browser implementing the APIs differently. For example, Chrome may allow you to call stop() on a source node before it’s been started but Safari will throw an error and crash your program. On the other hand, sometimes Safari will fail to initialize an AudioContext but won’t throw an error and Chrome would throw an error. In many browsers now, you are unable to start playing audio without a user interaction first. This is just the tip of the iceberg, but Sonorous handles all of this for you. It’s abstracted by the library, so your code can remain clean and concise.
Memory Management
Audio buffers, once decoded, can be huge. Unfortunately, audio files must be decoded in order to be played back. And with Howler, as soon as the sound is loaded, the decoded buffer is kept in memory at runtime. 60 seconds of uncompressed audio translates to 23MB of memory at runtime. If your program contains just 20 minutes worth of audio files, all loaded at once, that’s over 460MB of memory which is more than enough to cause an iOS Safari tab to instantly crash.
Sonorous gives you the option of storing the encoded buffer or the decoded buffer when the sound loads. If you store the encoded buffer, it would only be decoded when play() is called. Once the play operation finishes, the decoded buffer would be removed again. There might be a lag before playing, if the audio file is large, but this solution offers more control over how much memory this library uses when not playing a sound.
Reflection
Looking Back
Ultimately, developing Sonorous.js was a great move for eko. It is currently being used in our audio plugin, which powers all sound effects for all videos on our platform. There have been hardly any audio-related bugs since Sonorous.js was integrated into the eko platform. Before it was released, eko would often get 4-5 audio bugs per month. In May of 2020, I introduced Sonorous.js at JSVidCon, to great success. Head to eko.com and play any video to see Sonorous.js in action!
Moving Forward
WebAudio is extremely powerful, albeit complex to use. Right now, Sonorous.js is only using a small subset of WebAudio’s capabilities. It should be expanded to include things like spatial (3D) audio or audio filters. It would also be interesting to further explore memory management and offer even more fine grained control over the audio buffers.