Many Degrees Of Freedom - navigating the current landscape of 3D Audio
It's a great time to be an audio designer. VR/AR/MR/XR (any others?) have unleashed a wealth of opportunity for innovative sound design strategies in a 3D virtual world. Tools and technologies for implementing truly immersive audio, from personalized HRTF profile generation to in-workstation (and even in-game) 360-degree spatialization and rendering, now abound in the marketplace.
This was especially evident at the recent Audio Engineering Society (AES) conference in New York a few weeks ago, where a massively oversubscribed seminar track on spatial audio was featured, as well as multiple game and film tracks that came loaded with information and insights on how to leverage core expertise in these 'traditional' fields to the emerging workflows of the 'three R's of the immersive realities.
Immersive Audio: A Rough Guide
As a presenter on a few of these tracks, I felt it incumbent upon me to piece together something of a 'traveler's map' of 3D audio, partly for my own sanity and as an aide-memoire while discussing the vicissitudes of working in this new field, and also for the benefit of the audience - many of whom were there to learn what all the fuss is about. To be fair, working out the lay of the land felt much like creating some improbable urban transit system, but with far fewer years of planning and a significantly smaller budget:
There's always an inherent danger in creating a static taxonomy of a living ecosystem like this, in that it has already evolved beyond the snapshot the moment it is taken (indeed, this map is probably wildly inaccurate by now, less than a month after it was drafted). As such, the terms 'non-exhaustive', 'not mutually exclusive' and 'work in progress absolutely apply. But it's still useful to attempt such a task. Or so I tell myself.
The Wild West of Workflows
Poring through the various combinations of offerings of the purveyors of fine audio tech in this space, what became immediately obvious was the preponderance of 3D audio solutions that are geared to making the transition from traditional linear workflows to fully immersive formats (compared to solutions built for those who are already 'there'). This makes sense - after all, there's a lot to be gained from integrating solutions into an industry that already has its foot firmly in the door, production-wise.
FB Spatial Workstation has certainly been leading the charge in this field, with a nice cross-platform end-to-end solution for 360 video, but there are a number of fast followers that are offering nuanced tweaks and improvements over the FB version. AudioEase 360Pan Suite, for example, has a nifty UI that allows you to precisely place sounds within the ambisonic space and audition the results with concomitant head-tracking. Whether or not these are simply incremental gains at the expense of a 'one size fits all' workflow, time will tell as the market evolves.
One particularly interesting aspect of the who's who of 3D Audio is the scale gamut that it runs, from newcomers with an interesting approach like Dear VR to the behemoths of the audio industry like Dolby with their AtmosVR engine to, well - folks you wouldn't necessarily expect to make an audio solutions list, like Google. Overall, there's a kind of wild-west feel about the influx of audio gunslingers mozying into town en masse, each proclaiming to be the best around. Granted, it's an order of magnitude more navigable of a landscape than say, a year ago - where a few of my own forum queries would elicit headscratch-y responses even from the folks building the tools. The fact that it's now sort of possible to map out the space now shows how far we've come.
Follow the money...
But even despite this rapid maturing of the field, for the audio professional who needs to cut through the noise, where on earth to start? Well, it depends on a few things. Firstly, and possibly most importantly - my advice would be to look at what your clients are working with. Very often, and this is especially true of interactive/game formats, the toolkit that's available to you is going to be dictated by the overall development environment that the client is working in.
To be clear, this isn't a hard and fast rule - if you have a great handle on the interoperability of the system your client is working with and your preferred modus operandi, and you can assure them that your approach isn't going to break everything - you might be in with a shot of aligning them with your workflow, as well as reaping the added benefit of some serious consultant-style kudos!
Towards the future and the hybrid format.
Obviously, if you're working in pure linear formats like YouTube360, this will to a great extent guide your choice of software, but where it gets really interesting (at least in this writer's opinion) is in the hybrid format. Unity 2017.1 has now incorporated its long-awaited 'Timeline' feature, which combines the power of a linear workflow with an interactive engine.
Combining the storytelling power of the cinematic format with the feeling of 'experiential agency' afforded by game engines is potentially a paradigm-shifting move. Interactive movies (think Ready Player One) or even fully generative experiences based on stochastic/AI experience parameters are likely to change the way we think about immersive content. As such, the opportunity to integrate what we know about adaptive audio in a spatial context is very compelling, and will deeply enrich the VR experience.