PROJECT OVERVIEW

This project is my Master thesis of the MSc in Audio and Music Technology at the University of York, and it has Abbey Road studios as Industry Partner. I started to think about it approximately one month ago, when I was assigned to this topic. However, now I'm starting to research full-time, and I will be working on it until the end of August. 

If I tell you that the formal title is: "DAW based spatial audio creation using smartphones", probably your reply will be something like: "Sounds cool, but... what does it mean?" Let's start from the begging then:

background

"DAW based..."
A Digital Audio Workspace (DAW) is a generic software categorisation within we can find commercial software like ProTools (image), Cubase, Reaper, LogicPro etc... With them, the user has an environment to sequence, record, edit, and mix audio or music.

​This kind of software can manage inputs and outputs of an external soundcard (to work with analogue equipment, for instance), or use third-party effects (also called plug-ins) to generate or modify sound within the same environment. Therefore, a DAW is the base of all digital studios, from the smallest home-studio to Abbey Road's studio one. A DAW-based tool, thus, is whatever tool that can interact with a DAW (generally in a plug-in).

Pro Tools | First's GUI - Avid

"...Spatial audio creation..."
Spatial audio is a really vague name but, for now, just assume that spatial audio means the sound is perceived in the space (as the real life). Other ways to say it is 3D audio or immersive sound. Spatial audio can be reproduced using an array of speakers (like Dolby Atmos), but it is also possible to reproduce through headphones. Do you want to try it? What about going to the virtual barbershop?

​Several commercial brands like Waves, Facebook, or SSA, as well as independent developers like Matthias kronlachner or Ambisonic ToolKit, have released different tools to work with (and create) spatial audio. Some of these tools are used, for instance, to place sources in the space, to add spatial reverberation, or to apply other effects. Moreover, most of these tools are also DAW based, and then, why and what do I want to research in this area?
​

Font: ambiX - Ambisonic plug-in suite

​"...using smartphones."
Indeed, the last part of the sentence is the answer. At it is shown in the image above (from the ambix plug-ins suite) all of the previously presented tools are using 2D user interfaces to work with 3D audio. Even though they simulate 3D objects, such as the ambix encoder (bottom-right), sometimes it is difficult to work with them, especially concerning elevation, width or distance.

​"And what's happening with the smartphones?" Well, nowadays almost everyone has at least one smartphone with them; And, among others, it has lots of sensors like the gyroscope, the accelerometer or the camera that could be used to control spatial audio. An app then, can interpret these sensors information, and send OSC messages (a standard network communication for sound) to a plug-in in the DAW. After that, the plugin can translate the received information to place a sound in the 3D space, change the reverberation, the EQ, or whatever else.

​Now that we know the background and hopefully the title is better understood, let's talk about the specific aims and objectives of the project:​

aims and objectives

​The main objective is developing an iPhone app which lets the user control spatial audio parameters in a DAW. Therefore, also a plug-in needs to be developed to receive the app messages and process the audio.
With these objectives, the project has two research areas:

Spatial audio workflow
Before controlling spatial audio parameters with a smartphone, we have to know which parameters we want to control in our plug-in. Therefore, the first research area is the spatial audio workflow, where we can find questions like:

• Which are the typical Fx used to work with spatial audio? (Eq, Comp, Reverb...)
• Are they used individually in each source? Or as a global effect? In series or parallel?
• Is there any pattern in the order they are used?
• Which Fx are necessary and which are used sporadically?
• Are the users happy to use generic effects, or they prefer to use third-party effects?

Spatial audio control using smartphones
Finally, the second research area is related to gesture control and smartphones. An iPhone has lots of sensors and Augmented Reality capabilities to explore and thus, there are also several questions to ask:

• ​(Basic) Could a smartphone be used to control spatial audio? And how?
• ​​Which Fx are more likely to be controlled using these capabilities?
• ​Is it practical to have an app on the smartphone, and a plug-in in the computer, or is it better to use just one of them at once?
• Which is the best way to control spatial audio? Using AR and the camera? Using small movements? Moving the iPhone around the user? Just an extension of the plug-in user interface?

summary

To summarise, the main concept of the project is to enable the user to control spatial audio parameters in a DAW environment, but without using only the mouse and the keyboard. Instead, investigate new and easier ways to control it using the different capabilities of smartphone sensors.

A good example of spatial audio creation without the keyboard and the mouse is this product called DearVR Spatial connect. It is using a VR headset and VR controllers to move sound sources in a VR environment. This is what they call 'the mixing console of the future'.