Over the past few months I’ve taken several opportunities to blog about the role that music and sound may play in the virtual reality systems and games that have become famous in the media of late, and which will begin to hit retail during the holidays this year. Today I encountered a very interesting research paper that warns of a possible problem that may face game developers as they attempt to deliver three-dimensional audio for virtual reality experiences. I explored some issues regarding three-dimensional game audio in my book, A Composer’s Guide to Game Music, and as game composers, I thought that audio for VR might be of particular interest to us. So, I’m going to share what I learned about the issue from this research paper, including a conclusion which may indicate an imminent problem for some VR gamers.
All of the popular VR systems rely on headphones for audio delivery, but only one (the Oculus Rift) will include built-in headphones as a part of the system. The rest will allow the consumer to use their own headphones, and even the Oculus Rift allows for its attached headphones to be removed so that the user can replace them with their own “high quality” headphones.
The Oculus Rift, shipping with detachable headphones.
So, here’s where things start to get tricky.
What do the words “high quality headphones” mean to the modern gamer? Well, the gaming website Kotaku held a survey last year so that its hardcore gaming community could vote to determine the very best gaming headphones. The winner (by a wide margin) was the Astro A50 7.1 Wireless Surround Sound headset, followed by the Logitech G930 Wireless 7.1-Channel Surround Sound headset. Two surround-sound models had come out on top. Of the other headsets in the survey, most were stereo rather than surround, and the only other surround-sound headset in the survey was wired rather than wireless. Clearly, the community had told us what “high quality” meant – and that was a surround sound experience.
The Astro A50 Wireless 7.1 Surround Sound Headset.
Now, here’s where we hit upon the problem, and it’s explored in the paper “Challenges of the Headphone Mix in Games,” written by Aristotel Digenis (lead audio programmer with FreeStyleGames), who presented his paper in February 2015 at the Audio Engineering Society’s International Conference on Audio for Games in London. Virtual reality games will be offering binaural audio to simulate a fully three-dimensional listening environment. While binaural audio can present an awesome level of immersion and realism, the technology of binaural sound isn’t the same as that of surround sound. In fact, they’re fundamentally different. If gamers have opted to use their own “high quality” surround sound headsets, then they may be experiencing a lower-quality sound environment than the game developers intended.
Many of the highest quality surround sound headphones include the ability to process an incoming non-surround audio signal into a compatible surround-sound mix (essentially imitating surround sound by virtue of some built-in digital signal processing). If this processing were applied to the binaural soundscape of a virtual reality game, the effect would cause the immersive quality of the audio to deteriorate rather than improve. Gamers would be left wondering why their stellar top-of-the-line headphones are making their VR game sound lousy.
So far I haven’t heard any reps from the three VR system manufacturers address this issue, and gamers should definitely be warned that “high quality” headphones for VR will need to be stereo, rather than surround. VR enthusiasts who are hoping for the ultimate virtual reality experience may need to purchase some excellent stereo headphones, if they don’t already own them. Without a warning about this issue, some VR gamers may be set up for a nasty sonic surprise.
Winifred Phillips is an award-winning game music composer with more than 11 years of experience in the video game industry. Her projects include Assassin’s Creed Liberation, God of War, the LittleBigPlanet franchise, and many others. She is the author of the award-winning bestseller A COMPOSER’S GUIDE TO GAME MUSIC, published by the Massachusetts Institute of Technology Press. Follow her on Twitter @winphillips.
The Electronic Entertainment Expo is upon us once again, so I’ll be spending this blog exploring what we can expect to see and learn that’s most relevant to the field of game audio from this year’s big convention.
Virtual Reality
The impending releases of three virtual reality systems should make things especially interesting on the E3 show floor, and it will be awesome to see and hear what these systems have to offer. Let’s take a look at what we might expect from the three top VR systems, as well as a possible surprise VR reveal that might happen next week.
The newest model of the Oculus Rift, the famous Crescent Bay, offers 3D audio through a set of built-in headphones. Here’s an interview that Oculus CEO Brendan Iribe gave to Gamecrate during the ever-popular Consumer Electronics Show 2015 about the new VR audio features of the Oculus Rift.
HTC Vive
The HTC Vive doesn’t currently offer built-in headphones, but the developer assures us that the final consumer version will offer integrated 3D audio. The current model offers the user the option to connect their own high-end headphones to the Vive. E3 attendees may get to see how aurally immersive that can be by playing Arizona Sunshine, a game designed for the Vive and set in a genre so famous and pervasive that its appearance in the VR world was inevitable: the apocalyptic zombie shooter. The game was announced on May 21st by its developer, Vertigo Games, and it’s a good bet that the game could be showing on the E3 exhibit floor. Here’s a look at a trailer for Arizona Sunshine:
The Electronic Entertainment Expo is an opportunity for consumer audio hardware manufacturers to show off most of their top products, so let’s take a look at what this year’s exhibitors are offering.
Ready or not, virtual reality is coming! Three virtual reality headsets are on their way to market and expected to hit retail in either late 2015 or sometime in 2016. These virtual reality systems are:
VR is expected to make a big splash in the gaming industry, with many studios already well underway with development of games that support the new VR experience. Clearly, VR will have a profound impact on the visual side of game development, and certainly sound design and voice performances will be impacted by the demands of such an immersive experience… but what about music? How does music fit into VR?
At GDC 2015, a presentation entitled “Environmental Audio and Processing for VR” laid out the technology of audio design and implementation for Sony’s Project Morpheus system. While the talk concentrated mainly on sound design concerns, speaker Nicholas Ward-Foxton (audio programmer for Sony Computer Entertainment) touched upon voice-over and music issues as well. Let’s explore his excellent discussion of audio implementation for a virtual space, and ponder how music fits into this brave new virtual world.
Nicholas Ward-Foxton, during his GDC 2015 talk.
But first, let’s get a brief overview on audio in VR:
3D Positional Audio
All three VR systems feature some sort of positional audio, meant to achieve a full 3D Audio Effect. With the application of the principles of 3D Audio, sounds will always seem to be originating from the virtual world in a realistic way, according to the location of the sound-creating object, the force/loudness of the sound being emitted, the acoustic character of the space in which the sound is occurring, and the influences of obstructing, reflecting and absorbing objects in the surrounding environment. The goal is to create a soundscape that seems perfectly fused with the visual reality presented to the player. Everything the player hears seems to issue from the virtual world with acoustic qualities that consistently confirm an atmosphere of perfect realism.
To get a greater appreciation of the power of 3D audio, let’s listen to the famous “Virtual Barber Shop” audio illusion, created by QSound Labs to demonstrate the power of Binaural audio.
Head Tracking and Head-Related Transfer Function
According to Nicholas Ward-Foxton’s GDC talk, to make the three-dimensional audio more powerful in a virtual space, the VR systems need to keep track of the player’s head movements and adjust the audio positioning accordingly. With this kind of head tracking, sounds swing around the player when turning or looking about. This effect helps to offset an issue of concern in regards to the differences in head size and ear placement between individuals. In short, people have differently sized noggins, and their perception of audio (including the 3D positioning of sounds) will differ as a result. This dependance on the unique anatomical details of the individual listener is known as Head-Related Transfer Function. There’s an excellent article explaining Head-Related Transfer Function on the “How Stuff Works” site.
Head-Related Transfer Function can complicate things when trying to create a convincing three-dimensional soundscape. When listening to identical binaural audio content, one person may not interpret aural signals the same way another would, and might estimate that sounds are positioned differently. Fortunately, head tracking comes to the rescue here. As Ward-Foxton explained during his talk, when we move our heads about and then listen to the way that the sounds shift in relation to our movements, our brains are able to adjust to any differences in the way that sounds are reaching us, and our estimation of the spatial origination of individual sounds becomes much more reliable. So the personal agency of the gaming experience is a critical element in completing the immersive aural world.
Music, Narration, and the Voice of God
Now, here’s where we start talking about problems relating directly to music in a VR game. Nicholas Ward-Foxton’s talk touched briefly on the issues facing music in VR by exploring the two classifications that music may fall into. When we’re playing a typical video game, we usually encounter both diegetic and non-diegetic audio content. Diegetic audio consists of sound elements that are happening in the fictional world of the game, such as environment sounds, sound effects, and music being emitted by in-game sources such as radios, public address systems, NPC musicians, etc. On the other hand, non-diegetic audio consists of sound elements that we understand to be outside the world of the story and its characters, such as a voice-over narration, or the game’s musical score. We know that the game characters can’t hear these things, but it doesn’t bother us that we can hear them. That’s just a part of the narrative.
VR changes all that. When we hear a disembodied, floating voice from within a virtual environment, we sometimes feel, according to Ward-Foxton, as though we are hearing the voice of God. Likewise, when we hear music in a VR game, we may sometimes perceive it as though it were God’s underscore. I wrote about the problems of music breaking immersion as it related to mixing game music in surround sound in Chapter 13 of my book, A Composer’s Guide to Game Music, but the problem becomes even more pronounced in VR.When an entire game is urging us to suspend our disbelief fully and become completely immersed, the sudden intrusion of the voice of the Almighty supported by the beautiful strains of the holy symphony orchestra has the potential to be pretty disruptive.
The harpist of the Almighty, hovering somewhere in the VR world…
So, what can we do about it? For non-diegetic narration, Ward-Foxton suggested that the voice would have to be contextualized within the in-game narrative in order for the “voice of God” effect to be averted. In other words, the narration needs to come from some explainable in-game source, such as a radio, a telephone, or some other logical sound conveyance that exists in the virtual world. That solution, however, doesn’t work for music, so it’s time to start thinking outside the box.
Voice in our heads
During the Q&A portion of Ward-Foxton’s talk, an audience member asked a very interesting question. When the player is assuming the role of a specific character in the game, and that character speaks, how can the audio system make the resulting spoken voice sound the way it would to the ears of the speaker? After all, whenever any of us speak aloud, we don’t hear our voices the way others do. Instead, we hear our own voice through the resonant medium of our bodies, rising from our larynx and reverberating throughout our own unique formant, or acoustical vocal tract. That’s why most of us perceive our voices as being deeper and richer than they sound when we hear them in a recording.
Ward-Foxton suggested that processing and pitch alteration might create the effect of a lower, deeper voice, helping to make the sound seem more internal and resonant (the way it would sound to the actual speaker). However, he also mentioned another approach to this issue earlier in his talk, and I think this particular approach might be an interesting solution for the “music of God” problem as well.
Proximity Effect
“I wanted to talk about proximity,” said Ward-Foxton, “because it’s a really powerful effect in VR, especially audio-wise.” Referencing the Virtual Barber Shop audio demo from QSound Labs, Ward-Foxton talked about the power of sounds that seem to be happening “right in your personal space.” In order to give sounds that intensely intimate feeling when they become very close, Ward-Foxton’s team would apply dynamic compression and bass boost to the sounds, in order to simulate the Proximity Effect.
The Proximity Effect is a phenomenon related to the physical construction of microphones, making them prone to add extra bass and richness when the source of the recording draws very close to the recording apparatus. This concept is demonstrated and explained in much more depth in this video produced by Dr. Alexander J. Turner for the blog Nerds Central:
So, if simulating the Proximity Effect can make a voice sound like it’s coming from within, as Ward-Foxton suggests, can applying some of the principles of the Proximity Effect make the music sound like it’s coming from within, too?
Music in our heads
This was the thought that crossed my mind during this part of Ward-Foxton’s talk on “Environmental Audio and Processing for VR.” In traditional music recording, instruments are assigned a position on the stereo spectrum, and the breadth from left to right can feel quite wide. Meanwhile, the instruments (especially in orchestral recordings) are often recorded in an acoustic space that would be described as “live,” or reverberant to some degree. This natural reverberance is widely regarded as desirable for an acoustic or orchestral recording, since it creates a sensation of natural space and allows the sounds of the instruments to blend with the assistance of the sonic reflections from the recording environment. However, it also creates a sensation of distance between the listener and the musicians. The music doesn’t seem to be invading our personal space. It’s set back from us, and the musicians are also spread out around us in a large arc shape.
So, in VR, these musicians would be invisibly hovering in the distance, their sounds emitting from defined positions in the stereo spectrum. Moreover the invisible musicians would fly around as we turn our heads, maintaining their position in relation to our ears, even as the sound design elements of the in-game environment remain consistently true to their places of origin in the VR world. Essentially, we’re listening to the Almighty’s holy symphony orchestra. So, how can we fix this?
One possible approach might be to record our music with a much more intimate feel. Instead of choosing reverberant spaces, we might record in perfectly neutral spaces and then add very subtle amounts of room reflection to assist in a proper blend without disrupting the sensation of intimacy. Likewise, we might somewhat limit the stereo positioning of our instruments, moving them a bit more towards the center. Finally, a bit of prudently applied compression and EQ might add the extra warmth and intimacy needed in order to make the music feel close and personal. Now, the music isn’t “out there” in the game world. Now, the music is in our heads.
Music in VR
It will be interesting to see the audio experimentation that is surely to take place in the first wave of VR games. So far, we’ve only been privy to tech demos showing the power of the VR systems, but the music in these tech demos has given us a brief peek at what music in VR might be like in the future. So far, it’s been fairly sparse and subtle… possibly a response to the “music of the Almighty” problem. It is interesting to see how this music interacts with the gameplay experience. Ward-Foxton mentioned two particular tech demos during his talk. Here’s the first, called “Street Luge.”
The simple music of this demo, while quite sparse, does include some deep, bassy tones and some dry, close-recorded percussion. Also, the stereo breadth appears to be a bit narrow as well, but this may not have been intentional.
The second tech demo mentioned during Ward-Foxton’s talk was “The Deep.”
The music of this tech demo is limited to a few atmospheric synth tones and a couple of jump-scare stingers, underscored by a deep low pulse. Again, the music doesn’t seem to have a particularly wide stereo spectrum, but this may not have been a deliberate choice.
I hope you enjoyed this exploration of some of the concepts included in Nicholas Ward-Foxton’s talk at GDC 2015, along with my own speculation about possible approaches to problems related to non-diegetic music in virtual reality. Please let me know what you think in the comments!
Winifred Phillips is an award-winning game music composer with more than 11 years of experience in the video game industry. Her projects include Assassin’s Creed Liberation, God of War, the LittleBigPlanet franchise, and many others. She is the author of the award-winning bestseller A COMPOSER’S GUIDE TO GAME MUSIC, published by the Massachusetts Institute of Technology Press. Follow her on Twitter @winphillips.
