Welcome to the third (and final) article in this three-part discussion of how video game composers (like us) can make strategy gamers smarter! We’ve been exploring the best ways that the music of game composers can help strategy gamers to better concentrate while making more sound tactical decisions. During this discussion, I’ve shared my personal perspective as the composer for the popular Dragon Front strategy game for VR.
In part one, we discussed the concept of ‘music-message congruency,’ so if you haven’t read that article yet, you can read it here. In part two, we explored the meaning of ‘cognition-enhancing tempo’ – you can read that article here. Please make sure to read both those articles first and then come back.
Are you back? Awesome! Let’s launch into a discussion of the third technique for increasing the smarts of strategy gamers!
Tension-regulating affect
In psychology, the term ‘affect’ refers to emotion, particularly in terms of the way in which such emotional content is displayed. Whether by visual or aural means, an emotion can not be shared without some kind of ‘affect’ that serves as its mode of communication from one person to another. When we’re happy, we smile. When we’re angry, we frown.
Welcome back to our three-part discussion of how video game composers (such as ourselves) can make strategy gamers smarter! In these articles, we’re looking at ways in which our music can enhance concentration and tactical decision-making for players engrossed in strategic gameplay. Along the way, I’ve been sharing my personal experiences as the composer for the Dragon Front strategy game for virtual reality. Over the course of these articles we’ll be covering three of the top concepts that pertain to the relationship between music and concentration. In part one, we discussed the concept of ‘music-message congruency,’ so if you haven’t read that article yet, please go check it out and then come back.
Are you back now? Good! Let’s move on to the second big technique for increasing the smarts of strategy gamers!
Cognition-enhancing tempo
As video game composers, we create music in a wide variety of tempos designed to support the energy of play and the pacing of the game’s overall design. From leisurely tracks that accompany unstructured exploration to frenetic pieces that support the most high-stakes combat, our music is planned with expert precision to shape the excitement level of players and keep them motivated as they progress.
Can video game composers make you smarter? Well, video gaming can be a pretty cerebral activity, requiring astute problem-solving skills and disciplined concentration in order to excel. That’s especially true for any game built around strategic and/or tactical gameplay, such as real-time or turn-based strategy, tactical shooters, multiplayer online battle arenas (MOBAs), and online collectible card strategy games. To succeed in these types of games, players must assess the current situation and formulate a plan that accounts for future developments and variables. Without this type of tactical forward-thinking gameplay, a gamer has little chance to win. So, can music enable gamers to think tactically, stay focused and make smart decisions? Over the next three articles, I’ll try to answer that question, while exploring the role of music in enhancing the concentration of strategic/tactical gamers.
Along the way, we’ll be taking a look at some scholarly research on the subject, consulting the opinions of experts, and I’ll be sharing my experiences creating the music for the recently released Dragon Front strategy game from High Voltage software. We’ll check out some music tracks I composed for the popular Dragon Front game (pictured at the top of this article), and we’ll discuss methods for supporting and enhancing concentration for strategic/tactical game players. But first, let’s take a closer look at the Dragon Front game.
My work as a video game composer has lately included some projects for virtual reality games (more info on that in the coming months), and as a result I’ve been thinking a lot about the awesome potential of VR, and have also been writing lots of articles on the subject. Earlier this month I began a two-part article that focuses on the experience of the end user, and the gear with which they’ll be enjoying our video game music and audio content (you can read part one here). So, let’s now continue our discussion about the new generation of headphones designed specifically for VR!
In this article, we’ll be discussing two headphone models:
Entrim 4D
Plantronics RIG 4VR
So let’s get underway!
Entrim 4D headphones
This March at the famous SXSW convention in Austin, Samsung showed off a piece of experimental technology promising to bring a new dimension of immersion to virtual reality. It’s designed specifically to complement their popular Samsung Gear VR device, and it works by virtue of electrodes that send electrical signals right into the wearer’s head! As if virtual reality itself weren’t futuristic enough, now we’re talking about a device that zaps us to make the VR feel more real! It’s called Entrim 4D (pictured right). We’re talking about it here because (among other things) Entrim 4D is a pair of audio headphones built specifically for VR.
In this article I’d like to turn our attention towards the experience of the end user, and specifically, the primary interface with which users will be enjoying our audio content. So, let’s talk about headphones! More specifically, let’s talk about the newest incarnation of this device… headphones built specifically for VR!
This will be a two-part article (since there’s a lot of ground to cover!) In part one, we’ll be discussing these two headphone models:
Since I’ve been working recently on music for a Virtual Reality project (more info in the coming months), I’ve been thinking a lot about VR technology and its effect on the creative process. Certainly, VR is going to be a great environment in which to be creative and perform tasks and skills with enhanced focus, according to this article from the VR site SingularityHub. I’ve written in this blog before about the role that music and sound will play in the Virtual Reality gaming experience. It’s clear that music will have an impact on the way in which we experience VR, not only during gaming experiences, but also when using the tools of VR to create and be productive. With that in mind, let’s consider if the opposite statement may also be true – will VR impact the way in which we experience music, not only as listeners, but also as video game composers?
Simple VR technologies like the popular Google Cardboard headset can be a lot of fun – as I personally experienced recently (photo to the left). However, they offer only the rudimentary visual aspects, which omits some of the most compelling aspects of the VR experience. When motion tracking (beyond simple head movement) is added to the mix, the potential of VR explodes. Over the next three articles, we’ll be exploring some interesting possibilities created by the motion tracking capabilities of VR, and how this might alter our creative process. In the first article, we’ll have some fun exploring new ways to play air guitars and air drums in the VR environment. In the second article, we’ll take a look at ways to control virtual instruments and sound modules that are folded into the VR software. And finally, in the third article we’ll explore the ways in which VR motion tracking is allowing us to immersively control our existing real-world instruments using MIDI. But first, let’s take a look at the early days of VR musical technology!
In a new report released January 5, 2016, the research analysis firm SuperData issued a forecast of the future of Virtual Reality gaming in the coming year. Among the results: 5.1 billion dollars are predicted to be spent on VR hardware in 2016, and 55.8 million consumers will have adopted some version of a VR platform by year’s end. The report also predicts that inexpensive VR gaming on mobile devices will prove the most popular in the short-term, dominating the market in 2016. The report also suggests that small indie studios may benefit by jumping into VR development early (since the top publishers are proving to be a bit more reticent). These are awesome times to be in the video game industry, and there will certainly be lots to learn as we go boldly into the world of VR. In this blog, I’ve collected information about upcoming video game conferences – some that are already famous and some that are brand new. These events might help us to learn more about our role in the creation of VR music and audio.
Since the game audio community is abuzz with popular excitement about the impending arrival of virtual reality systems, I’ve been periodically writing blogs that gather together top news about developments in the field of audio and music for VR. In this blog we’ll be looking at some resources that discuss issues relating to artistry and workflow in audio for VR:
We’ll explore an interesting post-mortem article about music for the VR game Land’s End.
We’ll be taking a closer look at the 3DCeption Spatial Workstation.
We’ll be checking out the Oculus Spatializer Plugin for DAWs.
Designing Sound for Virtual Reality
In these early days of VR, postmortem articles about the highs and lows of development on virtual reality projects are especially welcome. Freelance audio producer and composer Todd Baker has written an especially interesting article about the audio development for the Land’s End video game, designed for the Samsung Gear VR system.
Here, you see me trying out the Samsung Gear VR, as it was demonstrated on the show floor at the Audio Engineering Society Convention in 2015.
Todd Baker is best known for his audio design work on the whimsical Tearaway games, and his work as a member of the music composition team for the awesome LittleBigPlanet series. His work on Land’s End for Ustwo Games affords him an insightful perspective on audio for virtual reality. “In VR, people are more attuned to what sounds and feels right in the environment, and therefore can be equally distracted by what doesn’t,” writes Baker. In the effort to avoid distraction, Baker opted for subtlety in regards to the game’s musical score. Each cue began with a gentle fade-in, attracting little notice at first so as to blend with the game’s overall soundscape in a natural way.
This past weekend, the Audio Engineering Society held its annual North American convention in the Jacob Javits Center in New York City. I was participating as an AES speaker, but I also knew that AES includes an exhibit floor packed with the best professional audio equipment from all the top manufacturers, and I didn’t want to miss that! So, in between my game audio panel presentation on Saturday, and the Sunday tutorial talk I gave on the music system of the LittleBigPlanet franchise, I had the pleasure of searching the show floor for what’s new and interesting in audio tech. Here are some of the attractions that seemed most interesting for game audio folks:
One of the most interesting technologies on display at AES this year was Fraunhofer Cingo – an audio encoding technology developed specifically to enable mobile devices to deliver immersive sound for movies, games and virtual reality. Cingo was developed by the institute responsible for the MP3 audio coding format. According to Fraunhofer, the Cingo technology “supports rendering of 3D audio content with formats that add a height dimension to the sound image, such as 9.1, 11.1 or other channel combinations.” This enables mobile devices to emulate “the enveloping sound of movies, games or any other virtual environment.” While I was there, Fraunhofer rep Jennifer Utley gave me the chance to demo the Cingo technology using the Gear VR headset, which turns Samsung mobile phones into portable virtual reality systems. The sound generated by Cingo did have an awesome sense of spatial depth that increased immersion, although I didn’t personally notice the height dimension in the spatial positioning. Nevertheless, it was pretty nifty!
Most visual artists in the game industry are familiar with a concept known as the “Uncanny Valley,” but it isn’t a problem that typically occupies the attention of sound designers and game music composers. However, with the imminent arrival of virtual reality, that situation may drastically change. Audio folks may have to begin wrestling with the problem right alongside their visual arts counterparts. I’ll explore that issue during the course of this blog, but first let’s start with a basic definition: what is the Uncanny Valley?
Here’s the graphic that is typically shown to illustrate the Uncanny Valley concept. The idea is this: human physical attributes can be endearing. We like human qualities when we see them attached to inhuman things like robots. It makes them cute and relatable. However, as they start getting more and more human in appearance, the cuteness starts going away, and the skin-crawling creepiness begins. The ick-factor reaches maximum in an amorphous no-man’s land right before absolute realism would theoretically be attained. In this realm of horrors known as the “Uncanny Valley,” we see that the appearance of the human-like creature is not close enough to be real, but close enough to be really disturbing. Don’t take my word for it, though. Here’s a great video from the Extra Credits video series that explores the meaning of the Uncanny Valley in more detail:
So, now we’ve explored what the Uncanny Valley means to visual artists, but how does this phenomenon impact the realm of audio?
Spatial Audio – Reconstructing Reality or Creating Illusion?
The idea of an audio equivalent for the Uncanny Valley was suggested by Francis Rumsey during a presentation he gave in May 2014 at the Audio Engineering Society Chicago Section Meeting, which took place at Shure Incorporated in Niles, Illinois. Francis Rumsey holds a PhD in Audio Engineering from the University of Surrey and is currently the chair of the Technical Council of the Audio Engineering Society. His talk was entitled “Spatial Audio – Reconstructing Reality or Creating Illusion?”
Francis Rumsey, chair of the AES Technical Council
In his excellent 90 minute presentation (available for viewing in its entirety by AES members), Francis Rumsey explores the history of spatial audio in detail, examining the long-term effort to reach perfect simulations of natural acoustic spaces. He examines the divergent philosophies of top audio engineers who approach the problem from a creative/artistic point of view, and acousticians who want to solve the dilemma mathematically by virtue of a perfect wave field synthesis technique. Along the way, he asks if spatial audio is really meant to recreate the best version of reality, or instead to conjure up an entertaining artistic illusion? This leads him to the main thesis of his talk:
Sound Design in VR: Almost Perfect Isn’t Perfect Enough
Rumsey suggests that as spatial audio approaches the top-most levels of realism, it begins to stimulate a more critical part of the brain. Why does it do this? Because human listeners react very strongly to a quality we call “naturalness.” We have a great depth of experience in the way environmental sound behaves in the world. We know how it reflects and reverberates, how objects may obstruct the sound or change its perceived timbre. As a simulated aural environment approaches perfect spatial realism and timbral fidelity, our brains begin to compare the simulation to our own remembered experiences of real audio environments, and we start to react negatively to subtle defects in an otherwise perfect simulation. “It sounds almost real,” we think, “but something about it is strange. It’s just wrong, it doesn’t add up.”
Take as an example this Oculus VR video demonstrating GenAudio’s AstoundSound 3D RTI positional 3D audio plugin. While the audio positioning is awesome and impressive, the demo does not incorporate any obstruction or occlusion effects (as the plugin makers readily admit). This makes the demo useful for us in examining the effects of subtle imperfections in an otherwise convincing 3D aural environment. The imperfections become especially pronounced when the gamer walks into the Tuscan house, but the sound of the outdoor fountain continues without any of the muffling obstruction effects one would expect to hear in those circumstances.
Voice in VR: The Uncanny Valley of Spatial Voice
During the presentation, Rumsey shared some of the research from Glenn Dickins, the Technical Architect of the Convergence Team at Dolby Laboratories. Dickins had applied the theory of the Uncanny Valley to vocal recordings. The sound of the human voice in a spatial environment is exceedingly familiar to us as human beings, much in the same way that human appearance and movement are both ingrained in our consciousness. Because of this familiarity, vocal recordings in a spatial environment such as 3D positional audio can be particularly vulnerable to the Uncanny Valley effect. Very small and subtle degradation in the audio output of a spatially localized voice recording may trigger a sense of deep-rooted unease.
Glenn Dickins of Dolby Laboratories
As we embark on three dimensional audio environments for virtual reality games, the sorts of sound compression typically used in video game design may become problematic, particularly in relation to voice recordings in games. While a typical gamer might not recognize that a vocal recording had been compressed, the gamer might nevertheless feel that there was something “not quite right” in the sound of the character’s voices. Compression of audio subtly changes the vocal sound in ways that are usually unnoticeable, but may become disruptive in a VR aural environment in which imperfections have the potential to nudge the audio into the Uncanny Valley.
Music in VR: Some Good News
While I’ve talked in this blog before about the importance of defining the role that music should play in the three-dimensional aural environment of a virtual reality game, Francis Rumsey offers an entirely different viewpoint in his talk. He thinks that when it comes to music, listeners don’t really care about spatial audio. That might be good news for game composers, because this may mean that music may play no role in the Uncanny Valley effect.
Describing a study that was conducted to determine how both naive and experienced listeners perceived spatial audio, Rumsey showed that when it came to listening to music, the spatial positioning wasn’t considered tremendously important. Sound quality was held to be absolutely crucial, but this desire was neither heightened nor lessened by spatial considerations. So does this mean that when it comes to music, listeners have an enhanced suspension of disbelief? Are they willing to accept music into their VR world, even if it isn’t realistically positioned within the 3D space? If so, then this would mean that non-diegetic music (i.e. music that isn’t occurring within the fictional world of the game) may not need to be spatially positioned as carefully as either voice or sound design elements of the aural environment. This may prove useful to audio teams, who may turn to music as a reassuring agent in the soundscape, binding the aural environment together and promoting emotional investment and immersion. However, music’s role in virtual reality may not conform to the way in which listeners react to spatially positioned music in other situations. At any rate, the issue certainly needs further study and experimentation to clarify the role that non-diegetic music should play in a VR game.
For other types of music in VR, the situation may be much simpler. Music doesn’t always have to occupy the traditional “underscore” role that it typically serves during gameplay. In a “music visualizer” VR experience, spatial positioning may become entirely unnecessary, because the music is serving the purpose of pure foreground entertainment (much the same way that music entertains listeners on its own). Here’s a preview of a musically-reactive virtual world in the upcoming “music visualizer” game Harmonix Music VR, created by the developer of the famous and popular game series Rock Band and Dance Central:
In Conclusion
Rumsey concluded his talk with the observation that near accurate may be worse than not particularly accurate… in other words, if it’s supposed to sound real, then it had better sound perfectly real. Otherwise, it might be better to opt for a stylized audio environment that exaggerates and heightens the world rather than faithfully reproducing it. I hope you enjoyed this blog, and please let me know what you think in the comments below!
Winifred Phillips is an award-winning video game music composer whose most recent project is the triple-A first person shooter Homefront: The Revolution. Her credits include five of the most famous and popular franchises in video gaming: Assassin’s Creed, LittleBigPlanet, Total War, God of War, and The Sims. She is the author of the award-winning bestseller A COMPOSER’S GUIDE TO GAME MUSIC, published by the Massachusetts Institute of Technology Press. As a VR game music expert, she writes frequently on the future of music in virtual reality video games. Follow her on Twitter @winphillips.
In psychology, the term ‘affect’ refers to emotion, particularly in terms of the way in which such emotional content is displayed. Whether by visual or aural means, an emotion can not be shared without some kind of ‘affect’ that serves as its mode of communication from one person to another. When we’re happy, we smile. When we’re angry, we frown.
My work as a video game composer has lately included some projects for virtual reality games (more info on that in the coming months), and as a result I’ve been thinking a lot about the awesome potential of VR, and 
This March at the famous 
Recently, I’ve been composing music for some VR projects (more info on that soon), so I’ve been giving virtual reality a lot of thought, and have simultaneously been writing some articles here on the subject. Those have included a discussion of 
Since I’ve been working recently on music for a Virtual Reality project (more info in the coming months), I’ve been thinking a lot about VR technology and its effect on the creative process. Certainly, VR is going to be a great environment in which to be creative and perform tasks and skills with enhanced focus, 
Simple VR technologies like the popular Google Cardboard headset can be a lot of fun – 
