Since I’ve been working recently on music for a Virtual Reality project (more info in the coming months), I’ve been thinking a lot about VR technology and its effect on the creative process.  Certainly, VR is going to be a great environment in which to be creative and perform tasks and skills with enhanced focus, according to this article from the VR site SingularityHub.  I’ve written in this blog before about the role that music and sound will play in the Virtual Reality gaming experience.  It’s clear that music will have an impact on the way in which we experience VR, not only during gaming experiences, but also when using the tools of VR to create and be productive.  With that in mind, let’s consider if the opposite statement may also be true – will VR impact the way in which we experience music, not only as listeners, but also as video game composers?

Simple VR technologies like the popular Google Cardboard headset can be a lot of fun – as I personally experienced recently (photo to the left).  However, they offer only the rudimentary visual aspects, which omits some of the most compelling aspects of the VR experience.  When motion tracking (beyond simple head movement) is added to the mix, the potential of VR explodes.  Over the next three articles, we’ll be exploring some interesting possibilities created by the motion tracking capabilities of VR, and how this might alter our creative process.  In the first article, we’ll have some fun exploring new ways to play air guitars and air drums in the VR environment. In the second article, we’ll take a look at ways to control virtual instruments and sound modules that are folded into the VR software.  And finally, in the third article we’ll explore the ways in which VR motion tracking is allowing us to immersively control our existing real-world instruments using MIDI. But first, let’s take a look at the early days of VR musical technology!

Early experiments

In the realm of science fiction, motion tracking is depicted as a slick, ultra-cool affair featuring futuristic graphics and lots of elaborate gestures that are tracked with pinpoint accuracy.  Long before the technology had reached consumers, sci-fi movies like Minority Report were offering up visions of motion tracking that looked like this:

Looks awesome, but what does it have to do with music makers like ourselves?  Well, virtual reality and motion tracking in connection to musical performance has been a subject of interest in scholarly research for awhile now.  Long before Palmer Luckey cobbled together his first Oculus Rift prototype in his parent’s garage in 2011, researchers were looking at VR as a way to revolutionize how music is created and played.  In 2005, a paper entitled “Experiments with Virtual Reality Instruments” examined how virtual instruments could be controlled using motion tracking inside a VR environment.

In their experiments they used a virtual environment called EVE (pictured left), consisting of a room of walls covered with projected images rendered three dimensional via the use of stereo shutter glasses.  In this environment, they experimented with virtual xylophones, synthesizers, guitars, and an abstract virtual membrane designed to emit different tones depending on where it was struck.  These experiments showed that users could have a lot of fun playing virtual instruments in a VR environment, and the virtual guitar was a particular hit with the participants in the study.  However, the limitations and costliness of the EVE environment prevented these experiments from reaching any practical applications.

Today’s VR headsets have transcended those limits, so let’s take a look at what can be done with today’s VR tech!

Air Guitar 2.0

The spectacle of a truly accomplished and creative air guitar performance is a wonder to behold, and VR is adding a new dimension to this time-honored discipline… but before we get into that, let’s first explore the arcane mysteries of air guitar showmanship.  For those of us who have already shredded an imaginary guitar, no further explanation is needed. But for those who have not done so… the situation must be immediately remedied.  Here is a useful instructional video on the art and science of air guitar:

So, now that we all have a deeper appreciation for the techniques of air guitar performance, let’s discuss how Virtual Reality has brought about a revolutionary change to this venerable tradition.  While we’ll still be playing our imaginary guitars by wiggling our fingers in the air, VR now allows us to actually produce the sound of a guitar by doing this.  Created by software developer Zach Kinstner of Aesthetic Interactive, the VR Guitar app lets us strum the glowing strings of a virtual guitar inside the VR space.  The app works with the Leap Motion controller, which was first created to add motion control to computer software.

Since its inception, Leap Motion has now been reconfigured to work with VR headsets, employing a new software driver called Orion.  In order to enable the motion tracking capabilities of Leap Motion Orion, the user affixes a small USB device containing two cameras and three infrared LED lights to the front of the headset, allowing the Leap Motion to detect the user’s hand movements and relay these back to the VR application.  Here’s a video that shows Leap Motion Orion in action within a demo application called “Blocks VR.”

Now that we’ve seen what Leap Motion Orion can do with some simple blocks, what can it do for us air guitar enthusiasts?  Well, with a decidedly Tron-like aesthetic and an interface that has us positioning our hands forward rather than at the traditional “guitar-holding” angles, the VR Guitar may interfere a little with the normal showmanship of an air guitar performance.  That being said, the ability to actually produce a guitar sound could quite possibly offset the visual drawbacks, and with a bit of creativity I’m sure a clever user could find ways to make performances look suitably rock & roll.  Here’s a video produced by Kinstner to demonstrate his VR Guitar app:

Air Drums

While air guitarists get all the glory and the groupies, air drummers are certainly not in short supply.  Does VR offer anything exciting for them? Actually, it does… although perhaps we should take a moment to appreciate the fine art of air drumming before we see how VR will enhance the experience.  Famous comedian Rowan Atkinson (best known as the indomitable Mr. Bean) has admirably demonstrated air drumming in this memorable comedy sketch:

So, how will VR possibly top that?  By putting the experience of playing magical invisible drums in all our hands.

With the Aerodrums app, an aspiring VR air drummer can use a set of drumsticks with silver balls attached, allowing a Sony PlayStation 3 Eye camera to track the movements of the sticks and relay those movements to the Aerodrums application. Aerodrums was developed by Yann Morvan and Richard Lee, former researchers at the Trinity College Dublin Vision and Visualization lab.  “We tried to stay as close as possible to the drum kit when designing Aerodrums,” says Yann Morvan in an interview with Drum Magazine.  “We wanted existing drummers to be able to adapt easily, and for people who want to learn drums to do so using Aerodrums.”

While Aerodrums was initially developed to run on a traditional PC, it really comes into its own when it’s running on the Oculus Rift.  While wearing the VR headset, we can see the virtual drum set arrayed all around us, and we can play it in the same way that we’d play an actual drum set… without the tactile feedback, of course.  The software seems to compensate for this by showing the drums pulsing a bit when the virtual sticks strike them, giving a sense of the moment of impact. Here’s a video showing Aerodrums being demonstrated at the NAMM 2016 convention:

Music in the Air

So, now we’ve taken a look at how musical performance and improvisation might be undertaken within VR, and how motion tracking technologies make it possible to play imaginary instruments in a virtual world.  In the next blog post, we’ll be exploring how to play actual instruments using VR technologies… more on that next time!

Winifred Phillips is an award-winning video game music composer whose most recent project is the triple-A first person shooter Homefront: The Revolution.  Her other credits include five of the most famous and popular franchises in video gaming: Assassin’s Creed, LittleBigPlanet, Total War, God of War, and The Sims.  She is the author of the award-winning bestseller A COMPOSER’S GUIDE TO GAME MUSIC, published by the Massachusetts Institute of Technology Press.  As a VR game music expert, she writes frequently on the future of music in virtual reality video games.  Follow her on Twitter @winphillips.