Welcome to part two of our ongoing exploration of some interesting possibilities created by the motion tracking capabilities of VR, and how this might alter our creative process as video game composers.
In part one we discussed how motion tracking lets us be awesome air guitarists and drummers inside the virtual space. In this article, we’ll be taking a look at how the same technology will allow us to make interesting music using more serious tools that are incorporated directly inside the VR environment – musical instruments that exist entirely within the VR ‘machine.’
Our discussion to follow will concentrate on three software applications: Soundscape, Carillon, and Lyra. Later, in the third article of this ongoing series, we’ll take a look at applications that allow our VR user interfaces to harness the power of MIDI to control some of the top music devices and software that we use in our external production studios. But first, let’s look at the ways that VR apps can function as fully-featured musical instruments, all on their own!
Soundscape
Let’s start with something simple – a step sequencer with a sound bank and signal processing tools, built for the mobile virtual reality experience of the Samsung Gear VR.
I got a chance to demo the Samsung Gear VR during the Audio Engineering Society Convention in NYC last year, and while it doesn’t offer the best or most mind-blowing experience in VR (such as what we can experience from products like the famous Oculus Rift), it does achieve a satisfying level of immersion. Plus, it’s great fun! The Soundscape VR app was built for Samsung Gear VR by developer Sander Sneek of the Netherlands. It’s a simple app designed to enable users to create dance loops using three instruments from a built-in electro sound library, a pentatonic step sequencer that enables the user to create rhythm and tone patterns within the loops, and a collection of audio signal processing effects that let the user warp and mold the sounds as the loops progress, adding variety to the performance.
This kind of app isn’t particularly new, and the creative tools are limited in their scope and flexibility, but the addition of a virtual reality environment (pictured left) has the potential to give the experience a more tactile immediacy that some musicians may find creatively inspiring.
“As a musician, music should always play an important role, since in VR audio makes 50% of the experience,” Sneek comments in an interview with The Creators Project. “Music and specifically creating music via software has always been a major interest of mine.”
Here’s a video demonstrating the software in action:
Carillon
Traditionally speaking, a Carillon is a large array of interconnected bells housed inside a church’s bell tower, usually played via a keyboard and pedals. It’s a grand instrument with an ancient tradition. In 2015, two Stanford University professors set out to create a highly stylized virtual version of this instrument that could be played via motion tracking within the Oculus Rift. Dr. Rob Hamilton of Stanford University and Chris Platz (Stanford’s Visiting Scholar at the Center for Computer Research in Music and Acoustics) collaborated on the project.
“Each performer’s avatar stands on a platform within the Carillon, looking up at the giant spinning rings in the center.” Dr. Hamilton describes the Carillon during an interview with the Leap Motion Blog in which he paints a picture of how the virtual instrument looks to the musicians interacting with it inside the Oculus Rift. “Directly in front of them are a smaller set of rings with which they can interact – holding a hand over a ring or set of rings selects them, swiping gestures left/right, up/down, away/toward start the rings spinning.”
Swiping at the rings of the elaborate virtual instrument (pictured left) produces many sounds associated with the traditional Carillon, but these reverberant tones are actually generated by the Pure Data Music Engine – a programming environment used often for generative music applications. By employing the Oculus Rift and the Leap Motion device for motion tracking, Hamilton and Platz created a new, intriguing way to interact with a sonic construct assembled in Pure Data.
“The idea of a mechanical instrument that responds to performers’ motions has been something we really enjoy,” commented Dr. Hamilton. “As we all become more comfortable with technology in our daily lives, it will seem less and less strange to see it blending with traditional musical performance practice, instrument design and concert performance.”
Here’s a video showing the Carillon in a live performance. The musicians are Dr. Rob Hamilton and Stanford Assistant Professor Ge Wang:
Lyra VR
On January 28th 2016, the winners of the Leap Motion 3D Jam 2.0 Awards were announced. Leap Motion 3D Jam is organized by the makers of the popular Leap Motion device for movement tracking in VR. This year, 1st prize in the competition went to an innovative musical instrument application called Lyra VR.
In Lyra VR, musicians and composers can manipulate a built-in library of sounds and loops to create their own compositions using a three dimensional interface within the Oculus Rift (pictured right). Unlike Carillon, which is limited to a single instrument, Lyra VR presents an array of choices and possibilities. The application was designed by Team Metanaut, led by Taiwanese VR developer Dilun Ho and an international team of developers that includes electronic musician Jean Marais (better known as MoShang). With floating buttons and many-sided geometrical shapes that glow softly in neon colors reminiscent of the world of Tron, Lyra VR is an immersive musical playground. While the instrument library may not provide a broad variety of sounds, the gestural control method could possibly afford users a sense of creative freedom and inspiration. Here’s a video of Lyra VR in action:
Music Inside the Machine
So, now we’ve explored some interesting applications that turn our VR headsets into standalone musical instruments with virtual interfaces we can manipulate with a wave of our hands, making us feel like expert VR musicians. In the third and blog post, we’ll look at how VR can reach out beyond the confines of its own applications and pull our preexisting musical instruments into the virtual world via the power of MIDI!
Winifred Phillips is an award-winning video game music composer whose most recent project is the triple-A first person shooter Homefront: The Revolution. Her credits include five of the most famous and popular franchises in video gaming: Assassin’s Creed, LittleBigPlanet, Total War, God of War, and The Sims. She is the author of the award-winning bestseller A COMPOSER’S GUIDE TO GAME MUSIC, published by the Massachusetts Institute of Technology Press. As a VR game music expert, she writes frequently on the future of music in virtual reality video games. Follow her on Twitter @winphillips.