Welcome back to our three article series dedicated to collecting and exploring the ideas that were discussed in five different GDC 2017 audio talks about interactive music! These five speakers shared ideas they’d developed in the process of creating interactivity in the music of their own game projects. We’re looking at these ideas side-by-side to cultivate a sense of the “bigger picture” when it comes to the leading-edge thinking for music interactivity in games. In the first article, we looked at the basic nature of five interactive music systems discussed in these five GDC 2017 presentations:
Okay, so let’s now contemplate some simple but important questions: why were those systems used? What was attractive about each interactive music strategy, and what were the challenges inherent in using those systems?
By video game music composer Winifred Phillips | Contact | Follow
The 2017 Game Developers Conference could be described as a densely-packed deep-dive exploration of the state-of-the-art tools and methodologies used in modern game development. This description held especially true for the game audio track, wherein top experts in the field offered a plethora of viewpoints and advice on the awesome technical and artistic challenges of creating great sound for games. I’ve given GDC talks for the past three years now (see photo), and every year I’m amazed at the breadth and diversity of the problem-solving approaches discussed by my fellow GDC presenters. Often I’ll emerge from the conference with the impression that we game audio folks are all “doing it our own way,” using widely divergent strategies and tools.
This year, I thought I’d write three articles to collect and explore the ideas that were discussed in five different GDC audio talks. During their presentations, these five speakers all shared their thoughts on best practices and methods for instilling interactivity in modern game music. By absorbing these ideas side-by-side, I thought we might gain a sense of the “bigger picture” when it comes to the current leading-edge thinking for music interactivity in games. In the first article, we’ll look at the basic nature of these interactive systems. We’ll devote the second article to the pros and cons of each system, and in the third article we’ll look at tools and tips shared by these music interactivity experts. Along the way, I’ll also be sharing my thoughts on the subject, and we’ll take a look at musical examples from some of my own projects that demonstrate a few ideas explored in these GDC talks:
I was pleased to give a talk about composing music for games at the 2016 Game Developers Conference (pictured left). GDC took place this past March in San Francisco – it was an honor to be a part of the audio track again this year, which offered a wealth of awesome educational sessions for game audio practitioners. So much fun to see the other talks and learn about what’s new and exciting in the field of game audio! In this blog, I want to share some info that I thought was really interesting from two talks that pertained to the audio production side of game development: composer Laura Karpman’s talk about “Composing Virtually, Sounding Real” and audio director Garry Taylor’s talk on “Audio Mastering for Interactive Entertainment.” Both sessions had some very good info for video game composers who may be looking to improve the quality of their recordings. Along the way, I’ll also be sharing a few of my own personal viewpoints on these music production topics, and I’ll include some examples from one of my own projects, the Ultimate Trailers album for West One Music, to illustrate ideas that we’ll be discussing. So let’s get started!
As a speaker in the audio track of the Game Developers Conference this year, I enjoyed taking in a number of GDC audio sessions — including a couple of presentations that focused on the future of interactive music in games. I’ve explored this topic before at length in my book (A Composer’s Guide to Game Music), and it was great to see that the game audio community continues to push the boundaries and innovate in this area! Interactive music is a worthwhile subject for discussion, and will undoubtedly be increasingly important in the future as dynamic music systems become more prevalent in game projects. With that in mind, in this blog I’d like to share my personal takeaway from two sessions that described very different approaches to musical interactivity. After that, we’ll discuss one of my experiences with interactive music for the video game Spore Hero from Electronic Arts (pictured above).
Baldursson began the presentation by explaining why an intelligent music system for games can be a necessity. “We basically want an intelligent music system because we can’t (or maybe shouldn’t really) precompose all of the elements,” Baldursson explains. He describes the conundrum of creating a musical score for a game whose story is still fluid and changeable, and then asserts, “I think we should find ways of making this better.”
This past weekend, the Audio Engineering Society held its annual North American convention in the Jacob Javits Center in New York City. I was participating as an AES speaker, but I also knew that AES includes an exhibit floor packed with the best professional audio equipment from all the top manufacturers, and I didn’t want to miss that! So, in between my game audio panel presentation on Saturday, and the Sunday tutorial talk I gave on the music system of the LittleBigPlanet franchise, I had the pleasure of searching the show floor for what’s new and interesting in audio tech. Here are some of the attractions that seemed most interesting for game audio folks:
One of the most interesting technologies on display at AES this year was Fraunhofer Cingo – an audio encoding technology developed specifically to enable mobile devices to deliver immersive sound for movies, games and virtual reality. Cingo was developed by the institute responsible for the MP3 audio coding format. According to Fraunhofer, the Cingo technology “supports rendering of 3D audio content with formats that add a height dimension to the sound image, such as 9.1, 11.1 or other channel combinations.” This enables mobile devices to emulate “the enveloping sound of movies, games or any other virtual environment.” While I was there, Fraunhofer rep Jennifer Utley gave me the chance to demo the Cingo technology using the Gear VR headset, which turns Samsung mobile phones into portable virtual reality systems. The sound generated by Cingo did have an awesome sense of spatial depth that increased immersion, although I didn’t personally notice the height dimension in the spatial positioning. Nevertheless, it was pretty nifty!
This is a continuation of my blog series on the top audio middleware options for game music composers, this time focusing on the psai Interactive Music Engine for games, developed by Periscope Studio, an audio/music production house. Initially developed as a proprietary middleware solution for use by Periscope’s in-house musicians, the software is now being made available commercially for use by game composers. In this blog I’ll take a quick look at psai and provide some tutorial resources that will further explore the utility of this audio middleware. If you’d like to read the first four blog entries in this series on middleware for the game composer, you can find them here:
The name “psai” is an acronym for “Periscope Studio Audio Intelligence,” and its lowercase appearance is intentional. Like the Elias middleware (explored in a previous installment of this blog series), the psai application attempts to provide a specialized environment specifically tailored to best suit the needs of game composers. The developers at Periscope Studio claim that psai’s “ease of use is unrivaled,” primarily because the middleware was “designed by videogame composers, who found that the approaches of conventional game audio middleware to interactive music were too complicated and not flexible enough.” The psai music engine was originally released for PC games, with a version of the software for the popular Unity engine released in January 2015.
psai graphical user interface
Both Elias and psai offer intuitive graphical user interfaces designed to ease the workflow of a game composer. However, unlike Elias, which focused exclusively on a vertical layering approach to musical interactivity, the psai middleware is structured entirely around horizontal re-sequencing, with no support for vertical layering. As I described in my book, A Composer’s Guide to Game Music, “the fundamental idea behind horizontal re-sequencing is that when composed carefully and according to certain rules, the sequence of a musical composition can be rearranged.” (Chapter 11, page 188).
Music for the psai middleware is composed in what Periscope describes as a “snippets” format, in which short chunks of music are arranged into groups that can then be triggered semi-randomly by the middleware. The overall musical composition is called a “theme,” and the snippets represent short sections of that theme. The snippets are assigned numbers that best represent degrees of emotional intensity (from most intense to most relaxed), and these intensity numbers help determine which of the snippets will be triggered at any given time. Other property assignments include whether a snippet is designated as an introductory or ending segment, or whether the snippet is bundled into a “middle” group with a particular intensity designation. Periscope cautions, “The more Middle Segments you provide, the more diversified your Theme will be. The more Middle Segments you provide for a Theme, the less repetition will occur. For a highly dynamic soundtrack make sure to provide a proper number of Segments across different levels of intensity.”
Here’s an introductory tutorial video produced by Periscope for the psai Interactive Music Engine for videogames:
Because psai only supports horizontal re-sequencing, it’s not as flexible as the more famous tools such as Wwise or FMOD, which can support projects that alternate between horizontal and vertical interactivity models. However, psai’s ease of use may prove alluring for composers who had already planned to implement a horizontal re-sequencing structure for musical interactivity. The utility of the psai middleware also seems to depend on snippets that are quite short, as is demonstrated by the above tutorial video produced by Periscope Studio. There could be some negative effects of this structure on a composer’s ability to develop melodic content (as is sometimes the case in a horizontal re-sequencing model). It would be helpful if Periscope could demonstrate psai using longer snippets that might give us a better sense of how musical ideas might be developed within the confines of their dynamic music system. One can imagine an awesome potential for creativity with this system, if the structure can be adapted to allow for more development of musical ideas over time.
And here is some gameplay video that demonstrates the music system of Mount & Blade II – Bannerlord:
Winifred Phillips is an award-winning video game music composer whose most recent project is the triple-A first person shooter Homefront: The Revolution. Her credits include five of the most famous and popular franchises in video gaming: Assassin’s Creed, LittleBigPlanet, Total War, God of War, and The Sims. She is the author of the award-winning bestseller A COMPOSER’S GUIDE TO GAME MUSIC, published by the Massachusetts Institute of Technology Press. As a VR game music expert, she writes frequently on the future of music in virtual reality video games. Follow her on Twitter @winphillips.
Welcome back to my blog series that offers tutorial resources exploring game music middleware for the game music composer. I initially planned to write two blog entries on the most popular audio middleware solutions (Wwise and FMOD), but since I started this blog series, I’ve been hearing buzz about other middleware solution and so I thought it best to expand the series to incorporate other interesting solutions to music implementation in games. This blog will focus on a brand new middleware application called Elias, developed by Elias Software. While not as famous as Wwise or FMOD, this new application offers some intriguing new possibilities for the creation of interactive music in games.
If you’d like to read the first three blog entries in this series, you can find them here:
Elias stands for Elastic Lightweight Integrated Audio System. It is developed by Kristofer Eng and Philip Bennefall for Microsoft Windows, with a Unity plugin for consoles, mobile devices and browser-based games. What makes Elias interesting is the philosophy of its design. Instead of designing a general audio middleware tool with some music capabilities, Eng and Bennefall decided to bypass the sound design arena completely and create a middleware tool specifically outfitted for the game music composer. The middleware comes with an authoring tool called Elias Composer’s Studio that “helps the composer to structure and manage the various themes in the game and bridges the gap between the composer and level designer to ease the music integration process.”
Here’s the introductory video for Elias, produced by Elias Software:
The interactive music system of the Elias middleware application seems to favor a Vertical Layering (or vertical re-orchestration) approach with a potentially huge number of music layers able to play in lots of combinations. The system includes flexible options for layer triggering, including the ability to randomize the activation of the layers to keep the listening experience unpredictable during gameplay.
As a middleware application designed specifically to address the top needs of game music composers, Elias is certainly intriguing! The software has so far been used in only one published game – Gauntlet, which is the latest entry in the awesome video game franchise first developed by Atari Games for arcade cabinets in 1985. This newest entry in the franchise was developed by Arrowhead Game Studios for Windows PCs. We can hear the Elias middleware solution in action in this gameplay video from Gauntlet:
Elias Software recently demonstrated its Elias middleware application on the expo floor of the Nordic Game 2015 conference in Malmö, Sweden (May 20-22, 2015). Here’s a look at Elias’ booth from the expo:
Since Elias is a brand new application, I’ll be curious to see how widely it is accepted by the game audio community. A middleware solution that focuses solely on music is definitely a unique approach! If audio directors and audio programmers embrace Elias, then it may have the potential to give composers better tools and an easier workflow in the creation of interactive music for games.
At least, that was my impression a couple of months ago when I attended the audio track of the Game Developers Conference. Setting a new record for attendance, GDC hosted over 24,000 game industry pros who flocked to San Francisco’s Moscone Center in March for a full week of presentations, tutorials, panels, awards shows, press conferences and a vibrant exposition floor filled with new tech and new ideas. As one of those 24,000 attendees, I enjoyed meeting up with lots of my fellow game audio folks, and I paid special attention to the presentations focusing on game audio. Amongst the tech talks and post-mortems, I noticed a lot of buzz about a subject that used to be labeled as very old-school: MIDI.
This was particularly emphasized by all the excitement surrounding the new MIDI capabilities in the Wwise middleware. In October of 2014, Wwise released its most recent version (2014.1) which introduced a number of enhanced features, including “MIDI support for interactive music and virtual instruments (Sampler and Synth).” Wwise now allows the incorporation of MIDI that triggers either a built-in sound library in Wwise or a user-created one. Since I talk about the future of MIDI game music in my book, A Composer’s Guide to Game Music, and since this has become a subject of such avid interest in our community, I thought I’d do some research on this newest version of Wwise and post a few resources that could come in handy for any of us interested in embarking in a MIDI game music project using Wwise 2014.1.
The first is a video produced by Damian Kastbauer, technical audio lead at PopCap games and the producer and host of the now-famous Game Audio Podcast series. This video was released in April of 2014, and included a preview of the then-forthcoming MIDI and synthesizer features of the new Wwise middleware tool. In this video, Damian takes us through the newest version of the “Project Adventure” tutorial prepared by Audiokinetic, makers of Wwise. In the process, he gives us a great, user-friendly introduction to the MIDI capabilities of Wwise.
The next videos were produced by Berrak Nil Boya, a composer and contributing editor to the Designing Sound website. In these videos, Berrak has taken us through some of the more advanced applications of the MIDI capabilities of Wwise, starting with the procedure for routing MIDI data directly into Wwise from more traditional MIDI sequencer software such as that found in a Digital Audio Workstation (DAW) application. This process would allow a composer to work within more traditional music software and then directly route the MIDI output into Wwise. Berrak takes us through the process in this two-part video tutorial:
Finally, Berrak Nil Boya has created a video tutorial on the integration of Wwise into Unity 5, using MIDI. Her explanation of the preparation of a soundbank and the association of MIDI note events with game events is very interesting, and provides a nicely practical application of the MIDI capability of Wwise.
Welcome back to my series of blogs that collect some tutorial resources about game music middleware for the game music composer. I had initially intended to publish two blog entries on this subject, focusing on the most popular audio middleware solutions: Wwise and FMOD. However, since the Fabric audio middleware has been making such a splash in the game audio community, I thought I’d extend this series to include it. If you’d like to read the first two blog entries in this series, you can find them here:
Fabric is developed by Tazman Audio for the Unity game engine (which enables game development for consoles, PCs, mobile devices such as iOS and Android, and games designed to run within a web browser). Here’s a Unity game engine overview produced by Unity Technologies:
The Fabric middleware is designed to expand the audio capabilities of the Unity game engine. The complete product manual for the Fabric middleware is available online. The video tutorials that I’m featuring below were created by two game audio professionals who have very generously walked us through the use of the software. If you’d like a more nuts-and-bolts overview of the software features of Fabric, you can find it here.
The first video was shot in 2013 during the Konsoll game development conference in Norway, and gives an overview of the general use of Fabric in game audio. The speaker, Jory Prum, is an accomplished game audio professional whose game credits include The Walking Dead, The Wolf Among Us, Broken Age, SimCity 4, Star Wars: Knights of the Old Republic, and many more.
Making a great sounding Unity game using Fabric
In the next two-part video tutorial, composer Anastasia Devana has expanded on her previous instructional videos about FMOD Studio, focusing now on recreating the same music implementation strategies and techniques using the Fabric middleware in Unity. Anastasia Devana is an award-winning composer whose game credits include the recently released puzzle game Synergy and the upcoming roleplaying game Anima – Gate of Memories.
Fabric and Unity: Adaptive Music in Angry Bots – Part 1
Fabric and Unity: Adaptive Music in Angry Bots – Part 2
In a previous blog post, we took a look at a few tutorial resources for the latest version of the Wwise audio middleware. One of the newest innovations in the Wwise software package is a fairly robust MIDI system. This system affords music creators and implementers the opportunity to avail themselves of the extensive adaptive possibilities of the MIDI format from within the Wwise application. Last month, during the Game Developers Conference in the Moscone Center in San Francisco, some members of the PopCap audio development team presented a thorough, step-by-step explanation of the benefits of this MIDI capability for one of their latest projects, Peggle Blast. Since my talk during the Audio Bootcamp at GDC focused on interactive music and MIDI (with an eye on the role of MIDI in both the history and future of game audio development), I thought that we could all benefit from a summation of some of the ideas discussed during the Peggle Blast talk, particularly as they relate to dynamic MIDI music in Wwise. In this blog, I’ve tried to convey some of the most important takeaways from this GDC presentation.
“Peggle Blast: Big Concepts, Small Project” was presented on Thursday, March 5th by three members of the PopCap audio team: technical sound designer RJ Mattingly, audio lead Jaclyn Shumate, and senior audio director Guy Whitmore. The presentation began with a quote from Igor Stravinsky:
The more constraints one imposes, the more one frees oneself, and the arbitrariness of the constraint serves only to maintain the precision of the execution.
This idea became a running theme throughout the presentation, as the three audio pros detailed the constraints under which they worked, including:
A 5mb memory limit for all audio assets
Limited CPU
2.5mb memory allocation for the music elements
These constraints were a result of the mobile platforms (iOS and Android) for which Peggle Blast had been built. For this reason, the music team focused their attention on sounds that could convey lots of emotion while also maintaining a very small file size. Early experimentation with tracks structured around the use of a music box instrument led the team to realize that they still needed to replicate the musical experience from the full-fledged console versions of the game. A simple music-box score was too unsatisfying, particularly for players who were familiar with the music from the previous installments in the franchise. With that in mind, the team concentrated on very short orchestral samples taken from the previous orchestral session recordings for Peggle 2. Let’s take a look at a video from those orchestral sessions:
Using those orchestral session recordings, the audio team created custom sample banks that were tailored specifically to the needs of Peggle Blast, focusing on lots of very short instrument articulations and performance techniques including:
pizzicato
marcato
staccato
mallets
A few instruments (including a synth pad and some orchestral strings) were edited to loop so that extended note performances became possible, but the large majority of instruments remained brief, punctuated sounds that did not loop. These short sounds were arranged into sample banks in which one or two note samples would be used per octave of instrument range, and note tracking would transpose the sample to fill in the rest of the octave. The sample banks consisted of a single layer of sound, which meant that the instruments did not adjust their character depending on dynamics/velocity. In order to make the samples more musically pleasing, the built-in digital signal processing capability of Wwise was employed by way of a real-time reverb bus that allowed these short sounds to have more extended and natural-sounding decay times.
The audio team worked with a beta version of Wwise 2014 during development of Peggle Blast, which allowed them to implement their MIDI score into the Unity game engine. The composer, Guy Whitmore, composed the music in a style consisting of whimsically pleasant, non-melodic patterns that were structured into a series of chunks. These chunks could be triggered according to the adaptive system in Peggle Blast, wherein the music went through key changes (invariably following the circle of fifths) in reaction to the player’s progress. To better see how this works, let’s watch an example of some gameplay from Peggle Blast:
As you can see, very little in the way of a foreground melody existed in this game. In the place of a melody, foreground musical tones would be emitted when the Peggle ball hit pegs during its descent from the top of the screen. These tones would follow a predetermined scale, and would choose which type of scale to trigger (major, natural minor, harmonic minor, or mixolydian) depending on the key in which the music was currently playing. Information about the key was dropped into the music using markers that indicated where key changes took place, so that the Peggle ball would always trigger the correct type of scale at any given time. The MIDI system did not have to store unique MIDI data for scales in every key change, but would instead calculate the key transpositions for each of the scale types, based on the current key of the music that was playing.
The presentation ended with an emphasis on the memory savings and flexibility afforded by MIDI, and the advantages that MIDI presents to game composers and audio teams. It was a very interesting presentation! If you have access to the GDC Vault, you can watch a video of the entire presentation online. Otherwise, there are plenty of other resources on the music of Peggle Blast, and I’ve included a few below:
The 2017 Game Developers Conference could be described as a densely-packed deep-dive exploration of the state-of-the-art tools and methodologies used in modern game development. This description held especially true for the game audio track, wherein top experts in the field offered a plethora of viewpoints and advice on the awesome technical and artistic challenges of creating great sound for games. I’ve given GDC talks for 
I was pleased to give 
As a speaker in the audio track of the 
