As a speaker in the audio track of the Game Developers Conference this year, I enjoyed taking in a number of GDC audio sessions — including a couple of presentations that focused on the future of interactive music in games. I’ve explored this topic before at length in my book (A Composer’s Guide to Game Music), and it was great to see that the game audio community continues to push the boundaries and innovate in this area! Interactive music is a worthwhile subject for discussion, and will undoubtedly be increasingly important in the future as dynamic music systems become more prevalent in game projects. With that in mind, in this blog I’d like to share my personal takeaway from two sessions that described very different approaches to musical interactivity. After that, we’ll discuss one of my experiences with interactive music for the video game Spore Hero from Electronic Arts (pictured above).
Baldursson began the presentation by explaining why an intelligent music system for games can be a necessity. “We basically want an intelligent music system because we can’t (or maybe shouldn’t really) precompose all of the elements,” Baldursson explains. He describes the conundrum of creating a musical score for a game whose story is still fluid and changeable, and then asserts, “I think we should find ways of making this better.”
This is a continuation of my blog series on the top audio middleware options for game music composers, this time focusing on the psai Interactive Music Engine for games, developed by Periscope Studio, an audio/music production house. Initially developed as a proprietary middleware solution for use by Periscope’s in-house musicians, the software is now being made available commercially for use by game composers. In this blog I’ll take a quick look at psai and provide some tutorial resources that will further explore the utility of this audio middleware. If you’d like to read the first four blog entries in this series on middleware for the game composer, you can find them here:
The name “psai” is an acronym for “Periscope Studio Audio Intelligence,” and its lowercase appearance is intentional. Like the Elias middleware (explored in a previous installment of this blog series), the psai application attempts to provide a specialized environment specifically tailored to best suit the needs of game composers. The developers at Periscope Studio claim that psai’s “ease of use is unrivaled,” primarily because the middleware was “designed by videogame composers, who found that the approaches of conventional game audio middleware to interactive music were too complicated and not flexible enough.” The psai music engine was originally released for PC games, with a version of the software for the popular Unity engine released in January 2015.
psai graphical user interface
Both Elias and psai offer intuitive graphical user interfaces designed to ease the workflow of a game composer. However, unlike Elias, which focused exclusively on a vertical layering approach to musical interactivity, the psai middleware is structured entirely around horizontal re-sequencing, with no support for vertical layering. As I described in my book, A Composer’s Guide to Game Music, “the fundamental idea behind horizontal re-sequencing is that when composed carefully and according to certain rules, the sequence of a musical composition can be rearranged.” (Chapter 11, page 188).
Music for the psai middleware is composed in what Periscope describes as a “snippets” format, in which short chunks of music are arranged into groups that can then be triggered semi-randomly by the middleware. The overall musical composition is called a “theme,” and the snippets represent short sections of that theme. The snippets are assigned numbers that best represent degrees of emotional intensity (from most intense to most relaxed), and these intensity numbers help determine which of the snippets will be triggered at any given time. Other property assignments include whether a snippet is designated as an introductory or ending segment, or whether the snippet is bundled into a “middle” group with a particular intensity designation. Periscope cautions, “The more Middle Segments you provide, the more diversified your Theme will be. The more Middle Segments you provide for a Theme, the less repetition will occur. For a highly dynamic soundtrack make sure to provide a proper number of Segments across different levels of intensity.”
Here’s an introductory tutorial video produced by Periscope for the psai Interactive Music Engine for videogames:
Because psai only supports horizontal re-sequencing, it’s not as flexible as the more famous tools such as Wwise or FMOD, which can support projects that alternate between horizontal and vertical interactivity models. However, psai’s ease of use may prove alluring for composers who had already planned to implement a horizontal re-sequencing structure for musical interactivity. The utility of the psai middleware also seems to depend on snippets that are quite short, as is demonstrated by the above tutorial video produced by Periscope Studio. There could be some negative effects of this structure on a composer’s ability to develop melodic content (as is sometimes the case in a horizontal re-sequencing model). It would be helpful if Periscope could demonstrate psai using longer snippets that might give us a better sense of how musical ideas might be developed within the confines of their dynamic music system. One can imagine an awesome potential for creativity with this system, if the structure can be adapted to allow for more development of musical ideas over time.
And here is some gameplay video that demonstrates the music system of Mount & Blade II – Bannerlord:
Winifred Phillips is an award-winning video game music composer whose most recent project is the triple-A first person shooter Homefront: The Revolution. Her credits include five of the most famous and popular franchises in video gaming: Assassin’s Creed, LittleBigPlanet, Total War, God of War, and The Sims. She is the author of the award-winning bestseller A COMPOSER’S GUIDE TO GAME MUSIC, published by the Massachusetts Institute of Technology Press. As a VR game music expert, she writes frequently on the future of music in virtual reality video games. Follow her on Twitter @winphillips.
At least, that was my impression a couple of months ago when I attended the audio track of the Game Developers Conference. Setting a new record for attendance, GDC hosted over 24,000 game industry pros who flocked to San Francisco’s Moscone Center in March for a full week of presentations, tutorials, panels, awards shows, press conferences and a vibrant exposition floor filled with new tech and new ideas. As one of those 24,000 attendees, I enjoyed meeting up with lots of my fellow game audio folks, and I paid special attention to the presentations focusing on game audio. Amongst the tech talks and post-mortems, I noticed a lot of buzz about a subject that used to be labeled as very old-school: MIDI.
This was particularly emphasized by all the excitement surrounding the new MIDI capabilities in the Wwise middleware. In October of 2014, Wwise released its most recent version (2014.1) which introduced a number of enhanced features, including “MIDI support for interactive music and virtual instruments (Sampler and Synth).” Wwise now allows the incorporation of MIDI that triggers either a built-in sound library in Wwise or a user-created one. Since I talk about the future of MIDI game music in my book, A Composer’s Guide to Game Music, and since this has become a subject of such avid interest in our community, I thought I’d do some research on this newest version of Wwise and post a few resources that could come in handy for any of us interested in embarking in a MIDI game music project using Wwise 2014.1.
The first is a video produced by Damian Kastbauer, technical audio lead at PopCap games and the producer and host of the now-famous Game Audio Podcast series. This video was released in April of 2014, and included a preview of the then-forthcoming MIDI and synthesizer features of the new Wwise middleware tool. In this video, Damian takes us through the newest version of the “Project Adventure” tutorial prepared by Audiokinetic, makers of Wwise. In the process, he gives us a great, user-friendly introduction to the MIDI capabilities of Wwise.
The next videos were produced by Berrak Nil Boya, a composer and contributing editor to the Designing Sound website. In these videos, Berrak has taken us through some of the more advanced applications of the MIDI capabilities of Wwise, starting with the procedure for routing MIDI data directly into Wwise from more traditional MIDI sequencer software such as that found in a Digital Audio Workstation (DAW) application. This process would allow a composer to work within more traditional music software and then directly route the MIDI output into Wwise. Berrak takes us through the process in this two-part video tutorial:
Finally, Berrak Nil Boya has created a video tutorial on the integration of Wwise into Unity 5, using MIDI. Her explanation of the preparation of a soundbank and the association of MIDI note events with game events is very interesting, and provides a nicely practical application of the MIDI capability of Wwise.
In a previous blog post, we took a look at a few tutorial resources for the latest version of the Wwise audio middleware. One of the newest innovations in the Wwise software package is a fairly robust MIDI system. This system affords music creators and implementers the opportunity to avail themselves of the extensive adaptive possibilities of the MIDI format from within the Wwise application. Last month, during the Game Developers Conference in the Moscone Center in San Francisco, some members of the PopCap audio development team presented a thorough, step-by-step explanation of the benefits of this MIDI capability for one of their latest projects, Peggle Blast. Since my talk during the Audio Bootcamp at GDC focused on interactive music and MIDI (with an eye on the role of MIDI in both the history and future of game audio development), I thought that we could all benefit from a summation of some of the ideas discussed during the Peggle Blast talk, particularly as they relate to dynamic MIDI music in Wwise. In this blog, I’ve tried to convey some of the most important takeaways from this GDC presentation.
“Peggle Blast: Big Concepts, Small Project” was presented on Thursday, March 5th by three members of the PopCap audio team: technical sound designer RJ Mattingly, audio lead Jaclyn Shumate, and senior audio director Guy Whitmore. The presentation began with a quote from Igor Stravinsky:
The more constraints one imposes, the more one frees oneself, and the arbitrariness of the constraint serves only to maintain the precision of the execution.
This idea became a running theme throughout the presentation, as the three audio pros detailed the constraints under which they worked, including:
A 5mb memory limit for all audio assets
Limited CPU
2.5mb memory allocation for the music elements
These constraints were a result of the mobile platforms (iOS and Android) for which Peggle Blast had been built. For this reason, the music team focused their attention on sounds that could convey lots of emotion while also maintaining a very small file size. Early experimentation with tracks structured around the use of a music box instrument led the team to realize that they still needed to replicate the musical experience from the full-fledged console versions of the game. A simple music-box score was too unsatisfying, particularly for players who were familiar with the music from the previous installments in the franchise. With that in mind, the team concentrated on very short orchestral samples taken from the previous orchestral session recordings for Peggle 2. Let’s take a look at a video from those orchestral sessions:
Using those orchestral session recordings, the audio team created custom sample banks that were tailored specifically to the needs of Peggle Blast, focusing on lots of very short instrument articulations and performance techniques including:
pizzicato
marcato
staccato
mallets
A few instruments (including a synth pad and some orchestral strings) were edited to loop so that extended note performances became possible, but the large majority of instruments remained brief, punctuated sounds that did not loop. These short sounds were arranged into sample banks in which one or two note samples would be used per octave of instrument range, and note tracking would transpose the sample to fill in the rest of the octave. The sample banks consisted of a single layer of sound, which meant that the instruments did not adjust their character depending on dynamics/velocity. In order to make the samples more musically pleasing, the built-in digital signal processing capability of Wwise was employed by way of a real-time reverb bus that allowed these short sounds to have more extended and natural-sounding decay times.
The audio team worked with a beta version of Wwise 2014 during development of Peggle Blast, which allowed them to implement their MIDI score into the Unity game engine. The composer, Guy Whitmore, composed the music in a style consisting of whimsically pleasant, non-melodic patterns that were structured into a series of chunks. These chunks could be triggered according to the adaptive system in Peggle Blast, wherein the music went through key changes (invariably following the circle of fifths) in reaction to the player’s progress. To better see how this works, let’s watch an example of some gameplay from Peggle Blast:
As you can see, very little in the way of a foreground melody existed in this game. In the place of a melody, foreground musical tones would be emitted when the Peggle ball hit pegs during its descent from the top of the screen. These tones would follow a predetermined scale, and would choose which type of scale to trigger (major, natural minor, harmonic minor, or mixolydian) depending on the key in which the music was currently playing. Information about the key was dropped into the music using markers that indicated where key changes took place, so that the Peggle ball would always trigger the correct type of scale at any given time. The MIDI system did not have to store unique MIDI data for scales in every key change, but would instead calculate the key transpositions for each of the scale types, based on the current key of the music that was playing.
The presentation ended with an emphasis on the memory savings and flexibility afforded by MIDI, and the advantages that MIDI presents to game composers and audio teams. It was a very interesting presentation! If you have access to the GDC Vault, you can watch a video of the entire presentation online. Otherwise, there are plenty of other resources on the music of Peggle Blast, and I’ve included a few below:
The use of third-party audio middleware in game development is a slow-growth trend that will doubtless become more influential in the future, so I thought I’d devote my next two blog entries to some recent video tutorials produced by a few intrepid game audio pros who have stepped forward to help the community.
This first blog is devoted to Wwise, and the tutorials come to us courtesy of Michael Kamper, Senior Audio Developer at Telltale Games. With over 16 years of experience in audio production, Michael has served as Audio Director for The Bureau: Xcom Declassified, Bioshock 2 DLC, and Bioshock 2, among others. Michael has also enjoyed a successful career as a feature film sound designer for such movies as Mission Impossible III, The Day After Tomorrow, Legally Blonde, and many more. His experience in television includes sound design for Xena: Warrior Princess, Hercules: The Legendary Journeys and Profiler.
In the following two-part video tutorial, Michael generously details his Wwise workflow during music implementation for The Bureau: Xcom Declassified:
Wwise Interactive Music Demo – The Bureau – Part 1 – Switches
Wwise Interactive Music Demo – The Bureau – Part 2 – Segments/RTPCs
Just came back from a fantastic experience speaking at the Montreal International Game Summit 2014!
Montreal is a beautiful city, and that’s reflected in the fantastic rainbow-tinted windows of the convention center where the summit was held – the Palais des congrès de Montréal.
The weather was relatively warm while I was there, but I spent most of my time at the summit… although I did enjoy the city views from the enormous walls of windows.
This year’s summit was more vibrant than ever, and the fun began in the wide hallways where attendees could test their video game trivia knowledge by taking part in “The Game Masters” quiz show. I wasn’t brave enough to compete, but I had to get a picture of the set:
The show floor was very exciting this year, with a lot of the activity centering around the two Oculus Rift stations. My attention, though, was caught by two things. First — the AudioKinetic booth, where the Wwise middleware was on display:
And second, this big green guy who was hulking inside the Ubisoft booth. He looks brutish, but don’t let that fool you — he’s a real charmer.
Here’s the big schedule of sessions that was posted at the event. My speech was towards the end of the second day of the summit, right before the MIGS Brain Dump (which is kind of similar to a GDC rant).
My talk was titled, “Music, the Brain, and the Three Levels of Immersion.” It was a great audience!
I had a wonderful time sharing some ideas about the role that music can play in helping gamers to achieve immersion. I’d first explored these ideas in my book, A Composer’s Guide to Game Music, and it was such a joy to explore these ideas with such an enthusiastic audience!
I’ll be posting a video excerpt from my talk soon. It was wonderful to speak at MIGS 2014, and thanks to all the creative and inspiring people I met this year in Montreal – it was a tremendous pleasure!
Instead of focusing on the yearly salaries/earnings of audio professionals, the survey concentrated on the money generated by the music/sound of individual projects. Each respondent could fill out the survey repeatedly, entering data for each game project that the respondent had completed during the previous year. The final results of the survey are meant to reflect how game audio is treated within different types of projects, and the results are quite enlightening, and at times surprising.
The financial results include both small-budget indie games from tiny teams and huge-budget games from behemoth publishers, so there is a broad range in those results. Since this is the first year that the GameSoundCon Game Audio Industry Survey has been conducted, we don’t yet have data from a previous year with which to compare these results, and it might be very exciting to see how the data shifts if the survey is conducted again in 2015.
Some very intriguing data comes from the section of the survey that provides a picture of who game composers are and how they work. According to the survey, the majority of game composers are freelancers, and 70% of game music is performed by the freelance composer alone. 56% of composers are also acting as one-stop-shops for music and sound effects, likely providing a good audio solution for indie teams with little or no audio personnel of their own.
A surprising and valuable aspect of the survey is to be found in the audio middleware results, which show that the majority of games use either no audio middleware at all, or opt for custom audio tools designed by the game developer. This information is quite new, and could be tremendously useful to composers working in the field. While we should all make efforts to gain experience with audio middleware such as FMOD and Wwise, we might keep in mind that there may not be as many opportunities to practice those skills as had been previously anticipated. Again, this data might be rendered even more meaningful by the results of the survey next year (if it is repeated), to see if commercial middleware is making inroads and becoming more popular over time.
Expanding upon this subject, the survey reveals that only 22% of composers are ever asked to do any kind of music integration (in which the composer assists the team in implementing music files into their game). It seems that for the time being, this task is still falling firmly within the domain of the programmers on most game development teams.
The survey was quite expansive and fascinating, and I’m very pleased that it included questions about both middleware and integration. If GameSoundCon runs the survey again next year, I’d love to see the addition of some questions about what type of interactivity composers may be asked to introduce into their musical scores, how much of their music is composed in a traditionally linear fashion, and what the ratio of interactive/adaptive to linear music might be per project. I wrote rather extensively on this subject in my book, and since I’ll also be giving my talk at GameSoundCon this year about composing music for adaptive systems, I’d be very interested in such survey results!
The GameSoundCon Game Audio Industry Survey is an invaluable resource, and is well worth reading in its entirety. You’ll find it here. I’ll be giving my talk on “Advanced Composition Techniques for Adaptive Systems” at GameSoundCon at the Millennium Biltmore Hotel in Los Angeles on Wednesday, October 8th.
Many thanks to Brian Schmidt / GameSoundCon for preparing this excellent survey!
A few days ago, I downloaded and installed the latest version of a software package entitled FMOD Studio and was pleasantly surprised to discover that an oversight had been corrected. It’s not unusual for software updates to correct problems or provide additional functionality, but this update was especially satisfying for me. The makers of FMOD Studio had added the “Music” section to the software manual.
A brief explanation: FMOD Studio is a software application designed by Firelight Technologies to enable game audio professionals to incorporate sound into video games. The application focuses solely on audio, and is used in conjunction with game software. In essence, FMOD Studio is folded into the larger construct of a game’s operational code, giving the overall game the ability to do more sophisticated things with the audio side of its presentation.
When FMOD Studio was initially released in August of 2012, the manual did not include information about the music capabilities of the software. Admittedly, the majority of FMOD Studio users are sound designers whose interests tend to focus on the tools for triggering sound effects and creating environmental atmospheres. That being said, many composers also use the portions of the FMOD Studio application that are specifically designed to enable the assignment of interactive behaviors to music tracks. It was a bit puzzling that the manual didn’t describe those music tools.
One of the biggest competitors to FMOD Studio is the Wwise software from Audiokinetic. Wwise offers much of the same functionality as FMOD, and in working with the software one of the things I really like about it is its documentation. Audiokinetic put a lot of thought and energy into the Wwise Fundamentals Approach document and the expansive tutorial handbook, Project Adventure. Both of these documents discuss the music features of the Wwise software, offering step-by-step guidance for the creation of interactive music systems within the Wwise application. This is why the omission of any discussion of the music tools from the FMOD manual was so perplexing.
It’s true that many of the music features of the FMOD Studio software are also useful in sound design applications, and some are similar in their function to tools described in the sound design portions of the manual. Firelight Technologies may have assumed that those portions of the manual would be sufficient for all users, including composers. However, composers are specialists, and their priorities do not match those of their sound design colleagues. In using the FMOD Studio tools, the needs of composers would be sharply different from those driving the rest of the audio development community. Wwise understood this from the start, but FMOD seemed to be following a philosophy that hearkened back to the early days of the game industry.
In those days, the audio side of a game was often created and implemented by a single person. This jack-of-all-trades would create all the sound effects, voice-overs and music. Nowadays, the audio field is populated by scores of specialists. It makes sense for FMOD Studio to acknowledge specialists such as composers in their software documentation, and I’m very glad to see that they’ve just done so. If you’d like to learn more about FMOD Studio, you can see a general overview of the application in this YouTube video:
As a speaker in the audio track of the Game Developers Conference this year, I enjoyed taking in a number of GDC audio sessions — including a couple of presentations that focused on the future of interactive music in games. I’ve explored this topic before at length in my book (A Composer’s Guide to Game Music), and it was great to see that the game audio community continues to push the boundaries and innovate in this area! Interactive music is a worthwhile subject for discussion, and will undoubtedly be increasingly important in the future as dynamic music systems become more prevalent in game projects. With that in mind, in this blog I’d like to share my personal takeaway from two sessions that described very different approaches to musical interactivity. After that, we’ll discuss one of my experiences with interactive music for the video game Spore Hero from Electronic Arts (pictured above).
(pictured left) is the audio director for Icelandic game development studio CCP Games, responsible for the EVE Online MMORPG. Together with Professor Kjartan Olafsson of the Iceland Academy of Arts, Baldursson presented a talk at GDC 2016 on a new system to provide “Intelligent Music For Games.”
