Being a very Visual person that I am, I think in perspectives and colors but what I often forget is there is an underlying auditory process that happens that support my mental images. Though most of us don't notice or pay attention, sound plays a very important part in our lives. The audio engineers are some of the unsung heroes in a most productions.
Big visual effects shots don't have any grounding without sound effects, any VFX shot is only at its 75% potential without sound. Even when there is nothing happening or what we think is silence, there is still some atmospheric sound that gives the scenario some dimension or we could call it 'the sound of silence'.
I don't think there is ever a place that doesn't trigger your auditory senses, unless you are in vacuum where you wouldn't survive in any case. To think of it, in movies to depict vacuum, they have an ominous sound that subconsciously tells you that there is nothing.
My brother, Varun is a sound engineer for a well known Audio company in Mumbai. Varun has been an audiophile from his young age he always had a knack with music and instruments, he recently gave a
talk on 'Sound as a Storyteller' at the India Game Developer Summit Bangalore 2010, he makes some interesting observations and points through his talk. Below you'll find a summary of his talk, definitely worth a read.
'Sound as a Storyteller'
I make sounds for a living. I create sounds out of nothing and mix and record music for a variety of media. Over the past 2 odd years I’ve had the chance to work on games - on a variety of them. Although, I have been involved largely on a creative level. I receive the game and a brief and provide sound, music and dialogues. I’ve had very little to do with the technicalities of implementation. So what I’m going to talk about today is what I’m best at - being creative and telling a story through sound.
I’m going to talk on two topics. One - Communication. Using sound as an effective communication tool - to enhance game play and make the experience for the gamer all the more enjoyable. And second, I will briefly go over the asset structure and how we do what we do.
So starting with communication. We all know that sound is one of our 5 senses. Along with sight, smell, touch and taste it helps us understand the environment around us. From the beginning of time sound has played an important role in understanding our environment. Birds tweeting would mean a danger free environment, while silence or low rumbles could mean approaching danger or disaster. These ideas have trickled down even today. That is why a happy scene in a movie might have birds tweeting in the background and definitely not a scary rumble. So this tells us that there’s a lot our brain understands from what we hear - and many of these processes happen sub consciously. And if we understand these clues and implement them into a game such that we leave hints both on a sub conscious and conscious level - then, we can create an environment that is engaging to the player without him/her realizing it. And isn’t that what we all want as developers and publishers? A game that’s engaging and sells?
Lets start with a question. What is sound?
There are easy answers that can be found in a dictionary. The first one - ‘sound is a traveling wave which is an oscillation of pressure’..etc..etc which is a purely scientific explanation. The second one is what interests me. “The sensation stimulated in organs of hearing by such vibrations”. This is going back to what I was talking about. Sound is and can be used as a form of communication because it directly appeals to our brain.
If sound does communicate, how does it work? When we hear a sound - either consciously or sub consciously - our brain refers to a catalogue of previously learned information. It refers back to our past. These could range from memories, emotions, what we see and hear in the media - especially in today’s age - and also most importantly what we have learnt from society. We take clues from our environment and then make decisions based on our past. These responses could range from emotional to practical, total disregard or in most cases a mix of these. For example - when you hear an airplane pass over - you properly identify it as an airplane because you’ve heard it tens of times in your past. But on the other hand if you hear an alien spacecraft your mind might have difficulty making references because it might be a sound you’ve never heard before.
So this tells us: Sound is an effective communication stream. It can give us a sense of realness or fantasy. And if it communicated efficiently then it can help the player suspend disbelief easily. But also, sound is not king. We constantly refer to our visual stream for confirmation of what we hear.
How does this apply to gaming? We get to play God. We get to make decisions for the player and therefore have the power to steer the story and gameplay in the direction we want. We have control. We can make the player feel what we want them to feel - either with strong sound effects or music. But does this mean we put in as much information as possible? No, our brain has limits. It can’t possibly take in infinite amounts of information. This brings us to the question - how do you find what you need?
To understand the importance of getting the right information out effectively, lets look at three scenarios. An ideal situation, then one of overload, and the third of underplay. Lets look an
Ideal situation. Here we have the visual and sound or auditory information balanced out. They feed off each other, influence each other but don’t overload the player with information. They lead the player along. The results of such a scenario would be: enhanced story telling, game play and realism or surrealism - which ever you are trying to achieve. Now, lets look at a scenario that is very easy to accomplish - one where there is a lot of visual and auditory information. You have strong visual detail, a thousand sound effects, a heavy background score and dialogues back to back. Overload. This leads to confusion and there fore miscommunication which leads to less involvement which in turn makes it harder for the player to suspend disbelief and eventually leads to the player hitting the mute button therefore making my job - useless!
The third scenario - Underplay. Lack of enough information can lead to loss of interest and can make game play less immersive. This although is still better than overload and is definitely more workable.
So how do we avoid clutter and a scenario of overload? Planning, more planning and..eh..planning! It is important to get every body involved with the game early on. And yes everybody does include the sound team. We’re not the last minute guys! By bringing in everybody early on you have the chance for different departments to influence each other. How important is sound? Is it vital? Can it push game play in some ways the visuals won’t be able to? There’s a lot that can pre determined in the planning stage. But as we all know, plans are never set in concrete. Leave enough room and buffer for change. While planning you need to think of ways in which the streams can compliment each other. Predetermine what the scene/level/character/moment require both visually and aurally. Think if you need sound to convey things beyond visuals...can they underline certain moments? Can they take over from visuals completely?
So the point I’m trying to make here is that each department and stream cannot exist by itself. Maximum communication to the player can be achieved first if there is maximum communication between the visual and aural stream both during development and game play.
Next I’m going to talk about two real life scenarios. The first is called the cocktail party effect. We’ve all been in noisy environments and have found that we can consciously tune and focus on the sounds that are important to us. In a noisy party you can still manage to have a conversation with someone and tune out the noise around you. This is called the cocktail party effect. We can make conscious decisions. So what relationship does this have to game play? A game is not reality. It has fixed boundaries preset by the designer. And we want the player to follow the right clues into unraveling the story or understanding the game. As we have seen before we have to make the decisions for the player before hand. So the decisions in sound too have to be made. For example if its a war RPG we would want the player to first hear his own sounds - say breathing, communication, gun sounds or sounds in the environment that directly communicate with him - like ammo pick up sounds. Next in priority would be the enemy sounds - sounds of approach or gun fire. Third in priority would be the environment sounds - maybe distant gunfire..vehicles..etc. All three levels of sound are important in placing the character in the right space. But for example if the environment sounds are as loud or louder that the player’s own sounds it can lead to confusion. So it is important to simulate the environment for the player and lead them to make the right decisions.
Next is a little but important concept in avoiding clutter. It’s called the 2.5 rule. The 2.5 rule is something Walter Murch - a brilliant sound designer and film editor who’s worked on some ground breaking films - thought about. During his years in working with film he noticed that the brain could only understand two and a half themes at a time. The first two themes will have the audience actively involved, while the half theme gives passive information that is important in suspending disbelief. An example of this would be an intense argument between two characters. The visuals would occupy the first theme since it holds maximum active information, the second theme would be the dialogues since it gives information about what the scene is like. The half theme would be the music that would add to the emotional level of the scene. Other information like footsteps or environment sound might be ignored. This is when we go back to planning. It is important to make the right choices early on.
Moving on to sound assets and what they are about -
Our ways in working with sound does come from what we’ve learnt in the past in working in linear media formats like film. Sound is usually split up into three broad departments: Dialogues, Music and Sound Effects. I prefer to split up sound effects further into ‘Designed Sounds’ and ‘Real Sounds’.
Starting with Dialogues. They can be anything from in game character or NPC dialogues to those in cut scenes. They are important because they are a direct communication link to the player and can probably be most effective in communicating information when compared to any other stream. This too needs to go back to the planning stage where character sketches are built and scripts are written. This information is then used to cast the right voice. There is a lot our brains can understand from only hearing a voice. Therefore the more the voice matches the character in game, better will be the communication to the player. Once scripts and characters are written, the actors are recorded. After which, the correct ‘takes’ are edited and processed to what they need to sound like for the game.
Music is the most important emotional tool available to us. It can push the player to feel they way we want them to feel. For this too time needs to be spent planning and sharing ideas about the style/genre and expectations. Referencing can help communicate ideas better to the composer. Post which the score or songs will be written, recorded and mixed to the specifications of the game.
