GDC 2010: Researcher Bruce Phillips' detailed study of 18,000 players' habits, multiplayer usage, and achievement acquisition is helping Microsoft Game Studios make better games.
Who Was There: Bruce Phillips (User Researcher, Microsoft).
What They Talked About: Many people take their Xbox 360 achievements seriously. Nobody takes them more seriously than Bruce Phillips--because studying achievements is part of his job. As head of Microsoft's player-behavior research project, the Ph.D. in psychology spends his time studying how players play games on Xbox Live. By looking at how--and when--thousands of gamers rack up achievements, choose multiplayer modes, and slog through single-player campaigns, Microsoft is collecting information it hopes will help developers make better games.
Phillips gets his data from "The Panel," a group of roughly 18,000 volunteers drawn from Microsoft Game Studios' beta-testing program. Though about half are in the US, Panel members are spread all over the world, including such far-off regions as India, South America, Indonesia, and Sub-Saharan Africa. All are over 18 years old, and all willfully let Microsoft have access to their Xbox Live information in exchange for early access to games and free software.
In return, Microsoft gets data--mountains of it. Phillips' team knows every single Panel member's achievement record, which lets them track all manner of player behavior. They know when Panel members stop playing a game, they know how many multiplayer hours they've logged, they know when they're online, and they know when they're offline. The one thing they don't know is how Panel members interact with their friends, since that would violate the privacy of said friends who are not Panel members.
One might think all this data is being monitored via some sinister, supersecret, WOPR-like supcomputer that acquires it through some secret back door in Xbox Live. That would be wrong, according to Phillips. He said the information is collected via HTML parsing of public gamertag profiles on Xbox.com and via an XML feed provided by the Microsoft Community Developer Program. Though it's not open to the public, any developer can apply to get access to the tool to access the XML feed, and Phillips said third-party publishers also track data from there. The applications collect data every 15 minutes, and data of online players every five minutes.
Phillips said his first job after setting up the Panel was to make sure its data was accurate. So he compared its general gameplay statistics to those of Xbox Live, which boasts more than 23 million users. Both the Panel and overall XBL user base played the same games in the same amounts, with popularity bar graphs of the two being nearly identical. Phillips showed off a chart of achievement completion rates, which also showed the two in sync.
SINGLE-PLAYER STATS & GAME COMPLETION
Confident that the Panel data was statistically sound, Phillips and his team began to use it to answer questions about gamers' habits. The first metric was game completion. Using achievements as markers, Microsoft can determine how far members of its Panel get in a single-player campaign before quitting--and what difficulty they played the game at. By finding out how much of a game consumers play until they quit, the company hopes it can isolate the issue that caused players to stop--and keep it out of any future Microsoft Game Studios titles.
As an example, Phillips showed the completion rates of two games--Halo 3 and Battlefield: Bad Company. His team measured the number of players who earned the first-chapter campaign-completion achievement in each game and how many completed the game. Some 93.22 percent of Halo 3 players finished the first chapter on normal difficulty, compared to 71.82 percent who finished the campaign.
By contrast, Battlefield: Bad Company saw only 39 percent of players finish the campaign in its entirety. He showed that each campaign-completion achievement saw a marked drop in continuing players--the first chapter was 70 percent, the second was 58 percent, and the third was 51 percent. "That's a very steep drop-off early on," he said. Phillips said Rainbow Six Vegas 2 had an even higher rate of attrition.
Microsoft can also track players' preferences via the achievement meta-data included with achievements. They can tell if the achievement is awarded due to completion, collection, questing, or gameplay. This also helps their in-house development team figure out what kinds of missions to put into games, since they can see which are the most popular.
Another graph presented by Phillips showed how a game--in this case Call of Duty: World at War--is played for the first 30 days. He presented a graph that showed that virtually all single-player achievements are earned in the first four to five days after a game is first played. After that, Zombie mode picked up speed, leaving co-op campaign a distant fourth after multiplayer.
MULTIPLAYER UNDER A MICROSCOPE
Unfortunately for single-player purists, by far the most popular game modes are multiplayer. Phillips and his team track the Panel's online play with a metaphorical microscope, tracking the popularity of each game's various modes and how they are used over time. Team Deathmatch dwarfed all other multiplayer modes for Call of Duty: World at War, with the others changing places at the bottom as people would try out different modes.
Microsoft also measures which multiplayer maps the Panel plays. Phillips displayed a dizzyingly detailed chart that showed what percentage of the Panel played particular World at War maps, when they played them, and in which modes they played them.
The rate at which achievements are earned is also closely studied. For Modern Warfare 2 multiplayer, people earn about five achievements on average in the first 30 minutes, but then it levels off, with five achievements earned every 1.5 hours. Phillips said Modern Warfare 2 was an example of a game that did a good job rewarding players for continuing to play by continuing to dole out achievements, albeit at a slower rate.
"GAME TENURE" & DATING DLC
One of the most important concepts Phillips and his team have come up with by studying achievement data is "game tenure." This is the duration of time between the day players first fire up a game in their Xbox and the last time they play it.
The vast majority of game tenure loss happens in the first few days, due to people not liking the game or renting the game. Then it flattens out, since people keep coming back to play it over and over again. One of the games with the best tenure was Rock Band 2, and another was Call of Duty: World at War, with a whopping 132 days. The game with the highest game tenure? Geometry Wars: Retro Evolved, which players are still frequently playing nearly four and a half years after its release.
Why is game tenure important? Because according to Phillips' research, DLC is more successful when it is released while players are still actively playing a game. That's why so many DLC packs are being released within weeks, days, or even hours of a game's launch. A good recent example of this (not provided by Phillips) is Dragon Age, which had DLC on day one and a full-fledged expansion, Awakening, just four months after the game hit the market. (A sequel also appears to be scheduled less than a year and a half after the original debuted.)
Though Phillips' team definitely knows how to detect patterns of behavior from Xbox Live data, its ultimate job is summed up in a simple question: "Why?" Why do players stop playing when they do? Why do they prefer one multiplayer mode over another? Why will they spend hours pursuing one achievement but leave others untouched?
To answer those questions, the Panel is bombarded with surveys about their behavior. These surveys are sent out automatically and correspond to behavior triggers in a game. Most interestingly, they can be sent out days, weeks, or months after the event occurred, allowing developers at Microsoft Game Studios to pick the brains of players while fashioning a sequel or DLC for a particular title.
Quote: "Now, for the cool stuff."--Phillips, showing a slide of Arthur Fonzarelli from Happy Days.
Takeaway: Many people consider achievements "nerd cred." Microsoft and other publishers consider them clues to be closely studied on how to make a better game.
If they actually studies the online habits of a widespread audience they would have realised gamers like to create their own rules in online multiplayer matches. Forza 3 would be a enjoyable online experience instead of those stupid bloody "hoppers", I like to race better players than myself as it helps me get better, instead i get lumped in a lobby of ass-wipes who see my team pre-fix and proceed to take me out on track or boot me from the lobby. The only thing microsoft are interested in is taking your money and giving you crap in return.
To say that Microsoft invented achievements is ridiculous. Games had achievements well before MS made the system we know have. Every innovation comes from another idea. MS does deserve credit for what they moved it to
@armydominican69 yea i know the PS3 has trophies but they got that idea from the 360. My eyes lit up the first time I got an achievement, I'm pretty sure everyone else did too. The trophy system is just not the same because xbox live users love to show off their gamerscore, can anybody agree?
@ -Saigo- i couldn't agree with you more, there is a really good article about the success of warcraft and how the vp of blizzard uses strategy to master the tenure business... also love your avatar pic
Definitely an interesting thing to study...but if M$ was interested in creating a game with the longest "Gaming Tenure" they should be studying World of Warcraft.
I actually chose a 360 over a PS3 simply cause of Xbox Live and the achievements...Microsoft brought back points in a whole new way. Yall know yall like to hear that PLOP! and then "Achievement"
This is actually pretty cool. While I own a PS3 and not an Xbox, it's great to see that someone is studying what we like.
I didn't know companies went this far to analyze gamer behavior :/ I guess if it means a better product in the end that's great lol, but still creepy.
cuz it's their business and they want to improve all the time and be more profitable. thought this article was interesting. go WAW!
So... they actually pay some Ph.D. a ton of money for all this? Hell, if MS matched my current wages, I could give them tons of advice based on what all the achievement site members complain about and it would probably be more useful than what net benefit they get out of this guy. All of that "great info" that's in the article should have a big DUH tag attached to it.
You guys realize that it isn't big brother right? Maybe I'm missing sarcasm here but its a volunteer program. A study. So they can (hopefully) constantly improve things.
As an aside, I totally dig that they referenced how the World at War Nazi Zombies mode was more popular than the multiplayer. Nobody anywhere (other than Treyarch) talks about Nazi Zombies AT ALL. I personally thought it was better than the Left 4 Dead rubbish and started to worry that my friends and I were the only ones who played or cared about the Nazi Zombies add-on.
Just remember that they are only tracking those in the Beta program, those who agreed to be tracked. Big Brother isn't watching everyone. But yeah, it sounds like a pretty sweet deal. And I like getting as many achievements as I can that are "within range" (reasonably attainable if I pay attention, not the crazy ones where you have to jump through hoops or spend enormous time to get).
"Microsoft is collecting information it hopes will force players to buy more games." There, I fixed it for you guys.
Wow I never thought of achievements as trackers for gaming behavior. Be careful what you play now that big brother watches.
Wow, early games and free software, just for allowing them to know everything about what games you play, what modes and for how long? That's not even fair, lol. Gaining lots for losing nigh nothing.
"nerd cred" - How is it nerd cred when you can just boost them with some other loser to make it easy to pull off? I like the idea to study them to find out where people quit playing and why. They should include profile information where players can review games they have played, similar to the my nintendo network that would hopefully give clues to the puzzle. Keep it tied to the gamercard. Let us rate the achievements and give feedback. As it stands right now though, most achievements seem to be little more than a fake reward for fake bragging rights or a goofy way to pretend you have stats about behavior where the researcher gets to make all the assumptions regardless of the circumstances.
That was very interesting I would love to have known about that job type before I gave up on psychology to focus on music.
I used to play a lot to unlock achievements (unlocking more than 250G a day), but after a while it loses the purpose of playing the game for just the fact that the game is fun. So I stopped focusing on the achievements, I only collect them if it makes sense, and doesn't ruin the experience.
they need to put more than that range of 50 achievements standard on a game and 1000 gs points.. they should stop being lazy and move that up to 2000 and 100 achievements
As a psychologist working at a place I hate, and given my love of video games, I have to say that I want his job.
yeah yeah. Im calling it. When multiplayer achievements pick up, they will be on a logarithmic scale
i hope not, kameo sucked lol. but i tell you what they can look at my achievement data, if it means i get to play some game betas lol
Waldkrieger Posted Mar 12, 2010 12:44 pm GMT @Worm209 While your comment is obviously sarcastic and unrealistic I will say: Maybe you are unaware. But, some people actually enjoy to just set down, relax, escape, and play a game without a bunch of over the top challenge. Life is quite challenge enough. I am a hardcore gamer, but I personally do not enjoy playing games on harder settings. Why? Because that is not why I play games. I play them to relax, enjoy, escape. Not get pissed off and redo sections 50 times. That's just a waste of time and id rather go read a book. So, having games split into two sections, one for the easier, one for the harder, is actually a good thing. This way both sides are happy as there are different types of players. The same should go for achievements. One section for single players, one section for multi-players. Your gamercard could reflect that you are a more single player person and unlocked all single player achievements. -------------------------------------------------- Yes games should cater for both types of player in some way or other, i never said they shouldnt. I was suggesting that if they continue going the way they ARE going, it is only the casual player that is catered for. Not the other way round (which you thought i was saying) I just tell it like it is
I played fallout for over 200 hours. Not necessarily to get all of the achievements (which I did) but becuase the world was believable and intense. Not for a million dollars would i suffer through the halo 3 campaign again just to collect al the skulls or whatever...same goes for odst.
interesting. i'd think this would be a cool topic to go further into. and do the same for trophies of course. and steam.
Interesting read. Makes sense about why some games have DLC only weeks to months after release. Used to wonder about that, but this research explains it.
I was hoping they were using the aggregate achievement data to improve games (or make games better suited to gaming tendencies). This is good for the industry and good for us gamers. Finally somebody's paying attention to our gaming habits and game makers have tangible proof of their suspicions. I would totally be on the XBL panel.
Stats and videogames... some of my favorite things, interesting read. Ive turned off notifications for Achvs though, and i dont even think of them until after ive beaten a game. Just my personal preferance, but it feels much more natural to me.
I'm one of the rare people who only care about single player. Most multiplayer is tacked on and unneeded.
I hope the conclusion they gather from all of this is more multiplayer and less single player hours. If that happens, I'll stop playing video games.
The best achievements I have come across in a game have got to be the original Mass Effect. They were tough, but they were accessible at the same time. I still only have 485G on it, and I've completed it twice.
@Worm209 While your comment is obviously sarcastic and unrealistic I will say: Maybe you are unaware. But, some people actually enjoy to just set down, relax, escape, and play a game without a bunch of over the top challenge. Life is quite challenge enough. I am a hardcore gamer, but I personally do not enjoy playing games on harder settings. Why? Because that is not why I play games. I play them to relax, enjoy, escape. Not get pissed off and redo sections 50 times. That's just a waste of time and id rather go read a book. So, having games split into two sections, one for the easier, one for the harder, is actually a good thing. This way both sides are happy as there are different types of players. The same should go for achievements. One section for single players, one section for multi-players. Your gamercard could reflect that you are a more single player person and unlocked all single player achievements.
Content you might like…
343 Industries' new shooter making its tournament debut at Major League Gaming Fall Championship from November 2-4; four vs four tournament offers $40,000 in prizes.
- Sep 27, 2012
Users who looked at this article also looked at these content items.
Avalanche Studios co-founder says developer's ambition is for action, not moments that make players cry; steampunk-style game on hold. Full Story
- Posted May 15, 2013 11:33 pm AEST
4A Games creative director Andrew Prokhorov thanks Jason Rubin for telling the studio's story, but says, "We deserve the ratings we get." Full Story
- Posted May 17, 2013 5:44 am AEST