One of the more interesting things to me about modern game design and implementation, particularly of online games, is the ability to heavily instrument them.  For example, tom5760 and I were talking the other day about Team Fortress 2 and how Valve tracks every death in great detail so they can do fancy things like plot heat maps of kill zones on levels or chart kill rates for all the weapons.  Clearly that’s critical for any sort of online, repeat play game like TF2 where you need to keep adding and continually balancing content to keep people coming back.

But it’s also really interesting and useful in general, providing great data for learning and adjusting future game design, and something that never even really occurred to me as a realistic possibility writing games for the late-90s landscape.  The technology and connectivity was only just beginning to be in place for it.

Gold Leader is only instrumented lightly, but it is hooked up with Google Analytics and it’s interesting examining the data (these numbers below are rounded slightly, for no real reason).

Players

Right now the game’s been up for about 60 hours.  Some 140 people have loaded the game a bit over 260 distinct times, and actually started playing a bit under 400 times.  Definitely not lighting the Internet on fire, but it’s still pretty cool to me at least that about 200 people have now played when you factor in FGL commenters.

Somewhat more interesting is how they’re doing.  Here’s the progression of players:

• 45 have beat the Sensor Grid (60 times).
• 20 have beat the Assault Command and Interceptors (25 times).
• 15 have beat the Minelayers (15 times).
• 10 have beat the narrative mission (the whole game) (10 times).
• 7 of them have given the survival mode a try (7 times).

As a fine grained input on the game design, nearly everyone is failing by missing a mission objective rather than dying.  Only 20% of the failed attempts have been at 0 lives, the rest all have lives remaining when the mission was reported as failed (missed too many Sensors, etc.).  I’m not totally sure what to make of that—is one way or the other in terms of getting players to try again? I have no idea—but it’s really cool to know and think about, and if I were going to do an update would provide a clear update on what to adjust if I wanted to make the game easier, e.g., upping the mission failure threshold but not worrying about the number of lives.

Progression

The total and unique events for the Assault Command and Interceptors are identical and it’s not possible to skip the former and go to the latter, so it seems that anybody who can beat the Assault Command “boss” can beat the Interceptors.  That’s actually really interesting to me, because I don’t consider the boss much of a challenge, it’s basically just an old school “learn the pattern and play safe” fight, and the Interceptors mission can definitely be challenging.

Similarly, most people who can get through the Minelayers can get through to and beat the final Command Group, the next mission.  That I’m happy about, the balancing seems about right.  It’s not 100%, so there is a progression in difficulty.  But, it is only about 75% so it’s not a ridiculous additional challenge.

Only one or two other players (besides myself and tom5760) have achieved a perfect run (no deaths).  Stupidly I’m not explicitly tracking this, but I can sort of figure it out.  One I know for sure because he was logged into GameShed and recorded the achievement.  He’s actually the number one player on the site (has earned the most achievement and high score awards), so that’s not too surprising.  The other is actually a friend of mine who posted such a high score to Mochi that I can’t believe he didn’t beat the mission in one run.  I’m not even sure it’s possible to do otherwise while scoring above ~2750 or so, and he’s just over 3000.

Long Game Repeats

Unfortunately I’m not really logging how far people have played into the survival mode.  I’m sure it’s too hard, even I can’t survive for long, but it would be interesting data to have.

From the data it’s hard to tell people’s behavior once they get into the late stages without putting together some analysis scripts on the raw logs.  “Unique” here means a session, so they could be coming back some time later.  Based on the scoreboard however, if they are they’re mostly not playing up to the same point, though some are.

Difficulty

In general though, the big lesson, really the confirmation, is that the game is pretty hard.  Only 25% of players have made it through the first mission, and only on about 15% of the plays.  Toward the objective of hooking people in and keeping them playing and coming back, it’s clearly too hard.  In so far as that limits the reward the sponsor is getting out of the game and the audience the game and Rocketship Games gains, that’s unfortunate.

However, beyond that concern I can’t say I’m particularly troubled or upset about it.  Many of the reviewers on FGL noted that it was fairly hard, and it was obvious from my own testing.  I’ve never been a fan of the intentionally super hard games out there now (e.g., precision platformers like Super Meat Boy), but early on in development I found myself unexpectedly adopting a definite attitude that the difficulty was part of the aesthetic of the game and its homage to classic early shooters.  To some extent that’ll cost future efforts by hampering the popularity and visibility of this one, but clearly the difficulty objective has been met: Pretty hard—too hard for “casual” gamers—but certainly accessible for more “serious” gamers.

In any event, fantastic data to collect, and this experience has really convinced me to put a lot more thought toward instrumentation in future efforts, particularly in playtesting.  I hadn’t thought I’d be able to get much data so I hadn’t worried about it much here, but in the end enough people beta-tested on FGL that I probably could have gotten a fair bit of data while still in development.  Something to consider in tuning future games.