Chewing on Some Stats

Hi everyone!

It’s finally time for this long overdue post on stats, and big apologies for the delay.  I’ve been pretty sick since Pax, contracting the famous and dreaded “Pax Plague.”  Who knew going to Pax would be an occupational hazard of sorts 😀

Ship Usage and Win Rates

With every update, we’d track various match as well as usage stats, and with the last update, it was no different.  Ship stats were of course most important because we made balance changes specifically pertaining to ships – namely the Squid, Spire, and Pyramidion.  A rationale behind the balance adjustments was based on ship usage data to start.  Looking at non-novice matches, Pyramidion before the last update was the clear cut most overused ship in the game, while the Squid and the Spire were most under-used.  Pyramidion also had a disproportionately high win rate, while the Squid and the Spire had disproportionately low win rates.

Since the update, we’ve been pulling up ship usage and win rates on a weekly basis in non-novice matches to see how usage and win rates migrated over time, and here are the summary data over the course of about a month:

• Goldfish: 12322 matches (22.5%), 5975 wins (48.5% win rate)
• Junker: 7969 matches (14.5%), 3819 wins (47.9%)
• Squid: 4892 matches (8.9%), 2496 wins (51.0%)
• Galleon: 5135 matches (9.4%), 2412 wins (47.0%)
• Spire: 6216 matches (11.3%), 3634 wins (58.5%)
• Pyramidion: 14510 matches (26.5%), 7057 wins (48.6%)
• Mobula: 3765 matches (6.9%), 2014 wins (53.5%)

Note:  Duration – from day of update to end of March

The first thing that jumps out is that, even after the last update, the Pyramidion was still the most used ship in the game, with the Goldfish trailing close behind.  Squid and Spire’s usage rates increased, but only to 8.9% and 11.3% respectively. Mobula is now the least used ship at 6.9%.  If we are to look at over usage rates, the Pyramidion and the Goldfish would still be overused right now, and the Squid, Spire, Galleon, and Mobula would under-used.  The Junker is the only ship at the expected ship usage rate of 14%.

If we look at win rates, we feel that all the ships are relatively well balanced at the moment, with the win rate of each ship within +/- 3% of 50%.  The Pyramidion’s win rate declined after the update, but it is still close to 50%, which we think is still relatively healthy given it’s high usage.  Spire’s win rate increased to 58.5%, leading all ships.  Other ships with over 50% win rates are the Mobula and the Squid.  We might be seeing a bit of bias in ship selection affecting win rates here, as the ship with highest win rates also have lower usage rates.  The ship with the lowest win rate right now is the Galleon, and it also has a lower usage rate.  In future balance adjustments, we might need to look at the Galleon.

Note:  The reason why we did not factor in novice match stats here is because novice matches only have the Pyramidion, the Goldfish, and the Junker as available ship options.  Therefore the data would be skewed had we included novice match stats.  Going forward, we could also look at how ship usage and win rate correlate with people’s MMR data.  The one thing we have tracked over time was usage in competitive tournaments by skilled veteran players, and that seemed to have coincided with overall usage data prior to the last update.

Match Stats

Duration:  2/1-3/31/15

• Mean Queue Wait Time (to get into a match lobby from queue): 50s
• Median Queue Wait Time: 20s
• Mean Wait Time in Match-Made Lobby:  4min 11s.

As you can probably guess, the average is highly affected by some really long waits. How long?

~1.5% of the players waited for more than 5 minutes. Among these players, 63% of them would eventually get into games.  If we look at wait time at or greater than 3 minutes, that represented about 5% of the player base in matchmaking queue.  In summary, in the vast majority of instances, wait time in queue seem fast or at least reasonable.  Yet, in some cases, the wait time can be intolerably long.

Upon further investigation of quite a few specific cases of long queue times, a pattern emerged.  In the majority of the long queue wait time cases we investigated, they invariably had to do with crew form, and more specially crew form of multiple ships and crews.  In these cases, especially for higher skilled players, the match maker would have a harder time finding balanced opponents because there weren’t enough players of similar skill levels available in queue at the time.  And because we made more stringent considerations for balance, the crew forms of multiple ships ended up waiting longer.  For the next update, we’ve made a change in the matchmaking system to allow multi-ship crew forms to split up if they feel like they’ve waited too long.  This would allow people to stay in their crew forms and continue to queue for the same match without exiting queue and redoing their crew forms.

As to the all important balance, we’ve established the match balance threshold value, above which the “underdog” team’s win rate could decrease to 35% or lower.  Currently, the average balance values of matches fall pretty comfortably within this threshold, while over 75% of the matches are statistically well-balanced.  However, this does mean then that up to 25% of the matches can be less than well-balanced.   While we are always striving to improve balance, we did have to make compromises for the sake of sociability and social features in game.  We also had to strike that median between balance and wait time.  In short, we’ve always had to juggle a number of different opposing forces and try to find the optimal solutions, and acknowledging that perfect balance every match is a nirvana that we’d like to strive for while realizing that we may never get there.

Having said that, we’re making a number of match system adjustments for the next update(s).  Firstly, we’re introducing an “Underdog” feature.  If balance discrepancy exceeds the threshold value for a given match, we’ll give a badge of courage for the “underdog” team, and the team members would have stats forgiveness (ex. losses would not be counted) and can earn achievements as well as titles. Being an underdog then becomes a challenge that people can take head on.

There will no longer be rematch limits.  In the match end flow, rematch voting would be made more prominent, and a majority greater than 60% of votes in favor of rematch ought to be attained in order for rematch to occur.  This way we could better ensure that the rematch is really reached by a majority.  By extension, when people return to crew form from match end, they will now be with their crew instead of the entire team.  While the correlation between these changes and balance is more indirect, it does address complaints from a number of players and can potentially augment ease of use and community at the same time.

Next, crew slots now no longer accept the pilot role in all cases.  We felt that this change was necessary for balance as well as improving team play.  As there really isn’t tangible benefits for pilots to be in crew slots.  This change was made again based on feedback of numerous players.  Also, solo queuing novice players entering non-novice matches would no longer be placed in the captain position.  We wanted to make sure novice players in non-novice matches have an easier time learning the game from veteran players.

Lastly, we made modifications to the match list in Custom Game to improve balance, and in concurrent player troughs, the match list would open up to all matches.  With more consideration for balance, we felt more comfortable making the adjustment in this case.

Summary

We’ll always be diligent in listening to everyone’s feedback as well as in finding ways to use data to improve the game.  While we do, we also have a mission to continue to create and to innovate on the game.  This could at times affect a person’s use case, and as a result, we’ll need to do our best to understand it and account for it.  Over the past 2.5  years, we’ve been constantly innovating on the game. We’ve added in numerous features and systems, introduced new weapons and skills, and carved out better specialization for each ship, weapon, and skill,  all to make the game a better, more diverse, and more interesting experience than it was when we first launched.  Innovation, however, is a 2-way street.  While we never want to be complacent, we also ask our great players to not be complacent either, and to continue to partner with us on this wild and incredible ride. We could not have gotten this far without your amazing support and feedback.