02 Sep 2023

# Analyzing rating systems for foosball outcomes

## Color me disappointed

I'm in the possession of data from around 26'000 matches of foosball played over a span of 9
years. This data was captured with a web application that also maintains an Elo score for each
player, ranking them on a leaderboard. Naturally, one has to wonder if
Elo is the optimal rating system
for this particular game. But first, what *is* a good rating system in the first place?

A rating systems affords a player a score. This score should in some way reflect on the skill
of the player: A player with a better score is expected to beat a player with a worse score
more often than not. The ability to compare scores then implies that we can build a ranking.
Since a ranking of players should accurately predict the outcome of a match, we can call a
rating system *better*, if it can correctly predict a higher proportion of games than a
different system.

With an extensive data set at hand, it should be easy to explore the performance of a rating system. For every history match in the data set, we compare the prediction of the ranking up to that match against the real outcome, then enter this match into the rating system and continue. At the end we can calculate how many matches have been predicted correctly by the ranking.

Let's take a simple rating system as an example: We compute for each player a simple win rate
of every match they played. A player with no matches played starts out with a win rate of 0.
When predicting the outcome of a match, we simple average the win rate of both players on a
team and weigh them against each other: The team with the higher average win rate is expected
to win. With our dataset this yields a correct prediction rate of `72.7%`

.

## The twist

Unsurprisingly, there are more sophisticated (and supposedly more accurate) rating systems out there. I looked at TrueSkill and Weng-Lin in particular due to their ability to treat 2v2 games, rather than just 1v1 for systems like Elo.

Imagine my shock when these also yield a correct prediction in just `73.3%`

and
`71.2%`

of all matches. Tuning their parameters did not achieve significant improvements.

My best guess is that there is just so much randomness in the outcomes of our foosball games that more sophisticated systems just can't produce a better result than the primitive win rate. Sorry to leave you hanging on such an unsatisfying conclusion. Ok bye.