Agena Company Logo

Agena Company Logo

Beating the bookies: How Bayesian nets predicted Spurs’ results with consistent accuracy

There must be at least 20 million people in the UK who fancy themselves as experts at predicting the results of football matches. But, as the bookmakers’ increased profits also confirm, it seems that those brave enough to put their money where their mouths are, are not especially accurate with their predictions. In fact, it has been shown that even if you do some very fancy statistical analysis of previous relevant data for each match as the basis for your predictions, you will still not be consistently accurate to beat the bookies.
Now a paper to be published in the scientific journal Knowledge Based Systems by Norman Fenton, Martin Neil and Adrian Joseph [1] shows that, with the right level of knowledge about a particular team integrated into a special type of probability model (called a Bayesian Network), you can achieve consistently accurate predictions that are good enough to beat the bookies.
This story actually starts back in late 1995. Norman Fenton (CEO of Agena and Professor of Computing at Queen Mary University of London), together with colleague Martin Neil (CTO of Agena and Reader in Risk at Queen Mary University), was grappling with the problem of how to accurately predict the number of bugs in software systems. They found that using traditional statistical techniques was never accurate enough for the kind of systems they had to assess. They were looking for a method that would enable them to combine objective data (such as that relating to previous known bugs) with much more subjective data like the ‘skill of the coders’. Eventually they found that a new method, called Bayesian networks, enabled them to combine these different types of information in a statistically rigorous way. The improved results they achieved for software bug prediction were so significant that their paper on this subject [2] is in the top 1% most cited papers in computer science, while the models (now incorporated as an example model in the AgenaRisk software called [3]) have been used with great success by companies like Philips and Motorola.
But in 1995 Norman Fenton was not so sure about how effective Bayesian networks could be. The proponents of Bayesian networks had argued that the best results are achieved when you build your model based on the knowledge of a real expert in the subject you want to make predictions about. Fenton felt the only subject in which he was a real expert was Tottenham Hotspur. So he set about building a simple Bayesian net model to predict Spurs results based around what he regarded as a number of key factors (like the combination of certain players and positions, the quality of the opposition, and the venue). He built the model in late 1995 with a view to testing it out over the next few matches. One of the unique things about Bayesian net models is that they do not make firm predictions, but rather produce a probability for everything that is uncertain or unknown. So, for each game, the model produced the probabilities for win, draw and lose. Armed with this knowledge he found that the predictions were not only accurate but provided enough information to beat the bookies hands down, even allowing for their ‘mark-up’. This is because the probabilities enable you to determine situations where the bookies’ odds might be in your favour. 
Because that model was dependent on particular players (Teddy Sheringham and Darren Anderton were two key examples) its longevity was limited to two season 1995-96 and 1996-1997.  The work may have stayed forgotten, but in 2001 Fenton and Neil took on a PhD student Adrian Joseph at Queen Mary who was specializing in machine learning, which is all about methods that automatically ‘learn’ from data. Thanks to Adrian having access to relevant tools and algorithms, there was a unique opportunity for a direct comparison between the Bayesian net model and a range of machine learning methods. Such studies are relatively rare and the results and lessons learnt would be of interest to researchers outside of this particular domain (even those readers who have no interest in Spurs or football in general). Whereas the Bayesian net model was ‘fixed’ in the sense that its structure was predetermined by the expert and could not learn from new results, the machine learning models had no prior expert input but are continually updated in the light of new results. There were just two seasons with 76 results to learn from. The machine learners were fed a mass of information about each match (including far more variables than were used in the expert model). The expectation was that the machine learners would eventually outperform the expert model after enough results had been ‘learnt’.
Yet, after subjecting all the models to the most rigorous analysis (which is described in the paper) none of them came close to outperforming the expert model (although the researchers discovered interesting variations in the machine learners that also makes the paper a valuable contribution to that community). This confirms a belief that Fenton and Neil have become increasingly convinced of over the years: that expert-built Bayesian network models provide better predictions than can be achieved by any methods based on pure data analysis alone. Indeed, Agena has been exploiting this power of Bayesian networks with great success in applications ranging from air traffic management to predicting operational risk in banks.

So, if you want to predict your own team’s results in the coming season (even taking account of possible food poisoning scams!) Bayesian networks could be the answer.

  1. Joseph  A¤, Fenton NE, Neil M, “Predicting football results using Bayesian nets and other machine learning techniques” To appear Knowledge Based Systems (published by Elsevier), 2006
  1. Fenton NE and Neil M, ''A Critique of Software Defect Prediction Models'', 25(5) IEEE Transactions on Software Engineering, 675-689,1999.
  1. AgenaRisk (Bayesian network software package): available from www.agenarisk.com