Fooled by Randomness?

There is a book by Nassim Nicholas Taleb called 'Fooled by Randomness' which I have referenced several times on this blog, and whose importance has very much returned to the fore given recent events at Charlton.

The key conclusions of the book (which I urge you to read), are that humans tend to overestimate 'causality', and view the world as more 'explainable' than it actually is.

When Charlton's Board chose to dismiss Alan Pardew, it is vital that they were not being 'fooled by randomness'.

I have been very adamant of course that it was not randomness that explained our lowly position, and it seems the Board agreed. However if we were wrong, then it would have potentially very detrimental effects on the club's future.

Moreover, if results under Phil Parkinson (or his permanent successor) suddenly improve, then it would be easy (and perhaps wrong) to suddenly proclaim the new man as some type of managerial genius, and Pards as a fool.

It is well-known that when teams sack the manager of an underperforming side, the subsequent results under the new leader tend to improve. Spurs under Harry Redknapp would appear to be a very obvious and recent example of this phenomenon. Pards himself seemed to have a similar impact when he first joined Charlton.

Very few Spurs fans would suggest that Juande Ramos would have presided over the same set of results, yet all one may be witnessing is a simple 'reversion to the mean'.

Moreover, form has momentum (a team's most recent results provide predictive value), yet the catalyst for a sudden run of good or bad form is likely the mere reversion, not some silver bullet like the arrival of a new manager.

When Ramos departed, Spurs were languishing on 2 points from 8 games. Even the most pessimistic Spurs cynic would have concluded that it was most unlikely that they'd have finished the season under Ramos on just 10 points, yet that is what their points-per-game implied.

In short, it is impossible to conclude definitively that Redknapp is responsible for the club's turnaround. Random factors will have played a part too, yet one suspects 'Arry would be loathed to admit it.

That amazing point they secured at Arsenal galvanised their season, but the manager does not influence the outcome at 4-2 with injury time looming. His excellent record at Pompey suggests he does add considerable value over time, but quite patently in this case, not yet.

When I argued that Pardew's reign at Charlton had reached a natural end, I tried to present it not merely in terms of no wins from his last eight games, but as relegation form over the equivalent of a full season.

I placed this form into the context of what a reasonable expectation should have been over that period, given resources at the (well-remunerated) manager's disposal.

Clearly the more data points one uses, the less likely it is that random factors will override non-random ones. I thus concluded that 49 points from 46 games was unacceptably bad for such a well-paid individual, and likely not randomly explained.

Pardew has now gone, and Parkinson's reign has begun with a seemingly undeserved defeat. With Saturday's game being described as a relegation six-pointer, there is clearly a risk that fans misguidedly make conclusions on Parkinson's suitability for the role full-time, at full-time. This would be a mistake.

One particular statistic from the last 9 games (4 home, 5 away) is an amazing outlier, yet it goes a long way to explaining the miserly three points we've accumulated over that period: we've conceded the first goal in all of them. This statistic is extraordinary.

Approximately 1 in 11 matches finish goalless (expressed in odds as 10/1). In the Championship meanwhile given the relative closeness of most match-ups, it is reasonable to conclude (again ignoring a profit margin) that the odds of the home team scoring the first goal are typically say 4/5 and the away team, 15/8.

Thus if you wanted to place a £100 bet before the away game at Cardiff that Charlton would concede the first goal in its next nine games, you'd have collected a whopping £129,096 and still retain the option to roll it forward to Saturday at 15/8!

Nobody did place that bet, but I highlight it to demonstrate just how incredibly unusual this run is. It is more unusual than doing ten coin flips, and getting a head on each of them.

Given the expected number of points a team will take from a game drops precipitously as soon as that first goal hits your net, a sudden turnaround in form under Parkinson is just as likely to be driven by the same high degree of randomness, if it comes after we've begun scoring the first goal again!

After all, if the only piece of information that you had on our last 9 games was that we'd conceded the first goal in all of them, your best guess for the total points we'd have accumulated would likely not be much more than the 3 we did pick up. It is surprisingly rare how often teams win matches from behind.

As I've noted before, we scored the first goal in 7 of our first 10 games accumulating a respectable, albeit not earth-shattering 13 points. We contrived to lose 3 of those 7 which is also extremely unusual, but anyhow perhaps it's just a weird old season.

If Parkinson was given 10 games as caretaker boss, and if he accumlated 13 points, I suspect most fans (and the Board) might conclude he was worthy of earning the job full-time.

Yet as I've hopefully demonstrated, this might be nothing more than a return to a more regular distribution of the first goal, which realistically Parkinson will not be able to affect.

He might claim otherwise, but he this would be a prime example of so-called 'confirmation bias'. Eventually the coin flipper will turn up a tail.

Anyone who'd seen those 9 games however, would probably not conclude that we had mere randomness to blame. The fact that we can't pick up markers in our penalty box, that we are overly reliant on flaky loan signings, and that Luke Varney can't hit a cow's backside with a banjo are all clearly factors too, which Pards surely did not do enough to address.

In conclusion, the factors which I urged the Board to use when evaulating Pardew's future, should be the same as those used to evaluate Parkinson's. Short-term results should not be key amongst them, just as I don't believe ultimately they were under Pards (the rot started a year ago).

Instead, the Board should decide whether Parkinson can improve those aspects of the team that he does have influence over, and does he have the organisational skills and vision to build all aspects of the playing side of the club for the long-term. In other words, is he a manager in whom we can have confidence, whichever division we begin 2009/10 in (and beyond)?

I dearly hope we wallop Southampton on Saturday, but let's not read too much into it if we do; it's highly unlikely Parkinson will have had anything to do with it yet. In short, we will have been fooled by randomness.