In the latter years of the war, the US Navy wanted to improve its bomber fleet, so they commissioned a study. They looked at planes which had returned from missions, and looked at where they were most damaged. They found that bullet and flak damage to their wings and fuselage was the most common, so they planned to increase armour protection over those areas.

But a statistician, Abraham Wald, pointed out that this prescription missed a vital point. The reason that it was possible to see the damage on those planes was because those planes had successfully made it home. The planes that were hit on the engines were much more likely simply to have fallen into the sea. The data suffers from “survivorship bias”. Wald recommended that armour be added to the engines, rather than the wings and fuselage.

From there, to ‘secrets of my success’ management books. Dominic Cummings, this year’s Prime Minister’s Brain, has apparently been telling Downing Street aides to read one such, High Output Management, by the former head of Intel, Andrew Grove.

First things first: I haven’t read it and I’m not going to. It may be very interesting, but I have a long backlog of silly grimdark sci-fi to get through. Perhaps, therefore, this is the one book that really does contain the hidden secrets of leadership.

I’m wary, though. There is a long tradition of How I Made My Millions books being 200+ pages of unexamined survivorship bias: essentially, books written by pilots who happened to be hit on the fuselage rather than the engine, supposedly revealing the secrets of survival.

There are many thousands of start-up companies set up every year. Some of them will happen to make it big, but most will go bust. The founders of those don’t write books. It’s as if you took 10,000 people, all wearing different-coloured hats, and asked them each to toss a coin. Then you exclude the 5,000 or so who got tails. Then you do it again, and again. After ten throws, on average, you’d have nine or ten people who threw ten heads in a row. Then you get them to write books about how the secret of throwing ten heads in a row is wearing a taupe hat with vermillion stripes.

In his book Standard Deviations, the economist Gary Smith examined two earlier business books of this kind: Good to Great by Jim Collins, which looked at eleven of the best-performing companies up to that date, and looked at the characteristics they shared; and In Search of Excellence, which did the same thing with 43 “excellent” companies.

Smith pointed out that after the books were published, almost exactly half of the companies they examined started to underperform against the market. It was survivorship bias. Picking out winners with hindsight is easy; your model is useless unless it can predict future success.

As I say, I haven’t read Cummings’s favourite; maybe it has, indeed, discovered the correct colour hat you need to wear if you want to throw more heads. But my own suspicion is that it’s only very slightly more useful than asking a lottery winner up on stage at a graduation ceremony to share the secrets of his success.

---

Tom Chivers is a science writer. His second book, How to Read Numbers, is out now.

TomChivers

---