Book Review: The Signal and the Noise, by Nate Silver

Five stars out of five.

I remember watching the TV series Quantum Leap years ago, starring Scott Bakula and Dean Stockwell who traveled through time and lived at the whim of ever-changing Bayesian probabilities. Whatever predicament they encountered has associated chance of solution or death as spit out by Stockwell’s computer named Ziggy. As the conditions changed, say, a bad guy presents himself, Stockwell would translate the changing probabilities: “Ziggy says you have a 85% chance of dying.” With a bad guy vanquished, it would change, “Ziggy now says you have only a 15% chance of dying.”

Nate Silver’s The Signal and the Noise explains the phenomenon of prior probability that subsequently changes with conditions that was first presented as Bayes Theorem three hundred years ago. Silver is a professional poker player who made his “f**k you” money by selling software to analyze baseball statistics. Recently he became a phenomenon of sorts by successfully predicting all 50 state elections in 2012 and currently writes the always informational “five-thirty-eight” blog for the New York Times.

The Signal and the Noise is a more comprehensive discussion of Bayesian probabilities and how they affect almost everything humans do. An engineer friends says that every high school kid should read Silver’s book just to get an idea of how probability works and to hone critical thinking skills. I cannot disagree.

Okay, Silver is concerned with more than the theory of Thomas Bayes, he begins by discussing the huge explosion of data available to humans beginning with the Gutenberg printing press and evolving to the electronic databases today. He also points out that humans are unduly flawed because we too often seek confirmation of our own ideas instead of critically parsing all this information. Political pundits are notorious for

stoking the confirmation biases of their customers, which has been shown by the likes of Dick Morris and Karl Rove claiming to “have the math” only to be embarrassingly wrong on the outcome of recent elections. But they aren’t embarrassed-- they aren’t paid to be correct, only to confirm bias.

Silver also uses the baseball player Dustin Pedroia as an example of the failure of Big Data. On paper, or more accurately, using computer stats, a guy like Pedroia should never have made it into Major League baseball: he’s too small, too slow, not a very powerful hitter and can’t throw very well. Instead of failing, Pedroia is an all-star and a winner. What gives? There are intangibles, data is never able to be completely known. All the stats in the world cannot always be interpreted perfectly.

Separating what’s important (the signal) from confusing interference (the noise) is the key to forecasting any natural event or human phenomenon. We are good at forecasting some things like weather and hurricanes, but poor at predicting other things like earthquakes and terror attacks. Silver is an excellent explainer of how we know things and the limitations of that knowledge. He uses a wide-ranging array of interesting stories: from bird flu and climate to Donald Rumsfeld and poker. Refreshingly, I find no ideological or political preferences in his discussion.

This book is highly recommended.