Skip to Content

Swing states are dead, but did Big Data pull the trigger?

Menno van Doorn
November 13, 2012

A swing state: “A state in the US in which democratic and republican candidates both have a good chance of winning”. But what if predictions are so accurate that you get all 50 states right? That’s the end of the swing state, because there’s nothing to swing anymore. The man who killed them is Nate Silver. This is what he predicted and there is no need to show the map with actual results, because that would be the same map. Nate Silver is a data cruncher and author of The Signal and the Noise: Why So Many Predictions Fail — But Some Don’t.  He’s a former baseball statistician. He’s the one who showed that when you look at the data it’s relatively easy to predict who is going to win. Silver is also the one that proved that a lot of what was going on in the mass media was just data entertainment. The pundits are the “sultans of swing” and people love to be entertained, but it has nothing to do with real predictive analyses. We don’t have to be entertained about “will Florida go the Democrats”. If you want to know for sure you better look at the statistics. Data pulled the trigger of the swinging of states, but was it Big Data or Small Data? The best way to describe what Nate Silver did was to interpret polls predictively. If you’re a data cruncher yourself you might be interested to read the posting on his blog in which he explains his methodology. In short: he used other predictions to predict. The most important input for his predictions was curated data coming from the other polls. By weighting and adjusting the polls and not looking at the outliers of the different polls but the consensus he was able to do the trick. Some factors: Recency – More recent polls are weighted more heavily Sample size – Polls with larger samples receive more weight Pollster rating – Pollsters committed to disclosure and transparency standards receive more weight Trendline adjustment – If old polls haven’t been replaced, they are adjusted to reflect the overall trendline House effects – Some polls tilt right, some left, and this adjustment mitigates those effects Likely voter adjustment – Polls of likely voters are given a lot of credence The Big Data fundamentalists say this is not Big Data. The fact that Silver used his laptop to do the analysis was seen as the final proof. But does it matter? Maybe a decade from now we conclude that Big Data was just a word that helped us looking a data in a different way. It helped us to predict in an early stage who’s going to win. The word “swing state” disappears because “a good chance of winning” is redefined. Nothing else to conclude than that the real victim of (big) data is entertainment and excitement of what could happen. Here’s a video by the New York Times on how Nate Silver got it right

About the author

Director and Trend Analyst VINT | Netherlands
Menno is Director of the Sogeti Research Institute for the Analysis of New Technology (VINT). He mixes personal life experiences with the findings of the 19 years of research done at the VINT Research Institute. Menno has co-authored many books on the impact of new technology on business and society. This is the list of the books and research project he has worked on: Making IT-Governance Work Ope

    Comments

    2 thoughts on “Swing states are dead, but did Big Data pull the trigger?

    1. It’s a very nice theory. Nate Silver got all of the predictions right. Does that make him The Guru? Does that make his method The Best or infallible?
      I personally like to look at Big Data. In this case it’s the Huge number of predictions that was made. If you make so many predictions, one of them is bound to be right. In Nate The Guru? Or is he just the one that got lucky this time around?

    2. You’re making sort off the same point Silver is making in his book. He is critical when it comes to making predictions and Big Data. Don’t think he wants to be seen as a Big Data Guru.

    Leave a Reply

    Your email address will not be published. Required fields are marked *