Why your Big Data is Useless?

Big Data is going to change everything and help us to gain a greater understanding of the world around us.  With the volumes of data we are collecting, we will be able to predict everything from illness, maintenance, weather, customer needs and more.

Well… maybe.

Kitman Labs is the “World’s First Athlete Optimization System” which transforms multiple sources of data into information which helps keeps professional athletes from getting injuries.  They are working with professional sports team around the world to make sure that their top stars are on the pitch as much as possible.  With their products, teams can monitor athletes to make sure they get the best performance possible.  However, their data does not come from just one club or sport.  When Kitman predicts an issue with an athlete, it is based on data from all the clubs and sports they work with.

Let’s say a professional soccer team has 50 professionals, how many injury data points could this one club generate?  Big Data is only useful when you collect a variety of data.  If we only collect data about things which happen a lot, we will only be able to predict these common events.  If we want to predict other events, we need lots of variety in the data.  If we want to predict injuries, we need lots of data on injuries, not data on perfectly fit Athletes.  Kitman collects data from lots of sports clubs and as a result, has more data on injuries than anybody else.

So to take advantage of Big Data you need to collect more data variety.  For example:

• If you want to predict if your staff will find another job, I wouldn’t just look at your own internal HR data. I would analyze LinkedIn data and finding out how many similar resources change their role and why.
• If you want to find the best marketing campaign for your product, don’t look at your own campaigns and sales. Look at your competitors and buy shop Point of Sale data which shows how competitors sales changed.

Also:

• Consider “as a service” products which give you indirect access to large datasets of bad data.
• If you are collecting and storing your own data, combine it with as many external datasets as you can and try to have as much variety in the data as possible.
• Use data to support your decision-making. Don’t use data to make your decision (see one of my other blogs here)