English | EN Big Data – Then and Now
Jul 4, 2016
When I first started my career in IT in the mid 90’s I was a Data Analyst. I absolutely loved it. I felt like Sherlock Holmes trying to find the one key ‘clue’ that would explain everything. It was a natural fit for me as I am very inquisitive and love to seek answers. I believe that no matter how great business is, it can always be better if we can just find the answer in the data.
Specifically, as a data analyst I worked with Point of Sales (POS) data from retail giants Wal-Mart and Target.
Wal-Mart was at the forefront of big data long before it even had a name. In the mid 1980’s they decided to build their own Consumer Behavior software that reported on this new technology called Barcodes. This was no easy feat; according to Frontline they started on development in 1985 and invested 4 billion dollars building it over the next seven years. They officially launched “Retail Link” in 1992. It was truly revolutionary. All of a sudden, suppliers had access to all of their inventory and sales data by SKU, by hour, by store. It was a massive amount of data and it was powerful.
As an analyst, I combined the sales and inventory data with weather patterns, major events, holidays and trends in the news. I also performed market basket analysis (aka Affinity Analysis) to see what products sold well together and when. I analyzed pricing, packaging and shelving strategies to see what trends and what worked and what didn’t.
It’s crazy when I think back to all I accomplished with so much data and ‘primitive’ analytical tools like Excel and SQL. I wonder what my job would look like with today’s tools and techniques?
Back then I didn’t have the tools to sift through so much data and information so I would start with a question and a hypothesis, such as “Will sales increase in years where Easter is earlier in the year?”, and then examine the data to see if it was true or not. It was cumbersome and time-consuming.
Compare that to today’s Data Mining that looks at the data and then presents patterns and outliers.
What used to be impossible for a human is now accomplished by technology. And not just any technology but fast, powerful machines that can learn and adapt.
My POS data fit nicely in CSV files with clear column labels and consistent data, whereas today’s most insightful data is often unstructured and come from many sources such as online blogs and social posts and photos.
While I had to wait until I had actual sales data to make changes to inventory, pricing, and shelving, today’s analysts can make changes proactively based on external leading indicators. For example if a recipe for Raspberry Chipotle Chicken suddenly starts trending on Pinterest the smart grocer will not only place the three key items near each other but will also add a QR code near the shelf with a link to the recipe online.
With online and social data, the amount of data only continues to grow. So does the need for analysts. McKinsey Global Institute predicts that “by 2018, the U.S. alone may face a 50 percent to 60 percent gap between supply and requisite demand of deep analytic talent.”
Data Source: Bloomberg.com
The data doesn’t lie.
About the author
