Big Data – Then and Now


5 critical factors to turn data into insightWhen I first started my career in IT in the mid 90’s I was a Data Analyst. I absolutely loved it. I felt like Sherlock Holmes trying to find the one key ‘clue’ that would explain everything. It was a natural fit for me as I am very inquisitive and love to seek answers. I believe that no matter how great business is, it can always be better if we can just find the answer in the data.


Specifically, as a data analyst I worked with Point of Sales (POS) data from retail giants Wal-Mart and Target.


Wal-Mart was at the forefront of big data long before it even had a name. In the mid 1980’s they decided to build their own Consumer Behavior software that reported on this new technology called Barcodes. This was no easy feat; according to Frontline  they started on development in 1985 and invested 4 billion dollars building it over the next seven years. They officially launched “Retail Link” in 1992. It was truly revolutionary. All of a sudden, suppliers had access to all of their inventory and sales data by SKU, by hour, by store.  It was a massive amount of data and it was powerful.


As an analyst, I combined the sales and inventory data with weather patterns, major events, holidays and trends in the news. I also performed market basket analysis (aka Affinity Analysis) to see what products sold well together and when. I analyzed pricing, packaging and shelving strategies to see what trends and what worked and what didn’t.


It’s crazy when I think back to all I accomplished with so much data and ‘primitive’ analytical tools like Excel and SQL. I wonder what my job would look like with today’s tools and techniques?


Back then I didn’t have the tools to sift through so much data and information so I would start with a question and a hypothesis, such as “Will sales increase in years where Easter is earlier in the year?”, and then examine the data to see if it was true or not. It was cumbersome and time-consuming.


Compare that to today’s Data Mining that looks at the data and then presents patterns and outliers.


What used to be impossible for a human is now accomplished by technology. And not just any technology but fast, powerful machines that can learn and adapt.


My POS data fit nicely in CSV files with clear column labels and consistent data, whereas today’s most insightful data is often unstructured and come from many sources such as online blogs and social posts and photos.


While I had to wait until I had actual sales data to make changes to inventory, pricing, and shelving, today’s analysts can make changes proactively based on external leading indicators. For example if a recipe for Raspberry Chipotle Chicken suddenly starts trending on Pinterest the smart grocer will not only place the three key items near each other but will also add a QR code near the shelf with a link to the recipe online.


With online and social data, the amount of data only continues to grow. So does the need for analysts. McKinsey Global Institute predicts that “by 2018, the U.S. alone may face a 50 percent to 60 percent gap between supply and requisite demand of deep analytic talent.”


Data Source:


The data doesn’t lie.

Susan Thayer


Susan is an experienced CRM and digital marketing consultant with nearly 15 years of managing discovery, implementation and strategic use of CRM solutions and related programs. Prior to joining the consulting world, she served as director of marketing and other leadership roles for leading businesses in both the B2B and Consumer Goods industries. Past corporate employment includes digital marketing and e-commerce for a leading global auto parts corporation, medical products and a website design firm. Susan is a current board member for the Junior League of Lincoln and President of her HOA. She is a past board member for the American Marketing Association of Lincoln and past Public Relations Officer for the state of Arizona Toastmasters.

More on Susan Thayer.

Related Posts

Your email address will not be published.

  1. Parthajit July 12, 2016 Reply

    good read

  2. Анна October 5, 2016 Reply

    All this information needs to be available. But you also need the ability to understand what other data might be related to this. What other people bought with this data, for instance.  So, governance takes on an even broader meaning, so that the big data lake becomes something that folks can ‘fish’ from, in a useful and efficient way.