Did you know that from the engines to the landing gear to the autopilot system and the altimeter, almost every part of a Boeing 787 Dreamliner jumbo jet is connected to the internet? As a result, a 787 generates more than half a terabyte of data per flight. To put that in perspective, it would take an entire month for 500 cellphone users – calling, texting, browsing, taking selfies, video streaming, tweeting, etc. – to burn through as much data as a single 787 generates in just one flight. Welcome to the brave new world of connected devices! According to a recent IDC study, worldwide data creation is expected to grow to an astounding 163 Zettabytes by 2025. That’s ten times the data generated in 2017. (1) And business data is growing even faster. As a result, a key driver of competitive advantage in today’s world is the ability to generate actionable insights from this overwhelming amount of data that every organization is swamped with. To enable this, many large enterprises are building customized, in-house data platforms. This, however, can be a tough nut to crack as there are certain basic requirements that such data platforms must meet. For a start, such platforms should be scalable, fault intolerant and able to work with a wide variety of data sources. There is also an implicit expectation that such platforms will allow self-service analytics and enable re-usability. As the platform matures, this will help reduce the development lifecycle. So, at its core, while a platform provides business with the capability to view and analyze data in ways it possibly couldn’t in the past, it also needs to be able to create a collaborative eco-system which will enable developers and the business community to come together and leverage lessons learnt and cut down on time-to-market. Because of the complexity of building such a platform, there are several challenges that both the development team as well as the project sponsors will face when working on this. There are a number of typical problems I have seen teams encounter over and over again. Hence, in this post I will talk about some of these forces that could potentially slow down or even stall such initiatives, and some strategies to tackle them.
About the author
Gopikrishna Aravindan is an experienced professional services leader who has a passion for everything technology starting from ideating concepts to delivering large-scale technology transformation solutions while taking ownership of everything in between. He has a Masters degree in Information Systems from Carnegie Mellon University. He started his consulting career at Deloitte and has served seve