Sustainable development for self-service analytics platform

Did you know that from the engines to the landing gear to the autopilot system and the altimeter, almost every part of a Boeing 787 Dreamliner jumbo jet is connected to the internet? As a result, a 787 generates more than half a terabyte of data per flight. To put that in perspective, it would take an entire month for 500 cellphone users – calling, texting, browsing, taking selfies, video streaming, tweeting, etc. – to burn through as much data as a single 787 generates in just one flight. Welcome to the brave new world of connected devices! According to a recent IDC study, worldwide data creation is expected to grow to an astounding 163 Zettabytes by 2025. That’s ten times the data generated in 2017. ⁽¹⁾ And business data is growing even faster. As a result, a key driver of competitive advantage in today’s world is the ability to generate actionable insights from this overwhelming amount of data that every organization is swamped with. To enable this, many large enterprises are building customized, in-house data platforms. This, however, can be a tough nut to crack as there are certain basic requirements that such data platforms must meet. For a start, such platforms should be scalable, fault intolerant and able to work with a wide variety of data sources. There is also an implicit expectation that such platforms will allow self-service analytics and enable re-usability. As the platform matures, this will help reduce the development lifecycle. So, at its core, while a platform provides business with the capability to view and analyze data in ways it possibly couldn’t in the past, it also needs to be able to create a collaborative eco-system which will enable developers and the business community to come together and leverage lessons learnt and cut down on time-to-market. Because of the complexity of building such a platform, there are several challenges that both the development team as well as the project sponsors will face when working on this. There are a number of typical problems I have seen teams encounter over and over again. Hence, in this post I will talk about some of these forces that could potentially slow down or even stall such initiatives, and some strategies to tackle them.

Quick wins:

Large data engagements mean huge upfront costs in terms of spinning up servers, licenses and associated team effort. With a bill of this size, there can be pressure from stakeholders to deliver software components that’s valuable to the business. While, such engagements are usually agile and almost always set up with a CI/CD ecosystem to deliver releases continuously and frequently, it may not help if the final output is of no substantial value to business. To address this challenge, teams should plan right upfront about quick wins that can be achieved. A quick win is a feature that can be implemented in the least amount of time and delivers the most value to business. Development team will need to work closely with the Product Owner or a subject matter expert to understand and plan use cases that can deliver quick wins to the business, in a form that is usable i.e. as an end-to-end integrated solution. A desirable side effect to such an approach is that by testing the platform with features of lesser complexity, the development team is in a better place to optimize the platform for more complex and perhaps critical use cases.

Clear Vision:

There is a classic case of disconnect between business and technical teams when the final product is delivered. While agile processes have product demo ceremonies, they often end up turning a development team only event. Engaging business at a later point in the development cycle increasing risk of gaps in business and IT alignment which could result in costly amendments. On the other hand, development team may not get sufficient traction from business, if business see no value in engagement at early development stages. To ensure alignment, business and development team should jointly develop a shared vision for the platform that is focused, desirable and easily communicable. Sometimes a high vision statement allows the enforcement of some critical requirements through the entire cycle of development.

Building trust with user community:

Just like other complex system implementations, data platforms are not without errors in initial releases. Even one unhappy user can damage reputation earned through several weeks or months of development. This may result in an undesirable situation, from which the team and its leadership may find hard to recover. This is why features should be released to a closed set of trusted users. This way even when issues are identified with the software, there is a clear process and a more productive environment supporting bug fixing. The release manager should ensure that user receive timely communication on short term and long terms releases so that users can makes themselves. Additionally, clear articulation of business benefits of using the tool/ feature in their daily operations will help build an amicable relationship that the development team can use in their favor. These are the top 3 basic strategies, which can help avoid a number of issues when building in-house data platforms. Are there any common issues that you or your team has been struggling with? Feel free to share in the comments section below, as I’d love to hear from you. Featured image source