IoT Proof of Concept Fatigue

Internet of Things (IoT) is possibly one of the hottest technology trends today. One of the most circulated predictions about IoT states that there will be around 50 billion connected devices in the world by 2020. According to IDC, by 2021, IoT will drive spending of almost 1.4 trillion US Dollars worldwide. Any way you look at it, IoT is here to stay and its adoption is becoming ubiquitous and all-pervasive whether we like it or not.

However, the widespread IoT related optimism notwithstanding, there’s another aspect of IoT that tends to gets overlooked and ignored by the numbers above. This is the fact that a vast majority of IoT projects today turn out to a non-starter, i.e. they never go beyond the initial discovery or Proof of Concept (PoC) phase.  According to some analysts, the figure stands at 75% of all IoT projects undertaken today.

The key question, then, is this: how do we go from a point where 75% of all IoT projects are failing to a world where there are 50 billion connected devices by 2020? For this to be at all possible, we need to reverse this trend and we need to do it fast. And in order to do so, we first need to understand what is the root cause of these failures.

I have spent a lot of time reflecting on these questions and based on my experience I believe a that key achieving success is not just accomplishing a successful technical PoC but also demonstrating business value – i.e. creating a successful Proof of Value (PoV). Companies usually start their IoT journey by first conducting a POC with the assumption that when this initiative is successful they will move on to full-fledged production. And yet, based on our experience in Capgemini we have seen that this is not the case. More often than not even though the PoC is a technical success, the full-fledged implementation or rollout is not undertaken to lead to a number of aborted attempts at IoT adoption. We believe the roots of this failure can generally be traced back to how this POC was set-up and executed.

At Capgemini, we have seen that most of the time IoT POCs suffer from major failure fatigue. We have observed six main points which lead to such failures. These are:

1) Failure to demonstrate business value:

Surprisingly, even among the successful IoT projects only very few of them bring in actual business value. At Capgemini, we have found that a majority of our clients are struggling with how to make money from their IoT projects. It turns out that the majority of IoT Projects and in particular POCs when they begin to tend to have an excessive focus on proving the technical feasibility of a project rather than its business feasibility. However, we need to remember that technology is merely an enabler to a bigger goal, which in most cases is profitability.

This issue arises because POCs typically are the responsibility of R&D Departments. That’s where they begin, and business hardly ever gets involved. This is a big mistake.

Just to give you an example, we recently worked with a Medical Devices company manufacturing X-ray machines. Our brief was to conduct a POC to predict the failure of X-Ray tubes with an accuracy of at least of 96%. The idea was ensuring that company could consistently predict with high degree of accuracy when the component was going to fail. We worked with our clients R&D Department and were able to meet the technical requirements fairly easily. However, what was a lot tougher was proving it’s business value.

This happened because we interacted with the business team fairly late in the process. This was the team that would have to take it to the end-customers to ensure that the hospitals would have almost zero-downtime in their radiology division, thus improving customer satisfaction and increasing the utilization of their X-ray machines. Convincing the business team of these benefits turned out to be a much larger task than merely proving the project’s technical feasibility.

Based on our experience, our recommendation in order to avoid such issues is to involve the business side right at the beginning of every project in order to work out use cases which are easily demonstrable, add value and can be taken to the customer to get a quick win. So, in essence, focus on setting up a Proof of Value (PoV) which takes a holistic look the technical and business feasibility in order to move past the initial pilot.

2) Relying on poor quality or inaccurate data:

Normally, when we start a POC we often get input data given to us by the customer. Such data could be based on past data from their customer service centers or log data or data generated by their internal systems. In order to prove the use case, this data needs to be of extremely high quality. However, due to a multiplicity of factors, we often find that this is not the case.

Additionally, when we generate data during the POC, we are often pressurized to use low-quality sensors. This is mostly due to budgetary constraints put on the POC. Clients often feel that once the project moves from POC to actual production, then they will get better sensors. This ignores the fact that using low-quality sensors will give low-quality data and this, in turn, might lead to incorrect inferences. Even if the POC based on such data turns out to be successful, there is no guarantee that the actual project will work out along the same lines.

Take, for instance, the case where we worked with a set-top box company in North America. They were sourcing their set-top boxes from 6 different vendors located in China and Taiwan. For the PoC we used that data that was given by the client. In the later stages of the POC, we found that the actual data from some of the vendors were very different from that being used for our POC and this led to a number of anomalies in our results.

Another example is when we were working with an Indian home appliances company which needed IoT enabled appliances at a delta cost of more than 2 USD. Given the budget, we were asked to source the sensors from the local market which had very low tolerance for some of the parameters like temperature and vibrations. This gave us erroneous results and ultimately had to be replaced by higher quality sensors.

The learning here is that one has to be very careful about the data on which the POC is based.

3) Testing for scalability through simple extrapolation:

When testing for scale during a POC, it is extremely important to use appropriate simulators in order to get accurate results. Simple extrapolation will not do. Unfortunately, this is often the case. So if a team has 50 devices and wants to test for 5000, there is a temptation to just copy the same data 10 times to arrive at the results. Unfortunately, this is not the way 5000 real devices might work and in order to correctly replicate that behavior one needs a simulator.

The issue is that using a simulator increases the cost of the project. Also, if the team is geographically distributed, simulators often have to be shipped to various location. This also increases the time taken for the POC. It is to avoid this that teams often take shortcuts by simply extrapolating data for a larger number of devices.

However, this tendency needs to be avoided if one is to have any degree of confidence in the POC results.

4) PoCs / PoVs are under-budgeted in terms of both cost and time:

One commonality underlying each of the above points is that POCs work under very tight time as well as cost constraints. Due to this, the technology decisions for these POCs are often very rushed. However, our experience at Capgemini is that a quick shot POC is usually very short-lived. If you do a POC in anything like 8-10 weeks and expect the results to dictate your long-term strategy, you might actually start a project which hurts you in the later stages. This is because 8-10 weeks is simply not enough to do the kind of due diligence that is required for a long-term strategy.

I saw an example of this in one of our recent engagements where we were working with a smart toilet company from Australia. They had made exactly this kind of a mistake with their previous vendor and rushed through a POC. This had led to a tech decision that was hurting them. Due to this, they were in a situation where they could not scale their project to more than 5000 toilets per installation per server. This, in fact, had restricted their ability to bid for smart cities projects in Australia. They came to us to remedy this and we did a POC for close to 16 weeks to come up with the results necessary to support their business in the long term.

5) Involving only the R&D department from the client side:

Although I have already talked about this, it needs to be called out in a separate point. In fact, I can’t stress this enough- we absolutely need the involvement of a multi-disciplinary team both from customer’s side as well as from our own organization right from the beginning of an IoT POC project. Our observation is that most of the time, a project is won by the sales team with the target customer coming from either the business or the R&D division. However, very rarely do we get a project for a POC which starts with the involvement of both Departments. Such projects usually meet the expectation of only one party while both parties are absolutely a must to make it into production.

For instance, we were working with a company making pneumatic pumps and did a POC for one of their plants in North America. One part of this involved measuring the vibration analysis for a specific type of plant. Unfortunately, it was only after the completion of the POC that we were given inputs by the business team and realized that they had a roadmap for the product which was going to add more features apart from vibration. This ideally, we should have tracked multiple parameters during the POC for it to provide actual lasting value to the client.

Thus, from now on, we always insist on the involvement of a multi-disciplinary team before we start any POC. Of course, we often get clients who resist this in which case we make it clear that there could be multiple deficiencies at a later stage and the chances of making it to production might then be minimal.

6) Neglecting security during POCs:

IoT security has been talked about a lot and is undoubtedly one of its most critical aspects. Unfortunately, this is an aspect that often gets overlooked during the POC phase as teams start looking at security only during the actual implementation. However, the security architecture is key and there have often been security holes even for successful projects in this field.

Examples of such problems include Hospira which had to recall some of their IoT-enabled infusion pumps as they could lead to a drug overdose.  Similarly, nest the company which deals with home automation systems discovered that people could hack into their systems and damage them.

Thus, security is crucial for IoT-enabled devices and you need to address these issues during the POC. Otherwise, expecting architecture which was originally built for a POC to address such security concerns is not realistic.

At Capgemini, we believe that these are some of the main points which lead to a failure of IoT POCs and ultimately of the projects themselves. We believe that as IoT adoption picks up rapidly in the coming years (at least according to the forecasts), both us as well as our clients to take such points into account if we are to improve the success rate of IoT projects in the future.

In conclusion, there is much that IoT promises but to fully reap all of its benefits we must ensure that we select the initial use cases carefully taking into consideration the various aspects discussed above – namely  Business Value, Security, Quality of components and data and realistic expectations of about the time frame in which the initial project can be accomplished.

Proof of Value (PoV) which takes a holistic look the technical and business feasibility and takes into consideration long-term aspects of scalability and sustainability is required. In order to move past the initial pilot, we must make sure to avoid the pitfalls pointed out above.