Curse of the Magic Box, or Why You Need to Educate your Users

Do you know which major risk you might come across during a data science project? Having your team do such amazing work that the resulting model looks, well, like magic.

Does your model predict the right values? Yes.

Is it stable? Very much so.

Does it automatically take context changes into account? Smoothly, yes, and on its own.

So, why is it not being used on a daily basis? Why the vague answer of « well, we need to test it a bit more before we really use it» from the client’s technical teams?

Well, because in the case of a data science project, what is ultimately being asked of us is to craft a tool that will reliably produce actionable insights from the client’s data. And this is exactly the point: a reliable tool. And a magic box is no tool. A tool will be used in production, be extended, and generate new needs. A magic box will make for a cool demo, at most.

What is the difference between the two? It lies in the capacity of its users to understand the underlying logic, if not the details at least the governing principles. It is a question of, as one customer puts it, «wanting to see the gears turn to trust they work well».

Therefore, it is our belief that most data science projects must include a data science course for our customers, covering:

• What do we mean by data science? By AI? Deep learning?
• What can each set of techniques do?
• What can they not do?
• What are the main technical approaches to data science, when are they applicable?
• What is the goal of each type of modeling approach, and what are the success criteria?
• And lots, lots, lots of hands-on examples, from post-it based mockups to actual code exercises

When can we tell our explanation has been sufficient for trust be possible? I would argue it is when the user develops an intuition of how the model might react to changes, which is why we strive to include members of the customer’s team in our development task force (there are other reasons, but this is an important one).

So when we hear a customer say «well, the model is behaving in an odd manner, I did not expect it to do that», we know we’re in the right direction to deliver a trusted tool.