WHY ARE PEOPLE VULNERABLE TO RISK IMPLICITNESS?

In the previous blog, I shared my observations from sessions in cross-functional teams. I noticed that colleagues often struggle to articulate risks with enough clarity. Sometimes, vague notes were recorded, which later lost context and became meaningless. This led to missed opportunities to refine both our quality and test strategies. Ultimately, if risks aren’t made explicit, quality and test strategies suffer, and teams risk missing crucial actions needed for successful software delivery.

What does research suggest about why people exhibit this behaviour? What factors increase implicit risks? I will discuss the cognitive effects that relate to this behaviour. Note that the discussed effects do not mean that we are not intelligent. Rather, the opposite. In many situations it’s an evolutionary benefit to process information quickly. We can process information at lightning speed, allowing us to navigate the world efficiently[1]. We can walk, talk and think at the same time. Moreover, the heuristics our brain uses simplify decisions, but as a result we are vulnerable to cognitive biases. In modern complex work environments, implicitness is a risk you cannot afford to take. Being blind on the way to a production release should tell that you’re not in control. In these situations, you want to trigger System 2 thinking, so it can monitor the quality of the output of System 1[2]. Let’s now dive into the biases that increase the likelihood of risk implicitness.

Illusion of Control | The false belief that we can fully influence outcomes within complex systems

Illusion of Control is an overconfidence bias. We think we have quality under control, however that is not the case. Due to the Illusion of Control[3], we tend to believe we can influence software quality to such a degree that it won’t go wrong: confidence replaces verification. Quality is experienced beyond your control. Product quality is decided by the product owners, project managers, and end-users.

Despite our efforts, historical events[4] keep telling us that software still can contain production-breaking bugs. Small mistakes and assumptions can negatively influence the capability and outcome of software. As a cross-functional team member, you are dependent on others to know what to build. Moreover, you are not in control of how colleagues such as architects, other developers, testers and operations deliver quality. However, they directly influence how code is written or corrected.

You might think that this overconfidence bias won’t happen to you. That you will not build software that works incorrectly. However, recognize that you are not in control of the complex systems we build and work with. This matters because quality is complex. People have different interpretations, and it’s mostly a tacit process. It is never fully expressed in explicit requirements or verbal statements. Moreover, perception of quality can change. As said, you are not in control of how others experience quality. Stakeholders might think they expressed the desired quality adequately, but they never tell the whole story. The Illusion of Explanatory Depth is also relevant here. Which is discussed later in this article.

Introspection Illusion | Trusting your own thoughts can lead to overlooked risks and misplaced confidence

Introspection Illusion explains that people tend to believe they have access to the origins of their mental outcomes. However, self-assessment does not necessarily provide accurate self-knowledge[5]. One might think they are rational, but our thought process is not as trustworthy as we think. As a result, we have unjustified trust in our memory and moral compass. Evidence suggests that environmental factors, such as time pressure, have greater impact and we might forget or miss risks because of it.

Moreover, the illusion can also lead to underestimating the insights of others. This is due to the bias that people overweigh their own thoughts, while discounting thoughts from others as unreliable or biased[6]. Your own introspection is more vivid and feels more reliable. However, this mistaken accuracy leads people to believe their understanding is deeper than others’, even while it is not. As a result, they might mistake confidence for correctness.

You can see the effects of this illusion happen during refinement sessions. A premature dismissal of valuable insights from others might harm the refinement of a feature. That insight from your colleague might result in an unforeseen risk, or a clarification of a requirement.

Another potential outcome in cross-functional teams is the tendency to overlook valuable perspectives provided by colleagues from different areas of expertise. If you disregard the needs of, e.g., a tester, the team risks suboptimal solutions which might damage testability.

While spending several minutes discussing an item, it may lead you to believe that you fully understand it, but there is often more to uncover. For example, colleagues can offer valuable perspectives regarding the required quality of the backlog item. Even if you feel certain you understand all associated risks, it is important not to underestimate potential gaps in shared understanding. This misconception is common, especially among product owners, who may prioritize addressing multiple backlog items during refinement sessions rather than pausing to thoroughly evaluate risks that could be present within a single item.

Illusion of Explanatory Depth | Believing we understand complex topics deeply, only to realise our knowledge is superficial when asked to explain them

Another cognitive phenomenon is the Illusion of Explanatory Depth. It tells us that people tend to believe that they know exactly how complex things work. Such as a toilet, a zipper or the economy. But when they are asked to explain it in detail, they notice that it’s harder than they thought[7]. This bias explains why writing out detailed, non-ambiguous, and complete requirements or user stories is hard. We talk about quality all the time. But when writing it down, in detail, you recognize it’s more difficult than thought at the beginning. So, don’t rely on someone else’s knowledge: don’t borrow confidence to inflate your own understanding.

We often confuse our familiarity with a subject for a deep understanding of it. You think you understand the complete system your team works on? Have you ever tried to model it? What components are there? How does it work internally? What business rules are there? How do these work? How does it interact with other systems? Try it and see if you fall to this overconfidence effect. If your knowledge is superficial, you should be aware that unknown risks might remain.

The Knowledge Community at the center | People mistake access to others’ knowledge for personal understanding

All the cognitive effects described so far, reinforce each other and come together in The Knowledge Community. The Knowledge Community is the idea that no individual holds more than a tiny fraction of the knowledge required to understand or operate complex systems[8]. This effect establishes the Illusion of Explanatory Depth: people mistake access to the Knowledge Community for individual possession of that knowledge.

Ever wondered how complex enterprise software systems keep running? Individuals within the enterprise only have a fraction of knowledge of such systems. No individual can tell you how the whole system works. Only the collective knowledge of all individuals can tackle that question.

Within quality engineering & testing, we also rely on other people’s expertise to build the right quality at the right time. The trick is knowing who knows what. Instead of knowing everything yourself. Leverage the community’s knowledge. Thus, when dealing with risks, use the knowledge of the community. Close knowledge gaps and identify risks from external perspectives. You don’t want to keep important risks that ‘live’ within the community implicit.

Quality can be a joint responsibility

The negative effects and biases from the Knowledge Community raise questions about the statement that ‘quality emerges from joint responsibility’. One person cannot hold the community’s quality knowledge. So how does your team manage quality and knowledge?

Ownership/decision rights

• Who decides what is adequate quality?
• Who decides what to do with risks?
• Who prioritizes risks?
• Who decides to release to production?
• Who is ultimately responsible for the product?

Risk discovery & learning loops

• When and how are the important risks discovered?
• Is risk analysis a continuous process?
• What does your culture look like when things go bad?
• What do your feedback loops look like?

Knowledge externalization

• Is your tester the integration point for knowledge?
• How are we learning, accessing, and distributing knowledge as a team?
• What external memory do we use?
• How do we deal with repeated discussions, historical context and onboarding?
• How is the needed expertise in our team handled?

Testing/automation misconception

• Are testers testing, or are they overloaded by treating them as the carrier of quality?
• Are testers unjustly taking quality ownership?
• Is automation (in testing) helping us, or a burden?

Many of you know that the cross-functional team does not hold accountability for product quality and decide what product quality should be. They are, however, responsible for the delivery of the product quality. Think of quality measures aimed at building quality in. Questions that help focus on ownership are:

• Why are you doing Scrum?
• Why do we have a separate team for infrastructure?
• Why are architects not part of the team?
• Why are operations outside the team?
• Why don’t we speak with end users?

Problems with software quality are rarely technical failures. They are failures to integrate a distributed Knowledge Community. Below is a table that can be used to identify Knowledge Community problems.

To summarize, these cognitive biases and fallacies are why teams keep walking blindfolded into risk. Not because risks are absent, but because our brains are wired to keep them implicit. Cognitive biases make us feel confident, even when our understanding is partial. In Knowledge Communities, that confidence is amplified: access to expertise feels like possession of understanding. As a result, teams believe risks are “covered” when they are merely discussed, nodded at, or assumed to be understood by someone else. But quality and test strategies depend on risks being explicit. When risks remain implicit, teams optimise in the dark: they test the wrong things, decide too late, and discover a lack of quality only after it has already escaped into the later phases of the software delivery lifecycle. Explicit risk is leverage. It is the moment the blindfold comes off, allowing teams to deliberately steer quality, focus effort, and make conscious trade‑offs instead of accidental ones.

Teams can use various quality measures to reduce implicit risks. In my next blog, I’ll recommend some actions you can take and experiment with.

[1] https://thebrainmarketer.com/the-unconscious-brain-the-hidden-engine-behind-our-decisions/

[2] Daniel Kahneman (2011). Thinking, Fast and Slow.

[3] Thompson SC (1999). “Illusions of Control: How We Overestimate Our Personal Influence”

[4] Categories for convicted person and suspect were configured the wrong way around

[5] D.C.Demetre (2023), What Is the Introspection Illusion.

[6] Wilson, Timothy D. (2002). Strangers to ourselves: discovering the adaptive unconscious. 

[7] Rozenblit L, Keil F. The misunderstood limits of folk science: an illusion of explanatory depth.

[8] Frank Keil – Doubt, Deference, and Deliberation: Understanding and Using the Division of Cognitive Labor