THE MEDICAL AI THAT DOESN’T KNOW WHAT IT DOESN’T KNOW

A problem far from solved

When shown an incorrect suggestion presented as coming from an AI, even experienced radiologists can be led astray. This is what a study published in Radiology in 2023 showed (Dratsch et al.). This is not a matter of competence: it is a well-known human bias, called automation bias.

So the real danger is not the error: no model is perfect. It is the confident error, the one nobody thinks of checking.

What this article brings

An early lesson learned from OvaScan, our research prototype for segmenting ovarian lesions on ultrasound:

1. Uncertainty helps spot errors. Failed contours are, more often than not, the ones where the model is most uncertain. More often than not, not always.
2. Where you measure matters. Measured on the lesion and its boundary, uncertainty spots errors far better than when averaged over the whole image.
3. Show the radiologist the doubt, not a percentage. In OvaScan, uncertainty is displayed as a map of the areas where the AI hesitates, next to the contour, before any validation.

Performance and calibration: two different questions

In medical AI, we measure a lot: precision, recall, Dice. But another question matters just as much:

When my model says “I am 90% sure”, is it right 90% of the time?

That is calibration. And it does not follow from accuracy. Guo et al. (2017) showed that modern deep networks, more accurate than their predecessors, are also more often overconfident.

In oncology, this is critical. A model can be excellent on average and far less reliable on rare cases: malignant tumours, scarce in the data, and ambiguous lesions. Yet these are precisely the cases that matter most.

The quiet cousin of hallucinations

A lot is said about large language models (LLMs) that confidently invent a reference: that is a hallucination. An imaging model that draws a wrong contour with the same assurance invents nothing; it is simply wrong while being sure of itself. It is not the same problem, but it is a close relative: in both cases, the model does not signal that it might be wrong.

And in both cases, measuring its uncertainty helps spot the error. For LLMs, an answer that changes from one attempt to the next often betrays a hallucination (Farquhar et al., 2024).

What already exists

• Monte Carlo Dropout (Gal & Ghahramani, 2016): dropout is kept active at inference and the prediction is repeated several times. If the answers diverge, the model is uncertain. Simple and widely used, but it is an approximation, and the inference cost is multiplied by the number of passes.
• Deep Ensembles (Lakshminarayanan et al., 2017): several independently trained models. Often better calibrated, but costly to train and deploy.
• Test-time augmentation (Wang et al., 2019, in medical segmentation): the model is shown several slightly transformed versions of the same image, and its disagreements are measured. No retraining needed.
• Conformal prediction (Angelopoulos & Bates, 2021): formal statistical guarantees, whatever the model. Promising, still rarely used routinely in medical imaging.

What we observed in OvaScan

Before even talking about calibration, a simpler question arises: when the model is uncertain, is it right to be?

In OvaScan, we asked this very concrete question:

does the model’s uncertainty actually spot its own errors?

On the validation images of the public MMOTU dataset, failed contours are a minority. And more often than not, they are also the ones where the model is most uncertain. More often than not, not always: a few errors slip through without the model hesitating, and some successful contours also come with doubt. Uncertainty is a useful signal, not a guarantee.

We also compared several ways of measuring this doubt: a single prediction, several flipped views of the same image, and Monte Carlo Dropout. On this data, and in our setting, heavier methods such as MC Dropout did not perform better than a simple approach. What matters most lies elsewhere.

Two public validation images, with the uncertainty map displayed in OvaScan. Case 1: the AI and the expert draw almost the same contour (Dice 0.97, i.e. 97% overlap with the expert), and the doubt is limited to a thin rim on the boundary. Case 2: the AI gets a large part of the lesion wrong (Dice 0.62)… and that is precisely where it signals its doubt. Here, doubt is estimated by comparing four flipped views of the same image; a single prediction gives comparable results. Examples chosen for illustration.

What matters most: where you measure. Whatever the method, uncertainty averaged over the whole image spots errors much less well. Measured on the lesion and its boundary, it is far more informative. In practice, the model is mostly uncertain at the boundary of the lesion, where the human eye may hesitate too.

In the application, these areas of doubt are shown to the radiologist. They know where to look before validating, and they always sign off.

What we do not claim: OvaScan is a research prototype, not yet a clinical tool. These results are preliminary: a single public, single-centre dataset, too few failed contours to draw firm conclusions, and an indicator that is not yet calibrated in the strict sense.

Open questions

Even on our own data, the question is not settled. Few failed contours, a single dataset, a single team of annotators: our early results are encouraging, but they are not enough to say which uncertainty measure is the most reliable, nor whether it remains so on rare or very small lesions. We will need more images, several centres, and ideally contours from several radiologists for the same image.

Does calibration survive a change of domain? Ovadia et al. (2019) showed that, in general, it does not: uncertainty degrades when the data shifts. The question remains wide open in oncological ultrasound: does uncertainty that is reliable on the training data remain so on another machine, in another hospital, with another imaging modality? We will come back to this in a future article.

Boundary or content? Doubt about the edge of a lesion and doubt about whether a lesion is there at all do not carry the same clinical meaning. The distinction between “aleatoric” uncertainty (image noise) and “epistemic” uncertainty (what the model has not learned), formalised for computer vision by Kendall & Gal (2017), is a starting point. Turning it into a signal that is useful to the radiologist is still work in progress.

Detecting “out-of-distribution” images: another machine, another hospital, a rare condition. The model should flag it instead of answering with the same confidence. This is far from solved.

Linking uncertainty to clinical cost: doubt about healthy tissue and doubt about a suspicious area do not weigh the same. There is no standard method yet for this weighting.

What it changes

A model that says “I’m not sure” is more useful than a model that is confidently wrong.

In medicine, acknowledged doubt is information. It tells the radiologist “look here”. It keeps the physician at the centre of the decision. It turns AI from an oracle into a tool for dialogue.

Good medical AI is not the one that is always right. It is the one that knows when it might be wrong.

Do you work on segmentation or uncertainty in medical imaging, or are you a radiologist to whom these questions speak? Let’s discuss in the comments, or get in touch directly.

References

• Gal & Ghahramani (2016). Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. ICML.
• Lakshminarayanan, Pritzel & Blundell (2017). Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. NeurIPS.
• Guo, Pleiss, Sun & Weinberger (2017). On Calibration of Modern Neural Networks. ICML.
• Kendall & Gal (2017). What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? NeurIPS.
• Wang et al. (2019). Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks. Neurocomputing.
• Ovadia et al. (2019). Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift. NeurIPS.
• Farquhar, Kossen, Kuhn & Gal (2024). Detecting hallucinations in large language models using semantic entropy. Nature.
• Angelopoulos & Bates (2021). A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification. arXiv.
• Zhao et al. (2022). MMOTU: A Multi-Modality Ovarian Tumor Ultrasound Image Dataset for Unsupervised Cross-Domain Semantic Segmentation. arXiv.
• Dratsch et al. (2023). Automation Bias in Mammography: The Impact of Artificial Intelligence BI-RADS Suggestions on Reader Performance. Radiology.