English | EN Building trust into enterprise AI
Generative AI makes it easy to build an assistant that sounds convincing. Building one an that organisation can trust is much harder.
Retrieval-augmented generation (RAG) is often presented as the solution to hallucinations: connect a language model to approved company documents, and its answers become grounded. Yet RAG can still retrieve the wrong passage, overlook the right one or generate a claim that its sources do not support. Fluency can hide all three failures.
The real question is therefore not simply, “Can the assistant answer?” It is:
When should the system answer, when should it abstain, and what evidence supports that decision?
Why a single quality score is not enough
Many RAG evaluations produce aggregate scores for relevance, faithfulness or answer quality. These measures are useful, but an average can hide important differences between successful and failed cases.
Coverage adds another perspective. It measures how often the system’s results include the right evidence or a correct answer across the questions it is expected to handle. However, neither an average score nor a coverage target is enough to decide whether the system can be trusted in practice. A risk owner must also ask:
– Where did the failure originate?
– How often does the system meet its intended level of coverage?
– Under which conditions should it stop answering?
– What evidence can be retained for review?
The first question is diagnostic: where did the failure originate? A RAG system has at least two distinct failure points. The retriever may fail to find the right source, or the language model may generate the wrong answer despite receiving relevant evidence. Treating the pipeline as one black box makes remediation harder.
TRAQ—Trustworthy Retrieval Augmented Question Answering—offers a more rigorous way to evaluate these two stages [1]. Using a labelled calibration set and a chosen error level, TRAQ constructs:
1. a set of passages intended to contain relevant evidence with high probability; and
2. a set of semantically distinct answers intended to contain a correct answer with high probability.
In this framework, a 90% coverage target does not mean that each displayed answer has a 90% probability of being correct. Under the method’s assumptions, it means the prediction sets contain a correct answer at least 90% of the time across the target distribution.
This statistical guarantee is useful evidence, but it does not decide whether a particular answer should be shown to the user. The organisation must still define when the system should answer, abstain or escalate to a human.
From research to enterprise control
Imagine an employee asking an HR assistant:
Can I carry unused leave into next year?
A conventional chatbot might retrieve an outdated policy, mix rules from two countries and still sound authoritative. A governed system should:
1. Retrieve relevant passages.
2. Generate answers grounded in those passages.
3. Apply calibrated retrieval and generation thresholds.
4. Verify that the source is authoritative and applicable.
5. Answer, abstain or escalate—and record why.
If the evidence is weak, contradictory or out of scope, “I don’t know” is not a failure. It is a control. The assistant can explain the limitation, show the available sources and route the question to a specialist.
For consequential decisions—such as employment, legal, health or financial outcomes—the assistant should support a qualified human, never become the final decision-maker.
Reliable AI begins with reliable calibration data
The effectiveness of the TRAQ approach depends on the calibration set.
It must be labelled, representative of the intended use and sufficiently consistent with future questions. If policies, user populations or query patterns change, the assumptions behind the calibration set may no longer hold.
The difficult work is therefore not limited to model engineering. Domain experts must help define correct answers, ambiguous cases and out-of-scope questions. In an HR context, that means involving the people responsible for policy interpretation before making any reliability claim.
Retrieval and generation need separate controls
An answer can fail at either stage of a RAG pipeline. During retrieval, the system may return the wrong document or miss the relevant one. During generation, the model may produce an incorrect or unsupported answer even when it receives the right evidence. Measuring the two stages separately turns evaluation into diagnosis. Once the source of the failure is clear, teams can apply the right fix: improve document indexing, metadata or chunking for retrieval failures; adjust prompts or answer thresholds for generation failures.
Trust has an efficiency cost
Higher coverage generally requires retaining more passages or candidate answers. Multiple generations and semantic comparison also consume more compute. Organisations must decide what level of risk justifies that cost and whether a smaller or local model can meet the requirement.
The right target is not “the most advanced model available.” It is the lightest architecture that meets the defined quality and risk threshold.
Trustworthy AI is AI that knows its boundary
Enterprise adoption will not be sustained by more persuasive demonstrations alone. It will depend on whether organisations can define, test and govern the boundary between useful automation and unacceptable risk.
The most trustworthy RAG is not the one that always has an answer. It is the one that can show why an answer is supportable—and knows when to say, “I don’t know.”
[1] Shuo Li, Sangdon Park, Insup Lee and Osbert Bastani, “[TRAQ: Trustworthy Retrieval Augmented Question Answering via Conformal Prediction](https://aclanthology.org/2024.naacl-long.210/),” NAACL 2024
