English | EN 
LLM-as-judge has quickly become a practical way to evaluate AI systems. It helps when there is no simple answer key for qualities such asfaithfulness, helpfulness, tone, completeness, or whether important caveats were preserved.
But there is a problem: Teams often validate the AI system carefully and then trust the judge without validating the judge itself.
That turns the judge into an untested test oracle.
A judge is a measurement instrument
An LLM judge is not “just another prompt.” It is part of your evaluation system. If the judge is inconsistent, biased, or drifting, every score it produces becomes questionable.
A polished dashboard can still be built on unreliable measurements.
Common judge failures are already well known. A model may prefer the first answer it sees in a pairwise comparison. It may reward longer answers even when they are less correct. It may favour outputs that resemble its own style. It can also be impressed by confident formatting while missing factual errors.
These are not rare edge cases. They are predictable failure modes and they should be tested.
Calibrate before you trust
Before using an LLM judge for release decisions, build a small human-labelled evaluation set. The examples should reflect real production cases and include near-misses, not only obviously good and obviously bad outputs.
Then compare the judge with trusted human or domain-expert decisions.
Start simple. Binary pass/fail judgments are often easier to interpret than vague 1–5 scores. Ask the expert to explain why an example passed or failed, then use those critiques to improve the rubric.
The goal is not to prove that the judge is perfect. The goal is to understand when it is reliable enough for the decisions you want it to make.
When the judge disagrees with humans, study the disagreement. It usually points to one of three things: a judge defect, an unclear rubric, or a genuinely difficult case.
That analysis is often more valuable than the overall score.
Treat judge changes as breaking changes
An LLM judge has several moving parts: the model, model version, prompt, rubric, decoding settings and scoring anchors.
Any of them can change the measurement.
If a provider updates the model or someone edits the rubric, the new score may not be comparable with last month’s score. Judge configuration should therefore be versioned just like application code.
A useful pattern is to keep two judges:
- A frozen judge for regression testing and release gating.
- A current judge for experiments and improvements.
When the current judge is promoted, re-score a fixed anchor set and document the difference. Do not silently change the measurement instrument and pretend the historical numbers still mean the same thing.
Use deterministic checks first
Not every quality check needs an LLM.
Schema validation, valid JSON, required fields, citation presence, pattern-based secret detection and many policy checks can often be handled with deterministic rules or classifiers.
Use the LLM judge where interpretation is genuinely required: faithfulness to retrieved context, tone, preservation of caveats, nuanced completeness or semantic quality.
A simple evaluation cascade can therefore be:
Deterministic checks → Lightweight classifier → LLM judge
This reduces cost, latency and unnecessary variance.
Decompose the rubric
“Rate this answer from 1 to 5 for quality” is difficult to defend and even harder to debug.
Instead, break quality into smaller questions:
- Did the answer address every part of the request?
- Are factual claims supported by the available context?
- Are limitations and caveats clearly stated?
- Does the answer introduce unsupported information?
These smaller judgments are easier to test, explain and improve.
The takeaway
LLM-as-judge is powerful, but it should not be trusted by default.
Validate it against human judgment. Test known biases. Version the complete judge configuration. Run deterministic checks before expensive judge calls. Use clear, decomposed rubrics. Revalidate whenever the judge changes.
If you cannot explain how well your judge agrees with trusted human decisions, you do not yet have a reliable evaluation system.
You have a number.