English | EN 
Introduction
Tokens are the billing units for LLM APIs and a proxy for compute when you self-host. Your input context and output length determine cost per call. Testing burns tokens fast, especially with long contexts, few-shot prompts, and adversarial runs – creating pressure to under-test. This post explores what tokens mean in practice, how token costs distort testing, and foundational strategies to counteract these pitfalls.
What “tokens” mean in practice
A token is a chunk of text (often part of a word). API providers bill per input token (system prompt, instructions, retrieved context, user message) and per output token (the model’s response). If you self-host, tokens still correlate with compute time and memory bandwidth -> longer prompts/responses mean more attention steps and higher GPU time.
Rough mental model:
- 1 token ≈ ¾ of a word (varies by tokenizer and language).
- Short prompts (quick chat/QA): a few hundred input tokens.
- Typical app prompts (RAG over a couple of retrieved chunks, a few rules, short output): ~1k–3k tokens total.
- Heavy contexts (multi-doc grounding, few-shot exemplars, longer outputs): ~4k–20k+ tokens total.
Modern models support anywhere from 8k up to hundreds of thousands of tokens of context. Bigger windows enable richer tasks, but they scale cost linearly and can mask retrieval/prompt design issues if you rely on them too much. Treat tokens like a budgeted resource – in product and in testing.
How token costs distort testing (and how to counter it)
The subtle “bad incentives”
- Skipping long-context tests because they’re expensive – exactly where recency bias, context mixing, and tool-call drift show up.
- Shrinking samples to save money – metrics look stable but miss variance and regressions.
- Judging with only cheap models – issues appear at the deployment tier, not the cheap judge.
- Avoiding adversarial/red-team runs – iterative by nature (more tokens), yet they catch critical failures.
- Testing format, not truth – cheap syntactic checks without verifying grounded correctness.
The constructive “good incentives”
- Prompt/context compression and retrieval boundaries to keep contexts lean.
- Tiered pipelines: cheap pre-checks -> mid-tier judging -> expensive target-model E2E when it matters.
- Adaptive allocation: spend more tokens where risk and uncertainty are highest.
A tiered, token-aware test pipeline
Design to spend smart, not uniformly:
- Static & deterministic checks (near-free model): Policy/PII patterns, schema/JSON validation, tool contract checks. Fail fast before any heavy calls.
- Embedding-based pre-evals (low-cost model): Semantic similarity to gold answers to catch obvious misses without generation.
- Small-/mid-tier judge (medium-cost model): Use a smaller/cheaper model (or a lighter configuration) to score rubric items (faithfulness, completeness, structure). Route only borderline cases upward.
- Target-model E2E evals (high-cost model): Use your deployment-tier model for realistic long-context, tool-use, retrieval, and adversarial scenarios. Include strict structured-output validation.
- Canary & post-deploy monitoring (steady-cost model): Small, continuous slices to detect drift; feed incidents back into risk weights and budgets.
Policy: Don’t skip the highest tier for high-risk flows (external users, compliance exposure). Save elsewhere via compression, routing, and caching.
LLM Judge
In LLM testing, a judge is the component (often another LLM, sometimes a human or a deterministic checker) that evaluates a model’s output against criteria. Think of it as your automated QA reviewer that assigns scores or labels so you can compute metrics at scale without reading every output manually.
What a judge does:
- Scores against a rubric: e.g., faithfulness (grounded in retrieved evidence), completeness (answers all parts), format/structure (valid JSON), safety/policy compliance, clarity/style.
- Classifies outcomes: pass/fail, correct/incorrect, policy violation/no violation, schema valid/invalid.
- Ranks or compares: pairwise comparisons of two candidate answers to decide which is better.
- Explains briefly: provides a short reason (optional) to aid debugging and auditability.
Types of judges:
- Deterministic judges: schema validators, regex/PII detectors, unit tests for tool outputs – fast, cheap, reliable for structural checks.
- LLM judges (cheap tier): smaller/cheaper models used for coarse screening; good for high-volume, lower-stakes scoring.
- LLM judges (strong/target tier): the same tier as your deployment model, used for final gate on high-risk flows.
- Human judges: reserved for critical cases (compliance, legal, sensitive content) or to calibrate LLM judges.
Why use LLM-as-a-judge?
- Scale: You can score thousands of outputs quickly.
- Consistency: Rubric-driven prompts enforce repeatable criteria.
- Cost control: Route easy cases to a cheap judge; escalate only the ambiguous/high-risk ones.
Reliability & bias considerations (and fixes):
- Model bias: Judges may prefer longer or more fluent answers.
Fix: Cap output length; require evidence; use faithfulness-first rubrics. - Self-judging bias: A model judging its own output can be lenient.
Fix: Use different judge model; blind the judge to the generator’s identity. - Rubric ambiguity: Vague criteria produce noisy scores.
Fix: Use explicit, example-backed rubrics with clear pass/fail thresholds. - Overfitting to style: Pretty formatting wins over correctness.
Fix: Prioritize grounded correctness and schema validity before style.
A simple judge prompt template (illustrative):
| System: You are a strict evaluator. Score the candidate answer against the rubric. Only return valid JSON with fields: {“faithfulness”:0-1,”completeness”:0-1,”format_valid”:true/false,”policy_violation”:true/false,”explanation”:string} User: [QUESTION] {{question}} [EVIDENCE] {{retrieved_chunks}} [CANDIDATE_ANSWER] {{answer}} [RUBRIC] – Faithfulness: Is every claim supported by the evidence? No external facts allowed. – Completeness: Does it answer all parts of the question? – Format: If the task requires JSON/YAML, is it valid and matches the schema? – Policy: Any disallowed content or missing disclaimers? Return JSON only. If evidence is insufficient for a claim, lower faithfulness. “ |
Use deterministic validators (schema/PII) before the LLM judge to save tokens and to prevent style from masking correctness.
Conclusion & What’s Next
Token economics affects how we build and test LLM systems. Ignore costs and you burn money; obsess over costs and you under-test. The answer is solid engineering: compress what’s safe, tier your tests, and spend where risk is high. In the next part, we’ll dive into budgeting, optimization techniques, cost estimation, and actionable test patterns to help you build robust, cost-effective LLM test pipelines.