English | EN 
Introduction
In Part 1, we explored why token economics matter in LLM testing and how to avoid the traps of under-testing. Now, let’s get practical: this post covers budgeting, optimization techniques, cost estimation, and essential test patterns for building a robust, token-aware test pipeline.
Budgeting basics
You don’t necessarily need complex formulas; here are some sample planning ranges for pass-rate style metrics:
- For rough ±5% confidence → ~400–600 cases.
- For tighter ±2–3% → ~1,000–2,000 cases.
- For very tight ±1–1.5% → ~4,000–8,000 cases.
These are rules of thumb (depend on actual pass rates), but they keep claims honest. If someone proposes 100 cases with ±2% precision, that’s a red flag.
Practical rule: Spend tokens until the next test tier wouldn’t materially reduce uncertainty in high-risk areas. Reduce spend in low-risk areas first (smaller samples, cheaper judges, compressed prompts).
Controls that prevent corner-cutting
- Minimum standards by risk class:
- External / Compliance / Mission-critical: All tiers required; include long-context and adversarial coverage; judge with deployment model for final gate.
- Internal / Low impact: Smaller samples; more reliance on cheaper judges.
- Coverage thresholds: Long-context slices; tool-chains (including failure paths); structured outputs with schema checks; retrieval failure modes.
- Budget transparency: Dashboards show tokens per tier, cost per metric, sample sizes, and precision achieved.
- Exceptions with guardrails: If tests are reduced, document the risk, add compensating controls (canary + fast rollback), and schedule a follow-up run.
Practical optimization techniques (works across models)
- Prompt compression: Summarize prior turns; compress reusable instructions; remove redundant examples; prefer few high-quality exemplars to many mediocre ones.
- Retrieval boundaries: Cap top-k results, restrict chunk sizes, and use sliding windows. Don’t let retrieval silently balloon context.
- Caching & reuse: Cache system prompts, policy blocks, and any deterministic tool responses. De-duplicate retrieved chunks.
- Multi-judge routing: Cheap judge first for coarse scoring; route only uncertain or high-risk cases to the expensive tier.
- Early stopping: If interim results are clearly above/below thresholds, stop and reallocate spend to weaker areas.
A simple cost estimator you can drop into CI
Replace placeholders with your actual provider prices and typical token sizes. Defaults below reflect a moderate test case (compact RAG, a few rules, short-to-medium answer).
| # Token cost calculator for LLM test runs # Adapt: pricing (per token), token sizes, and sample counts. # — Pricing (€/token) — # Example tiers (indicative): # – Low-cost API tier: ~0.0000008 to 0.0000015 per input token # – Mid-tier API: ~0.0000020 to 0.0000040 per input token # – Premium API: ~0.0000060 to 0.0000200 per input token # Output tokens often cost the same or slightly more; set both explicitly. pin = 0.0000030 # € per input token (example mid-tier) pout = 0.0000040 # € per output token (example mid-tier) # — Typical token sizes (moderate RAG test case) — systemtokens = 400 # policies + system prompt instructionstokens = 250 # task/rubric instructions retrievaltokens = 900 # concatenated RAG chunks (top-k bounded) usertokens = 300 # user query & context outputtokens = 450 # expected model response length Tin = systemtokens + instructionstokens + retrievaltokens + usertokens Tout = outputtokens # — Test set sizes by tier — nstatic = 0 # static checks (no LLM cost) nembed = 1500 # embedding pre-evals (separate cost; exclude here) njudgelo = 2000 # cheap judge count njudgehi = 800 # expensive/target-model count # Cost per case by tier (assuming same token sizes; adjust if you compress more at lower tiers) costpercase = pin Tin + pout Tout totalcost = (njudgelo + njudgehi) * costpercase print(f”Input tokens per case: {Tin}”) print(f”Output tokens per case: {Tout}”) print(f”Cost per case: €{costpercase:,.4f}”) print(f”Total test run cost: €{totalcost:,.2f}”) # What-if explorations: # – Reduce retrievaltokens (e.g., 900 -> 600) and re-run. # – Route more to low-cost judge (e.g., njudgelo +200, njudgehi -200). # – Shorten outputtokens for non-critical cases. |
How to use it:
Run with your actual averages so everyone sees real money at stake. Try “what-ifs”: fewer retrieved chunks, shorter outputs for low-risk cases, more routing to the cheap judge—to hit a budget without gutting high-risk tiers.
Test patterns to include (don’t skimp)
- Long-context slices: Sample early/middle/late segments; use sliding windows to catch recency bias and context mixing.
- Tool-call bursts & error handling: Simulate timeouts, partial data, and out-of-order returns; verify graceful recovery.
- Structured output contracts: Strict JSON/YAML schemas and type constraints; auto-validate before judge scoring.
- Hallucination stress: Withhold facts, add distractors; require citations/tool-backed evidence.
- Policy & safety guardrails: Deterministic filters first; then model-level guardrails; include jailbreak/adversarial prompts.
- Multi-judge triangulation: Cheap judge for coarse screening; strong judge for ties and high-risk paths; human-in-the-loop for critical cases.
A practical release playbook
- Classify features by risk (External, Compliance, Mission-critical, Internal).
- Set token budgets & coverage per class (use a calculator + the sample-size guide).
- Enforce the tiered pipeline with compression, retrieval boundaries, caching.
- Dashboard spend & precision; block releases that miss minimums.
- Canary with rollback; monitor drift & incidents; feed learnings back into risk weights and budgets.
Conclusion
Token economics fundamentally shapes how we build and test LLM systems. The key is to compress what’s safe, tier your tests, and spend where risk is high. Make costs visible and part of your gatekeeping so corners aren’t cut in silence.
End of Part 2.
If you missed Part 1, check it out for the foundational principles behind these strategies. Together, these posts provide a comprehensive guide to cost-effective, robust LLM testing.