English | EN 
Introduction
The defining characteristic that elevates an AI agent above a traditional machine learning model is agency—specifically, the capacity to autonomously select and use external tools. In a medical diagnostic context, these tools might include a database to fetch electronic health records (EHRs), an image processing pipeline to calculate tumor volumes, or an algorithmic calculator to compute clinical risk scores (such as the CHA₂DS₂-VASc score for stroke risk).
However, giving an LLM-based agent access to an external toolkit introduces a critical vulnerability: tool-use fallibility. In the clinical sandbox, an agent can fail catastrophically not by misinterpreting pixels, but by making malformed API calls, passing incorrect clinical variables to external software, or hallucinating the very existence of a medical tool. For researchers building dependable AI, securing the tool-interface boundary is the new battleground for safety.
1. The Anatomy of Tool-Use Failure in Medicine
In non-medical applications, a broken tool call results in an annoying software crash or an unrendered webpage. In healthcare, it leads to erroneous diagnostic reporting. When an autonomous agent attempts to interact with external clinical software, failures typically manifest in three ways (Winston and Just, 2025):
- Argument Hallucination: The agent invokes a legitimate external tool (e.g., a glomerular filtration rate calculator) but populates the input parameters with incorrect data extracted from the patient’s record—such as substituting serum creatinine levels with blood urea nitrogen (BUN).
- API Protocol Violations: The agent generates structurally malformed code or JSON payloads that violate the syntax required by the hospital’s database, causing silent data truncation or uncaught runtime errors.
- Imaginary Tool Creation: Faced with a difficult clinical query, the agent hallucinates a non-existent medical guideline API or a specialized calculator, proceeding through its reasoning path under the false assumption that this tool successfully validated its hypothesis.
2. Why Agents Misinterpret the Toolkit
The root of this problem stems from the fundamental misalignment between the stochastic nature of large language models and the deterministic requirements of software interfaces (APIs).
Agents use text-based descriptions (often written in natural language or docstrings) to understand what a tool does and how to call it. When a clinical case becomes highly complex, the model’s attention mechanism can experience cognitive overload. The nuances of strict variable typing (e.g., expecting an integer in milligrams vs. a float in grams) get lost in the semantic context (Patil et al., 2024; Chen et al., 2024). The model prioritizes generating text that looks like a valid tool call over ensuring absolute syntactic and logical precision.
3. Engineering Solutions: Constrained Decoding and Formal Verification
To prevent agents from misusing their toolkits, researchers are shifting away from relying solely on the agent’s internal “good behavior” and are instead implementing rigorous computer science guardrails.
Grammar-Guided and Constrained Decoding
One of the most powerful defenses against malformed tool use is constrained decoding. By leveraging frameworks that enforce Context-Free Grammars (CFGs) directly at the token-generation level, we can mathematically restrict what the model can output (Beurer-Kellner, Fischer and Vechev, 2024; Dong et al., 2025). If the agent decides to call an API, the system forces the model’s next tokens to strictly adhere to a valid JSON schema or code syntax. The model is physically incapable of hallucinating an illegal parameter or an unmapped tool name because those tokens are blocked during generation.
Semantic Type Verification and Sandboxing
Before an agent’s tool request is executed against a real clinical database, it must pass through an intermediary validation sandbox. This layer performs semantic type-checking. For instance, if an agent attempts to pass a blood pressure value of “120/80” into an API that strictly expects an isolated systolic integer, the sandbox intercepts the call, generates an explicit error message, and feeds it back into the agent’s reasoning loop, prompting it to self-correct before any diagnostic harm is done (Ruan et al., 2024; Sun et al., 2024).
4. Grounding Guidelines via Retrieval-Augmented Generation (RAG)
Tool misuse also extends to clinical guidelines (e.g., RECIST criteria for tumor response or ESC guidelines for cardiology). To prevent agents from hallucinating these rules, researchers integrate vector databases containing verified medical textbooks and official society guidelines. By forcing the agent to fetch and reference specific, immutable document chunks before calling an action, we replace the model’s unreliable parametric memory with verified, structured truth.
Conclusion
Giving diagnostic AI systems the autonomy to use tools is a prerequisite for achieving true clinical reasoning, but unconstrained autonomy is a liability. By wrapping agentic loops in strict syntactic guardrails, grammar-guided decoding, and robust semantic sandboxes, we can ensure that these digital clinicians interact with external software with the same precision and care expected of any medical professional.
References
Beurer-Kellner, L., Fischer, M. and Vechev, M. (2024) ‘Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generation’, in Proceedings of the 41st International Conference on Machine Learning. Vienna, Austria: PMLR. Available at: https://openreview.net/forum?id=pXaEYzrFae.
Chen, Z., Du, W., Zhang, W., Liu, K., Liu, J., Zheng, M., Zhuo, J., Zhang, S., Lin, D., Chen, K. and Zhao, F. (2024) ‘T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step’, in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Bangkok, Thailand: Association for Computational Linguistics, pp. 9510–9529. doi: 10.18653/v1/2024.acl-long.515.
Dong, Y., Ruan, C.F., Cai, Y., Xu, Z., Zhao, Y., Lai, R. and Chen, T. (2025) ‘XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models’, in Proceedings of the 8th Annual Conference on Machine Learning and Systems (MLSys 2025). Santa Clara, CA. Available at: https://mlsys.org/virtual/2025/3235.
Patil, S.G., Zhang, T., Wang, X. and Gonzalez, J.E. (2024) ‘Gorilla: Large Language Model Connected with Massive APIs’, in Advances in Neural Information Processing Systems, vol. 37. Curran Associates, Inc. doi: 10.52202/079017-4020.
Ruan, Y., Dong, H., Wang, A., Pitis, S., Zhou, Y., Ba, J., Dubois, Y., Maddison, C.J. and Hashimoto, T. (2024) ‘Identifying the Risks of LM Agents with an LM-Emulated Sandbox’, in The Twelfth International Conference on Learning Representations (ICLR 2024). Available at: https://arxiv.org/abs/2309.15817.
Sun, J., Min, S.Y., Chang, Y. and Bisk, Y. (2024) ‘Tools Fail: Detecting Silent Errors in Faulty Tools’, in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Miami, Florida, USA: Association for Computational Linguistics, pp. 14272–14289. doi: 10.18653/v1/2024.emnlp-main.790.
Winston, C. and Just, R. (2025) ‘A Taxonomy of Failures in Tool-Augmented LLMs’, in 2025 IEEE/ACM International Conference on Automation of Software Test (AST). IEEE, pp. 125–135. doi: 10.1109/AST66626.2025.00019.