English | EN 
In a medical retrieval-augmented generation (RAG) system, the final generator turns retrieved documents into a usable answer. This is a very specific task. General-purpose leaderboards rarely test it, because they often focus on general reasoning or coding. The top-ranked model may therefore not be the best choice for a medical RAG pipeline. Selection should start from the need, then compare relevant candidates under the same conditions. This guide gives a four-step approach.
1. Define the need
A specific task requires a clear definition of the expected result.
Specify the medical specialty, language, users, context window, response format, and target latency. Also define the hardware, privacy, access, and licensing constraints.
Check access to the model weights and code separately, as well as the usage rights. Different parts of the same model package may use different licenses. These licenses may be unclear or incompatible. Review the exact model version, not only the model family.
2. Start with established model branches
Our synthesis of three reviews of medical RAG shows a strong presence of the LLaMA and Mistral families. When only general-purpose models are counted, they represent 65.1% of the observed uses. This result does not identify a single winner. It highlights ecosystems that have already been tested, documented and integrated.
Some of the models shown are older. This is expected: models that have been available for longer have had more time to be tested, cited, and included in reviews. They are therefore better established in the literature. This older group provides a trusted base of mature model branches. The next step is to look within these branches for models that match the need.
3. Use the taxonomy to narrow the search
First choose between a general-purpose model and a medically specialized model. A general-purpose model is often more flexible. A specialized model is useful when its language, medical specialty, and adaptation data closely match the use case.
Figure 2. Taxonomy of open-weight LLMs used as final generators in medical RAG systems.
Then narrow the search with the taxonomy categories: a regionally or linguistically targeted model, an encoder-decoder architecture, a derivative assistant, a reasoning-oriented model, a mixture-of-experts (MoE) model, an efficient compact model, or a reference dense model. For medical models, distinguish general biomedical knowledge, medical reasoning, targeted clinical use, and encoder-decoder architectures.
This view avoids comparisons between models designed for different goals. A general-purpose or coding assistant may have excellent benchmark scores but still perform poorly when summarizing medical evidence.
Within each selected branch, choose several exact models. Record the version, checkpoint, parameter count, base or instruction-tuned variant, context window, quantization, license, and hardware requirements.
4. Compare models in the same RAG pipeline
Keep the same corpus, retrieval method, prompt, context, and generation settings. Evaluate criteria such as faithfulness to the sources, accuracy, coverage, citations, abstention and stability. Also test conflicts between the retrieved context and the model’s parametric knowledge. Finally, measure latency, memory use, throughput and cost.
Keep the selection up to date. A new model, a new version, or a more efficient technique can change the best trade-off.
A simple strategy is to compare emerging models with trusted reference models in the same category. Read metrics together with model parameters and evaluation conditions. Benchmarks are not always enough. Some developers highlight their most favourable scores, often from general-purpose or coding tasks. Testing on representative medical cases shows whether a reported improvement actually improves the target RAG system.
This article was co-written with Ethan DESCHILDRE (https://www.linkedin.com/in/ethan-deschildre-84177629a).