Pick any task involving extraction from real clinical text: diagnosis identification from notes, medication-condition mapping, MEAT validation, code assignment. A healthcare-specific medical language model outperforms the current frontier general-purpose LLMs on the published benchmarks. The maximum gap is 8.6 percentage points in accuracy; the average gap is smaller but consistent across the 13-benchmark suite (John Snow Labs benchmarks). The peer-reviewed comparison in Scientific Reports, 2025 found domain-specific medical LLMs showing "consistent performance advantages over general-purpose models" across a different evaluation framework.
The question I get asked most often by AI engineering leads at health plans is which model architecture to build on. The honest answer takes longer than the question. The short version: the architecture choice is upstream of every operational decision downstream of it, and the wrong choice compounds. For an AI engineering lead choosing the model architecture for an HCC coding workload, this is the load-bearing decision. Every accuracy point translates either to missed RAF capture or to audit clawback exposure.
The benchmark suite, briefly
The benchmark gap is measured across 13 medical AI evaluations spanning four task types: clinical natural language processing (MedNLI, BC2GM, BC5CDR), named entity recognition on clinical text (i2b2 corpora), medical reasoning (MedQA, PubMedQA), and hallucination detection in clinical applications.
The hallucination benchmarks are recent and worth citing individually:
- Med-HALT (arXiv 2307.15343)
- MedHallu (arXiv 2502.14302)
- MedHallBench (arXiv 2412.18947)
- The Hallucination Risk Scoring framework published in Algorithms, April 2026 (doi.org/10.3390/a19040315)
The Stanford HAI Holistic Evaluation of LLMs for Medical Applications covers 31 benchmarks across six models. The pattern is consistent: frontier general models do well on multiple-choice medical knowledge tests but lag on tasks requiring extraction and reasoning from real clinical text.
The 8.6 points isn't a cherry-picked result. It's a peak on a structural gap that holds across the structured medical evaluation landscape.
Three architectural reasons for the gap
Pretraining distribution. A frontier general model is pretrained on broad internet text. Clinical notes and biomedical literature are a fraction of a percent of that mix. The model picks up medical vocabulary at a surface level but has limited exposure to the conventions of clinical documentation: section structures, abbreviation patterns, negation cues, temporal markers. A healthcare-specific model is pretrained on a corpus weighted toward clinical text, biomedical literature, and structured medical resources. The priors are different from the start.
This is the part I'd over-communicate to anyone considering closing the gap with fine-tuning alone. Pretraining distribution sets a ceiling that downstream fine-tuning improves against but doesn't eliminate. If you want production-grade clinical extraction, you start the model on clinical text, not on Common Crawl.
Task-specific fine-tuning. General models are typically fine-tuned for conversational helpfulness. Healthcare-specific models are fine-tuned for clinical tasks: named entity recognition, relation extraction, ICD-10 mapping, MEAT validation. The output distribution is shaped to what the workflow actually needs.
Evaluation feedback loop. A general model improves against general benchmarks. A healthcare-specific model improves against medical benchmarks. Over multiple model generations, the gap on medical evaluations gets baked in. General models can improve substantially on MMLU between releases and barely move on Med-HALT.
What the gap means in HCC coding operations
A benchmark gap of 8.6 points doesn't translate one-to-one to operational accuracy. Benchmarks abstract workflow complexity. The directional implications, though, are consistent across deployments.
At equivalent confidence floors, healthcare-specific extraction pipelines typically show:
- Higher precision (fewer false positives at the same recall)
- Higher recall on the long tail of clinical patterns
- Lower hallucination rates on edge cases
For a 200,000-life Medicare Advantage plan, the difference between a 92% precision system and a 95% precision system at the same recall is a meaningful operational gap: thousands of additional codes survive audit instead of getting clawed back. The dollar impact runs in seven figures annually for mid-sized plans.
What closing the gap would take from a general LLM
If you start from a frontier general LLM and want to reach healthcare-specific extraction quality, the engineering work splits into four streams. I've watched teams take longer than they planned on each one.
Domain adaptation. Continue pretraining the general model on clinical text. Usually means tens to hundreds of billions of tokens of additional pretraining, requiring either substantial compute or access to a pre-adapted base model. Returns decrease with model size: a 70B model adapts less than a 7B model at the same compute budget.
Task-specific fine-tuning. Train task heads or LoRA adapters for clinical NER, relation extraction, and code assignment. The annotation work is significant, typically tens of thousands of clinical text annotations per task, done by qualified annotators with clinical training.
Negation and temporality. General models handle these probabilistically. To reach production quality, most deployments add explicit negation and temporality classifiers, either rule-based (ConText, NegEx) or learned from clinical text. This is the layer I'd put the most attention on. Most engineering teams under-invest here and pay for it at audit when false positives propagate downstream.
Cross-document reasoning. A general LLM given a single note has the context of that note. For HCC coding, the relevant clinical state usually distributes across encounters. Building the retrieval and coreference layer for cross-document reasoning is its own engineering project, separate from the model.
In aggregate, the engineering effort to close the gap with a general LLM is 18 to 36 months of focused work for a competent ML team. The market alternative, license a healthcare-specific stack, is months, not years.
The AI governance side of the decision
The architecture choice is a governance decision as much as a technical one. Three properties matter for risk-adjustment work over a multi-year audit horizon.
Auditability of the model. A healthcare-specific model trained against published medical benchmarks has an evaluation record an auditor can review. A general model used for clinical extraction places the auditability burden on the customer.
Stability of the model over time. General models update on a frequent cadence driven by general utility considerations. An update that helps general performance may incidentally regress clinical performance. A healthcare-specific model with controlled releases and clinical-task evaluation at each release has stability properties more compatible with the multi-year audit horizon of HCC coding.
Hallucination posture. The medical hallucination benchmark literature documents the gap, but the more important point for governance is process: which model and which version produced each suggestion, and what was the measured hallucination rate at release? The append-only audit log capturing this is what an AI governance review will ask for. The Pacific AI certification framework attests to these properties; Martlet AI is Pacific AI certified.
What I'd tell an AI lead choosing the architecture
Start from the workflow accuracy you need, work back to the benchmark gap that corresponds to it, and choose architecture accordingly. For most HCC coding workloads, the benchmark gap matters and healthcare-specific is the right architecture. For LLM applications where the workflow is general (summarization, drafting, conversational interfaces), the gap matters less and a general model with light fine-tuning may be the right call.
The trap I'd flag: assuming the gap will close with the next frontier release. It might, on some benchmarks. The healthcare-specific stack is moving in parallel on the same benchmarks. Architectural decisions made today on the assumption that general models will catch up tomorrow tend to age badly.
Where this lands operationally
The Martlet AI engine runs on the John Snow Labs healthcare-specific medical language model stack, the same stack benchmarked at the 8.6-point gap above and used in production by more than 500 enterprise healthcare and life-sciences organizations. It's deployed at WVU Medicine across 25 hospitals and powers John Snow Labs' own medical coding products.
For an AI engineering or governance lead evaluating model architecture, the choice is between a general model with substantial engineering work required to close the accuracy gap and a healthcare-specific model where the architecture closes the gap by default. The solution pages cover what production deployment looks like.
FAQ
Is the 8.6-point number going to be out of date as frontier models advance?
The frontier advances, and so does the John Snow Labs stack. Published benchmarks are refreshed periodically with current comparators. The gap has remained roughly stable over recent model generations.
Can fine-tuning close the gap?
Fine-tuning narrows the gap; it doesn't close it. Pretraining distribution sets the ceiling. Closing the gap requires pretraining on clinical text, not just fine-tuning.
What benchmarks should we use to evaluate a model for clinical extraction?
For risk adjustment specifically: MedNLI, BC2GM, BC5CDR for clinical NER and relation extraction, plus Med-HALT, MedHallu, and MedHallBench for hallucination. The Stanford HAI evaluation framework covers a broader set. Run the benchmarks on real clinical text from your own production environment if you can; published numbers establish the floor, but your data is where the model actually has to perform.
Does Pacific AI certification cover model accuracy?
Pacific AI certification covers governance properties: version control, controlled releases, bias and red-team testing, audit logging. Model accuracy is evaluated separately against benchmarks appropriate to the domain.
Does running healthcare-specific models require more compute than general models?
Generally no. Healthcare-specific models are often smaller than frontier general models at comparable accuracy on clinical tasks. Production deployments routinely run on single-GPU configurations.
What about retrieval-augmented generation as a way to make a general model work?
RAG helps with the cross-document reasoning failure mode by giving the model relevant context. It doesn't address negation handling, abbreviation disambiguation, section-aware reading, or hallucination resistance in the clinical domain. The residual gap remains substantial.
How do we measure hallucination rate in production?
Sample a fixed percentage of surfaced codes and reconcile against source documentation. Track per-code-family hallucination rates over time. Reviewer disagreement rate is a useful proxy when full reconciliation isn't feasible. The peer-reviewed Hallucination Risk Scoring framework in Algorithms gives a more formal approach.