When a virtual patient in a benchmark scenario tells an AI agent they're feeling feverish, or hopeless, the difference between a safe response and an unsafe one often comes down to clinical design — not just medical knowledge. As healthcare AI shifts from answering questions to completing tasks on behalf of patients — scheduling visits, managing prescriptions, triaging symptoms — we need benchmarks that measure what actually matters: whether these systems keep patients safe, follow clinical workflows, and help patients accomplish their goals across realistic, multiturn conversations. Primary care already guards against diagnostic errors, unsafe medication use, and gaps in follow-up; an agent acting on a patient's behalf should be evaluated on how it handles the same risks.
Today we're sharing PatientAgentBench, a reproducible, clinician-vetted evaluation standard built specifically for patient-facing AI agents in healthcare. We’re also releasing key findings about the ways in which even capable foundation models fall short, and we show how the findings can guide the design of safer agents.
Why existing benchmarks don't measure patient-facing agentic AI
Most healthcare AI benchmarks fall into two camps. Those in the first camp test medical knowledge in a static form, such as answers to medical-exam questions or responses to single-turn or short clinician-facing exchanges. Those in the second test agentic, tool-using agents, but on technical tasks done for providers rather than conversations with patients. Both are valuable, but neither captures what a patient-facing agent has to do: reason over a patient's health record across multiple turns and decide when to gather more information, when to act, and when to escalate, all while maintaining appropriate clinical safety boundaries.
A different challenge lies in how these benchmarks score. They typically rely on bespoke, per-conversation criteria — physician-written rubrics tied to specific, static sets of conversations. Such criteria don't generalize to new agents and new conversations. And once the dataset is published, it joins the sea of public data used for model training, so a model can effectively learn the benchmark’s answer key, inflating its scores through memorization rather than reasoning.
What we built
PatientAgentBench generates a synthetic patient chart and health record, a realistic clinical vignette derived from that record, and a patient agent that uses all that contextual information to converse with the health AI system under evaluation. The health AI system is itself an agent: a base model with a harness that governs how it reasons over the patient's context and uses the benchmark's stateful, simulated healthcare tools. According to the defined benchmark scenario, the agent must gather enough information about the patient’s condition across multiple turns, reason over the health record, determine the appropriate level of care, and execute healthcare workflows correctly.
An LLM-as-a-jury panel evaluates the health AI system using over 100 clinician-vetted criteria organized along six dimensions — clinical safety, triage quality, workflow accuracy, task completion, clinical helpfulness, and conversational quality. Unlike the bespoke per-conversation criteria of the traditional approach, these criteria are reusable: the same rubrics apply to any patient-facing healthcare conversation, with the evaluator interpreting each requirement against the patient's dynamically generated record and clinical context. Combined with fresh scenario generation, this makes the benchmark extensible to new clinical domains, new patient populations, and new models without additional physician annotation. It also helps prevent training contamination, since there's no fixed answer key to memorize.
The panel scores each conversation against the criteria and, for every dimension, returns a written explanation of what the agent did well or missed. Licensed clinicians have validated the automated evaluation by annotating a shared sample of conversations across all evaluated systems. In our experiments, their scores aligned strongly with the jury — on par with or exceeding human inter-annotator agreement on these tasks. The jury panel also exhibited a reassuringly conservative bias on safety-critical dimensions: the automated panel is more likely to flag a potential problem unnecessarily than to miss a real one.
All patient profiles, clinical narratives, and conversations are fully synthetic; no real patient health information is used at any stage.
What we found
We evaluated multiple families of frontier models on thousands of shared multiturn patient conversations. Out of the box, in a baseline agentic harness, even the most capable models fell short of the standard patient-facing care requires. Three findings stood out:
Triage is where models diverge most. The hardest cases weren't emergencies (which trigger strong safety protocols) but routine administrative requests from clinically complex patients — say, a pharmacy update from someone with multiple active medications and a documented mental-health concern. Most models processed these transactionally; few paused to screen for clinical risk. This produces a counterintuitive severity paradox: most agents actually score higher on clearly severe cases than on mild or routine ones, because an obvious emergency triggers stronger safety and triage behavior. The hardest cases are the routine ones that hide real risk — the same kind of case where studies show human diagnostic errors are most common.
Safety failures concentrated in specific patterns. The dominant pattern was crisis resource omission — for instance, recognizing suicidal ideation but failing to provide hotline information. A second was clinical-information fabrication: invented provider credentials, fake citations, claimed tool executions that never ran.
These findings aren't an indictment of any one model; they're a map of where the field needs to invest. They also reveal something important: model capability alone doesn't guarantee perfect clinical safety. More-capable models narrow clinical gaps but do not close them, and the strongest still leave clinically important cases unhandled. Such shortcomings surface only in sustained, tool-using conversations involving realistic patient records, which is exactly why static knowledge tests are not enough for systems approaching autonomous primary care. Beyond surfacing these clinical gaps, our benchmark's per-dimension scores and explanations provide a roadmap: they show precisely where to focus when iterating on model choice and where agent design must compensate for what model capability alone does not deliver.
Consistent with how responsible patient-facing AI should behave, these agents are designed to support, not replace, the patient's provider: rather than rendering definitive diagnoses, they provide educational information and route to clinicians.
Get started with PatientAgentBench
We're releasing the PatientAgentBench framework on GitHub. The release includes the synthetic-scenario generation pipeline, the healthcare sandbox with stateful tools, the dual-agent-conversation runner, and the LLM-as-a-jury evaluation system with all six rubric prompts. The framework generates fresh scenarios on demand from a configurable seed distribution rather than including a fixed dataset.
Researchers, healthcare organizations, and AI developers can use PatientAgentBench with predefined scenarios or expand it for their own use cases, evaluating their systems at scale to identify clinical safety gaps and target improvements. The configurable distribution over patient attributes also lets teams disaggregate results by specific subgroups, helping them identify why their own agents fall short. To get started, see the README in the repository. We welcome contributions and feedback as the field works toward evaluation standards worthy of the promise, and the stakes, of increasingly autonomous patient care.
Read the full paper: PatientAgentBench: A Benchmark Framework for Evaluating Patient-Facing Health AI Agents
