Guide · Core, Specialties, Pathways and documents

AI clinical decision support: what it can and cannot do for a hospital doctor

Written for
Hospital physicians, emergency doctors and residents
Reading time
7 minutes
Published
Updated

Written by the Sana Health team for clinicians. It summarises published guidance, is not a guideline and has not been peer reviewed. Check current local protocols before acting.

Key points

  • AI decision support can structure a case, apply named criteria and summarise evidence quickly, but it works only from what it is given.
  • High scores on medical examinations have not translated simply into better clinical reasoning; in one randomised trial, access to a large language model did not significantly improve physicians' diagnostic reasoning.
  • Large language models can produce confident statements that are wrong or fabricated, and over-reliance on any decision support introduces new errors.
  • A tool is best judged by evaluations on real patients, prospective trials and transparent reporting, not by examination benchmarks.
  • The clinician remains responsible for every decision, and local protocols take precedence over a general recommendation.

Two kinds of decision support

Clinical decision support is not new. Rule-based systems, such as drug-interaction alerts, dosing calculators and order sets, have been studied for decades. In systematic reviews they have improved the process of care more often than they have improved patient outcomes [11]. Large language models are different: they read free text and write free text, can respond to questions they were never specifically trained on, and are now being built into clinical tools [1, 2]. That flexibility is what makes them useful, and also what makes them harder to evaluate.

What it can do well

  • Organise a case and surface diagnoses or complications not yet considered.
  • Apply named scores and classification criteria consistently, when the inputs are supplied.
  • Summarise guidelines and trials quickly, and point to where the evidence is thin.
  • Lay out a plan in a form that is easy to check: doses, monitoring, what would change the plan.
  • Prepare a case for discussion with a senior colleague or a multidisciplinary meeting.

The evidence on whether this improves care is still early. Large language models now routinely exceed human performance on medical licensing examinations, but how that translates into clinical practice is poorly characterised [4]. In a randomised trial, physicians given access to a large language model did not show significantly better diagnostic reasoning than those using conventional resources [5]. Randomised trials in real clinical settings have begun, for example in subspecialist cardiology [6] and in the transition from primary to specialist care [7], and their results will matter more than benchmark scores.

What it cannot do

  • Examine the patient, or know anything that has not been typed or attached.
  • Know the hospital's antibiogram, formulary, staffing or local protocols.
  • Take responsibility for a decision.
  • Make sure that a fluent answer is also a correct one.

The last point needs emphasis. Large language models generate text by statistical prediction, and they can produce outputs that are factually incorrect, fabricated or internally inconsistent, a problem usually called hallucination [12]. In medicine this includes invented or mismatched references. Image interpretation is a particular concern: progress in models that read medical images has outpaced rigorous clinical validation [9]. And a systematic review of how large language models are evaluated in health care concluded that current approaches may not adequately identify where they are most useful [3].

A model trained mainly on literature from high-income countries may not reflect the disease patterns, resistance rates or drug availability of a hospital in India, Africa or South-East Asia. National and local guidance takes precedence.

The human factor: automation bias

Automation bias is the tendency to over-rely on an automated system. Although decision support usually improves performance overall, it can introduce new errors, both by prompting a wrong action and by lowering vigilance [8]. The risk is greatest when the clinician is tired, the answer looks authoritative, and the output agrees with a first impression.

  1. Form your own view of the case before reading the output.
  2. Read the reasoning, not only the conclusion, and check the facts it rests on.
  3. Check that a cited source exists and says what it is cited for, especially when it drives a decision.
  4. Be suspicious of an answer that fits the input too neatly, and of one that ignores an abnormal result.
  5. Disagree when the bedside picture says otherwise, and document your own reasoning.

How to judge an AI tool, or a study of one

  • Was it tested on real patient cases, or on examination questions and vignettes [3, 4]?
  • Was the evaluation prospective, and ideally randomised, with clinical outcomes [5, 6, 7]?
  • Is the study reported against a recognised standard, such as the TRIPOD-LLM reporting guideline [10]?
  • Were the patients, settings and languages similar to yours?
  • Can every reference in its answers be opened and checked?

Where Sana Health fits

Sana Health is a web application for practising clinicians. Each of its 38 specialist engines follows the way that specialty works a case: it applies named criteria and scores, puts time-critical problems first and lays its report out for that workflow. Every answer ends with a reference list read from PubMed, ClinicalTrials.gov and, for named medicines, US product information on DailyMed at the time the question is asked. The model does not write that list, so its links cannot be invented, and any PubMed link the model writes itself is checked against PubMed. Retracted papers are excluded.

That addresses invented references; it does not remove the other limits above. A real paper can still be cited for something it does not quite say, a case can be described incompletely, and US drug labels do not match every country's licensing. The report supports clinical judgement and does not replace it.

A working rule

Treat AI decision support as a well-read colleague who has never met the patient: quick, thorough, sometimes wrong, and always worth checking. Used that way, it can shorten the path to a well-referenced plan. Used as an authority, it adds a new way to make mistakes.

How Sana Health helps

The Diagnostic Reasoning engine works through a complex or undifferentiated case to a ranked differential, shows the reasoning behind it, lists the red flags and the tests that would change the picture, and returns a structured report, typically in under a minute. The report cites by number a reference list read from PubMed, ClinicalTrials.gov and, for named medicines, DailyMed when the question is asked. The model does not write that list. The clinician reads the sources that matter and makes the decision.

Questions

Can AI replace clinical judgement?

No. It can organise information, apply criteria and summarise evidence, but it cannot examine the patient, knows only what it is told, and can be confidently wrong. The clinician remains responsible for every decision.

Are references produced by AI chatbots reliable?

Not automatically. General-purpose large language models can generate references that look real but do not exist or do not support the statement. A reference list retrieved from a database avoids invented links, but the source should still be read when it drives a decision.

Is it safe to enter patient details into an AI tool?

Only after removing identifiers such as names, record numbers, dates of birth and exact dates, and only in line with hospital policy and national data protection law. Relative timings and ages are usually enough.

References

  1. Large language models in medicine (opens in a new tab)Nat Med, 2023
  2. Large Language Models in Medicine: The Potentials and Pitfalls : A Narrative Review (opens in a new tab)Ann Intern Med, 2024
  3. Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review (opens in a new tab)JAMA, 2025
  4. Knowledge-Practice Performance Gap in Clinical Large Language Models: Systematic Review of 39 Benchmarks (opens in a new tab)J Med Internet Res, 2025
  5. Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial (opens in a new tab)JAMA Netw Open, 2024
  6. A large language model for complex cardiology care (opens in a new tab)Nat Med, 2026
  7. An LLM chatbot to facilitate primary-to-specialist care transitions: a randomized controlled trial (opens in a new tab)Nat Med, 2026
  8. Automation bias: a systematic review of frequency, effect mediators, and mitigators (opens in a new tab)J Am Med Inform Assoc, 2012
  9. Vision-language models in diagnostic imaging: review of technical advances, clinical validation, and practical deployment (opens in a new tab)Int J Med Inform, 2026
  10. The TRIPOD-LLM reporting guideline for studies using large language models (opens in a new tab)Nat Med, 2025
  11. Computerized clinical decision support systems for chronic disease management: a decision-maker-researcher partnership systematic review (opens in a new tab)Implement Sci, 2011
  12. AI hallucinations in academic writing: implications for research integrity (opens in a new tab)Naunyn Schmiedebergs Arch Pharmacol, 2026