Guide · Core, Pathways and documents

How verified references work: why the links under an answer cannot be invented

Written for
Hospital physicians, emergency doctors and residents
Reading time
5 minutes
Published
Updated

Written by the Sana Health team for clinicians. It summarises published guidance, is not a guideline and has not been peer reviewed. Check current local protocols before acting.

Key points

  • Every Sana Health answer ends with a reference list read from PubMed, ClinicalTrials.gov and, for named medicines, DailyMed when the question is asked.
  • The model does not write the list, so the links in it cannot be invented.
  • Guidelines, systematic reviews, meta-analyses and randomised trials are searched first, and retracted papers are excluded.
  • The answer cites the list by number, and any PubMed link the model writes in the text is checked against PubMed.
  • Verification shows that each source is real and can be opened; whether it supports a decision for a particular patient remains the clinician's judgement.

The problem this solves

A reference list is only useful if every entry is a real paper that can be opened and read. Studies of general-purpose chatbots that write their own references have found high rates of fabricated and inaccurate citations in medical content [1, 2, 3], including when the chatbots were asked to reproduce the references of published systematic reviews [4]. A clinician cannot tell a plausible fabrication from a real paper without checking each one, which defeats the purpose of asking.

Sana Health takes a different approach to the list itself: it is retrieved, not generated.

Retrieval and generation

A language model produces text by predicting what is likely to come next. Asked for a reference, it can produce something with the shape of a citation (authors, a journal, a year, even a PMID) without any assurance that the paper exists. Retrieval works the other way round: a database is searched, and only the records it returns are listed.

A retrieved record can still be weak evidence, or less relevant than it first appears, but it cannot be imaginary. That is why the two parts of a Sana Health answer are produced differently: the text is written, and the list beneath it is retrieved.

Read from the databases, not written by the model

When a question is asked, the references are read from the databases at that moment. The language model writes the answer, but it does not write the reference list. Because each entry is a record returned by PubMed, ClinicalTrials.gov, DailyMed or the NCBI Bookshelf, the links in the list cannot be invented.

SourceWhat it addsWhen it appears
PubMedJournal articles: at least 40 records when the literature holds that manyEvery answer
ClinicalTrials.govRegistered trialsEvery answer, when relevant trials are registered
DailyMedOfficial US product information for the medicines namedWhen the question turns on named medicines
NCBI BookshelfReference chapters from StatPearls and GeneReviews, and LactMed and LiverTox monographsWhen a chapter covers the topic, or a named medicine is asked about in breastfeeding or liver injury

What is searched first

The search is weighted towards the evidence a clinician would want to see first. Guidelines, systematic reviews, meta-analyses and randomised trials are searched before other study types, including a set of leading general and specialty journals.

Retracted papers are excluded. Retracted work can otherwise keep circulating as if it were valid, often because readers do not know it has been withdrawn [5].

How the answer uses the list

  • The answer cites the reference list by number, so each statement can be traced to its source.
  • Any PubMed link the model writes in the body of the answer is checked against PubMed.
  • Each article in the list links to PubMed, to the journal's own page and, where one exists, to the free full text.
  • Product information comes from DailyMed, the US National Library of Medicine's collection of official labels.

Drug labels attached to answers are US labels. Licensed doses and indications differ between countries, and local guidance takes precedence.

What verification does and does not mean

Verification means that every entry in the list is a real record, retrieved from the database for this question, with working links to read it. It does not mean that every sentence in the answer is correct, or that a cited paper applies to the patient being treated. A trial may have studied a different population, a guideline may differ from local practice, and the strength of evidence varies from one record to the next. A large trial cited for a drug's benefit, for example, may have excluded patients with severe renal impairment: the reference is real, but it does not settle the question for a patient on dialysis.

Sana Health supports clinical judgement; it does not replace it, and the clinician remains responsible for every decision. For a decision that rests on a particular reference, the safe habit is the same as with any source: open the record, read the abstract or full text, and confirm that it says what the answer uses it for.

Checking a reference yourself

  1. Open the PubMed link and confirm the title, journal and year match the entry.
  2. Use the journal link or free full text to read beyond the abstract when the detail matters.
  3. Check the study type and date: a guideline, a systematic review and a single trial carry different weight.
  4. Confirm that the population, intervention and outcome match the question that was asked.

How Sana Health helps

In Rapid Consult, the short answer ends with an Evidence heading that singles out the two to four numbered records that best support it, each with one line on what it shows and a link. Below that sits the complete verified list: PubMed records led by guidelines, systematic reviews and trials, registered trials from ClinicalTrials.gov, US product information from DailyMed when a medicine is named, and reference chapters from the NCBI Bookshelf. Every journal article links to PubMed, to the journal and, where available, to the free full text; trials, labels and chapters link to their own records.

Questions

Can the references in a Sana Health answer be made up?

The reference list is read from PubMed, ClinicalTrials.gov and DailyMed when the question is asked, and the model does not write it, so its links cannot be invented. PubMed links that the model writes in the text are checked against PubMed.

Why are the attached drug labels from the United States?

Product information comes from DailyMed, which holds official US labels. Licensed doses and indications differ between countries, so local formularies and guidance take precedence.

Does a verified reference mean the answer is right?

No. It means the source is real and can be opened. Whether it supports the statement it is cited for, and whether it applies to the patient, is for the clinician to judge.

References