News
Newest first, each with its date. When something changes later, it gets an item of its own, and the earlier ones stay as they were.
Two medical students checked 85 items of an evidence map built with an earlier version of Julie against the source papers. Of their 51 flags, 8 were upheld, and the flags gathered where the machine’s own explanation disagreed with its answer rather than where the errors were.
Julie is getting a web version, which we are building with a highly respected software development company. This week its first test-ready version faced the same test as in our last article, from title and abstract screening to linking evidence to recommendations.
A published guideline graded one recommendation A and another B. Given the wording but not the grades, Julie drafted an A and a C: the order held, no false A appeared, and the second grade came out one step lower.
We checked each of the 895 citations in a clinical literature review drafted with a general-purpose AI assistant against the source it cites. In 27 the review got the source wrong, and in 103 more the sources carried only part of what it said.
Julie now prepares a human check of her screening decisions in any review, on records drawn from those she kept and those she excluded. The report shows how many of the eligible records she kept, and the terms are yours, set before the first sheet goes out.
Four published COPD trials went through data extraction, meta-analysis and GRADE certainty assessment to a manuscript draft. Two AI readers matched 43 and 42 of 48 printed numbers, and neither returned a value that differed from the paper.
Beside the systematic review, Julie now offers pairwise meta-analysis: from an analysis plan you sign to forest plots, GRADE certainty and the manuscript. Every pooling decision stays on the record, under a person’s name.
In a 300-record development sample from a review in progress, Julie’s screening kept 71 of the 72 records that three AI reviewers all judged eligible or possibly eligible, and 77 of the 84 that at least two did. The figures measure agreement with a machine reference, not accuracy.
A source check of four published COPD trials confirmed every numerical entry in an AI-assisted reference set. Six descriptive entries needed clarification.
You choose the standard for each project at the outset, because it shapes what is extracted, how it is rated and what you receive.
If you are running a systematic review, sitting on a guideline panel, or working in an evidence synthesis unit, tell us what you are working on.