How a verdict is made
Every claim goes through the same ten steps, whether it is about creatine or the pyramids. I print them on each investigation so you can see where a conclusion came from rather than taking it on trust.
- Find how people actually say it. I write down three or four real phrasings rather than a tidied-up version, because the tidy version is usually a different claim and answering it convinces no one.
- Trace where it came from. Most of these have a source: a study, a documentary, a guideline that got simplified, a marketing campaign. Naming the origin usually does more work than correcting the claim.
- Turn it into something a study could answer. Who, doing what, compared with what, measured how, over how long. If a claim cannot be written that way, it is usually a values question rather than an empirical one, and I say so instead of grading it.
- Run a search and publish it. The database, the terms, the filters and the exclusions, written out so you could repeat it and get the same set of papers.
- Read the full texts. Not the abstracts. An abstract is the authors’ own summary, and the limitations usually sit further in.
- Grade the evidence with the reasoning visible. Randomised trials start high and observational studies start low, then the level moves for named reasons: risk of bias, inconsistency, indirectness, imprecision, publication bias. Showing the arithmetic means a grade cannot be quietly adjusted to fit a conclusion.
- Work out who it applies to. A verdict is rarely portable. Who was actually enrolled, and how close that is to you, often matters more than the headline result.
- Name what has not been measured. Sex, genetics, dose, duration, source. Where a variable has not been studied properly, it gets its own section rather than a silent omission.
- Say what would change my mind. Each entry names the specific study that would move the verdict. If nothing could change it, I am not doing evidence appraisal.
- Put a review date on it. Research moves. Every entry carries the date I last examined it and the date it is next due, and I mark verdict changes when they happen.
For claims outside clinical medicine, step six swaps its domains for provenance, primary source, expert consensus and replication. The process stays the same; only the instrument changes.
What a verdict can say
A claim can come back true, partly true, unsettled, mostly false, false, or not covered. That last one means the evidence I found does not speak to the group being asked about, and saying so is more useful than guessing.
Educational information only. Fact Over Myth does not provide medical advice, diagnosis, or individualised nutrition care.
The evidence protocol
The rule I follow before any claim on this site gets a verdict. It exists so that two people running it on the same claim would land in roughly the same place, and so that you can check my work rather than take my word for it.
Step oneTurn the claim into a question I can search
A myth is a sentence, and a sentence is not searchable. So I rewrite it as a question with four named parts: who it is about, what the exposure is in a form somebody could measure, what it is being compared with even when the claim leaves that unstated, and what outcome is supposed to follow.
A claim that cannot survive this step is not a factual claim. It does not get a verdict. It gets a note saying why.
Step twoSearch, and write down what I searched
PubMed and MEDLINE first, through the NCBI interface, and the Cochrane Library when the question is about a treatment. I run the search in four strands rather than one string, because one string always misses something: trials in humans, then the epidemiology, then the mechanism, then a check for whether somebody has already reviewed this systematically.
No date limit on experimental and landmark work, because a good trial does not expire. Ten years back on the epidemiology, because that is where the current picture sits. Both choices get stated rather than assumed.
Every query string goes on the page, word for word, with the number of records it returned and the date I ran it. If you disagree with how I searched, you can run it yourself and see what I saw.
Step threeScreen against criteria written beforehand
Written before I look at the results, so I cannot bend them toward a conclusion I already like. A paper gets in if it studies people, if the exposure is the one in the claim, if the outcome is the one in the claim rather than a laboratory marker standing in for it, if the design can answer the question being asked of it, and if it is peer reviewed and not retracted.
It stays out if the outcome is a surrogate, if the population is materially different from the one in the claim, if the venue is predatory or the piece is a hypothesis rather than evidence, or:
- If I cannot retrieve enough of the paper to know what it found. That exclusion gets named on the page, because a paper I could not read is more interesting to you than one I simply chose not to use.
The counts are published: found, screened, read in full, used.
Step fourRank by what the design can prove
Not by how famous the journal is.
| Tier | Design | What it can establish |
|---|---|---|
| Tier 1 | Meta-analysis or systematic review of randomised trials | That the exposure causes the outcome, across many trials |
| Tier 2 | A single randomised controlled trial | That it caused the outcome, in the people studied |
| Tier 3 | Prospective cohort, or systematic review of observational studies | That the exposure predicts the outcome, with confounding still possible |
| Tier 4 | Case-control, cross-sectional, ecological | That the two occur together, in populations rather than in people |
| Tier 5 | Mechanism: cell, tissue or animal work. Case report. Narrative review. Expert opinion | How something could work. Never that it does |
A tier five paper never outranks a tier one paper. It can explain one, and it can make a weak finding easier to believe, and that is the whole of what it does.
Step fiveSynthesise from the top down
- Start at the highest tier present. Where a current systematic review exists, it is the spine and everything else is context around it.
- Where studies disagree, say which design wins and why, instead of splitting the difference.
- Keep the mechanism separate from the outcome, because they answer different questions.
- Give effect sizes with their intervals. “Significant” is not a finding.
- Say what the whole body of evidence cannot answer.
Step sixGrade the certainty, and show the arithmetic
Certainty starts High when the body of evidence is randomised trials and Low when it is observational. It comes down a level for each of risk of bias, inconsistency between studies, indirectness between what was studied and what was claimed, imprecision, and signs of publication bias. It goes up a level for a large effect, for a dose-response gradient, and for confounding that should have pushed the result the other way and did not.
The arithmetic is printed on the page. If you think one of my downgrades is wrong, you can see exactly which one to argue with.
And thenIt gets a date
Every verdict carries the date it was made and the date it is next due for review. A verdict with no date is an opinion wearing a lab coat. Anything here that stops being reviewed should stop being trusted.
What this protocol does not do. It does not make me right. It makes me checkable, which is a different promise and a more useful one. The searches are reproducible, the exclusions are named, the tiers are stated and the certainty arithmetic is shown. If I am wrong, this is how you find where.
Protocol version 1.0 · written 21 September 2026 · applied to every brief and investigation published after that date
