Methodology · Study Rating v1.1 · October 2026
How we rate research
Every study we cite gets a Study Rating out of 100. Tap the rating next to any study to see how it was scored. Not all research can be judged the same way. A medical question can be tested in a randomised trial, but a question about markets usually can't. So we use two scales, and every rating says which one it uses:
- Health and social science scale: health, fitness, medicine, and the psychology and sociology behind our friendship course.
- Investing research scale: research on markets, investors and personal finance.
What's the same on both scales
Both scales have the same six parts with the same points: design 30, risk of bias 20, size 15, independence 15, fit with the wider evidence 10, publication 10. They also use the same labels: Strong 80 and over, Good 65–79, Limited 50–64, Weak under 50. A retracted study scores 0. A Strong investing study is as well supported, for an investing question, as a Strong health study is for a health question. The scores aren't meant to be compared across subjects.
How a rating is made
- Two independent reviews record the facts for each study, with where each fact was found: design, size, who paid, the authors' declared conflicts, registration, corrections and how it sits with other research.
- Fixed rules turn the facts into points, so the same facts always give the same score.
- Where the two reviews disagree, we check the paper and record the deciding fact. Before rolling out, we tested this on 20 studies: the two reviews landed within 10 points of each other 90% of the time.
- Ratings are re-checked when a study is corrected or retracted, or when newer research changes the picture.
Independence
This part asks one question: was the research paid for, or written by, someone who sells a related product? The answers score 15 for public or charity funding with no relevant author ties, down to 0 for a relevant conflict that was hidden. Sometimes the authors' disclosures exist but we can't read them, for example because they sit behind a paywall. Then the study gets the average independence score of the studies we've rated (12 of 15), marked as not yet checked, rather than being penalised for what we couldn't see. When a paper we have read simply leaves out who paid for it, it scores 3.
What we deliberately leave out
Journal prestige, citation counts and the fame of the authors aren't part of the score. Prestigious journals still publish weak and retracted studies, and citation counts reward older and more fashionable work. Some research tools rank studies this way; we think it misleads, so we score what the study actually did. An author's record only counts through retractions and hidden conflicts.
The claim, not just the study
A rating is about one study. When we check a claim in a video, we also say how sure the whole body of research is, on a scale of High, Moderate, Low or Very low. We only call a claim “Supported” or “Contradicted by strong evidence” when that certainty is Moderate or High; anything less is “Mixed or uncertain”. A video is never marked down because its creator disagrees with an official body, only when strong evidence says otherwise.
What we don't rate
Patient-information pages, statistics releases, calculators, news stories and pure theory (such as a mathematical formula) don't get a Study Rating. Where we can, we link the study or guideline behind them instead.
Health and social science scale
This scale is built on GRADE, the system the World Health Organization, Cochrane and the UK's NICE use to judge medical evidence. Randomised trials sit at the top because chance, not choice, decides who gets the treatment, which rules out most other explanations. Observational studies can show links, but people who choose a habit often differ in other ways. We use the same scale for the psychology and sociology research behind our friendship course, where small randomised lab experiments and surveys are common.
| Part | Points | How it's scored |
|---|---|---|
| Design | 30 | Systematic review of randomised trials 30 · randomised trial with 1,000+ people 26 · smaller randomised trial 22 · Mendelian randomisation (genetic) study 18 · systematic review of observational studies 18 · prospective cohort 16 · case-control 10 · cross-sectional survey or model 8 · lab, animal, case report or opinion 4. Guidelines: built on systematic reviews 26, without them 16; expert consensus 16. |
| Risk of bias | 20 | Low 20 · some concerns 12 · high 5 · can't be assessed 8. Judged with the standard tools (Cochrane RoB 2, ROBINS-I, AMSTAR 2): randomisation, blinding, dropouts, selective reporting, confounding. |
| Size | 15 | 10,000+ people or 500+ events 15 · 1,000+ or 100+ events 12 · 200+ or 30+ events 8 · fewer 4; minus 4 if the result is too imprecise to rule out benefit or harm. Guidelines score a neutral 10. |
| Independence | 15 | Public or charity funding, no relevant author ties 15 · public funding, some relevant ties declared 11 · industry-funded, academics in control 8 · industry-funded, sponsor involved in design or analysis 4 · not disclosed 3 · hidden relevant conflict 0. Disclosures we couldn't read: 12, marked not yet checked. |
| Fit with wider evidence | 10 | Agrees and independently replicated 10 · agrees, not yet replicated 7 · mixed picture 5 · conflicts with stronger evidence 2. |
| Publication | 10 | Peer-reviewed and registered in advance 10 · peer-reviewed 7 · preprint, report or guideline not in a journal 4 · minus 2 for a correction that changed results. |
Investing research scale
Markets can't be put into randomised trials, so on the health scale nearly all investing research would look like weak observational work. That would be misleading. Investing research has a different main risk: data mining. With enough historical data, some pattern always looks profitable by chance. In the US, published market patterns have shrunk sharply once they were published (McLean & Pontiff 2016), and researchers still disagree about how many survive careful re-testing: Hou, Xue & Zhang (2020) found most didn't, while Jensen, Kelly & Pedersen (2023) found most did. So this scale rewards what investing researchers themselves use to separate real effects from chance:
- Does the effect hold in other countries and periods (Berkin & Swedroe call this persistent and pervasive)?
- Does it survive after publication?
- Does it still work after costs and taxes?
- Is the statistical bar high enough for how many patterns were tried? Harvey, Liu & Zhu (2016) argue new findings should clear a t-statistic of about 3, not the usual 2.
| Part | Points | How it's scored |
|---|---|---|
| Design | 30 | Randomised or natural experiment (e.g. a pension rule change that switched people into saving) 26 · long-run study across many markets or asset classes 22 · long-run study of one market 18 · industry or index-provider data (e.g. fund-performance scorecards) 16 · a single backtest or simulation 8. Pure theory isn't rated. |
| Risk of bias | 20 | Low 20 · some concerns 12 · high 5 · can't be assessed 8. We check for data mining (many variations tried, weak statistical bar), survivorship bias (failed funds or delisted shares left out), look-ahead bias (using data no one had at the time) and ignoring costs and taxes. |
| Size | 15 | 50+ years of data or 20+ markets 15 · 25+ years or 10+ markets 12 · 10+ years 8 · less 4; minus 4 if the result isn't statistically clear. |
| Independence | 15 | The same as the health scale. Research by fund managers, index providers or anyone selling a related product counts as industry research, even when it's good. |
| Fit with wider evidence | 10 | Holds after publication and in other markets 10 · agrees with other research but not yet tested out of sample 7 · mixed, or shrank after publication 5 · contradicted by stronger evidence 2. |
| Publication | 10 | Peer-reviewed with data or code public, or independently replicated 10 · peer-reviewed 7 · working paper (e.g. NBER, SSRN) or industry report 4 · minus 2 for a correction that changed results. |
The certainty words (High to Very low) mean the same on this scale: how sure the whole body of research is. For example, the long-run evidence that low-cost index funds beat most active funds after fees is High. A newly published market pattern from one dataset starts at Low.
Why two scales, and why these
Other rating systems face the same problem. Education evidence toolkits rate strength with one simple symbol but explain the rules behind it on a separate page. US crime-prevention ratings use two independent reviewers and different rules for single programs and for meta-analyses. Fund ratings use the same medals across fund types while weighting the evidence differently. We copied what works: one familiar score and set of labels, two independent passes over every study (ours are AI passes; see who's behind the ratings below), and a published method for each kind of research. Where the field already has a standard, we use it (GRADE for health, the replication and out-of-sample tests above for investing) rather than inventing our own.
Who's behind the ratings
AI does the reading, published rules do the scoring, and a person owns the method. Every video and every study is read and scored by AI (Anthropic's Claude) against the rules published on these pages. For research, two independent AI passes record the facts of each study; a script turns those facts into points, so the same facts always give the same score; and where the passes disagree, we check the paper and record the deciding fact. In our checks so far, the two passes landed within 10 points of each other for 86–92% of studies, and in the pilot our evidence grades matched every published GRADE assessment we compared against. We spot-check by hand, fix what's wrong and log every change to the method.
The method is designed and run by Tim Hammacher, the founder: a former trader, data analyst and consultant at Quantium, the data science company, where he consulted for blue-chip Australian companies using big data. He has qualifications in actuarial science and economics, including statistics, mathematics and physics.
No paid placement, ever: payment or affiliate status never affects a rating.
Limits and challenges
A rating summarises a study; it isn't a substitute for reading it or for advice from your doctor or a licensed adviser. We can get facts wrong, especially where disclosures sit behind paywalls. If you think a rating is wrong, tell us with the source, and we'll re-check it and record any change.
Changes
- 22 Oct 2026, v1: health and social science scale; pilot on 20 studies.
- 22 Oct 2026, v1.1: disclosures we couldn't read score the average (12 of 15), marked not yet checked; investing research scale added.
- 23 Oct 2026: added who's behind the ratings (AI passes, the agreement figures and who runs the method).