How to Read a Supplement Study Without Getting Scammed

The supplement industry is a $150B market built on a simple trick: cite a study.

Marketers know that most people won't actually read the study. They just see a citation and assume the science backs the claim. In reality, most cited studies have one or more of the following problems:

• Sample sizes too small to detect real effects
• Funded directly by the company selling the supplement
• Measure the wrong outcomes
• Run for too short a time to be meaningful
• Report effect sizes so small they don't matter in practice

Here's the framework I use to score any supplement study in under 3 minutes. This is essentially the same framework we built into Q-SCI.

The 6 factors that determine study quality

1. Sample size (weight: 25%)

• Under 20 participants: Preliminary at best. Do not update your beliefs.
• 20–50 participants: Suggestive. Wait for replication.
• 50–100 participants: Reasonable. Findings can inform decisions but shouldn't be definitive.
• 100+ participants: Solid. Statistical power is real.

Many "breakthrough" supplement studies have 8–15 participants. That's insufficient to detect anything but massive effects, and even then, half of what you see is noise.

2. Study duration (weight: 20%)

• Under 2 weeks: Measuring acute effects. Doesn't tell you about real training adaptations.
• 4–8 weeks: Middle ground. Fine for some outcomes (strength), too short for others (body composition changes).
• 12+ weeks: Meaningful adaptation window.
• 6+ months: Excellent, but rare.

An 8-week strength study on 20 people is basically the industry ceiling. Anything shorter, discount heavily.

3. Design (weight: 20%)

Ranked from strongest to weakest:

1. Systematic review / meta-analysis of RCTs — highest quality
2. Randomized controlled trial (RCT) with double-blind and placebo — the gold standard
3. Single-blind RCT — decent
4. Cohort study — informative but confounded
5. Case-control — limited
6. Cross-sectional / observational — hypothesis-generating only
7. Case reports / anecdotes — worthless for causality

If a supplement's evidence base is entirely observational, treat all claims as speculation.

4. Funding source (weight: 20%)

This one is a heartbreaker.

Lesser et al. (2007, PLoS Medicine) found that studies funded by industry are approximately 4–8× more likely to report favorable results than independently funded studies of the same intervention. This is called sponsorship bias, and it's real and measurable.

• Independent (university, government, NIH): Full trust
• Foundation-funded (non-industry): Mostly trustworthy
• Industry-funded but transparent + preregistered: Reduced trust, still usable
• Industry-funded, non-preregistered, favorable outcome: Approach with heavy skepticism

The funding section is usually at the end of the paper under "Conflicts of Interest." If it says "the authors declare no conflicts of interest" but they work at a supplement company, they're lying by omission.

5. Effect size (weight: 10%)

A statistically significant effect can still be practically meaningless.

• 1–3% improvement: Below the noise floor of real-world training. Don't bother.
• 3–5% improvement: Marginal. Worth it only if cheap and safe.
• 5–10% improvement: Meaningful. This is where creatine sits.
• 10%+ improvement: Rare. Deserves scrutiny before believing.

Look for effect sizes (Cohen's d, percentage change) not just p-values. A p-value of 0.049 with a 0.3% improvement is a rounding error.

6. Replication (weight: 5%)

One study is a data point. A finding is only a finding after 3+ independent replications.

Before accepting any supplement claim, ask: has this been replicated by researchers not affiliated with the original team?

For creatine: yes, thousands of times. Believe it.
For most "revolutionary" new supplement: no. Wait.

Putting it together: the 0-100 scoring

Here's a simplified version of the Q-SCI scoring:

Sample size ...............  0-25 pts
Duration ..................  0-20 pts
Design ....................  0-20 pts
Funding ...................  0-20 pts
Effect size ...............  0-10 pts
Replication ...............  0-5 pts
                            --------
Total ..................... 0-100 pts

80-100: High-confidence evidence
60-79:  Reasonable evidence
40-59:  Suggestive but incomplete
20-39:  Weak, don't update beliefs
0-19:   Marketing dressed as science

Try it right now

Pick any supplement you're taking. Google "[supplement name] study" and open the first result. Score it against these six factors.

Most fail hard. Most people never check.

If you want the automation — paste any study into Q-SCI and it does the scoring for you, with breakdowns and explanations. It's the tool I wish I had when I started reading this literature.

The two claims worth remembering

1. Absence of evidence is not evidence of absence. A supplement might work — but if the evidence is weak, you're gambling, not investing.

2. The strongest evidence for a supplement working is if it has cheap, generic, replicated evidence spanning decades. Creatine. Caffeine. Whey. That's about it.

Everything else — until it accumulates real evidence — is somewhere between plausible and pure marketing.

---

Score your own supplement studies free at q-sci.org. More analyses on the blog.