Multiple Comparisons and False-Positive Findings in Longevity Studies

Who This Is Useful For

This page is useful for readers interpreting studies that report many ageing biomarkers, several health outcomes, repeated follow-up measurements, or results for multiple demographic or clinical subgroups. Such designs can answer important questions, but they also create a multiplicity problem when numerous results are screened for small p-values. [2] [8]

A false positive, or Type I error, occurs when a statistical test rejects a null hypothesis that is in fact true. A conventional significance level of 0.05 limits the long-run false-rejection probability for one valid test under the null model; it does not mean that a reported finding has a 5% probability of being false. [1]

Multiple comparisons change the question. If investigators run many tests and treat any p-value below 0.05 as a discovery, the probability of seeing at least one false-positive result across the set can be much greater than 5%. The relevant error rate therefore depends on what collection, or “family,” of inferences is being considered. [2] [3]

Why the False-Positive Opportunity Grows

For m independent tests whose null hypotheses are all true, each tested at level 0.05, the chance of at least one false rejection is 1 − (1 − 0.05)^m. It is about 40% for 10 tests and 64% for 20 tests. These values are illustrations, not universal predictions, because results from related biomarkers or repeated measurements are often correlated. [2] [7]

Correlation among tests can alter the exact probability, but it does not make the underlying issue disappear. Valid procedures differ in the dependence structures they permit, and the choice of method must match both the scientific question and the statistical properties of the tests. [4] [7]

Where Multiplicity Appears in Longevity Research

These sources of multiplicity are common in biomarker and clinical research. Ageing studies can be especially analysis-rich because candidate biomarkers span molecular, physiological, functional, and clinical domains, while validation may examine associations across several populations and outcomes. [7] [8] [9]

Family-Wise Error Rate and False Discovery Rate

Family-wise error rate (FWER) is the probability of making at least one false rejection within a defined family of hypotheses. The Bonferroni principle and Holm’s sequential procedure are designed to control this probability; Holm’s method provides strong FWER control without requiring the tests to be independent. [3] [7]

False discovery rate (FDR) instead concerns the expected proportion of false rejections among all rejected hypotheses. Benjamini and Hochberg introduced an FDR-controlling procedure that can offer greater power when the scientific aim is to identify signals from a large set, although its guarantees depend on stated assumptions. [4] [7]

The Definition of the Test Family Matters

A correction is only interpretable in relation to the hypotheses included in it. A paper might adjust all molecular markers together, adjust each biomarker panel separately, or adjust only the primary outcomes. Those choices protect different families and can produce different adjusted results from the same data. Statistical methods alone do not determine which family best represents the scientific claim. [2] [10]

The distinction between confirmatory and exploratory work is therefore important. A confirmatory claim is clearer when the primary outcome, model, subgroup strategy, and multiplicity procedure were specified before results were examined. Exploratory analyses remain useful for generating hypotheses, but their nominal p-values should not be read as if a single prespecified test had been conducted. [5] [6] [10]

Hidden Multiplicity and Selective Reporting

The comparisons visible in a results table may be fewer than the analyses that were possible or actually attempted. Decisions about exclusions, transformations, covariates, outcomes, stopping rules, and subgroups create researcher degrees of freedom. When only favorable combinations are reported, ordinary p-values no longer describe the full selection process that produced the published result. [5] [11]

This selective search for statistically significant specifications is often called p-hacking. Empirical work has found patterns consistent with its presence across scientific literatures, while also emphasizing that the prevalence and impact are difficult to quantify from published p-values alone. [11]

What Adjustment Changes—and What It Does Not

A multiple-testing procedure changes the rule used to classify statistical results; it does not repair confounding, measurement error, attrition, model misspecification, selective publication, or poor endpoint validity. It also does not turn statistical significance into evidence of a large or practically important effect. [1] [9]

Error control also involves a trade-off. More stringent thresholds generally reduce false rejections but can increase false negatives, particularly when sample sizes are small or effects are modest. FWER and FDR methods embody different balances, which is why the method and its rationale should be reported rather than represented simply as “corrected.” [3] [4] [7]

How to Read a Study With Many Comparisons

• Identify what was designated primary: primary outcomes and analyses carry a different evidential role from secondary or exploratory results when they were prespecified. [6] [10]
• Count the opportunities, not only the highlighted findings: outcomes, biomarkers, groups, visits, and model variants can all enlarge the analysis space. [2] [5]
• Check what family was adjusted: the phrase “multiple-testing correction” is incomplete unless the included hypotheses and controlled error rate are stated. [3] [4]
• Read effect sizes and intervals: multiplicity-adjusted significance does not establish biological magnitude, precision, or relevance to human ageing. [1] [9]
• Look for validation: confirmation in new samples or with prespecified tests helps distinguish a stable signal from one selected within a large search space. [6] [9]

What This Does Not Mean

• It does not mean every study with many outcomes is unreliable; multiplicity can be handled transparently through design, analysis, and appropriately limited claims. [2] [10]
• It does not mean an unadjusted secondary finding is necessarily false; it means its error rate must be interpreted in the context of the broader set of analyses. [12]
• It does not mean the strictest correction is always the scientifically correct one; different error criteria serve different inferential goals. [3] [4]
• It does not mean exploration should be avoided; transparent separation of exploration from confirmation preserves the value of both. [6]

Related Reading

• P-Values, Confidence Intervals, and Statistical vs Practical Significance
• Sample Size, Statistical Power, and Why Small Studies Mislead
• Preregistration and Registered Reports in Ageing Research
• Publication Bias and Why Positive Longevity Results Get More Attention

Summary

Multiple comparisons create more chances for apparently significant findings, whether the comparisons arise from numerous biomarkers, outcomes, subgroups, time points, or flexible analysis choices. FWER and FDR procedures address different versions of this problem, and their meaning depends on how the family of tests was defined. In longevity research, credible interpretation therefore requires attention to the full analysis space, prespecification, effect sizes, uncertainty, endpoint validity, and independent validation—not only whether an adjusted threshold was crossed. [2] [4] [6] [9]

References

1. Greenland, S., et al. (2016). Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology. https://pmc.ncbi.nlm.nih.gov/articles/PMC4877414/
2. Li, G., et al. (2017). An introduction to multiplicity issues in clinical trials: the what, why, when and how. International Journal of Epidemiology. https://academic.oup.com/ije/article/46/2/746/2622846
3. Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics. https://www.jstor.org/stable/4615733
4. Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x
5. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science. https://pubmed.ncbi.nlm.nih.gov/22006061/
6. Nosek, B. A., et al. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.1708274114
7. Menýhárt, O., Weltz, B., & Győrffy, B. (2021). MultipleTesting.com: a tool for life science researchers for multiple hypothesis testing correction. PLOS ONE. https://pmc.ncbi.nlm.nih.gov/articles/PMC8189492/
8. Moqri, M., et al. (2023). Biomarkers of aging for the identification and evaluation of longevity interventions. Cell. https://pmc.ncbi.nlm.nih.gov/articles/PMC11088934/
9. Moqri, M., et al. (2024). Validation of biomarkers of aging. Nature Medicine. https://www.nature.com/articles/s41591-023-02784-9
10. Dmitrienko, A., & D’Agostino, R. B. (2013). Traditional multiplicity adjustment methods in clinical trials. Statistics in Medicine. https://pubmed.ncbi.nlm.nih.gov/23529248/
11. Head, M. L., et al. (2015). The extent and consequences of p-hacking in science. PLOS Biology. https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002106
12. Rothman, K. J. (1990). No adjustments are needed for multiple comparisons. Epidemiology. https://pubmed.ncbi.nlm.nih.gov/2081237/