Sensitivity Analyses and Robustness Checks in Ageing Research
Key Takeaways
- Sensitivity analyses examine how results change when investigators vary assumptions, definitions, data-handling rules, or statistical models. [1] [2]
- A finding is robust when reasonable alternative analyses lead to materially similar conclusions, not merely when every analysis produces a p-value below a threshold. [1] [3]
- Ageing studies often need checks for selective attrition, missing data, competing mortality, outcome definitions, and model specification because these features can alter estimates. [5] [8] [9]
- Sensitivity analysis can reveal dependence on assumptions, but it cannot repair a fundamentally unsuitable design or identify which of several conflicting analyses is true. [1] [2]
Who This Is Useful For
This page is useful for readers evaluating studies of biological-age measures, cognitive or functional decline, age-related disease, and mortality. These studies commonly involve long follow-up, incomplete observations, several defensible outcome definitions, and modelling choices that may affect the reported association. [1] [5] [8]
Most statistical findings depend on choices and assumptions. Investigators decide how to define the study population, code exposures and outcomes, handle missing observations, select covariates, and specify the statistical model. A sensitivity analysis changes one or more of these elements to examine whether the main interpretation is stable. [1] [10]
The term robustness check is often used more broadly for an alternative analysis intended to test whether a conclusion depends heavily on a particular decision. Terminology varies between disciplines, so the useful questions are what was changed, why the alternative is credible, and how much the result changed. [1] [4]
What Robustness Means
Robustness concerns the stability of the substantive conclusion across reasonable analyses. The effect estimate may move while remaining compatible with the same broad interpretation, or a p-value may cross 0.05 even though the estimate changes very little. Reading robustness only as repeated statistical significance discards information about effect size and uncertainty. [1] [3]
Stability is also not proof of validity. Several analyses can reproduce the same bias if they share the same flawed measurements, selection process, or untested assumptions. Sensitivity analyses are most informative when each alternative corresponds to a plausible source of uncertainty rather than a set of cosmetic model changes. [2] [6]
Common Checks in Ageing Research
| Source of Uncertainty | Possible Check | What a Change May Indicate |
|---|---|---|
| Missing follow-up data | Compare complete-case results with multiple imputation or analyses using alternative missingness assumptions | The conclusion may depend on who remained observed and how unobserved values were represented |
| Selective attrition | Use inverse-probability-of-attrition weighting or vary assumptions about dropout | Loss of participants with poorer health or cognition may have distorted the longitudinal estimate |
| Competing mortality | Compare an analysis of cause-specific hazards with an analysis of cumulative incidence | The estimated rate among event-free people and the observed probability of an event may answer different questions |
| Outcome definition | Vary a frailty threshold, cognitive-decline definition, biomarker transformation, or follow-up window | The finding may be tied to one operational definition rather than a broader construct |
| Confounder adjustment | Compare prespecified, minimally adjusted, and extended models; quantify sensitivity to unmeasured confounding | The association may depend on adjustment decisions or residual confounding |
| Influential observations | Examine diagnostics and repeat the analysis with defensibly defined influential observations excluded | A small number of observations may have disproportionate influence on the estimate |
These checks address different threats and are not interchangeable. For example, multiple imputation addresses missing data under stated assumptions, whereas an E-value describes how strong an unmeasured confounder would need to be, on the risk-ratio scale, to explain away an association. [5] [7]
Missing Data and Selective Attrition
Longitudinal ageing cohorts lose participants through withdrawal, illness, inability to complete an assessment, and death. If continued participation is related to health or cognition, the people with observed follow-up may differ systematically from those who are no longer observed. [8] [11]
A complete-case analysis and a multiple-imputation analysis rely on different assumptions. Multiple imputation can preserve information and reduce bias under an appropriate imputation model, but it does not make missingness harmless; its validity depends on the variables included and the assumed missing-data mechanism. [5]
Ageing research offers a particularly clear example of informative attrition. In a study of smoking and cognitive decline, inverse-probability weighting was used to assess bias from survival and continued participation, illustrating how attrition checks can alter the interpretation of longitudinal change. [8]
Competing Risks and Survival Models
A competing event prevents the event of interest from occurring. Death from another cause, for example, can preclude a later diagnosis or disability outcome. This issue becomes prominent in older populations because competing mortality is more common and can materially affect absolute-risk estimates. [9] [12]
Cause-specific hazard models and Fine–Gray subdistribution hazard models do not simply provide two interchangeable versions of the same answer. They target different quantities, so comparing them is informative only when the research question and interpretation of each model are made explicit. [9]
Alternative Specifications
Observational estimates can vary with choices about covariates, transformations, exclusions, and model form. An analysis of many clinical, environmental, and physiological variables in relation to mortality demonstrated substantial variation across alternative adjustment sets, including reversals in the direction of some associations. [3]
Specification-curve and multiverse approaches make this analytical flexibility visible by displaying results across a defined set of reasonable specifications. Their value depends on whether the included specifications are scientifically defensible and whether the set was chosen transparently rather than constructed after seeing the preferred result. [4] [10]
Unmeasured Confounding and Quantitative Bias Analysis
Conventional adjustment cannot remove confounding by variables that were not measured adequately. Quantitative bias analyses instead ask how an estimate would change under explicit assumptions about the strength or distribution of unmeasured confounding, misclassification, or selection bias. [2] [6]
The E-value is one summary for unmeasured confounding on the risk-ratio scale. It estimates the minimum association strength that an unmeasured confounder would need with both exposure and outcome, conditional on measured covariates, to explain away an observed association. It does not detect a confounder, establish that such a confounder exists, or address other biases such as selection and measurement error. [7]
How to Read a Robustness Section
| Question | Why It Matters | Warning Sign |
|---|---|---|
| Was the alternative analysis specified in advance? | Prespecification reduces the opportunity to select checks after seeing their results | The paper reports only favorable alternatives without explaining how they were chosen |
| Does the check address a plausible threat? | A useful check links an analytical variation to a specific assumption or bias | Many models are reported, but none examines the study's main vulnerability |
| How much did the estimate and interval change? | Magnitude and uncertainty are more informative than significance labels alone | Robustness is claimed solely because p-values remain below 0.05 |
| Are conflicting results explained? | Divergence can identify which assumption drives the conclusion | An unfavorable check is placed in an appendix and not discussed |
| Do all analyses share the same limitation? | Repeated models may not address common measurement or selection bias | Several regressions use the same poorly measured exposure and are called independent confirmation |
Good reporting identifies the primary analysis, the rationale for each alternative, the assumptions that changed, and the consequences for the study conclusion. Reporting only that results were “similar” makes the robustness claim difficult to evaluate. [1] [4]
When Results Disagree
Disagreement between analyses is evidence about dependence on assumptions, not automatically evidence that one analysis failed. The next step in interpretation is to identify which choice caused the change and whether one assumption is better supported by the design, measurement process, and research question. [1] [2]
A large change in effect size or direction generally indicates fragility. A small shift in a p-value near a conventional threshold may indicate nearly identical numerical evidence described with different binary labels. Confidence intervals and the range of estimates across analyses provide more context than a count of statistically significant results. [3] [4]
What This Does Not Mean
- It does not mean researchers should try every possible model; alternatives should be tied to plausible assumptions and a defined scientific question. [4] [10]
- It does not mean the analysis producing the smallest p-value or largest effect is the preferred analysis. [3] [4]
- It does not mean stable results are free from bias, because all checked models may share the same measurement or selection problem. [2] [6]
- It does not mean unstable findings are useless; instability can show where interpretation depends most strongly on an uncertain assumption. [1]
Practical Interpretation Examples
- If a biomarker association disappears after excluding early deaths: the result may be sensitive to pre-existing illness or reverse causation, but the exclusion also changes the target population and must be justified. [1] [2]
- If cognitive decline is weaker after attrition weighting: continued participation may have selected a healthier or cognitively different subset of the original cohort. [8]
- If a disease-risk estimate changes under a competing-risk model: determine whether the paper is estimating an event rate among those still event-free or the probability of the event over time. [9] [12]
- If only the p-value changes across adjustment sets: compare the actual effect estimates and confidence intervals before concluding that the scientific interpretation changed. [3]
Related Reading
Summary
Sensitivity analyses and robustness checks show how strongly a result depends on assumptions and analytic decisions. In ageing research, the most informative checks often concern missing observations, selective attrition, competing mortality, variable definitions, confounding, and model specification. A robust finding is one whose substantive interpretation remains reasonably stable across credible alternatives, while a fragile finding is one whose interpretation changes under plausible assumptions. Neither label substitutes for evaluating the underlying study design and measurement quality. [1] [2] [8] [9]
References
- Thabane, L., et al. (2013). BMC Medical Research Methodology. https://pmc.ncbi.nlm.nih.gov/articles/PMC3720188/
- Greenland, S. (1996). International Journal of Epidemiology. https://pubmed.ncbi.nlm.nih.gov/9027513/
- Patel, C. J., Burford, B., & Ioannidis, J. P. A. (2015). Journal of Clinical Epidemiology. https://pmc.ncbi.nlm.nih.gov/articles/PMC4555355/
- Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2020). Nature Human Behaviour. https://www.nature.com/articles/s41562-020-0912-z
- Sterne, J. A. C., et al. (2009). BMJ. https://pmc.ncbi.nlm.nih.gov/articles/PMC2714692/
- Lash, T. L., Abrams, B., & Bodnar, L. M. (2014). Epidemiology. https://pmc.ncbi.nlm.nih.gov/articles/PMC4306386/
- VanderWeele, T. J., & Ding, P. (2017). Annals of Internal Medicine. https://pubmed.ncbi.nlm.nih.gov/28693043/
- Weuve, J., et al. (2012). Epidemiology. https://pmc.ncbi.nlm.nih.gov/articles/PMC3237815/
- Austin, P. C., & Fine, J. P. (2017). Statistics in Medicine. https://pmc.ncbi.nlm.nih.gov/articles/PMC5698744/
- Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016). Perspectives on Psychological Science. https://pubmed.ncbi.nlm.nih.gov/27694465/
- Goudy, W. J. (1985). Journal of Gerontology. https://pubmed.ncbi.nlm.nih.gov/3989251/
- Cooper, H., et al. (2021). International Journal of Epidemiology. https://pubmed.ncbi.nlm.nih.gov/34109395/
This content is provided for educational purposes only and does not constitute medical advice.