Meta-Analysis: Motivation and Limitations

#math #datascience #statistics #tutorial

Adapted from an appendix of my MS thesis.

Meta-Analysis

Motivation

Science is generally assumed to be a cumulative process, and never in history have we had access to more evidence in the form of published research articles than we do today. In principle, this development should make us enthusiastic about the prospects of science. If science is cumulative, then more published research equals more evidence. Yet, it is not that easy. It is an inconvenient truth that the scientific process, when left to its own devices, will not automatically move us to the best of all possible worlds [1].

Meta-analysis is helpful for critically appraising bodies of evidence in their entirety. That is, as long as we acknowledge its own limitations and biases. A meta-analysis can be described as an analysis of analyses. In conventional studies, the units of analysis are a number people, places, or objects. In meta-analysis, primary studies themselves become the elements of analysis. The aim of meta-analysis is to combine, summarize, and interpret all available evidence related to a clearly defined research field or question. There are at least three distinct ways that evidence from multiple studies can be synthesized [1].

Traditional or narrative reviews are often written by experts and authorities of a research field. There are no strict rules on how studies in a narrative review have to be selected and how to define the scope of the review. There are also no fixed rules on how to draw conclusions from the reviewed evidence. Overall, this can lead to biases favoring the opinion of the author. Nevertheless, when written in a balanced way, narrative reviews can be helpful for readers to get an overall impression of the relevant research questions and available evidence of a field [1].

Systemic reviews try to summarize evidence using clearly defined and transparent rules. Research questions are determined beforehand, and there is an explicit, reproducible methodology through which studies are selected and reviewed. System reviews aim to cover all available evidence. They also asses the validity of evidence using predefined standards and present a synthesis of outcomes in a systemic way [1].

Meta-analyses can be seen as an advanced type of systemic review. However, there is one aspect which makes meta-analyses special. Meta-analyses aim to combine results from previous studies in a quantitative way. The goal of meta-analyses is to integrate quantitative outcomes reported in the selected studies into one numerical estimate. Compared to systemic reviews, meta-analyses often have to be more exclusive concerning the kind of evidence that is summarized. To perform a meta-analysis, it is usually necessary that studies used the same design and type of measurement, and delivered the same intervention [1].

Synthesis Limitations

Certain common pitfalls of meta-analyses can be associated with systemic problems of the scientific process, while others can be traced back to flaws of meta-analyses themselves [1].

The Apples and Oranges problem: Even with the strictest inclusion criteria, studies in a meta-analysis will never be absolutely identical. Meta-analysis means to calculate a numerical estimate which represents the results of all studies. Such an estimate becomes meaningless when studies do not share the properties that matter to a specific research question. Results of a meta-analysis allow us to pool effects and quantify if and how much this effect may vary across different settings. Variation between studies can often be unproblematic, and even insightful if it is correctly incorporated into the aims and problem specification of a meta-analysis [1].
The Garbage In, Garbage Out problem: The quality of evidence produced by a meta-analysis heavily depends on the quality of the studies it summarizes. if the results reported in our included findings are biased, or simply incorrect, the results of the meta-analysis will be equally flawed. This problem can be mitigated to some extent by assessing the quality or risk of bias of the included studies. Though, even the most rigorous meta-analysis will not always be able to balance this out. However, even such disappointing outcomes can be informative, and help guide future research [1].
The File Drawer problem: The file drawer problem refers to the issue that not all relevant research findings are published, and therefore missing in out meta-analysis. Not being able to integrate all evidence in a meta-analysis would be undesirable, but at least tolerable if we could safely assume that research findings are missing at random in the published literature. Unfortunately, they are not, there is good reason to believe that studies with negative or disappointing results are systematically underrepresented in the published literature and that there is a so called publication bias [1].
The Researcher Agenda problem: When defining the scope of a meta-analysis, searching and selecting studies, and ultimately pooling outcomes measures, researchers have to make a myriad of choices. Experts are deeply invested in the research area they are examining. As such, they may hold strong opinions about certain topics, and may intentionally or unintentionally influence the results in the direction that fits their beliefs. One way to reduce the researcher agenda problem is pre-registration of a detailed analysis plan before beginning with data collection [1].