Meta-Analysis: Between-Study Heterogeneity

Adapted from an appendix of my MS thesis.

Between-Study Heterogeneity

The extent to which true effect sizes vary within a meta-analysis is called between-study heterogeneity. For example, the random-effects model assumes that between-study heterogeneity causes the true effect sizes of studies to differ. It therefore includes an estimate of $\tau^ 2$ , which quantifies this variance in true effects. This allows us to calculate the pooled effect, defined as the mean of the true effect size distribution [1].

High heterogeneity can be caused by the fact that there are two or more subgroups of studies in our data that have a different true effect. In extreme cases, very high heterogeneity can mean that the studies have nothing in common, and that it makes no sense to interpret the pooled effect at all. Every good meta-analysis should not only report an overall effect but also state how trustworthy this estimate is. An essential part of this is to quantify and analyze the between-study heterogeneity [1].

Cochran’s $Q$

When we want to quantify between-study heterogeneity, the difficulty is to identify how much of the variation can be attributed to the sampling error, and how much to true effect size differences. Traditionally, meta-analysts have used Cochran’s $Q$ for this purpose. Cochran’s $Q$ is defined as a weighted sum of squares (WSS). It uses the deviation of each study’s observed effect $\hat{\theta}_ k$ from the summary effect $\hat{\theta}$ , weighted by the inverse of the study’s variance $w_ k$ [1].

$$Q = \sum_ {k=1}^ {K}w_ k(\hat{\theta}_ k-\hat{\theta})^ 2.$$

The amount to which individual effects deviate from the summary effect, the residuals, is squared. Because of the weighting by $w_ k$ , the value of $Q$ does not only depend on how much of $\hat{\theta}_ k$ deviates from $\hat{\theta}$ but also on the precision of studies. If the standard error of an effect size is low (and thus the precision is high), even small deviations from the summary effect will be given a higher weight, leading to higher values of $Q$ . The value of $Q$ can be used to check if there is excess variation in our data, meaning more variation than can be expected from sampling error alone. If this is the case, we can assume that the rest of the variation is due to between-study heterogeneity [1].

It is assumed that $Q$ will approximately follow a $\chi^ 2$ distribution with $K-1$ degrees of freedom where $K$ is the number of studies in our meta-analysis. That is, this assumption holds if effect size differences are only caused by sampling error. Thus the mean of a $\chi^ 2$ distribution with $K-1$ degrees of freedom tells us the value of $Q$ we can expect through sampling error alone [1].

Cochran’s $Q$ can be used to test if the variation in a meta-analysis significantly exceeds the amount we would expect under the null hypothesis of no heterogeneity. Although $Q$ is commonly used and reported in meta-analyses, it has several flaws. A practical concern is that $Q$ increases both when the number of studies $K$ , and when the precision (the sample size of a study) increases. Therefore, $Q$ and whether it is significant highly depends on the size of a meta-analysis, and thus its statistical power. From this it follows that we should not only rely on the significance of a $Q\text{-test}$ when assessing heterogeneity [1].

Higgins & Thompson’s $I^ 2$ Statistic

The $I^ 2$ statistic is directly based on Cochran’s $Q$ . It is defined as the percentage of variability in the effect sizes that is not caused by sampling error. $I^ 2$ draws on the assumption that $Q$ follows a $\chi^ 2$ distribution with $K-1$ degrees of freedom under the null hypothesis of no heterogeneity. It quantifies, in percent, how much the observed value of $Q$ exceeds the expected $Q$ value when there is no heterogeneity. The value of $I^ 2$ cannot be lower than 0%, so if $Q$ happens to be smaller than $K-1$ , we simply use 0 instead of a negative value [1].

$$I^ 2 = \frac{Q-(K-1)}{Q}.$$

It is common to use the $I^ 2$ statistic to report the between-study heterogeneity in meta-analyses, and the popularity of this statistic may be associated with the fact that there is a rule of thumb on how we can interpret it [1].

• $I^ 2=25\%$ : low heterogeneity

• $I^ 2=50\%$ : moderate heterogeneity

• $I^ 2=75\%$ : substantial heterogeneity.

The $H^ 2$ Statistic

The $H^ 2$ statistic is also derived from Cochran’s $Q$ , and similar to $I^ 2$ . It describes the ratio of the observed variation, measured by $Q$ , and the expected variance due to sampling error. The computation of $H^ 2$ is a little more elegant than the one of $I^ 2$ because we do not have to artificially correct its value when $Q$ is smaller than $K-1$ . Values greater than one indicate the presence of between-study heterogeneity. Compared to $I^ 2$ , it is far less common to find this statistic reported in published meta-analyses [1].

$$H^ 2 = \frac{Q}{K-1}.$$

Heterogeneity Variance $\tau^ 2$ & Standard Deviation $\tau$

As previously discussed, $\tau^ 2$ quantifies the variance of the true effect sizes underlying our data. When we take the square root, we obtain $\tau$ , which is the standard deviation of the true effect sizes. A great asset of $\tau$ is that it is expressed on the same scale as the effect size metric. The value of $\tau$ tells us something about the range of the true effect sizes. For example, we can calculate the 95% confidence interval of the true effect sizes by multiplying $\tau$ with 1.96, and then adding and subtracting this value from the pooled effect size [1].

Assessing Heterogeneity

Cochran’s $Q$ and whether it is significant highly depends on the size of a meta-analysis, and thus its statistical power. We should therefore no only rely on $Q$ when assessing between-study heterogeneity. $I^ 2$ , on the other hand, is not sensitive to changes in the number of studies in the analysis. It is also relatively easy to interpret. It is recommended to include $I^ 2$ with confidence intervals as a heterogeneity measure in a meta-analysis report [1].

However, despite its common use in the literature, $I^ 2$ is not a perfect measure for heterogeneity either. It still heavily depends on the precision of the included studies. $I^ 2$ is simply the percentage of variability not caused by sampling error $\epsilon$ . If our studies becomes increasingly large, the sampling error tends to zero, while at the same time, $I^ 2$ tends to 100% simply because the studies have a greater sample size. Since $H^ 2$ behaves similarly to $I^ 2$ , the same caveats also apply to this statistic [1].

The value of $\tau^ 2$ , on the other hand, is sensitive to the number of studies, and their precision. Yet, it is often difficult to interpret how relevant the amount of variance $\tau^ 2$ is from a practical standpoint. Prediction intervals (PIs) are a good way to overcome this limitation, giving us a range into which we can expect the effects of future studies to fall based on present evidence. In addition to reporting $I^ 2$ with confidence intervals, one should also report prediction intervals in meta-analyses [1].

To calculate the 95% prediction intervals around the overall effect $\hat{\mu}$ , we use both the estimated between-study heterogeneity variance $\hat{\tau}^ 2$ , as well as the standard error of the pooled effect $SE_ {\hat{\mu}}$ , to compute the standard deviation of the prediction interval $SD_ \text{PI}$ , using a $t\text{-distribution}$ with $K-1$ degrees of freedom [1].

$$\hat{\mu} \pm t_ {K-1,0.975}\sqrt{SE_ {\hat{\mu}}^ 2+\hat{\tau}^ 2} = \hat{\mu} \pm t_ {K-1,0.975}SD_ \text{PI}.$$

References

1. Harrer, Mathias, Cuijpers, Pim, Furukawa Toshi A, Ebert, David D (2021) Doing Meta-Analysis With R: A Hands-On Guide. Chapman & Hall/CRC Press.