DEV Community: Ethan Davis

Multilevel (Three-Level) Meta-Analysis

Ethan Davis — Sun, 12 Jul 2026 22:53:20 +0000

Multilevel Meta-Analysis

When people talk about multilevel meta-analysis, they are referring to three-level meta-analysis models. A three-level model contains three pooling steps. First, researchers themselves pool the results of individual participants in their primary studies, and report the aggregated effect size. Then, on level 2, these effect sizes are nested within several clusters denoted by $κ$ . These clusters can either be within individual studies or across subgroups of studies. Lastly, pooling the aggregated cluster effects leads to the overall true effect size $μ$ [1].

For an example of dependence introduced within a study, consider scientists conducting a study. They may have collected data from multiple sites, compared multiple interventions to one single control group, or used different questionnaires to measure the same outcome. In all these scenarios, we can assume that some kind of dependency is introduced within the reported study [1].

As an example of dependence introduced across studies, think of a meta-analysis that focuses on some psychological mechanism. This meta-analysis includes studies which were conducted in different cultural regions of the world. Depending on the type of psychological mechanism, it could be that results of studies conducted in the same cultural region are more similar compared to those conducted in a different culture [1].

The formula for the three-level model follows. $θ^_{ij}$ is an estimate of the true effect size $θ_{ij}$ . The term $ij$ can be read as some effect size $i$ nested in cluster $j$ . Parameter $κ_{j}$ is the average effect size in cluster $j$ , and $μ$ the overall average population effect. In contrast to the random-effects model, this formula contains two heterogeneity terms. One is $ζ_{(2) ij}$ , which stands for the within-cluster heterogeneity on level 2. The other is $ζ_{(3) j}$ , the between-cluster heterogeneity on level 3 [1].

(Level 1) θ^_{ij} = θ_{ij} + ϵ_{ij} (Level 2) θ_{ij} = κ_{j} + ζ_{(2) ij} (Level 3) κ_{j} = μ + ζ_{(3) j} θ^_{ij} = μ + ζ_{(3) j} + ζ_{(2) ij} + ϵ_{ij} .

Subgroup Analyses in Three-Level Models

We can add regression terms to a multilevel model, which leads to three-level mixed-effects model where $θ$ is the intercept and $β$ is the regression weight of a predictor variable $x$ [1].

θ^_{ij} = θ + β x_{i} + ζ_{(3) j} + ζ_{(2) ij} + ϵ_{ij} .

Robust Variance Estimation

A hierarchical model can provide a better representation of a dataset than a conventional meta-analysis, which assumes that all effect sizes are independent. But it is still a simplification of reality. In practice, there are often forms of dependence between effect sizes that are more complex than what we have captured so far in nested models. When several effect sizes in one study are based on the same sample, we expect their sampling errors (the $ϵ_{ij}$ terms in the equation) to be correlated [1].

The extended three-level architecture, the correlated and hierarchical effects (CHE) model, explicitly takes into account that some effect sizes within clusters are based on the same sample, and that their sampling errors are therefore correlated. In combination with the CHE model, the robust variance estimation (RVE) and its so-called sandwich estimator can be used to obtain robust confidence intervals and $p -values$ [1].

For the robust variance estimator of a meta-regression of the conventional random-effects model, we can rewrite the equation so that it models dependent effect sizes, and is expressed in matrix notation [1].

t_{j} = X_{j} β + u_{j} + e_{j} .

The equation tells us that effect sizes $t_{j}$ are predicted by regression weights $β$ associated with certain covariates $X_{j}$ . It also tells us that, besides the sampling error $e_{j}$ , there are random effects for each study $u_{j}$ , thus producing a mixed-effects meta-regression model. Say that $n_{j}$ is the number of effect sizes in some study $j$ . The effect sizes in study $j$ can then be written as the column vector $t_{j} = (t_{j, 1}, \dots, t_{j, n_{j}})^{⊤}$ . Similarly, $X_{j}$ is the design matrix containing the covariate values of some study $j$ , where $p - 1$ is the total number of covariates. The vector of regression coefficients $β = (β_{1} \dots, β_{p})^{⊤}$ contains no subscript $j$ , since it is assumed to be fixed across all studies [1].

X_{j} = ⎣ ⎡ x_{1, 1} ⋮ x_{n_{j}, 1} \dots ⋱ \dots x_{1, p} ⋮ x_{n_{j}, p} ⎦ ⎤ .

Based on this formula, we can estimate the meta-regression coefficients $β^$ . To calculate confidence intervals and conduct significance tests of the coefficients, we need an estimate of their variance $V_{β^}$ . This can be achieved using the robust sampling variance estimator otherwise known as the sandwich estimator [1].

V_{β^} = (j = 1 \sum J X_{j ⊤} W_{j} X_{j})^{- 1} (j = 1 \sum J X_{j ⊤} W_{j} A_{j} Φ_{j} A_{j} W_{j} X_{j}) (j = 1 \sum J X_{j ⊤} W_{j} X_{j})^{- 1} .

The true dependence structure of the effect sizes in some study $j$ is described by the $n_{j} \times n_{j}$ covariance matrix $Φ_{j}$ . Unfortunately, it is rarely known to what extent effect sizes are correlated within and across studies. Therefore, it is necessary to make a few simplifying assumptions in our model. The CHE model, assumes that there is a known correlation $ρ$ between effect sizes in the same study, and that $ρ$ has the same value across all studies in our meta-analysis [1].

The weights $W_{j}$ of each effect size takes the precision of effect size estimates into account before we can pool them. The optimal way to do this is to take the inverse of the variance $W_{j} = Φ_{j - 1}$ . As mentioned, the true values of $Φ_{j}$ are hardly even known, so an estimate based on our model $Φ^_{j - 1}$ is used [1].

The adjustment matrix $A_{j}$ ensures that the estimator provides valid results even when the number of studies in our meta-analysis is small. Say, 40 studies or less. The recommended approach is to use a matrix based on the bias-reduced linearization, or CR2 method [1].

Cluster Wild Bootstrapping

Another, and sometimes favorable way to test coefficients in our model are bootstrapping procedures. The cluster wild bootstrapping (CWB) method is well suited if the total number of studies $J$ is our meta-analysis is small. That is, compared to RVE, which can lead to overly conservative results in small samples [1].

The cluster wild bootstrap is based on the residuals of a null model fitted without any additional covariates. Residuals are transformed using an adjustment matrix $A_{j}$ , for example based on the CR2 method, to handle dependent effect sizes. A general algorithm for CWB looks like the following. Steps 3 to 5 are repeated $R$ times. The bootstrap $p -value$ can be derived as the proportion of times the bootstrap test statistic was more extreme than the one based on the original data [1].

The Cluster Wild Bootstrapping Algorithm [1]

Calculate the full model based on the original data, and derive the test statistic of interest.
Fit a null model based on the original data and extract its residuals $e$ .
For each study $j$ , draw a random value from a distribution, and multiply the residuals of $j$ by this random value.
Generate new, bootstrapped effect sizes by adding the transformed residuals to the predicted values of the null model based on the original data.
Fit the full model again, using the bootstrapped effect sizes, and calculate the test statistic again.

References

Harrer, Mathias, Cuijpers, Pim, Furukawa Toshi A, Ebert, David D (2021) Doing Meta-Analysis With R: A Hands-On Guide. Chapman & Hall/CRC Press.

Meta-Analysis: Meta-Regression

Ethan Davis — Sun, 12 Jul 2026 22:50:12 +0000

Meta-Regression

Usually, regression models are based on data comprising individual persons or specimens, for which both the independent $x$ and dependent $y$ values are measured. In meta-regression, this concept is applied to entire studies. The variable $x$ represents characteristics of studies, for example, the year in which it was conducted. Based on this information, a meta-regression model tries to predict $y$ , the study’s effect size [1].

In a conventional regression, we want to estimate the value $y_{i}$ of a person or specimen $i$ using a predictor (or covariate) $x_{i}$ with a regression coefficient $β$ [1].

y_{i} = β_{0} + β_{1} x_{i} .

In meta-regression, the variable $y$ we want to predict is the observed effect size $θ^_{k}$ of study $k$ [1].

θ^_{k} = θ + β x_{k} + ϵ_{k} + ζ_{k} .

This formula contains two extra terms, $ϵ_{k}$ and $ζ_{k}$ . The same terms found in the random-effects model from the equation. The first one, $ϵ_{k}$ , is the sampling error through which the effect size of a study deviates from its true effect. The second error, $ζ_{k}$ , denotes that even the true effect size of the study is only sampled from an overarching distribution of effect sizes. Since the meta-regression formula includes a fixed effect (the $β$ coefficient) as well as a random effect ( $ζ_{k}$ ), the model used in meta-regression is often called a mixed-effects model [1].

Meta-Regression with a Categorical Predictor

Subgroup analysis (from the companion Subgroup Analysis post) is nothing else than a meta-regression with a categorical predictor, where such categorical variables can be included through dummy-coding, such as the following [1].

D_{g} = {0 : 1 : Subgroup A Subgroup B

To specify a subgroup analysis in the form of a meta-regression, we simply have to replace the covariate $x_{k}$ with $D_{g}$ [1].

θ^_{k} = θ + β D_{g} + ϵ_{k} + ζ_{k} .

Meta-Regression with a Continuous Predictor

When people speak of a meta-regression, they usually think of models in which a continuous variable is used as the predictor. The aim of the meta-regression model is to find values of $θ$ and $β$ which minimize the difference between the predicted effect size and the true effect size of studies. Taking into account both the sampling error $ϵ_{k}$ and between-study heterogeneity $ζ_{k}$ , meta-regression tried to find a model that generalizes well, not only to the observed effect sizes but to the universe of all possible studies of interest [1].

An important detail about meta-regression models is that they can be seen as an extension of the random-effects model. The random-effects model is nothing but a meta-regression without a slope term. Since it contains no slope, the random-effects model simply predicts the same value for all studies. That is, the estimate of the pooled effect size $μ$ , which is equivalent to the intercept [1].

In the first step, the calculation of a meta-regression therefore closely resembles the one of a random-effects meta-analysis, in that the between-study heterogeneity $τ^{2}$ is estimated. In the next step, the fixed weights $θ$ and $β$ are estimated. In meta-regression, weighted least squares (WLS) is used to find the regression line that fits the data best. WLS makes sure that studies with a smaller standard error are given a higher weight [1].

If the meta-regression model fits the data well, the true effect sizes should deviate less from the regression line compared to the pooled effect $μ^$ . If this is the case, the predictor $x$ explains some of the heterogeneity variance in our meta-analysis. The fit of the meta-regression model can therefore be assessed by checking how much of the heterogeneity variance it explains [1].

The predictors included in the mixed-effects model should minimize the amount of residual heterogeneity variance denoted by $τ^_{MEM 2}$ . The $R_{* 2}$ index is used to quantify the percentage of variation explained by the meta-regression model. The asterisk is added to differentiate from the $R^{2}$ index used in conventional regressions, since meta-regression deals with true effect sizes instead of observed data points [1].

$R_{* 2}$ uses the amount of residual heterogeneity variance that even the meta-regression slope cannot explain. It is put in relation to the total heterogeneity that we initially found in our random-effects model $τ^_{REM 2}$ . Subtracting this fraction from 1 leaves us with the percentage of between-study heterogeneity explained by the predictor, and is expressed in the equation. We can also says that it is how much the mixed-effects model has reduced the heterogeneity variance compared to the initial random-effects pooling model, in percent [1].

R_{* 2} = 1 - \frac{τ ^ _{MEM 2}}{τ ^ _{REM 2}} = \frac{τ ^ _{REM 2} - τ ^ _{MEM 2}}{τ ^ _{REM 2}} .

Usually, we are not only in the amount of heterogeneity explained by the regression model but also if the regression weight of our predictor $x$ is significant. Both in conventional and meta-regression, the significance of a regression weight is commonly assessed through a Wald test. This involves calculating the test statistic $z$ , by dividing the estimate of $β$ through its standard error [1].

z = \frac{β}{S E _{β^}} .

Under the null hypothesis that $β = 0$ , this $z -statistic$ follows a standard normal distribution. This allows us to calculate a corresponding $p -value$ . Like in normal meta-analysis models, we can also use the Knapp-Hartung adjustment, which results in a test statistic based on the $t -distribution$ [1].

References

Harrer, Mathias, Cuijpers, Pim, Furukawa Toshi A, Ebert, David D (2021) Doing Meta-Analysis With R: A Hands-On Guide. Chapman & Hall/CRC Press.

Meta-Analysis: Subgroup Analysis

Ethan Davis — Sun, 12 Jul 2026 22:47:03 +0000

Subgroup Analysis

Outlier and influence analysis can help us identify which studies contribute to observed heterogeneity. They are performed after seeing the data, and often because of the results we found. These analyses may tell us that some study does not properly follow the expectations of our model, but not why this is that case. This makes it clear that we need a different approach, one that allows us to identify why a specific heterogeneity pattern can be found in our data [1].

Subgroup analyses, also known as moderator analyses, are one way to do this. They allow us to test specific hypotheses, describing why some type of study produces lower or higher effects than another. The idea behind subgroup analysis is that meta-analysis is not only about calculating an average effect size but that it can also be a tool to investigate variation in our evidence. In subgroup analyses, we see heterogeneity not merely as a nuisance but as interesting variation which may or may not be explainable by a scientific hypothesis [1].

In subgroup analyses, we hypothesize that studies in our meta-analysis do not stem from one overall population. Instead, we assume that they fall into different subgroups and that each subgroup has its own true overall effect. Our aim is to reject the null hypothesis that there is no difference in effect sizes between subgroups. The calculation of a subgroup analysis consists of two parts. First, we pool the effect in each subgroup. Subsequently, the effects of the subgroups are compared using a statistical test [1].

Pooling the Effect in Subgroups

The first part is straightforward, as the same criteria as those for a meta-analysis without subgroups apply. If we assume that all studies in a subgroup stem from the same population, and have one shared true effect, we can use the fixed-effect mode. However, as mentioned previously, it is often unrealistic that this assumption holds in practice, even when we partition our studies into smaller subgroups [1].

Therefore, the alternative is to use a random-effects model. This assumes that studies within a subgroup are drawn from a universe of populations, the mean of which we want to estimate. The difference to a normal meta-analysis is that we conduct several separate random-effects meta-analyses, one for each subgroup. This results in a pooled effect $μ^_{g}$ for each subgroup $g$ [1].

Since each subgroup gets its own separate meta-analysis, estimates of the $τ^{2}$ heterogeneity will also differ from subgroup to subgroup. In practice, however, the individual heterogeneity values $τ^_{g 2}$ are often replaced with a version of $τ^{2}$ that was pooled across subgroups. This means that all subgroups are assumed to share a common estimate of the between-study heterogeneity. This is done for practical reasons. When the number of studies in a subgroup is small, for example $k_{g} < 5$ , it is likely that the estimate of $τ^{2}$ will be imprecise. In this case, it is better to calculate a pooled version of $τ^{2}$ that is used across all subgroups [1].

Comparing the Subgroup Effects

In the next step, we assess if there is a true difference between the $G$ subgroups. The assumption is that the subgroups are different, meaning that at least one subgroup is part of a different population of studies. A way to test this is to pretend that the pooled effect of a subgroup is nothing more than the observed effect size of one large study. Similar to when assessing the heterogeneity of a normal meta-analysis, we want to know if difference in effect sizes exist only due to sampling error, or because of true differences in the effect sizes [1].

Therefore, we use the value of $Q$ to determine if the subgroup differences are large enough to not be explainable by sampling error alone. Pretending that the subgroup effects are observed effect sizes, we calculate the value of $Q$ . This observed $Q$ value is compared to its expected value assuming a $χ^{2}$ distribution with $G - 1$ degrees of freedom. If the observed value of $Q$ is substantially larger than the expected one, the $p -value$ of the $Q$ test will become significant. This indicates that there is a difference in the true effect sizes between subgroups [1].

Borenstein and Higgins (2013) argue that the subgroups we choose to analyze cannot always be seen as random draws from a universe of possible subgroups, but represent fixed levels of a characteristic we want to examine. Take employment status as an example. This features two fixed subgroups, “employed” and “unemployed”. Borenstein and Higgins call the model for subgroup analyses the fixed-effects (plural) model, where the word “plural” is added because we have to differentiate it from the standard fixed-effect model. It is also known in the literature as a mixed-effects model [1].

References

Harrer, Mathias, Cuijpers, Pim, Furukawa Toshi A, Ebert, David D (2021) Doing Meta-Analysis With R: A Hands-On Guide. Chapman & Hall/CRC Press.

Meta-Analysis: Forest Plots

Ethan Davis — Sun, 12 Jul 2026 22:43:42 +0000

Forest Plots

The most common way to visualize meta-analyses is through forest plots. See the figure for an example created from a dataset containing studies examining the effect of so-called “third wave” psychotherapies on perceived stress in college students. For each study, the standardized mean difference between a treatment and control group at post-test was calculated, and Hedges’ $g$ small sample correction was applied [1].

For each study, a graphical representation of the effect size is provided. This visualization shows the point estimate of a study on the horizontal axis. This point estimate is supplemented by a line, which represents the range of the confidence interval calculated for the observed effect size. Usually, the points estimate is surrounded by a square. The size of this square is determined by the weight of the effect size: Studies with a larger weight are given a larger square [1].

At the bottom of the plot, a diamond shape represents the average effect. The length of the diamond symbolizes the confidence interval of the pooled result on the horizontal axis. Typically, forest plots also include a vertical reference line, which indicates the point on the horizontal axis equal to no effect [1].

References

Harrer, Mathias, Cuijpers, Pim, Furukawa Toshi A, Ebert, David D (2021) Doing Meta-Analysis With R: A Hands-On Guide. Chapman & Hall/CRC Press.

Meta-Analysis: Between-Study Heterogeneity

Ethan Davis — Sun, 12 Jul 2026 22:37:59 +0000

Between-Study Heterogeneity

The extent to which true effect sizes vary within a meta-analysis is called between-study heterogeneity. For example, the random-effects model assumes that between-study heterogeneity causes the true effect sizes of studies to differ. It therefore includes an estimate of $τ^{2}$ , which quantifies this variance in true effects. This allows us to calculate the pooled effect, defined as the mean of the true effect size distribution [1].

High heterogeneity can be caused by the fact that there are two or more subgroups of studies in our data that have a different true effect. In extreme cases, very high heterogeneity can mean that the studies have nothing in common, and that it makes no sense to interpret the pooled effect at all. Every good meta-analysis should not only report an overall effect but also state how trustworthy this estimate is. An essential part of this is to quantify and analyze the between-study heterogeneity [1].

Cochran’s $Q$

When we want to quantify between-study heterogeneity, the difficulty is to identify how much of the variation can be attributed to the sampling error, and how much to true effect size differences. Traditionally, meta-analysts have used Cochran’s $Q$ for this purpose. Cochran’s $Q$ is defined as a weighted sum of squares (WSS). It uses the deviation of each study’s observed effect $θ^_{k}$ from the summary effect $θ^$ , weighted by the inverse of the study’s variance $w_{k}$ [1].

Q = k = 1 \sum K w_{k} (θ^_{k} - θ^)^{2} .

The amount to which individual effects deviate from the summary effect, the residuals, is squared. Because of the weighting by $w_{k}$ , the value of $Q$ does not only depend on how much of $θ^_{k}$ deviates from $θ^$ but also on the precision of studies. If the standard error of an effect size is low (and thus the precision is high), even small deviations from the summary effect will be given a higher weight, leading to higher values of $Q$ . The value of $Q$ can be used to check if there is excess variation in our data, meaning more variation than can be expected from sampling error alone. If this is the case, we can assume that the rest of the variation is due to between-study heterogeneity [1].

It is assumed that $Q$ will approximately follow a $χ^{2}$ distribution with $K - 1$ degrees of freedom where $K$ is the number of studies in our meta-analysis. That is, this assumption holds if effect size differences are only caused by sampling error. Thus the mean of a $χ^{2}$ distribution with $K - 1$ degrees of freedom tells us the value of $Q$ we can expect through sampling error alone [1].

Cochran’s $Q$ can be used to test if the variation in a meta-analysis significantly exceeds the amount we would expect under the null hypothesis of no heterogeneity. Although $Q$ is commonly used and reported in meta-analyses, it has several flaws. A practical concern is that $Q$ increases both when the number of studies $K$ , and when the precision (the sample size of a study) increases. Therefore, $Q$ and whether it is significant highly depends on the size of a meta-analysis, and thus its statistical power. From this it follows that we should not only rely on the significance of a $Q -test$ when assessing heterogeneity [1].

Higgins & Thompson’s $I^{2}$ Statistic

The $I^{2}$ statistic is directly based on Cochran’s $Q$ . It is defined as the percentage of variability in the effect sizes that is not caused by sampling error. $I^{2}$ draws on the assumption that $Q$ follows a $χ^{2}$ distribution with $K - 1$ degrees of freedom under the null hypothesis of no heterogeneity. It quantifies, in percent, how much the observed value of $Q$ exceeds the expected $Q$ value when there is no heterogeneity. The value of $I^{2}$ cannot be lower than 0%, so if $Q$ happens to be smaller than $K - 1$ , we simply use 0 instead of a negative value [1].

I^{2} = \frac{Q - ( K - 1 )}{Q} .

It is common to use the $I^{2}$ statistic to report the between-study heterogeneity in meta-analyses, and the popularity of this statistic may be associated with the fact that there is a rule of thumb on how we can interpret it [1].

$I^{2} = 25%$ : low heterogeneity
$I^{2} = 50%$ : moderate heterogeneity
$I^{2} = 75%$ : substantial heterogeneity.

The $H^{2}$ Statistic

The $H^{2}$ statistic is also derived from Cochran’s $Q$ , and similar to $I^{2}$ . It describes the ratio of the observed variation, measured by $Q$ , and the expected variance due to sampling error. The computation of $H^{2}$ is a little more elegant than the one of $I^{2}$ because we do not have to artificially correct its value when $Q$ is smaller than $K - 1$ . Values greater than one indicate the presence of between-study heterogeneity. Compared to $I^{2}$ , it is far less common to find this statistic reported in published meta-analyses [1].

H^{2} = \frac{Q}{K - 1} .

Heterogeneity Variance $τ^{2}$ & Standard Deviation $τ$

As previously discussed, $τ^{2}$ quantifies the variance of the true effect sizes underlying our data. When we take the square root, we obtain $τ$ , which is the standard deviation of the true effect sizes. A great asset of $τ$ is that it is expressed on the same scale as the effect size metric. The value of $τ$ tells us something about the range of the true effect sizes. For example, we can calculate the 95% confidence interval of the true effect sizes by multiplying $τ$ with 1.96, and then adding and subtracting this value from the pooled effect size [1].

Assessing Heterogeneity

Cochran’s $Q$ and whether it is significant highly depends on the size of a meta-analysis, and thus its statistical power. We should therefore no only rely on $Q$ when assessing between-study heterogeneity. $I^{2}$ , on the other hand, is not sensitive to changes in the number of studies in the analysis. It is also relatively easy to interpret. It is recommended to include $I^{2}$ with confidence intervals as a heterogeneity measure in a meta-analysis report [1].

However, despite its common use in the literature, $I^{2}$ is not a perfect measure for heterogeneity either. It still heavily depends on the precision of the included studies. $I^{2}$ is simply the percentage of variability not caused by sampling error $ϵ$ . If our studies becomes increasingly large, the sampling error tends to zero, while at the same time, $I^{2}$ tends to 100% simply because the studies have a greater sample size. Since $H^{2}$ behaves similarly to $I^{2}$ , the same caveats also apply to this statistic [1].

The value of $τ^{2}$ , on the other hand, is sensitive to the number of studies, and their precision. Yet, it is often difficult to interpret how relevant the amount of variance $τ^{2}$ is from a practical standpoint. Prediction intervals (PIs) are a good way to overcome this limitation, giving us a range into which we can expect the effects of future studies to fall based on present evidence. In addition to reporting $I^{2}$ with confidence intervals, one should also report prediction intervals in meta-analyses [1].

To calculate the 95% prediction intervals around the overall effect $μ^$ , we use both the estimated between-study heterogeneity variance $τ^^{2}$ , as well as the standard error of the pooled effect $S E_{μ^}$ , to compute the standard deviation of the prediction interval $S D_{PI}$ , using a $t -distribution$ with $K - 1$ degrees of freedom [1].

μ^\pm t_{K - 1, 0.975} S E_{μ^2} + τ^^{2} = μ^\pm t_{K - 1, 0.975} S D_{PI} .

References

Harrer, Mathias, Cuijpers, Pim, Furukawa Toshi A, Ebert, David D (2021) Doing Meta-Analysis With R: A Hands-On Guide. Chapman & Hall/CRC Press.

Meta-Analysis: Pooling Effect Sizes

Ethan Davis — Sun, 12 Jul 2026 22:28:54 +0000

Pooling Effect Sizes

Fixed-Effect Model

The fixed-effect model assumes that all effect sizes stem from a single, homogeneous population. It states that all studies share the same true effect size. According to the fixed-effect model, the only reason why a study $k$ ’s observed effect size $θ^_{k}$ deviates from $θ$ is because of its sampling error $ϵ_{k}$ . This sampling error causes the observed effect to deviate from the overall, true effect. We can describe the relationship as the following [1].

θ^_{k} = θ + ϵ_{k} .

In the formula of the fixed-effect model, the true effect size is symbolized by $θ$ , not $θ_{k}$ as in the equation. Previously, we only made statements about the true effect size of one individual study $k$ . The fixed-effect tells us that a study’s true effect size $θ_{k}$ , and the overall, pooled effect size $θ$ , are identical. The formula of the fixed-effect models tells us that there is only one reason why observed effect sizes $θ_{k}$ deviate from the true overall effect: because of the sampling error $ϵ_{k}$ . Furthermore, all things being equal, as the sample size becomes larger, the sampling error becomes smaller [1].

If we want to calculate the pooled effect size under the fixed-effect model, we therefore simply use a weighted average of all studies. To calculate the weight $w_{k}$ for each study $k$ , we can use the standard error, which we square to obtain the variance $s_{k 2}$ of each effect size. Since a lower variance indicates higher precision, the inverse of the variance is used to determine the weight of each study [1].

w_{k} = \frac{1}{s _{k 2}} .

Once we know the weights, we can calculate the weighted average, our estimate of the true pooled effect $θ^$ . This method is the most common approach to calculate average effects in meta-analyses. Since we use the inverse of the variance, it is often called inverse-variance weighting, or simply inverse-variance meta-analysis [1].

θ^= \frac{\sum _{k = 1 K} θ ^ _{k} w _{k}}{\sum _{k = 1 K} w _{k}} .

Random-Effects Model

The fixed-effect model assumes that all our studies are part of a homogeneous population. However, it is simply unrealistic that studies in a meta-analysis are always completely homogeneous. It is likely that we can anticipate considerable between-study heterogeneity in the true effects. The random-effects model assumes that there is not only one true effect size but a distribution of true effect sizes. The goal of the random-effects model is therefore not to estimate the one true effect size of all studies, but the mean of the distribution of true effects [1].

Similar to the fixed-effect model, the random-effects model starts by assuming that an observed effect size $θ^_{k}$ is an estimator of the study’s true effect size $θ_{k}$ , burdened by sampling error $ϵ_{k}$ [1].

θ^_{k} = θ_{k} + ϵ_{k} .

The fact that we use $θ_{k}$ instead of $θ$ is an important difference. The random-effects model only assumes that $θ_{k}$ is the true effect size of one single study $k$ . It stipulates that there is a second source of error, denoted by $ζ_{k}$ . This second source of error is introduced by the fact that even the true effect size $θ_{k}$ of study $k$ is only part of an over-arching distribution of true effect sizes with mean $μ$ [1].

θ_{k} = μ + ζ_{k} .

The random-effects model tells us that there is a hierarchy of two processes. The observed effect sizes of a study deviate from their true value because of the sampling error. But even the true effect sizes are only a draw from a universe of true effects, whose mean $μ$ we want to estimate as the pooled effect of our meta-analysis. We can express the random-effects mode in one line. This formula makes it clear that our observed effect size deviates from the pooled effect $μ$ because of two error terms, $ζ_{k}$ and $ϵ_{k}$ [1].

θ^_{k} = μ + ζ_{k} + ϵ_{k} .

A crucial assumption of the random-effects model is that the size of $ζ_{k}$ is independent of $k$ . That is, the size of $ζ_{k}$ is a product of chance, and chance alone. This is known as the exchangeability assumption of the random-effects model. All true effect sizes are assumed to be exchangeable in so far as we have nothing that could tell us how big $ζ_{k}$ will be in some study $k$ before seeing the data [1].

The challenge associated with the random-effects model is that we have to take the error $ζ_{k}$ into account. To do this, we have to estimate the variance of the distribution of true effect sizes. This variance is known as $τ^{2}$ . Once we know the value of $τ^{2}$ , we can include the between-study heterogeneity when determining the inverse-variance weight of each effect size. In the random-effects model, we therefore calculate an adjusted random-effects weight $w_{k *}$ for each observation [1].

w_{k *} = \frac{1}{s _{k 2} + τ ^{2}} .

Using the adjusted random-effects weights, we then calculate the pooled effect size using the inverse variance method, just as we do using the fixed-effect model [1].

θ^= \frac{\sum _{k = 1 K} θ ^ _{k} w _{k *}}{\sum _{k = 1 K} w _{K *}} .

There are several methods to estimate $τ^{2}$ , and it is an ongoing research question which of these estimators performs best for different kinds of data. Overall, estimators of $τ^{2}$ fall into two categories. Some, like the DerSimonian-Laird and Sidik-Jonkman estimator, are based on closed-form expressions. The restricted maximum likelihood, Paule-Mandel estimator, and empirical Bayes estimator find the optimal value of $τ^{2}$ through an iterative algorithm [1].

The Knapp-Hartung adjustments try to control for the uncertainty in our estimate of the between-study heterogeneity. While significance tests of the pooled effect usually assume a normal distribution known as Wald tests, the Knapp-Hartung method is based a $t -distribution$ . Applying a Knapp-Hartung adjustment is usually sensible. Several studies have shown that these adjustments can reduce the chance of false positives, especially when the number of studies is small [1].

References

Harrer, Mathias, Cuijpers, Pim, Furukawa Toshi A, Ebert, David D (2021) Doing Meta-Analysis With R: A Hands-On Guide. Chapman & Hall/CRC Press.

Meta-Analysis: Effect Sizes

Ethan Davis — Sun, 12 Jul 2026 22:26:20 +0000

Effect Sizes

To perform a meta-analysis, we have to find an effect size which can be summarized across all studies. In particular, the selected effect size measure for a meta-analysis should comparable, computable, reliable, and interpretable. An effect size is defined as a metric quantifying the relationship between two entities. It captures the direction and magnitude of this relationship. For example, correlations describe how well we can predict the values of a variable through the values of another and so can be seen as a form of effect size [1].

In mathematical notation, it is common to use the Greek letter $θ$ as the symbol for a true effect size. More precisely, $θ_{k}$ represents the true effect size of a study $k$ . Importantly, the true effect size is not identical with the observed effect size that we find in the published results of the study. The observed effect size $θ^$ is only an estimate of the true effect size. The observed effect size of study $k$ is therefore $θ^_{k}$ . Note that $θ^_{k}$ differs from $θ_{k}$ because of the sampling error $ϵ_{k}$ [1].

θ^_{k} = θ_{k} + ϵ_{k} .

We can assume that studies with smaller $ϵ$ will deliver a more precise estimate of the true effect size. When pooling the results of different studies, meta-analyses give studies with greater precision (less sampling error) a higher weight, because they are better estimators of the true effect. The question is how we can know how large the sampling error is [1].

Unsurprisingly, the true effect of a study $θ_{k}$ is unknown, and so $ϵ_{k}$ is also unknown. Often, however, we can approximate the expected sampling error. A common way to represent $ϵ$ is through the standard error (SE). The standard error is defined as the standard deviation of the sampling distribution. A sampling distribution is the distribution of a metric we get when we draw random samples with the same sample size $n$ from our population many times [1].

Within-Group Study

The arithmetic mean is probably the most commonly used central tendency measure, and can easily be pooled in a meta-analysis [1].

x ˉ = \frac{\sum _{i = 1 n} x _{i}}{n} .

The standard error of the mean is calculated as the ratio of the sample standard deviation $s$ and the square root of the sample size $n$ [1].

S E_{x ˉ} = \frac{s}{n} .

Between-Group Study

The between-group mean difference $M D_{between}$ is defined as the raw, unstandardized difference of the means for two independent groups. Between-group mean differences can be calculated when a study contained at least two groups, as is usually the case in controlled trials or other types of experimental studies. In meta-analyses, mean differences can only be used when all the studies measured the outcome of interest on exactly the same scale. The mean difference is defined as the mean of group 1, $x ˉ_{1}$ , minus the mean of group 2, $x ˉ_{2}$ [1].

M D_{between} = x ˉ_{1} - x ˉ_{2} .

The standard error can be obtained using the following formula where $n_{1}$ represents the sample size of group 1, $n_{2}$ the sample size of group 2, and $s_{pooled}$ the pooled standard deviation of both groups [1].

S E_{M D_{between}} = s_{pooled} \frac{1}{n _{1}} + \frac{1}{n _{2}} .

Using the standard deviation of group 1, $s_{1}$ , and group 2, $s_{2}$ , the value of $s_{pooled}$ can be calculated as follows [1].

s_{pooled} = \frac{( n _{1} - 1 ) s _{12} + ( n _{2} - 1 ) s _{22}}{( n _{1} - 1 ) + ( n _{2} - 1 )} .

Standardized Between-Group Study

The standardized between-group mean difference $SM D_{between}$ is defined as the difference in means between two independent groups, standardized by the pooled standard deviation $s_{pooled}$ . $SM D_{between}$ can be compared between studies, even if those studies did not measure the outcome of interest using the same instruments. In the literature, the standardized mean difference is also often called Cohen’s $d$ . In contrast to unstandardized mean differences, $SM D_{between}$ expresses the difference between two groups in units of standard deviations. This can be achieved by dividing the raw mean difference of two groups, $x ˉ_{1}$ and $x ˉ_{2}$ , through the pooled standard deviation $s_{pooled}$ of both groups [1].

SM D_{between} = \frac{x ˉ _{1} - x ˉ _{2}}{s _{pooled}} .

Standardized mean differences are often interpreted using the conventions of Cohen, however, these are rules of thumb at best. It is usually much better to interpret standardized mean differences based on their implications. For example, for many serious diseases even a very small statistical effect can still have a large impact on the population level [1].

$SM D \approx 0.20$ : small effect.
$SM D \approx 0.50$ : moderate effect.
$SM D \approx 0.80$ : large effect.

The standardized error of $SM D_{between}$ can be calculated using this formula where $n_{1}$ and $n_{2}$ are the sample sizes of group 1 and group 2 [1].

S E_{SM D_{between}} = \frac{n _{1} + n _{2}}{n _{1} n _{2}} + \frac{SM D _{between 2}}{2 ( n _{1} + n _{2} )} .

Effect Size Correction

The standardized mean difference has been found to have an upward bias when the sample of a study is small, especially when $n \leq 20$ . This small sample bias means that SMDs systematically overestimate the true effect size when the total sample size of a study is small, which is unfortunately often the case in practice. It is therefore sensible to correct the standardized mean differences of all included studies for small-sample bias, which produces an effect size called Hedges’ $g$ . In this formula, $n$ represents the total sample size of the study [1].

g = SM D \times (1 - \frac{3}{4 n - 9}) .

References

Harrer, Mathias, Cuijpers, Pim, Furukawa Toshi A, Ebert, David D (2021) Doing Meta-Analysis With R: A Hands-On Guide. Chapman & Hall/CRC Press.

Meta-Analysis: Motivation and Limitations

Ethan Davis — Sun, 12 Jul 2026 22:24:46 +0000

Meta-Analysis

Motivation

Science is generally assumed to be a cumulative process, and never in history have we had access to more evidence in the form of published research articles than we do today. In principle, this development should make us enthusiastic about the prospects of science. If science is cumulative, then more published research equals more evidence. Yet, it is not that easy. It is an inconvenient truth that the scientific process, when left to its own devices, will not automatically move us to the best of all possible worlds [1].

Meta-analysis is helpful for critically appraising bodies of evidence in their entirety. That is, as long as we acknowledge its own limitations and biases. A meta-analysis can be described as an analysis of analyses. In conventional studies, the units of analysis are a number people, places, or objects. In meta-analysis, primary studies themselves become the elements of analysis. The aim of meta-analysis is to combine, summarize, and interpret all available evidence related to a clearly defined research field or question. There are at least three distinct ways that evidence from multiple studies can be synthesized [1].

Traditional or narrative reviews are often written by experts and authorities of a research field. There are no strict rules on how studies in a narrative review have to be selected and how to define the scope of the review. There are also no fixed rules on how to draw conclusions from the reviewed evidence. Overall, this can lead to biases favoring the opinion of the author. Nevertheless, when written in a balanced way, narrative reviews can be helpful for readers to get an overall impression of the relevant research questions and available evidence of a field [1].

Systemic reviews try to summarize evidence using clearly defined and transparent rules. Research questions are determined beforehand, and there is an explicit, reproducible methodology through which studies are selected and reviewed. System reviews aim to cover all available evidence. They also asses the validity of evidence using predefined standards and present a synthesis of outcomes in a systemic way [1].

Meta-analyses can be seen as an advanced type of systemic review. However, there is one aspect which makes meta-analyses special. Meta-analyses aim to combine results from previous studies in a quantitative way. The goal of meta-analyses is to integrate quantitative outcomes reported in the selected studies into one numerical estimate. Compared to systemic reviews, meta-analyses often have to be more exclusive concerning the kind of evidence that is summarized. To perform a meta-analysis, it is usually necessary that studies used the same design and type of measurement, and delivered the same intervention [1].

Synthesis Limitations

Certain common pitfalls of meta-analyses can be associated with systemic problems of the scientific process, while others can be traced back to flaws of meta-analyses themselves [1].

The Apples and Oranges problem: Even with the strictest inclusion criteria, studies in a meta-analysis will never be absolutely identical. Meta-analysis means to calculate a numerical estimate which represents the results of all studies. Such an estimate becomes meaningless when studies do not share the properties that matter to a specific research question. Results of a meta-analysis allow us to pool effects and quantify if and how much this effect may vary across different settings. Variation between studies can often be unproblematic, and even insightful if it is correctly incorporated into the aims and problem specification of a meta-analysis [1].
The Garbage In, Garbage Out problem: The quality of evidence produced by a meta-analysis heavily depends on the quality of the studies it summarizes. if the results reported in our included findings are biased, or simply incorrect, the results of the meta-analysis will be equally flawed. This problem can be mitigated to some extent by assessing the quality or risk of bias of the included studies. Though, even the most rigorous meta-analysis will not always be able to balance this out. However, even such disappointing outcomes can be informative, and help guide future research [1].
The File Drawer problem: The file drawer problem refers to the issue that not all relevant research findings are published, and therefore missing in out meta-analysis. Not being able to integrate all evidence in a meta-analysis would be undesirable, but at least tolerable if we could safely assume that research findings are missing at random in the published literature. Unfortunately, they are not, there is good reason to believe that studies with negative or disappointing results are systematically underrepresented in the published literature and that there is a so called publication bias [1].
The Researcher Agenda problem: When defining the scope of a meta-analysis, searching and selecting studies, and ultimately pooling outcomes measures, researchers have to make a myriad of choices. Experts are deeply invested in the research area they are examining. As such, they may hold strong opinions about certain topics, and may intentionally or unintentionally influence the results in the direction that fits their beliefs. One way to reduce the researcher agenda problem is pre-registration of a detailed analysis plan before beginning with data collection [1].

References

Harrer, Mathias, Cuijpers, Pim, Furukawa Toshi A, Ebert, David D (2021) Doing Meta-Analysis With R: A Hands-On Guide. Chapman & Hall/CRC Press.

Statistical Analysis

Ethan Davis — Sun, 12 Jul 2026 17:54:44 +0000

Statistical Analysis

Internal Validity

Train and test splits, cross-validation, and the like seek to estimate the expected generalization performance of a learning procedure. Keeping test data rigorously independent from algorithm development minimizes the bias of this estimation. However, there are multiple sources of arbitrary variations in these estimates. A systematic study of machine learning benchmarks shows that their most important sources of variance are as follows [1].

Choice of test split: A given test set is an arbitrary sample of the actual population that we are trying to generalize to. As a result, the corresponding measure of performance is an imperfect estimate of the actual expected performance. Using multiple splits, and thus multiple test sets, improves the estimation, though it makes computing confidence intervals hard [1].
Hyperparameter optimization: The choice of hyperparameters is imperfect, for example, because of limited resources to tune there hyperparameters. Another attempt to tune hyperparameters would lead to a slightly different choice. Thus, benchmarks do not give an absolute characterization of a learning procedure but are muddied by imperfect hyperparameters [1].
Random seeds: Random choices in a learning procedure, such as initial weights, random dropout for neural networks, or bootstraps in bagging, lead to uncontrolled fluctuations in benchmarking results that do not characterize the ability of the algorithm to generalize to new data [1].

With all these sources of arbitrary variance, the question is whether given benchmarks of a learning procedure performance, or improvement, is likely to generalize reliably to new data or rather to be due to benchmarking fluctuations. To answer this question, we must account for estimation error in the expected generalization performance from the different sources of uncontrolled variance in the benchmarks. A simple statistical testing procedure can be described as follows [1].

Training and testing a prediction pipeline multiple times is needed to estimate the variability of the performance measure. The simplest solution is to do this several times while varying the arbitrary factors, such as split between the train and test sets or random initialization. The resulting set of performance measures is similar to bootstrap samples and can be used to draw conclusions on the distribution of performances in a test set. Confidence intervals can be computed using percentiles of this distribution. Two learning procedures can be compared by counting the number of times that one outperforms the other. Outperforming 75% of the times is typically considered as a reliable improvement [1].

Note these procedures do not perform classic null hypothesis significance testing, which is difficult here. In particular, the standard error across the various runs should not be used instead of the standard deviation. The standard error is the standard deviation divided by the number of runs. The number of runs can be made arbitrarily large given enough compute power, thus making the standard error arbitrarily small. But in no way does the uncertainty due to the limited test data vanish [1].

Furthermore, in repeated splits or cross-validation, the runs are not independent. Specifically, it is invalid to use a standard hypothesis test, such as a $t -test$ , across the different folds of a cross-validation. Another reason not to rely on null hypothesis testing is that their statistical significance only asserts that the expected performance, or improvement, is non-zero over a test population of infinite size. From a practical perspective, we care about meaningful improvements on tests sets of finite size, which is related to the notion of acceptance tests, as opposed to significance, in the Neyman-Pearson framework of statistical testing [1].

External Validity

The procedures described above characterize the expected error of a learning procedure applied on a given population. A related but different question is that of characterizing the error of a given predictive model on a study population. This is related to the notion of external validity, and is important for two reasons. First, it characterizes the specific predictive model that will be used in practice. Second, characterizing the model on the target population may be important, as it may differ markedly from the study population [1].

Indeed, the techniques in the previous description rely on splitting the initial dataset into training and testing (or validation) sets. Hence, these different sets are by construction drawn from the same population and have similar characteristics. They only demonstrate the ability of the model to generalize to new but similar data. For example, to better assess model utility, guidelines on evaluating clinical prediction models insist on external validation using data collected later in time, or in a different geographical area [1].

Testing whether a prediction model can generalize to dissimilar data is important as it is all too frequent that the study sample, on which the model was developed, does not represent the target population. Participants from on study may not be representative of the target population. This can be due to inclusion/exclusion criteria or due to uncontrolled biases. To assess generalization ability, a common practice is to use one or several additional datasets for testing, these datasets being acquired using different protocols and at different sites [1].

External validation of a predictive model relies on an independent test set and not cross-validation. Statistical testing thus amounts to deriving confidence intervals or null hypothesis significance testing for the metric of interest on this test set. A general and good option, applicable to all situations, is to approximate the sampling distribution of the metric of interest by bootstrapping the test set. When comparing two classifiers, a McNemar’s test can be useful to test whether the observed difference in errors can be explained solely by sampling noise [1].

References

Varoquaux, Gael, Colliot, Olivier (2023) Evaluating Machine Learning Models and Their Diagnostic Value. Springer US.

Cross-Validation: Training, Validation, and Test Splits

Ethan Davis — Sun, 12 Jul 2026 17:42:41 +0000

Cross-Validation

Training Phase

Performance should not be evaluated using the same data that was used for training. Therefore, the first step is to split the data into a training set and a testing set. This should be done before starting work on the data, whether it is training an ML model or even doing simple statistics for identifying interesting features. Note that we want the output variable distribution to be approximately the same in the training and testing sets. This is called stratification [1].

The split between train and test sets is arbitrary. With the same machine learning algorithm, two different data splits will lead to two different observed performances, both of which are noisy estimates of the expected generalization performance of prediction models built with this learning procedure. A common strategy to obtain better estimates consists of performing multiple splits of the whole dataset into training and testing sets, a so-called cross-validation loop [1].

For each split of cross-validation, a model is trained using the training set, and the performances are computed using the testing set. The performances over all the testing sets are then aggregated. $k -fold$ cross-validation consists of splitting the data into $k$ sets called folds of approximately equal size. It ensures that each sample in the dataset is used exactly once for testing. Splitting out 10-20% for the testing set is a good trade-off, which amounts to $k = 5$ or 10 [1].

Often an additional validation set is useful to make choices on the model to maximize prediction performance, such as, make changes on the architecture, tune hyperparameters, or perform early stopping. As the test set performance is our best estimate of prediction performance, we could double the test set as a validation set. However, in such a situation, the performances reported on the testing set will have an optimistic bias, since a data dependent choice has been made on this test set. There are two main solutions to this issue [1].

The first one is usually applied when the model training is fast and the dataset is of small size. It is called nested cross-validation. It consists of running two loops of cross-validation, one nested into the other. The inner loop serves for hyperparameter tuning or model selection, while the outer loop is used to evaluate the performance. Nested cross-validation should be preferred for very small datasets as it gives better testing power [1].

The second solution is to separate from the whole dataset the test set, which will only be used to evaluate the performances. Then, the remainder of the dataset can be further split into training data and data used to make modeling choices, called the validation set. The training and validation sets will be used in a cross-validation manner, and to experiment with different models and tune parameters. It is absolutely crucial that the test set is isolated at the very beginning, before any experiment is done It should be left untouched and used only at the end of the study to report the performances [1].

Data leakage denotes cases where some information from the training set has leaked into the test set. As a consequence, the estimation of the performances is likely to be optimistic. A first basic cause of data leakage is to use the whole dataset for performing various operations on the data. A very common example is to perform feature selection using the whole dataset and then to use the selected feature for model training. A similar situation is when dimensionality reduction is performed on the whole dataset [1].

References

Varoquaux, Gael, Colliot, Olivier (2023) Evaluating Machine Learning Models and Their Diagnostic Value. Springer US.

Bayesian Neural Networks

Ethan Davis — Sun, 12 Jul 2026 04:09:13 +0000

Bayesian Neural Network

Deep neural networks (DNNs) are usually trained using a regularized maximum likelihood objective to find a single setting of parameters. However, large flexible models like neural networks can represent many functions, corresponding to different parameter settings, which fit the training data well, yet generalize in different ways. This phenomenon is known as underspecification [1].

Considering all of these different models together can lead to improved accuracy and uncertainty representation. This can be done by computing the posterior predictive distribution using Bayesian model averaging. The main challenges in applying Bayesian inference to DNNs are specifying suitable priors, and efficiently computing the posterior, which is challenging due to the large number of parameters and the large datasets [1].

p (y ∣ x, D) = \int p (y ∣ x, θ) p (θ ∣ D) d θ where p (θ ∣ D) \propto p (θ) p (D ∣ θ) .

Consider the generalized MLP with $L - 1$ hidden layers and a linear output as described in the companion MLP post [1].

f (x; θ) = W_{L} (\dots φ (W_{1} x + b_{1})) + b_{L} .

The most common choice of priors is to use a factored Gaussian prior [1].

W_{ℓ} \sim N (0, α_{ℓ 2} I), b_{ℓ} \sim N (0, β_{ℓ 2} I) .

Initializing this model’s parameters at a particular random value is like sampling a point from this prior over functions. In the limit of infinitely wide neural networks, a neural network defines a Gaussian process (see the companion Gaussian process series) with a fixed kernel. Indeed, an MLP with one hidden layer, whose width goes to infinity, and which has a Gaussian prior on all the parameters, converges to a Gaussian process with a well-defined kernel. These kernels can be used in modeling non-stationary covariance structure [1].

Monte Carlo dropout (MCD) is a very simple and widely used method for approximating the Bayesian predictive distribution. Usually stochastic dropout layers are added as a form of regularization during training, and are turned off at test time. However, the idea of MCD is to also perform random sampling during testing [1].

More precisely, we drop out each hidden unit according to a $Bernoulli (p)$ distribution, and repeat this procedure $K$ times to create $K$ distinct models. We then create an equally weighted average of the predictive distribution for each of these models as shown below. One drawback of MCD is that it is slow at test time. However, this can be overcome by distilling the model’s predictions into a deterministic student network [1].

p (y ∣ x, D) \approx \frac{1}{K} k = 1 \sum K p (y ∣ x, θ^{k}) .

Another very simply approximation to the posterior is to only be Bayesian about the weights in the final layer. This is called the “Bayesian last layer” approximation. In more detail, let $z = f (x; θ)$ be the predicted outputs of the model before any optional final nonlinearity. We assume this has the form $z = w_{L ⊤} ϕ (x; θ)$ , where $ϕ (x)$ are the features extracted by the first $L - 1$ layers. Then, we can use standard techniques, such as Markov chain Monte Carlo (described in the companion MCMC series), to compute $p (w_{L} ∣ D) = N (μ_{L}, Σ_{L})$ , given $ϕ (\cdot)$ [1].

MCMC methods like Hamiltonian Monte Carlo (HMC) are generally considered to be the gold standard for posterior approximation, since they do not make strong assumptions about the form of the posterior. However, a significant limitation of standard MCMC procedures, including HMC, is that they require access to the full training set at each step. Stochastic gradient MCMC methods, such as Stochastic gradient Langevin dynamics (SGLD), operate instead using mini-batches of data, offering a scalable alternative [1].

Many conventional approximate inference methods, such as variational inference, focus on approximating the posterior $p (θ ∣ D)$ in a local neighborhood around one of the posterior modes. While this is often not a major limitation in classical machine learning, modern deep neural networks have highly multi-modal posteriors, with parameters in different modes giving rise to very different functions [1].

On the other hand, the functions in a neighborhood of a single mode may make fairly similar predictions. So using a local approximation to compute the posterior predictive will underestimate uncertainty and generalize more poorly. A simple alternative method is to train multiple models, and then to approximate the posterior using an equally weighted mixture of delta functions as follows where $M$ is the number of models. This approach is called deep ensembles [1].

p (θ ∣ D) \approx \frac{1}{M} m = 1 \sum M δ (θ - θ^_{m}) .

Once we have an approximation $q (θ ∣ D)$ of the parameter posterior $p (θ ∣ D)$ , we can use it to approximate the posterior predictive distribution [1].

p (y ∣ x, D) = \int p (y ∣ x, θ) p (θ ∣ D) d θ .

We often approximate this integral using Monte Carlo where $θ^{s} \sim q (θ ∣ D)$ [1].

p (y ∣ x, D) \approx \frac{1}{S} s = 1 \sum S p (y ∣ x, θ^{s}) .

References

Kevin P. Murphy (2023) Probabilistic Machine Learning: Advanced Topics. MIT Press.

Convolutional Neural Networks

Ethan Davis — Sun, 12 Jul 2026 04:05:52 +0000

Convolutional Neural Network

The convolutional neural network (CNN) is used to apply neural networks for images. In this case, we replace matrix multiplication with a convolution operation. The basic idea is to divide the input into overlapping 2D image patches, and to compare each patch with a set of small weight matrices, or filters, which represents part of an object [1].

This can be thought of as a form of template matching. We learn these templates from data. Since the templates are small (often just $3 \times 3$ or $5 \times 5$ ), the number of parameters needed for learning is small. Additionally, since we use convolution to do template matching, instead of matrix multiplication, the model will be translationally invariant. This is useful for tasks such as image classification, where the goal is to classify if an object is present regardless of its location [1].

The 1D convolution between two function $f, g : R^{D} \to R$ is defined as follows [1].

[f ⊛ g] (z) = \int_{R^{D}} f (u) g (z - u) d u .

Now suppose we replace the functions with finite-length vectors, which we can think of as functions defined on a finite set of points. Let $f$ be evaluated at the points ${- L, - L + 1, \dots, 0, 1, \dots, L}$ to yield the weight vector (also called a filter or kernel) $w_{- L} = f (- L)$ up to $w_{L} = f (L)$ . Now let $g$ be evaluated at the points ${- N, \dots, N}$ to yield the feature vector $x_{- N} = g (- N)$ up to $x_{N} = g (N)$ . Then the equation becomes the following. We see that we flip the weight vector $w$ , and then drag it over the the $x$ vector, summing up the local windows at each point [1].

[w ⊛ x] (i) = w_{- L} x_{i + L} + \dots + w_{- 1} x_{i + 1} + w_{0} x_{i} + w_{1} x_{i - 1} + \dots + w_{L} x_{i - L} .

When we do not flip $w$ first, then we get the cross correlation [1].

[w ⊛ x] (i) = w_{- L} x_{i - L} + \dots + w_{- 1} x_{i - 1} + w_{0} x_{i} + w_{1} x_{i + 1} + \dots + w_{L} x_{i + L} .

If the weight vector is symmetric, then cross correlation and convolution are the same. In the deep learning literature, the term “convolution” is often used to mean cross correlation. We can also evaluate the weights $w$ on the domain ${0, 1, \dots, L - 1}$ and the features $x$ on the domain ${0, 1, \dots, N - 1}$ , to eliminate negative indices [1].

[w ⊛ x] (i) = u = 0 \sum L - 1 w_{u} x_{i + u} .

In 2D, the equation becomes the following where the 2D filter $W$ has size $H \times W$ [1].

[W ⊛ X] (i, j) = u = 0 \sum H - 1 v = 0 \sum W - 1 w_{u, v} x_{i + u, j + v} .

We can think of 2D convolution as template matching, since the output at a point $(i, j)$ will be large if the corresponding image centered on $(i, j)$ is similar to $W$ . More generally, we can think of convolution as a form of feature detection. The resulting output $Y = W ⊛ X$ is therefore called a feature map [1].

In the figure we see that convolving a $3 \times 3$ image with a $2 \times 2$ filter results in a $2 \times 2$ output. This is called valid convolution, since we only apply the filter to valid parts of the input, and we do not let it slide off the ends. If we want the output to have the same size as the input, we can use zero-padding, where we add a border of zeros to the image. This is called same convolution [1].

Since each output pixel is generated by a weighted combination of inputs in its receptive field, neighboring outputs will be very similar in value, since their inputs are overlapping. We can reduce this redundancy and speedup computation by skipping every $s$ number of inputs. This is called strided convolution [1].

Convolution will preserve information about the location of input features, a property known as equivariance. In some cases we want to be invariant to the location. For example, when performing image classification, we may just want to know if an object of interest is present anywhere in the image. A simple way to achieve this is called max pooling, which computes the maximum over its incoming values. An alternative is to use average pooling, which replaces the maximum with the mean. A common design pattern is to create a CNN by alternating convolutional layers with max pooling layers, followed by a final linear classification layer at the end [1].

References

Kevin P. Murphy (2022) Probabilistic Machine Learning: An Introduction. MIT Press.
Dumoulin, Vincent, Visin, Francesco (2016) A guide to convolution arithmetic for deep learning. ArXiv e-prints.
Aphex34 (2015) Max pooling.

DEV Community: Ethan Davis

Multilevel (Three-Level) Meta-Analysis

Multilevel Meta-Analysis

Subgroup Analyses in Three-Level Models

Robust Variance Estimation

Cluster Wild Bootstrapping

References

Meta-Analysis: Meta-Regression

Meta-Regression

Meta-Regression with a Categorical Predictor

Meta-Regression with a Continuous Predictor

References

Meta-Analysis: Subgroup Analysis

Subgroup Analysis

Pooling the Effect in Subgroups

Comparing the Subgroup Effects

References

Meta-Analysis: Forest Plots

Forest Plots

References

Meta-Analysis: Between-Study Heterogeneity

Between-Study Heterogeneity

Cochran’s Q

Higgins & Thompson’s I2 Statistic

The H2 Statistic

Heterogeneity Variance τ2 & Standard Deviation τ

Assessing Heterogeneity

References

Meta-Analysis: Pooling Effect Sizes

Pooling Effect Sizes

Fixed-Effect Model

Random-Effects Model

References

Meta-Analysis: Effect Sizes

Effect Sizes

Within-Group Study

Between-Group Study

Standardized Between-Group Study

Effect Size Correction

References

Meta-Analysis: Motivation and Limitations

Meta-Analysis

Motivation

Synthesis Limitations

References

Statistical Analysis

Statistical Analysis

Internal Validity

External Validity

References

Cross-Validation: Training, Validation, and Test Splits

Cross-Validation

Training Phase

References

Bayesian Neural Networks

Bayesian Neural Network

References

Convolutional Neural Networks

Convolutional Neural Network

References

Cochran’s $Q$

Higgins & Thompson’s $I^{2}$ Statistic

The $H^{2}$ Statistic

Heterogeneity Variance $τ^{2}$ & Standard Deviation $τ$