SR Phase 7: Data Synthesis and Meta-Analysis

Your Role

Guide the user through the decision to pool, select appropriate models and effect measures, execute meta-analysis in R via bash, interpret heterogeneity and publication bias, and produce manuscript-ready results text.

Prerequisites

Extraction data (from Phase 5 CSV)
RoB judgments (from Phase 6)
Protocol analysis plan (from Phase 2)
R installed: check with R --version. If not installed, guide user through installation.

Workflow

Step 1: Decide Whether to Pool

ASK the user about the first outcome to analyze. CHECK these criteria:

Pool if ALL are true:

≥2 studies with compatible PICO
Outcomes measured at comparable time points (±20% of each other)
Sufficiently similar study designs and populations
Outcome definitions are compatible

Do NOT pool if ANY:

Studies differ substantially in population severity (e.g., mixing ISS >15 with ISS >25)
Outcomes defined differently (e.g., mortality at 28d vs 60d vs in-hospital)
Only 1 study available → report descriptively
Excessive clinical heterogeneity despite statistical homogeneity → narrative synthesis

DOCUMENT the decision: "We decided to [pool / not pool] because [reason]."

Step 2: Select Effect Measure and Model

| Outcome Type | Effect Measure | Notes | |-------------|----------------|-------| | Dichotomous (mortality, intubation, adverse events) | Risk Ratio (RR) | Preferred for EM/CC. OR overestimates when outcome is common (>10%). | | Continuous, same scale (LOS, ventilator days) | Mean Difference (MD) | Preserves original units. | | Continuous, different scales (pain scores, quality of life) | Standardized Mean Difference (SMD) | Hedges' g (corrects for small sample bias). | | Time-to-event (survival, time to extubation) | Hazard Ratio (HR) | Extract from Cox model or KM curves. | | Diagnostic accuracy | Sensitivity + Specificity separately; DOR if pooling | Use bivariate model or HSROC. |

Model Selection:

Default: Random-effects (REML) — Expect heterogeneity across EM settings and protocols.
Fixed-effect: Only if near-identical populations/interventions/outcomes (rare in EM).
DerSimonian-Laird: If REML fails to converge (<10 studies).
Peto OR: Only for very rare events (<1%) when OR is appropriate.

Step 3: Run Meta-Analysis in R

GENERATE and RUN the R script via bash.

First, create the data file from extraction data:

cat > /tmp/ma_data.csv << 'EOF'
study,event_t,n_t,event_c,n_c
Smith_2021,45,120,30,125
Jones_2020,88,200,65,210
Lee_2019,23,80,12,85
EOF

Then GENERATE and RUN the R script:

# scripts/meta_analysis.R
# Usage: Rscript meta_analysis.R --data /tmp/ma_data.csv --outcome "28-day mortality"

library(meta)
library(metafor)

args <- commandArgs(trailingOnly = TRUE)
data_file <- args[grep("--data", args) + 1]
outcome_name <- args[grep("--outcome", args) + 1]

dat <- read.csv(data_file)

# Run meta-analysis
ma <- metabin(
  event.e = event_t, n.e = n_t,
  event.c = event_c, n.c = n_c,
  studlab = study,
  data = dat,
  method = "REML",
  sm = "RR",
  hakn = TRUE,
  prediction = TRUE,
  overall = TRUE,
  random = TRUE,
  fixed = TRUE
)

# Print results
cat("\n=== META-ANALYSIS RESULTS ===\n")
cat("Outcome:", outcome_name, "\n\n")
cat("Number of studies:", length(ma$studlab), "\n")
cat("Total participants:", sum(ma$n.e) + sum(ma$n.c), "\n\n")

cat("--- Random-Effects Model ---\n")
cat("Pooled RR:", exp(ma$TE.random), "\n")
cat("95% CI:", exp(ma$lower.random), "-", exp(ma$upper.random), "\n")
cat("95% Prediction Interval:", exp(ma$lower.predict), "-", exp(ma$upper.predict), "\n")
cat("z =", round(ma$zval.random, 3), ", p =", format.pval(ma$pval.random, digits=3), "\n\n")

cat("--- Heterogeneity ---\n")
cat("I² =", round(ma$I2, 1), "%\n")
cat("τ² =", round(ma$tau2, 4), "\n")
cat("Q =", round(ma$Q, 2), "(df =", ma$df.Q, ", p =", format.pval(ma$pval.Q, digits=3), ")\n\n")

cat("--- Fixed-Effect Model ---\n")
cat("Pooled RR:", exp(ma$TE.fixed), "\n")
cat("95% CI:", exp(ma$lower.fixed), "-", exp(ma$upper.fixed), "\n\n")

cat("--- Individual Study Results ---\n")
for (i in 1:nrow(dat)) {
  cat(sprintf("%s: RR %.2f (%.2f-%.2f), Weight %.1f%%\n",
    ma$studlab[i], exp(ma$TE[i]), exp(ma$lower[i]), exp(ma$upper[i]), ma$w.random[i]))
}

# Generate forest plot
png(paste0("forest_plot_", gsub(" ", "_", outcome_name), ".png"),
    width = 800, height = 400 + 30 * nrow(dat))
forest(ma, leftlabs = c("Study", "Events", "Total", "Events", "Total"),
       rightlabs = c("RR", "95% CI", "Weight"),
       smlab = outcome_name)
dev.off()
cat("\nForest plot saved.\n")

# Funnel plot if >=10 studies
if (length(ma$studlab) >= 10) {
  png(paste0("funnel_plot_", gsub(" ", "_", outcome_name), ".png"))
  funnel(ma, studlab = TRUE)
  dev.off()
  cat("Funnel plot saved.\n")

  # Egger's test
  library(metafor)
  rma_obj <- rma(yi = ma$TE, sei = ma$seTE, method = "REML")
  egger <- regtest(rma_obj)
  cat("\n--- Publication Bias ---\n")
  cat("Egger's test:", "t =", round(egger$zval, 3), ", p =", format.pval(egger$pval, digits=3), "\n")
  if (egger$pval < 0.10) {
    cat("Asymmetry detected. Trim-and-fill recommended.\n")
    tf <- trimfill(rma_obj)
    cat("Adjusted estimate after trim-and-fill:\n")
    cat("RR:", exp(tf$beta), ", 95% CI:", exp(tf$ci.lb), "-", exp(tf$ci.ub), "\n")
  } else {
    cat("No significant asymmetry detected.\n")
  }
}

cat("\n=== DONE ===\n")

WRITE this as scripts/meta_analysis.R and RUN it:

Rscript scripts/meta_analysis.R --data /tmp/ma_data.csv --outcome "28-day mortality"

INTERPRET the output for the user.

Step 4: Heterogeneity Interpretation

Use this template to interpret I²:

Heterogeneity: I² = [X]% (95% CI: [low]-[high])
└── <25%: "Low heterogeneity — studies show consistent effects"
└── 25-50%: "Moderate heterogeneity — explore with pre-specified subgroup analyses"
└── 50-75%: "Substantial heterogeneity — pooled estimate should be interpreted cautiously" 
              + identify potential sources
└── >75%: "Considerable heterogeneity — meta-analysis may be misleading"
         + recommend exploring clinical/methodological differences
         + consider narrative synthesis instead

GENERATE narrative text: "There was [low/moderate/substantial/considerable] heterogeneity among studies (I² = X%, τ² = Y, Q p = Z). The 95% prediction interval ranged from [lower] to [upper], suggesting [clinical interpretation]."

Step 5: Prediction Interval Interpretation

For ≥3 studies, generate: "The 95% prediction interval ranges from [lower] to [upper]. This means that in a new clinical setting similar to the included studies, the true effect could range from [clinical interpretation of lower bound] to [clinical interpretation of upper bound]."

Step 6: Subgroup Analysis (Pre-Specified Only)

If the user has pre-specified subgroups, GENERATE and RUN:

# Add subgroup variable to data
dat$subgroup <- c("ED", "ICU", "ED", "ICU", "Pre-hospital")  # example

# Run subgroup meta-analysis
ma_sub <- update(ma, byvar = subgroup, tau.common = FALSE)
print(ma_sub)
cat("Test for subgroup differences: Q =", ma_sub$Q.b.random, ", p =", ma_sub$pval.Q.b.random, "\n")

INTERPRET the interaction test (p for interaction), NOT the within-group p-values:

p-interaction <0.10: "Effect differs by subgroup. Report separate estimates."
p-interaction >0.10: "No evidence of differential effect by subgroup."

Step 7: Sensitivity Analysis

Run these and report:

# 1. Exclude high RoB studies
# 2. Use fixed-effect model
# 3. Exclude outlier studies (leave-one-out)
# 4. Change effect measure (RR → OR)

For leave-one-out:

cat("\n--- Leave-One-Out Analysis ---\n")
for (i in 1:nrow(dat)) {
  ma_loo <- update(ma, subset = -i)
  cat(sprintf("Without %s: RR %.2f (%.2f-%.2f), I² = %.1f%%\n",
    ma$studlab[i], exp(ma_loo$TE.random), exp(ma_loo$lower.random),
    exp(ma_loo$upper.random), ma_loo$I2))
}

INTERPRET: "The estimate was [robust to / sensitive to] exclusion of [X]."

Step 8: Publication Bias Assessment

If ≥10 studies:

Funnel plot: visually assess asymmetry
Egger's test: p<0.10 suggests asymmetry
Trim-and-fill: adjusted estimate if asymmetry detected

GENERATE: "Egger's test for funnel plot asymmetry was [significant/not significant] (p=[value]). The funnel plot appeared [symmetric/asymmetric]. Trim-and-fill adjusted estimate [changed/did not change] the conclusion, suggesting [low/possible] risk of publication bias."

If <10 studies: "Publication bias could not be formally assessed (<10 studies available). This is a limitation."

Step 9: Narrative Synthesis (if meta-analysis is not possible)

If pooling is not appropriate, use structured narrative synthesis:

Group studies by similarity (population, intervention, outcome)
For each group: describe direction of effect, consistency, size of effect
Use vote-counting ONLY as a last resort (weak and misleading)
Consider effect direction plots or albatross plots

GENERATE: "Meta-analysis was not possible due to [reason]. We performed a narrative synthesis of [N] studies, which showed [summary of findings]."

Scripts

`scripts/meta_analysis.R`

Complete meta-analysis pipeline: random/fixed effects, forest plot, funnel plot, Egger's, trim-and-fill, leave-one-out, subgroups.

`scripts/heterogeneity_interpreter.R`

Takes I², τ², Q stats and generates interpretation narrative text.

Outputs to Generate

sr-meta-analysis-results.md — Full results with narrative interpretation
sr-forest-plot-summary.md — Forest plot data in manuscript format
sr-subgroup-results.md — Subgroup analysis results with interaction tests
sr-sensitivity-results.md — Sensitivity analysis comparison table
sr-publication-bias.md — Publication bias assessment report
forest_plot_*.png — Generated forest plot image
funnel_plot_*.png — Generated funnel plot image (if ≥10 studies)

Guardrails

Do NOT pool incompatible outcomes — Different time points, different scales, different definitions. Must be clinically justified.
Do NOT use fixed-effect as default — Random-effects is the default for EM/CC reviews. Fixed-effect only with strong justification.
Prediction interval MANDATORY for ≥3 studies — "More clinically useful than I² alone."
Do NOT over-interpret subgroup analyses — "Subgroup analyses are observational even within RCTs. Pre-specified only. p-interaction >0.10 = no evidence of difference."
Do NOT rely on I² alone — "I² can be misleading with few studies. Report τ² and 95% CI for I²."
Funnel plot requires ≥10 studies — Underpowered with fewer.
Show ALL sensitivity analyses — Even if results are unchanged. "None of these analyses changed the conclusion" is a valid finding.

Edge Cases

| Situation | Response | |-----------|----------| | Zero events in both arms | Use Mantel-Haenszel method with continuity correction (0.5). If all studies have zero events, report event rates only. | | Only 1 study | "Insufficient data for meta-analysis. Report descriptive summary." | | <3 studies | Can still pool, but prediction interval will be very wide. Interpret cautiously. | | Extreme heterogeneity (I² >90%) | Do NOT report pooled estimate as primary. Investigate sources: outlier studies, measurement differences, population differences. Consider narrative synthesis. | | Studies report different continuous outcomes for same construct | Use SMD (Hedges' g) to combine different scales. Interpret SMD in clinically meaningful units (0.2=small, 0.5=moderate, 0.8=large). | | Sparse data (total N < event count) | Rare in EM. Use Peto OR or exact methods. | | Intention-to-treat vs per-protocol | Pool ITT data as primary. Sensitivity analysis using per-protocol. |

GRADE Interface

For each outcome, produce a GRADE-ready summary:

Outcome: [name]
Studies: [N] ([total N] participants)
Effect estimate: RR [X.XX] (95% CI: [lower]-[upper])
Heterogeneity: I² = [X]%
RoB: [N] studies at high risk
Imprecision: [adequate/inadequate] OIS
Indirectness: [direct/indirect]
Publication bias: [not detected/suspected/unassessed]
GRADE starting level: [High (RCT) / Low (observational)]

This feeds directly into sr-grade Phase 8.

Handoff

When synthesis is complete:

Pass: pooled estimates + forest plot + funnel plot + GRADE-ready summaries
Summarize: "Synthesis complete. [X] outcomes pooled with [Y] studies each. [Narrative of main findings]. Ready for Phase 8: GRADE or Phase 9: Writing."
Reference: all numerical results + figures + interpretation text for manuscript