Narrative Synthesis (Cochrane Handbook Ch 12)

Your Role

Guide the user through alternative synthesis methods when meta-analysis of effect estimates is not possible. Apply the three acceptable methods from Ch 12 and avoid unacceptable vote counting.

---

Prerequisites

• Extraction data from Phase 5
• Confirmation that meta-analysis cannot be performed (use Ch 12.1 decision table)

---

Workflow

Step 1: Confirm Meta-Analysis Is Not Possible

USE Ch 12.1 Table 12.1.a decision table:

| Scenario | Action | |----------|--------| | Only 1 study | Report descriptively | | Incompletely reported outcomes/effects | Try to calculate effect + variance (Ch 6). If impossible → alternative method | | Different effect measures | Try to convert (Ch 6, Ch 10.6). If impossible → alternative method | | Bias concerns | Do not meta-analyze. Use structured reporting | | Clinical/methodological diversity | Consider subgroup MA or random-effects with prediction interval. Only reject MA if truly incomparable | | Statistical heterogeneity | Explore with subgroup/meta-regression. Only reject if I² > 75% and effects in both directions |

DOCUMENT the decision: "Meta-analysis was not possible because [reason from table]. We used [chosen alternative method]."

Step 2: Choose Method by Data Availability

USE Ch 12.2 Table 12.2.a decision tree:

What data do you have for each study?
├── Effect estimate + variance (SE/CI)
│   └── Can you meta-analyze? → Yes: do MA (Ch 10). No: use Method A (summarize effects)
├── P value + direction + sample size
│   └── Method B: Combine P values (Fisher's method)
├── Direction of effect only (no magnitude, no P)
│   └── Method C: Vote counting based on direction of effect
├── Nothing at all
│   └── Structured tabulation only (no formal synthesis possible)

Method A: Summarizing Effect Estimates

Use when effect estimates are available but variances are missing or incorrect.

CALCULATE:

• Median effect, interquartile range, range of observed effects across studies
• Box-and-whisker or bubble plot for visual display

REPORT: "Across X studies, the median effect was [median] (IQR: [Q1]-[Q3], range: [min]-[max])."

LIMITATION: Does not account for study size.

Method B: Combining P Values (Fisher's Method)

Use when only P values + direction of effect are available.

CONVERT to one-sided P values:

If effect consistent with hypothesis: p_one = p_two / 2
If effect opposite to hypothesis: p_one = 1 - p_two/2

CALCULATE Fisher's statistic:

χ² = -2 × Σ ln(p_one_i)
df = 2 × k studies
p = P(χ² > computed value | df)

INTERPRET:

• p < 0.05: "Evidence of an effect in at least one study"
• p ≥ 0.05: "No evidence of an effect across studies (but does not rule out effects in individual studies)"

RUN: python3 scripts/fisher_p_combine.py --p-values 0.04,0.12,0.56 --directions +,+,-

LIMITATION: Provides NO information on magnitude. Can reject null on basis of single study.

Method C: Vote Counting Based on Direction of Effect

Use when only direction is reported.

CATEGORIZE each effect as benefit or harm (direction only, IGNORING statistical significance).

CALCULATE:

• u = number of studies favoring intervention
• n = total studies
• p̂ = u/n (proportion favoring intervention)
• 95% CI for p̂ (Wilson or Jeffreys method)
• Sign test: p = 2 × P(X ≥ u | Bin(n, 0.5))

REPORT: "X of Y studies favored the intervention (p̂ = X/Y, 95% CI: [lower]-[upper])."

GENERATE harvest or effect direction plot.

LIMITATION: Provides no magnitude information. Does not weight by study size.

Step 3: Generate Visual Displays

RUN scripts for appropriate plots:

Albatross Plot (Method B)

Rscript scripts/albatross_plot.R --data p_values.csv --output albatross.png

Scatter of sample size vs P value, separated by direction, with effect size contours.

Harvest Plot (Method C)

Rscript scripts/harvest_plot.R --data directions.csv --output harvest.png

Bars grouped by direction, height weighted by sample size, annotated with RoB.

Structured Tabulation

GENERATE table: studies ordered by risk of bias, with columns: study, design, N, effect (direction/estimate), direction, notes.

Step 4: Document Methods and Limitations

Generate manuscript text: "We used [Method] for synthesis because [reason from Step 1]. This method provides [what it provides] but does not provide [limitations]. Therefore, [cautious interpretation]."

---

Scripts

scripts/fisher_p_combine.py

#!/usr/bin/env python3
import argparse, math, statistics

def combine_pvalues(p_one_sided, directions):
    if len(p_one_sided) != len(directions):
        raise ValueError("p-values and directions must match")
    # Convert two-sided to one-sided if needed
    p_one = []
    for p, d in zip(p_one_sided, directions):
        if d == '+':
            p_one.append(p / 2 if p <= 1 else p)
        elif d == '-':
            p_one.append(1 - p/2 if p <= 1 else p)
        else:
            p_one.append(p)
    chi2 = -2 * sum(math.log(max(p, 1e-300)) for p in p_one)
    df = 2 * len(p_one)
    from scipy.stats import chi2 as chi2_dist
    p_val = chi2_dist.sf(chi2, df)
    return chi2, df, p_val

Usage: python3 fisher_p_combine.py --p-values 0.04,0.12,0.56 --directions +,+,-

scripts/albatross_plot.R

Generates albatross plot (sample size vs P value with effect size contours). Uses only P value + direction + sample size per study.

scripts/harvest_plot.R

Generates harvest plot: bars grouped by effect direction, height weighted by sample size, annotated with study quality.

scripts/summarize_effects.py

Median, IQR, range of effect estimates across studies.

---

Assets

assets/method-selector.md

Decision tree for choosing between the three methods based on available data.

assets/tabulation-template.md

Structured results table template with ordering by risk of bias.

assets/limitations-template.md

Manuscript text templates for documenting limitations of each method.

---

Guardrails

1. "Vote counting based on statistical significance is UNACCEPTABLE — Ch 12.2.2.1: 'can lead to the wrong conclusion.'"
2. "Only one outcome per study per synthesis."
3. "Pre-specify the alternative method in the protocol. Do not choose after seeing the data."
4. "Do NOT privilege narrative synthesis over meta-analysis when meta-analysis is possible."
5. "Albatross plots are exploratory — they approximate effect size, they do not test it."
6. "The phrase 'narrative summary' without specifying the method is insufficient — be explicit about which method was used."

Handoff

→ sr-grade (certainty from synthesized evidence), → sr-writing (results section)