SR Phase 8: GRADE Certainty of Evidence Assessment

Your Role

Apply the GRADE framework to each outcome, generate a Summary of Findings (SoF) table, and produce manuscript-ready certainty-of-evidence text.

Prerequisites

Pooled estimates per outcome (from Phase 7)
RoB judgments (from Phase 6)
Study designs per outcome

Workflow

Step 1: Set Starting Level

| Study Design | Starting Certainty | |-------------|-------------------| | RCTs | High | | Observational (cohort, case-control) | Low | | Case series, case reports | Very Low | | Diagnostic accuracy (cross-sectional) | High (if consecutive, proper reference standard) | | Diagnostic accuracy (case-control) | Low |

Step 2: Apply 5 Downgrade Domains

For EACH domain, assess whether to downgrade:

1. Risk of Bias (-1 or -2)

USE: RoB judgments from Phase 6.

Most studies at low risk → no downgrade
Some concerns in >50% of studies → -1
High risk in >25% of studies → -2
EM/CC note: If blinding was impossible but outcomes are objective, do NOT downgrade.

2. Inconsistency / Heterogeneity (-1 or -2)

USE: I², τ², and prediction interval from Phase 7.

I² <25% AND overlapping CIs → no downgrade
I² 25-50% OR τ² >0.1 → -1 (consider rating down)
I² >50% AND non-overlapping CIs → -1
I² >75% AND prediction interval crosses null → -2

CHECK: "Is there a meaningful explanation for the heterogeneity? If yes (e.g., subgroup effect confirmed), do not downgrade."

3. Indirectness (-1 or -2)

Population matches your PICO → no downgrade
Differences in population, intervention, comparator, or outcome → -1
Multiple or major indirectness → -2

EM/CC common indirectness issues:

ED study finds used to guide ICU practice → possible indirectness
Surrogate outcome used (e.g., lab value instead of mortality) → -1
Short follow-up for long-term outcomes → -1

4. Imprecision (-1 or -2)

Total events ≥300 (dichotomous) OR total N ≥500 (continuous) → adequate OIS
95% CI crosses no effect line → -1
95% CI crosses clinically important benefit AND harm → -2
Total events <300 (dichotomous) → -1

EM/CC thresholds for clinical importance:

Mortality: RR <0.90 or >1.10 (10% relative change)
LOS: MD >1 day
Ventilator-free days: MD >2 days

5. Publication Bias (-1 or -2)

Funnel plot symmetric AND Egger's p>0.10 → no downgrade
Funnel plot asymmetric OR Egger's p<0.10 → -1
Strong suspicion + asymmetry → -2
<10 studies → "Unable to assess formally" → do not downgrade solely for this

Step 3: Apply Upgrade Domains (Observational Studies Only)

Observational studies start at Low. They can be upgraded:

1. Large Effect (+1 or +2)

RR >2 or <0.5 → +1
RR >5 or <0.2 → +2

2. Dose-Response Gradient (+1)

Evidence of dose-response relationship documented → +1

3. All Plausible Confounding Would Reduce Effect (+1)

"In observational studies, all plausible confounders would have reduced the observed effect" → +1

Step 4: Calculate Final Rating

Starting level: [High / Moderate / Low / Very Low]
Downgrades: -[N] for [reasons]
Upgrades: +[N] for [reasons]
Final: [High / Moderate / Low / Very Low]

Step 5: Generate GRADE Evidence Profile

For EACH outcome, produce:

## Outcome: [name]

| Domain | Judgment | Explanation |
|--------|----------|-------------|
| Study design | [RCT/Obs] | |
| Risk of bias | No downgrade / -1 / -2 | [specific reasons] |
| Inconsistency | No downgrade / -1 / -2 | I²=X%, τ²=Y |
| Indirectness | No downgrade / -1 / -2 | [specific reasons] |
| Imprecision | No downgrade / -1 / -2 | OIS met/not met, CI crosses null |
| Publication bias | No downgrade / -1 / -2 | [assessment details] |
| Large effect | +0 / +1 / +2 | RR= [value] |
| Dose-response | +0 / +1 | [evidence if any] |
| Confounding reduces effect | +0 / +1 | [assessment] |
| **FINAL CERTAINTY** | **[High/Moderate/Low/Very Low]** | |

Step 6: Generate SoF Table

Generate in this format:

## Summary of Findings Table

| Outcome | Studies (N) | Participants (N) | Effect Estimate (95% CI) | Absolute Effect per 1000 | Certainty |
|---------|-------------|------------------|--------------------------|-------------------------|-----------|
| [Outcome 1] | X RCTs | N | RR X.XX (X.XX-X.XX) | XX fewer/more | ⨁⨁⨁◯ Moderate |
| [Outcome 2] | Y obs | N | MD X.XX (X.XX-X.XX) | — | ⨁◯◯◯ Very Low |

For absolute effects, CALCULATE:

Dichotomous: Baseline risk × RR → risk difference per 1000
Continuous: MD is absolute effect

GRADE certainty symbols:

⨁⨁⨁⨁ = High
⨁⨁⨁◯ = Moderate
⨁⨁◯◯ = Low
⨁◯◯◯ = Very Low

Step 7: Generate Manuscript-Ready Certainty Text

Template: "The certainty of evidence for [outcome] was [rating], [downgraded/upgraded] due to [domains]. Our confidence in the effect estimate is [description]."

Examples:

High: "We are very confident the true effect lies close to that of the estimate."
Moderate: "We are moderately confident. The true effect is likely close to the estimate, but there is a possibility it is different."
Low: "Our confidence is limited. The true effect may be substantially different."
Very Low: "We have very little confidence. The true effect is likely substantially different."

Scripts

`scripts/grade_engine.py`

#!/usr/bin/env python3
"""GRADE certainty of evidence calculator."""

def grade_certainty(starting_level, downgrades, upgrades):
    """Calculate final GRADE rating.
    
    starting_level: 'high', 'low', 'very_low'
    downgrades: int (0-4)
    upgrades: int (0-3)
    """
    levels = {'high': 4, 'moderate': 3, 'low': 2, 'very_low': 1}
    reverse = {4: 'High', 3: 'Moderate', 2: 'Low', 1: 'Very Low'}
    
    if starting_level not in levels:
        return 'Very Low'
    
    level = levels[starting_level] - downgrades + upgrades
    if level > 4: level = 4
    if level < 1: level = 1
    
    return reverse[level]

def interpret_certainty(rating):
    interpretations = {
        'High': 'We are very confident the true effect lies close to that of the estimate.',
        'Moderate': 'We are moderately confident. The true effect is likely close to the estimate, but there is a possibility it is substantially different.',
        'Low': 'Our confidence is limited. The true effect may be substantially different from the estimate.',
        'Very Low': 'We have very little confidence. The true effect is likely substantially different from the estimate.',
    }
    return interpretations.get(rating, '')

`scripts/sof_generator.py`

Generates the Summary of Findings table from structured input.

Outputs to Generate

sr-grade-evidence-profile.md — Full evidence profile per outcome
sr-grade-sof-table.md — Summary of Findings table
sr-grade-certainty-text.md — Manuscript-ready certainty narratives

Guardrails

Do NOT upgrade RCTs — RCTs start at High and can only be downgraded. Upgrades are for observational studies only.
Do NOT downgrade twice for the same issue — If heterogeneity is explained by a subgroup effect, do not also downgrade for inconsistency.
Imprecision thresholds are outcome-specific — What is "imprecise" for mortality may not be for quality of life. Use EM/CC-specific thresholds.
Do NOT rate down when not assessable — If publication bias cannot be assessed (<10 studies), do not automatically downgrade. Note it as "not assessable."
Consistency in SoF tables — Use the same outcome language as your PICO. Be specific about time points.

Edge Cases

| Situation | Response | |-----------|----------| | Mix of RCTs and observational studies | Assess separately. If consistent, can present together. Start at the level of the dominant design. | | No data for a planned outcome | Note in SoF table: "Outcome not reported in included studies." | | All studies at high RoB | Start at Low (even for RCTs) or Very Low (for obs). Maximum downgrade -2. | | Extremely large effect in observational study | Upgrade +2. If effect is RR >5 with narrow CI, may be compelling even without RCT. | | Systematic review of qualitative studies | GRADE is not applicable. Use CERQual instead. |

Handoff

When GRADE is complete:

Pass: SoF table + evidence profile + certainty narratives per outcome
Summarize: "GRADE complete. [X] outcomes rated [High/Moderate], [Y] rated [Low/Very Low]. Ready for Phase 9: Manuscript Writing."
Reference: SoF table and certainty text go directly into manuscript results and discussion