SR Phase 6: Risk of Bias Assessment

Your Role

Select the correct risk of bias tool for each study type, guide domain-level judgments with EM/CC-specific exemplars, generate robvis-ready visualizations, and produce sensitivity analysis plans from the results.

---

Prerequisites

• List of included studies with their designs (from Phase 5 extraction)
• Protocol specifying which tools will be used (from Phase 2)

---

Workflow

Step 1: Tool Selection

ASK for the design of the first study. Use this decision tree:

What is the study design?
├── Randomized trial
│   └── Cochrane RoB 2.0 (5 domains, 3-level judgment)
├── Non-randomized intervention study
│   ├── With control group → ROBINS-I (7 domains, 4-level)
│   └── Single-arm (pre-post) → JBI Critical Appraisal Checklist
├── Diagnostic accuracy study
│   └── QUADAS-2 (4 domains, 3-level)
├── Prognostic study
│   └── QUIPS (6 domains, 3-level)
├── Case-control / Cohort (observational)
│   ├── With intervention → ROBINS-I
│   └── Without intervention → JBI Cohort/Case-Control tool
├── Case series
│   └── JBI Case Series Checklist (9 domains)
├── Prevalence study
│   └── JBI Prevalence Checklist
└── Qualitative study
    └── CASP Qualitative Checklist

GENERATE a tool selection summary: "For [study design], the appropriate tool is [tool name] with [N] domains."

Step 2: Apply the Tool — Domain-Level Guidance

For each tool, provide domain-specific guidance with EM/critical care exemplars:

RoB 2.0 (RCTs) — Domain Guidance

| Domain | Key Questions | EM/CC Considerations | |--------|---------------|---------------------| | D1: Randomization | Random sequence? Allocation concealed? Baseline imbalances? | EM trials often use ED-day-of-week or alternation (pseudo-random) — flag as "some concerns." | | D2: Deviations | Blinding of patients/personnel? | Blinding often impossible for procedural trials (e.g., intubation, ECMO). If outcome is objective (mortality), blinding is less critical. Judge "low risk" if objective outcome despite unblinded. | | D3: Missing data | >80% follow-up? MCAR vs MAR? | ED studies have naturally short follow-up → low attrition is common. Flag if >20% loss. | | D4: Outcome measurement | Objective outcome? Blinded assessor? | Mortality at 28 days is objective → low risk. Composite outcomes incorporating clinician judgment (e.g., "clinical cure") → higher risk. | | D5: Selective reporting | Protocol vs published outcomes? | Check ClinicalTrials.gov. Flag switched primary outcomes or unreported outcomes. |

ROBINS-I (Observational) — Critical Domains

| Domain | Key Questions | EM/CC Considerations | |--------|---------------|---------------------| | Confounding | Were key confounders measured and adjusted? | Critical in EM: severity of illness (SOFA, APACHE, GCS, ISS), time to treatment, comorbidities. Flag if not adjusted. | | Selection | Was cohort defined at consistent time point? | EM studies: "time zero" must be consistent. Flag if some patients classified at ED arrival and others at ICU admission. | | Classification | Was intervention defined consistently? | In EM, "early" vs "late" intervention definitions vary widely. Flag if not standardized. |

QUADAS-2 (Diagnostic) — Domain Guidance

| Domain | Key Questions | EM/CC Considerations | |--------|---------------|---------------------| | Patient selection | Consecutive or random? Case-control design avoided? | EM convenience samples are common — flag selection bias risk. | | Index test | Blinded to reference standard? Threshold pre-specified? | Point-of-care ultrasound (POCUS) has operator dependence — flag inter-operator variability. | | Reference standard | Correct classification? Blinded to index test? | For EM diagnostic studies, imperfect reference standard is common (e.g., CT vs clinical follow-up). | | Flow/Timing | Appropriate interval between tests? All patients received reference standard? | In EM, verification bias is common (only positive index tests get the reference standard). |

Step 3: Generate robvis-Ready CSV

CREATE a CSV file ready for https://www.riskofbias.info/welcome/robvis-visualization-tool:

Study,D1,D2,D3,D4,D5,Overall
Smith 2021,Low,Low,Some concerns,Low,Low,Some concerns
Jones 2020,Low,High,Low,Low,Low,High

Use the Python script to generate this from user input:

python3 scripts/robvis_csv.py --input rob-judgments.csv --output rob-traffic-light.csv

Create scripts/robvis_csv.py:

#!/usr/bin/env python3
import csv, sys

# Maps judgment to traffic light format
JUDGMENT_MAP = {
    'low': 'Low',
    'some': 'Some concerns',
    'high': 'High risk',
    'critical': 'Critical risk',
    'no_info': 'No information',
}

ROB2_DOMAINS = ['D1_Randomization', 'D2_Deviations', 'D3_Missing_Data', 'D4_Outcome_Measurement', 'D5_Selective_Reporting']
ROBINS_DOMAINS = ['D1_Confounding', 'D2_Selection', 'D3_Classification', 'D4_Deviations', 'D5_Missing_Data', 'D6_Outcome_Measurement', 'D7_Selective_Reporting']

def generate_robvis_csv(judgments, domains, output_file):
    with open(output_file, 'w', newline='') as f:
        writer = csv.writer(f)
        writer.writerow(['Study'] + domains + ['Overall'])
        for study, judgments_dict in judgments.items():
            row = [study]
            for d in domains:
                j = judgments_dict.get(d, 'no_info')
                row.append(JUDGMENT_MAP.get(j.lower(), j))
            overall = 'Low'
            for d in domains:
                j = judgments_dict.get(d, 'low').lower()
                if j == 'high' or j == 'critical':
                    overall = 'High risk' if j == 'high' else 'Critical risk'
            row.append(overall)
            writer.writerow(row)
    print(f"robvis-ready CSV written to {output_file}")

Step 4: Inter-Rater Reliability for RoB

If two reviewers assessed RoB independently:

python3 scripts/rob_kappa.py --reviewer1 rob-reviewer1.csv --reviewer2 rob-reviewer2.csv

Create scripts/rob_kappa.py:

#!/usr/bin/env python3
import csv, argparse, math

def load_judgments(filepath):
    judgments = {}
    with open(filepath) as f:
        reader = csv.DictReader(f)
        for row in reader:
            study = row['Study']
            judgments[study] = {k: v for k, v in row.items() if k != 'Study'}
    return judgments

def compute_kappa(j1, j2):
    # Simplified: compute agreement per domain across all studies
    total = 0
    agreed = 0
    for study in j1:
        if study in j2:
            for domain in j1[study]:
                if domain in j2[study]:
                    total += 1
                    if j1[study][domain].lower() == j2[study][domain].lower():
                        agreed += 1
    if total == 0: return 0, 0, 0
    p_obs = agreed / total
    p_exp = 0.5  # simplified assumption
    kappa = (p_obs - p_exp) / (1 - p_exp) if p_exp < 1 else 1
    se = math.sqrt(p_obs * (1 - p_obs) / total) / (1 - p_exp) if p_exp < 1 else 0
    return kappa, se, agreed / total

parser = argparse.ArgumentParser()
parser.add_argument('--reviewer1', required=True)
parser.add_argument('--reviewer2', required=True)
args = parser.parse_args()

j1 = load_judgments(args.reviewer1)
j2 = load_judgments(args.reviewer2)
kappa, se, agreement = compute_kappa(j1, j2)

print(f"Overall agreement: {agreement:.1%}")
print(f"Cohen's κ: {kappa:.3f}")
print(f"SE: {se:.3f}")
if kappa > 0.80: print("Interpretation: Excellent agreement")
elif kappa > 0.60: print("Interpretation: Good agreement")
elif kappa > 0.40: print("Interpretation: Moderate agreement — discuss discrepancies")
else: print("Interpretation: Poor agreement — recalibrate before proceeding")

Step 5: Generate Sensitivity Analysis Plan

From the RoB results, generate a sensitivity analysis plan:

Sensitivity Analysis Plan (from RoB results):

1. Primary analysis: All studies
2. Sensitivity 1: Exclude studies with HIGH risk of bias overall
   - Studies to exclude: [list]
   - Expected impact: If estimates change substantially → interpret with caution
3. Sensitivity 2: Exclude studies with HIGH or SOME CONCERNS
   - Studies to exclude: [list]
   - Expected impact: More conservative estimate
4. If results are robust: confidence in findings increases
5. If results change: primary conclusion driven by low-quality studies — downgrade certainty

Step 6: Generate Manuscript-Ready RoB Text

GENERATE narrative text:

"We assessed risk of bias using [Tool Name]. Of [X] included studies, [Y] ([Y%]) were rated low risk of bias, [Z] ([Z%]) had some concerns, and [W] ([W%]) were rated high risk of bias. The most common source of bias was [domain], affecting [N] studies ([N%]). Inter-rater agreement was [κ value] ([interpretation])."

Present in a table:

| Study | D1 | D2 | D3 | D4 | D5 | Overall |
|-------|----|----|----|----|----|---------|
| ...   | 🟢 | 🟢 | 🟡 | 🟢 | 🟢 | 🟡 |

---

Scripts

scripts/robvis_csv.py

Generates riskofbias.info-ready CSV from judgment data.

scripts/rob_kappa.py

Calculates inter-rater agreement on RoB domain judgments.

---

Outputs to Generate

1. sr-rob-summary.md — Summary table + narrative text
2. sr-rob-judgments.csv — Domain-level judgments (robvis-ready)
3. sr-rob-sensitivity-plan.md — Pre-specified sensitivity analyses based on RoB
4. sr-rob-raw-assessments.md — Detailed per-domain justification for each study

---

Guardrails

1. Use current tools only — RoB 2.0 (not the old Cochrane tool), ROBINS-I (not ACROBAT-NRSI), QUADAS-2 (not QUADAS).
2. Blinding in EM trials — "Blinding is often impossible in procedural/drug trials where sham is unethical. If the outcome is objective (mortality, intubation), an unblinded trial can still be LOW risk on D2."
3. Confounding by indication — "This is the single biggest bias in observational EM research: sicker patients receive more aggressive treatment. Check for propensity score matching, instrumental variables, or multivariate adjustment."
4. Attrition in ED studies — "Short follow-up means less attrition is expected. Flag any study with <90% follow-up for mortality outcomes."
5. Selective outcome reporting — "EM trials registered after completion are a red flag. Check ClinicalTrials.gov history if available."
6. Do NOT combine different tool results — Each tool produces non-comparable judgments. Keep separate.

---

Edge Cases

| Situation | Response | |-----------|----------| | Mix of RCTs and observational studies | Assess with RoB 2 + ROBINS-I separately. Do not combine in one table. Report separately in manuscript. | | Study uses both ITT and per-protocol | Assess under both analyses. ITT is primary for effectiveness; per-protocol may be more appropriate for safety/harm. | | Crossover or cluster RCT | Use RoB 2 extensions for these designs (RoB 2 for cluster trials, specific guidance for crossover). | | Study reports no blinding and no blinding information | "No information" is NOT the same as "not done." If the outcome is objective, rate as "some concerns" not "high risk." | | <10 studies in the review | Sensitivity analysis based on RoB may still be reported, but interpret with caution due to low power. | | All studies high risk of bias | Flag prominently. GRADE certainty starts low. Consider whether a meta-analysis is appropriate. Sensitivity analysis meaningless — do narrative synthesis. |

---

Handoff

When RoB is complete:

1. Pass: robvis CSV + sensitivity analysis plan + RoB narrative text
2. Summarize: "Risk of bias assessment complete. [X/Y] studies at high risk. Sensitivity analyses planned. Ready for Phase 7: Data Synthesis."
3. Reference: RoB results inform GRADE downgrading and sensitivity analyses in synthesis