Runbook Creation

Position in workflow: v0.8 Release Planning → v0.8 Runbook Creation → v0.8 Monitoring Setup

Purpose

Create step-by-step operational playbooks that enable anyone on-call to handle incidents, perform deployments, and execute maintenance tasks without requiring deep system knowledge.

Core Concept: Runbook as Insurance

A runbook is not documentation—it is operational insurance. When systems fail at 3 AM, the runbook is the difference between 5-minute recovery and 5-hour chaos.

Runbook Categories

| Category | Purpose | Trigger | |----------|---------|---------| | Incident Response | Handle production issues | Alert fires, user reports | | Deployment | Execute release procedures | Scheduled release | | Maintenance | Regular operational tasks | Scheduled windows | | Recovery | Restore from failures | Disaster scenario | | Escalation | Route to right people | Issue beyond capability |

Execution

1. Identify critical scenarios

   • What alerts from MON- require action?
   • What deployment steps from DEP- need procedures?
   • What RISK- entries need response plans?

2. Map each scenario to a runbook

   • One runbook per distinct scenario
   • Clear scope—what it covers and doesn't

3. Document step-by-step procedures

   • Numbered steps, no ambiguity
   • Include commands, links, contact info
   • Assume operator has minimal context

4. Define escalation paths

   • When to escalate
   • Who to contact
   • What information to provide

5. Add verification steps

   • How to confirm the issue is resolved
   • What metrics should normalize

6. Create RUN- entries with full traceability

RUN- Output Template

RUN-XXX: [Runbook Title]
Category: [Incident | Deployment | Maintenance | Recovery | Escalation]
Trigger: [What initiates this runbook]
Owner: [Team or role responsible]
Last Tested: [Date of last drill/use]

## Scope
- **Handles**: [What scenarios this covers]
- **Does NOT handle**: [Explicit exclusions]

## Prerequisites
- [ ] Access to [system/tool]
- [ ] Credentials for [service]
- [ ] Contact info for [team]

## Procedure

### Step 1: [Action Title]
[Detailed instructions]

Commands:
```bash
# Example command with placeholders

Verification:

• [ ] [How to confirm step succeeded]

Step 2: [Action Title]

[Detailed instructions]

Step N: [Final Action]

[Detailed instructions]

Escalation

• When to escalate: [Conditions that require help]
• Who to contact: [Name/role, contact method]
• What to provide: [Information needed for handoff]

Post-Incident

• [ ] Document incident timeline
• [ ] Update runbook if steps were wrong/missing
• [ ] Schedule post-mortem if severity > [threshold]

Linked IDs: [MON-XXX, DEP-XXX, RISK-XXX related]

**Example RUN- entries:**

RUN-001: Database Connection Pool Exhaustion Category: Incident Trigger: MON-005 alert (connection pool >90%) Owner: Backend Team Last Tested: 2025-01-15

Scope

• Handles: Connection pool saturation, slow queries causing pooling
• Does NOT handle: Database server crash (see RUN-010)

Prerequisites

• [ ] Access to AWS RDS console
• [ ] Database read credentials
• [ ] PagerDuty access for escalation

Procedure

Step 1: Verify Alert

Check current connection pool status:

Commands:

SELECT count(*) FROM pg_stat_activity WHERE state = 'active';
SELECT * FROM pg_stat_activity WHERE state = 'active' ORDER BY query_start;

Verification:

• [ ] Connection count matches alert threshold

Step 2: Identify Problematic Queries

Find long-running or blocked queries:

Commands:

SELECT pid, now() - pg_stat_activity.query_start AS duration, query
FROM pg_stat_activity
WHERE (now() - pg_stat_activity.query_start) > interval '5 minutes';

Step 3: Kill Problematic Queries (if safe)

Only kill queries that are clearly stuck:

Commands:

SELECT pg_terminate_backend(pid) FROM pg_stat_activity
WHERE pid = <problematic_pid>;

Verification:

• [ ] Connection count dropping
• [ ] MON-005 alert resolving

Step 4: Investigate Root Cause

• Check recent deployments (last 24h)
• Review application logs for query patterns
• Check for missing indexes on recent queries

Escalation

• When to escalate: Issue persists >15 minutes, data integrity concern
• Who to contact: Database Team Lead (Slack: @db-team, PagerDuty)
• What to provide: Timeline, queries identified, actions taken

Post-Incident

• [ ] Document incident timeline
• [ ] File ticket for query optimization if needed
• [ ] Update this runbook if steps were wrong/missing

Linked IDs: MON-005, DEP-001, RISK-008

RUN-002: Production Deployment Procedure Category: Deployment Trigger: Scheduled release, all DEP- criteria met Owner: DevOps Team Last Tested: 2025-01-20

Scope

• Handles: Standard production deployments
• Does NOT handle: Hotfix deployments (see RUN-003), Database migrations (see RUN-004)

Prerequisites

• [ ] All DEP- criteria verified (DEP-002, DEP-003)
• [ ] Staging deployment successful
• [ ] Release approval in deployment channel
• [ ] On-call engineer available for rollback

Procedure

Step 1: Pre-Deploy Announcement

Notify stakeholders of upcoming deployment:

Commands:

# Post to #deployments Slack channel
./scripts/notify-deploy.sh --env production --version ${VERSION}

Step 2: Create Deployment Checkpoint

Tag current production for rollback:

Commands:

git tag -a "pre-deploy-$(date +%Y%m%d-%H%M)" -m "Checkpoint before ${VERSION}"
git push origin --tags

Step 3: Execute Deployment

Deploy to production using CI/CD:

Commands:

# Trigger production deploy pipeline
./scripts/deploy.sh --env production --version ${VERSION}

Verification:

• [ ] Deployment pipeline green
• [ ] New version visible in health check endpoint

Step 4: Post-Deploy Validation

Run smoke tests and verify key flows:

Commands:

./scripts/smoke-test.sh --env production

Verification:

• [ ] All smoke tests pass
• [ ] Key UJ- flows verified manually
• [ ] Error rate within baseline (MON-001)

Step 5: Monitor for 15 Minutes

Watch dashboards for anomalies:

• Error rate dashboard
• Latency dashboard
• Business metrics dashboard

Verification:

• [ ] No new alerts
• [ ] Metrics within normal range

Escalation

• When to escalate: Smoke tests fail, error rate >2%, user reports
• Who to contact: On-call engineer, then Tech Lead
• What to provide: Deployment version, failure mode, logs

Post-Incident

• [ ] Post deployment success/failure to #deployments
• [ ] Update deployment log
• [ ] Schedule retro if issues encountered

Linked IDs: DEP-001, DEP-002, DEP-003, MON-001

## Runbook Quality Checklist

For each runbook, verify:

| Criterion | Question | Pass? |
|-----------|----------|-------|
| **Actionable** | Can someone follow this without asking questions? | |
| **Complete** | Are all steps documented with commands? | |
| **Verifiable** | Does each step have a verification check? | |
| **Scoped** | Is it clear what this does and doesn't cover? | |
| **Escalatable** | Is the escalation path defined? | |
| **Tested** | Has this runbook been tested in a drill? | |
| **Maintained** | Is there an owner who updates it? | |

## Runbook Types by MON- Alert

Map monitoring alerts to runbooks:

| Alert Type | Runbook | Response Time |
|------------|---------|---------------|
| **Critical** | Dedicated incident RUN- | <5 min |
| **Warning** | Shared investigation RUN- | <30 min |
| **Info** | Reference documentation | Next business day |

## Common Procedures to Document

| Category | Must-Have Runbooks |
|----------|-------------------|
| **Incident** | Service down, Performance degradation, Security incident |
| **Deployment** | Standard release, Hotfix, Rollback |
| **Maintenance** | Database backup, Log rotation, Certificate renewal |
| **Recovery** | Data restore, Failover, Service restart |

## Anti-Patterns

| Pattern | Signal | Fix |
|---------|--------|-----|
| **Too vague** | "Investigate the issue" | Add specific commands and checks |
| **Too long** | 50+ step runbook | Split into focused runbooks |
| **Outdated** | References deprecated tools | Add review date, assign owner |
| **No verification** | Steps without confirmation | Add verification after each step |
| **Assuming knowledge** | "You know how to do this" | Write for someone's first day |
| **No escalation** | Dead ends with no help path | Always define escalation |

## Quality Gates

Before proceeding to Monitoring Setup:

- [ ] All critical scenarios have runbooks (RUN-)
- [ ] Each MON- alert maps to a RUN- procedure
- [ ] DEP- rollback triggers link to RUN- procedures
- [ ] RISK- high/medium entries have response runbooks
- [ ] All runbooks have owners and last-tested dates
- [ ] Escalation paths defined for all runbooks

## Downstream Connections

| Consumer | What It Uses | Example |
|----------|--------------|---------|
| **Monitoring Setup** | RUN- procedures linked from alerts | MON-001 → RUN-001 |
| **On-Call Team** | RUN- as operational reference | Night incident → RUN-001 |
| **Post-Mortems** | RUN- gaps inform improvements | "Runbook missing step" → Update RUN-001 |
| **Training** | RUN- for new engineer onboarding | Run drills using RUN-002 |

## Detailed References

- **Runbook examples by category**: See `references/runbook-examples.md`
- **RUN- entry template**: See `assets/run-template.md`
- **Incident response framework**: See `references/incident-framework.md`