Oncall Playbooks

Skill Profile

(Select at least one profile to enable specific modules)

[ ] DevOps
[x] Backend
[ ] Frontend
[ ] AI-RAG
[ ] Security Critical

Overview

Playbooks and runbooks are essential tools for on-call engineers, providing step-by-step guidance for responding to common incidents. Well-written runbooks reduce Mean Time To Recovery (MTTR), enable consistent responses, and help less experienced engineers handle complex situations.

Core Principle: "Document once, execute many times. Make incident response repeatable and reliable."

Why This Matters

Faster Response: No need to figure out from scratch
Consistency: Same steps every time
Knowledge Sharing: Junior engineers can handle incidents
Reduced Stress: Clear guidance during high-pressure situations
Continuous Improvement: Update based on learnings
Onboarding Tool: New engineers learn system architecture

Core Concepts & Rules

1. Core Principles

Follow established patterns and conventions
Maintain consistency across codebase
Document decisions and trade-offs

2. Implementation Guidelines

Start with the simplest viable solution
Iterate based on feedback and requirements
Test thoroughly before deployment

Inputs / Outputs / Contracts

Inputs:
- Incident alerts and symptoms
- Monitoring and metrics data
- Service inventory and architecture
- Common incident patterns
Entry Conditions:
- Monitoring system is operational
- Runbook repository is accessible
- On-call engineers are trained
Outputs:
- Documented runbooks for common incidents
- Playbooks for complex scenarios
- Runbook index and organization
- Test results and maintenance logs
Artifacts Required (Deliverables):
- Runbook repository with proper structure
- Runbook index by symptom/alert/severity
- Test schedule and results
- Maintenance schedule
Acceptance Evidence:
- All critical incidents have runbooks
- Runbooks tested quarterly
- Runbooks linked from alerts
- MTTR improvement metrics
Success Criteria:
- All SEV0/1 incidents have tested runbooks
- Runbook accuracy > 90%
- MTTR reduced by 30%

Skill Composition

Depends on: Incident Triage, Severity Levels
Compatible with: Escalation and Ownership, Communication Templates
Conflicts with: Systems without proper monitoring
Related Skills:
- 41-incident-management/incident-triage - Initial assessment
- 41-incident-management/severity-levels - Severity classification
- 40-system-resilience/disaster-recovery - DR procedures
- 40-system-resilience/postmortem-analysis - Learning from incidents

Quick Start / Implementation Example

Review requirements and constraints
Set up development environment
Implement core functionality following patterns
Write tests for critical paths
Run tests and fix issues
Document any deviations or decisions

# Example implementation following best practices
def example_function():
    # Your implementation here
    pass

Assumptions / Constraints / Non-goals

Assumptions:
- Development environment is properly configured
- Required dependencies are available
- Team has basic understanding of domain
Constraints:
- Must follow existing codebase conventions
- Time and resource limitations
- Compatibility requirements
Non-goals:
- This skill does not cover edge cases outside scope
- Not a replacement for formal training

Compatibility & Prerequisites

Supported Versions:
- Python 3.8+
- Node.js 16+
- Modern browsers (Chrome, Firefox, Safari, Edge)
Required AI Tools:
- Code editor (VS Code recommended)
- Testing framework appropriate for language
- Version control (Git)
Dependencies:
- Language-specific package manager
- Build tools
- Testing libraries
Environment Setup:
- .env.example keys: API_KEY, DATABASE_URL (no values)

Test Scenario Matrix (QA Strategy)

Technical Guardrails & Security Threat Model

1. Security & Privacy (Threat Model)

Top Threats: Injection attacks, authentication bypass, data exposure

[ ] Data Handling: Sanitize all user inputs to prevent Injection attacks. Never log raw PII
[ ] Secrets Management: No hardcoded API keys. Use Env Vars/Secrets Manager
[ ] Authorization: Validate user permissions before state changes

2. Performance & Resources

[ ] Execution Efficiency: Consider time complexity for algorithms
[ ] Memory Management: Use streams/pagination for large data
[ ] Resource Cleanup: Close DB connections/file handlers in finally blocks

3. Architecture & Scalability

[ ] Design Pattern: Follow SOLID principles, use Dependency Injection
[ ] Modularity: Decouple logic from UI/Frameworks

4. Observability & Reliability

[ ] Logging Standards: Structured JSON, include trace IDs request_id
[ ] Metrics: Track error_rate, latency, queue_depth
[ ] Error Handling: Standardized error codes, no bare except
[ ] Observability Artifacts:
- Log Fields: timestamp, level, message, request_id
- Metrics: request_count, error_count, response_time
- Dashboards/Alerts: High Error Rate > 5%

Agent Directives & Error Recovery

(ข้อกำหนดสำหรับ AI Agent ในการคิดและแก้ปัญหาเมื่อเกิดข้อผิดพลาด)

Thinking Process: Analyze root cause before fixing. Do not brute-force.
Fallback Strategy: Stop after 3 failed test attempts. Output root cause and ask for human intervention/clarification.
Self-Review: Check against Guardrails & Anti-patterns before finalizing.
Output Constraints: Output ONLY the modified code block. Do not explain unless asked.

Definition of Done (DoD) Checklist

[ ] Tests passed + coverage met
[ ] Lint/Typecheck passed
[ ] Logging/Metrics/Trace implemented
[ ] Security checks passed
[ ] Documentation/Changelog updated
[ ] Accessibility/Performance requirements met (if frontend)

Anti-patterns / Pitfalls

⛔ Don't: Log PII, catch-all exception, N+1 queries
⚠️ Watch out for: Common symptoms and quick fixes
💡 Instead: Use proper error handling, pagination, and logging

Reference Links & Examples

Internal documentation and examples
Official documentation and best practices
Community resources and discussions

Versioning & Changelog

Version: 1.0.0
Changelog:
- 2026-02-22: Initial version with complete template structure