Production Readiness Checklist

This skill provides comprehensive checklists to ensure all code meets production-grade standards before deployment.

When to Use

Apply this skill when:

• Preparing code for production deployment
• Conducting final code review before merge
• Verifying system readiness for release
• Implementing quality gates in CI/CD
• Auditing existing production code
• Planning major feature releases
• Setting up new environments
• Establishing deployment procedures

Pre-Commit Checklist (Code Level)

Code Quality (10 items)

• [ ] Code follows golang-production-standards skill
• [ ] All functions have clear documentation comments
• [ ] No hardcoded values (use constants or config)
• [ ] No print statements (use structured logging)
• [ ] No commented-out code (delete or explain)
• [ ] Variable names are meaningful (no single letters except i, j)
• [ ] Function names describe exact behavior
• [ ] No TODO/FIXME comments without issue reference
• [ ] Imports are organized (stdlib, third-party, internal)
• [ ] File does not exceed 200 lines (except special cases)

Error Handling (8 items)

• [ ] All errors are wrapped with context (fmt.Errorf %w)
• [ ] No ignored errors (no blank _ assignment)
• [ ] Custom error types defined for domain errors
• [ ] Error messages are user-friendly
• [ ] No error information leaks (no secrets in messages)
• [ ] Panic only in main, never in libraries
• [ ] Error recovery implemented where needed
• [ ] Goroutine errors are properly handled

Testing Coverage (7 items)

• [ ] Unit tests exist for all public functions
• [ ] Test coverage >= 70%
• [ ] Edge cases and error scenarios tested
• [ ] Tests use table-driven pattern where applicable
• [ ] Mocking used appropriately (not over-mocked)
• [ ] Concurrent code tested with race detector
• [ ] Tests pass locally with go test -race -cover ./...

Security (8 items)

• [ ] No hardcoded secrets or credentials
• [ ] Passwords hashed with bcrypt (cost >= 12)
• [ ] All inputs validated at API boundary
• [ ] SQL injection prevented (parameterized queries)
• [ ] Path traversal prevented (validated file paths)
• [ ] No sensitive data logged (passwords, tokens, PII)
• [ ] TLS used for all external communications
• [ ] Authentication/authorization implemented

Pre-Review Checklist (Integration Level)

API Design (6 items)

• [ ] RESTful endpoints follow naming conventions
• [ ] Request/response DTOs used (not domain models)
• [ ] Error responses standardized
• [ ] Pagination implemented for large result sets
• [ ] API versioning clear (v1, v2, etc.)
• [ ] Documentation present (Swagger/OpenAPI)

Database (7 items)

• [ ] Migrations are versioned and tested
• [ ] Indexes created for frequently queried columns
• [ ] Foreign keys properly defined
• [ ] No N+1 queries (use preloading)
• [ ] Transactions used for multi-step operations
• [ ] Connection pool configured correctly
• [ ] Performance tested (<100ms typical queries)

Concurrency (5 items)

• [ ] Goroutine leaks prevented
• [ ] Race detector passes (go test -race)
• [ ] Context used correctly (not leaked)
• [ ] Timeouts set for all blocking operations
• [ ] Resource cleanup guaranteed (defer cleanup)

Kubernetes (6 items)

• [ ] K8s client nil-checked for test environments
• [ ] Resources labeled properly
• [ ] Retry logic for transient failures
• [ ] Graceful shutdown implemented
• [ ] Resource requests/limits defined
• [ ] Probes configured (startup, readiness, liveness)

Pre-Deployment Checklist (System Level)

Configuration Management (8 items)

• [ ] All config from environment variables
• [ ] No secrets in version control
• [ ] Config validation on startup
• [ ] Defaults sensible but explicit
• [ ] Config documented in README
• [ ] Multiple environment configs tested
• [ ] Feature flags implemented where needed
• [ ] Config hot-reload tested if supported

Logging and Monitoring (8 items)

• [ ] Structured JSON logging configured
• [ ] Log levels appropriate (Debug, Info, Warn, Error)
• [ ] Request IDs tracked through request lifecycle
• [ ] Metrics exposed at /metrics endpoint
• [ ] Health checks implemented (/health endpoint)
• [ ] Readiness/liveness probes work correctly
• [ ] Error rates monitored
• [ ] Performance metrics baseline established

Performance (6 items)

• [ ] API response time < 200ms (p95)
• [ ] Database queries < 100ms typical
• [ ] K8s API calls < 500ms
• [ ] Memory usage < 512MB per pod
• [ ] Startup time < 30 seconds
• [ ] Load tested with expected traffic

Security (10 items)

• [ ] Authentication implemented
• [ ] Authorization (RBAC) enforced
• [ ] Rate limiting enabled
• [ ] CORS configured correctly
• [ ] Security headers present
• [ ] Input validation enforced
• [ ] SQL injection prevention verified
• [ ] Secrets management configured
• [ ] TLS certificates valid
• [ ] Vulnerability scan passed (gosec)

Operations (8 items)

• [ ] Runbooks written for common issues
• [ ] Alert thresholds set and tested
• [ ] Rollback procedures documented
• [ ] Backup/restore tested
• [ ] Disaster recovery plan exists
• [ ] On-call documentation complete
• [ ] Incident response procedures defined
• [ ] Service dependencies documented

Pre-Release Checklist (Quality Gate)

Code Review Completion (5 items)

• [ ] Minimum 2 reviewers approved
• [ ] All review comments addressed
• [ ] No blocking comments remain
• [ ] Security review completed
• [ ] Architecture review completed

Testing Completion (8 items)

• [ ] Unit tests pass (100%)
• [ ] Integration tests pass (100%)
• [ ] Smoke tests pass
• [ ] Load tests pass
• [ ] Security tests pass
• [ ] No test skips without reason
• [ ] Coverage report reviewed
• [ ] Race detector clean

CI/CD Pipeline (8 items)

• [ ] All GitHub Actions workflows pass
• [ ] Build completes in < 5 minutes
• [ ] Docker image builds successfully
• [ ] Linting passes (golangci-lint)
• [ ] Format check passes (gofmt)
• [ ] Vet passes (go vet)
• [ ] Dependency check passes
• [ ] License check passes (if applicable)

Documentation (7 items)

• [ ] README.md updated
• [ ] API documentation updated
• [ ] Migration guide written (if applicable)
• [ ] Changelog entry added
• [ ] Code comments added for complex logic
• [ ] Architecture decision recorded
• [ ] Performance benchmarks updated

Deployment Readiness (8 items)

• [ ] Deployment plan documented
• [ ] Rollback plan documented
• [ ] Communication plan ready
• [ ] Stakeholders notified
• [ ] Maintenance window scheduled (if needed)
• [ ] Monitoring configured
• [ ] Logging configured
• [ ] Alerting configured

Production Deployment Checklist

Pre-Deployment (10 items)

• [ ] Backup taken
• [ ] Deployment plan reviewed with team
• [ ] Rollback procedure tested
• [ ] Database migrations tested in staging
• [ ] Feature flags disabled by default
• [ ] Circuit breakers configured
• [ ] Rate limits tested
• [ ] Load balancing configured
• [ ] DNS propagation planned
• [ ] Communication channels open

Deployment Execution (8 items)

• [ ] Deployment performed during planned window
• [ ] Deployment leader assigned
• [ ] Changes deployed incrementally
• [ ] Health checks passing after each step
• [ ] Logs monitored during deployment
• [ ] Metrics monitored during deployment
• [ ] Incidents tracked if any occur
• [ ] All steps documented in runbook

Post-Deployment (10 items)

• [ ] All services healthy
• [ ] No error rate spike
• [ ] Performance metrics normal
• [ ] User-facing features working
• [ ] Database queries responsive
• [ ] API latency acceptable
• [ ] Memory/CPU usage normal
• [ ] All probes returning healthy
• [ ] Alerts not triggering
• [ ] Team standby for 1 hour

Post-Release (8 items)

• [ ] Feature monitored for 24 hours
• [ ] Performance metrics stable
• [ ] Error rates normal
• [ ] User feedback positive
• [ ] No critical issues found
• [ ] Documentation updated with lessons learned
• [ ] Monitoring alerts tuned if needed
• [ ] Success communicated to stakeholders

Production Code Compliance

Skills Compliance Verification

Ensure code follows all applicable skills:

Mandatory Skills for All Code:
- golang-production-standards (required)
- error-handling-guide (required)
- security-best-practices (if handling user data)

Feature-Specific Skills:
- api-design-patterns (for API endpoints)
- database-best-practices (for database operations)
- kubernetes-integration (for K8s operations)
- testing-best-practices (for test code)
- package-organization (for new packages)
- file-structure-guidelines (for file organization)

Operations Skills:
- monitoring-observability (for logging/metrics)
- cicd-pipeline-optimization (for CI/CD)

Automated Checks

# Code quality checks
go vet ./...
golangci-lint run
gofmt -l .

# Security checks
gosec ./...
trufflehog filesystem ./

# Testing
go test -race -cover -timeout 30m ./...

# Build
go build ./cmd/api
go build ./cmd/scheduler

# Docker
docker build -t platform-go:latest .

# Compliance
grep -r "TODO\|FIXME" --include="*.go" internal/ cmd/ || true
grep -r "print\|println" --include="*.go" internal/ cmd/ || true

Common Failure Scenarios

API Latency High (> 200ms p95)

Checklist:

• [ ] Database queries analyzed (use slow query log)
• [ ] N+1 queries identified and fixed
• [ ] Indexes verified on queried columns
• [ ] Connection pool size verified
• [ ] Caching strategy reviewed
• [ ] Load test results analyzed
• [ ] Network latency checked
• [ ] Third-party API latency checked

Memory Usage High (> 512MB)

Checklist:

• [ ] Goroutine leaks detected with pprof
• [ ] Memory profiling run
• [ ] Large object allocations identified
• [ ] Cache eviction policies checked
• [ ] Database connection pool reviewed
• [ ] Resource cleanup verified
• [ ] GC tuning optimized
• [ ] Heap snapshot analyzed

Error Rate Spike (> 1%)

Checklist:

• [ ] Error logs analyzed for pattern
• [ ] Dependencies health checked
• [ ] Database connectivity verified
• [ ] Rate limits triggered?
• [ ] Circuit breaker states checked
• [ ] Resource exhaustion checked
• [ ] Configuration changes reviewed
• [ ] Network connectivity tested

Build Failure

Checklist:

• [ ] Compilation errors cleared
• [ ] Linting errors resolved
• [ ] Test failures investigated
• [ ] Docker build logs analyzed
• [ ] Dependency versions compatible
• [ ] Go version compatible
• [ ] CGO dependencies installed
• [ ] Build cache cleaned

Metrics to Monitor Post-Deployment

Availability Metrics

- Uptime percentage (target: 99.9%)
- Health check pass rate (target: 100%)
- Pod crash rate (target: 0%)
- Service availability (target: 99.9%)

Performance Metrics

- API response time p50 (target: <50ms)
- API response time p95 (target: <200ms)
- API response time p99 (target: <500ms)
- Database query time (target: <100ms)
- K8s API call time (target: <500ms)

Error Metrics

- Error rate (target: <0.1%)
- 5xx error rate (target: <0.01%)
- Timeout rate (target: <0.01%)
- Panic rate (target: 0%)

Resource Metrics

- CPU usage (target: <70%)
- Memory usage (target: <70%)
- Disk usage (target: <80%)
- Network I/O (monitor trends)

Production Standards Verification

All code must satisfy:

Code Quality:
- Golangci-lint: all checks pass
- Go fmt: all files formatted
- Go vet: no issues
- Coverage: >= 70%

Security:
- gosec: no high/critical issues
- trufflehog: no secrets found
- Dependencies: no known vulnerabilities

Performance:
- API: <200ms p95
- Database: <100ms
- Memory: <512MB per pod
- Startup: <30s

Testing:
- All tests pass
- Race detector clean
- Integration tests pass
- Load tests pass

Sign-Off Process

Before deployment, require sign-off from:

• [ ] Code Owner: Reviewed code changes
• [ ] Security Lead: Security review passed
• [ ] QA Lead: Testing complete
• [ ] DevOps Lead: Deployment plan reviewed
• [ ] Product Manager: Feature readiness confirmed

Emergency Rollback

If deployment issues occur:

1. Immediate Actions (< 5 minutes)

   • [ ] Alert team immediately
   • [ ] Stop deployment if in progress
   • [ ] Assess impact scope
   • [ ] Decide rollback or fix forward

2. Rollback Execution (< 30 minutes)

   • [ ] Execute rollback procedure
   • [ ] Verify previous version healthy
   • [ ] Monitor metrics return to normal
   • [ ] Document incident

3. Post-Incident (< 24 hours)

   • [ ] Root cause analysis
   • [ ] Prevention steps documented
   • [ ] Team retro/learning session
   • [ ] Updates to deployment procedure

---

Note: This checklist is comprehensive. Not all items apply to every release. Customize based on your risk profile and service criticality.