Resilience Patterns

Retry with Exponential Backoff

async function withRetry<T>(
  fn: () => Promise<T>,
  maxRetries = 3,
  baseDelay = 1000,
): Promise<T> {
  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    try {
      return await fn();
    } catch (error) {
      if (attempt === maxRetries) throw error;
      if (!isRetryable(error)) throw error;
      const delay = baseDelay * Math.pow(2, attempt) + Math.random() * 1000;
      await new Promise((r) => setTimeout(r, delay));
    }
  }
  throw new Error('Unreachable');
}

function isRetryable(error: unknown): boolean {
  if (error instanceof Error && 'status' in error) {
    const status = (error as { status: number }).status;
    return status === 429 || status >= 500;
  }
  return true;
}

Circuit Breaker

class CircuitBreaker {
  private failures = 0;
  private lastFailure = 0;
  private state: 'closed' | 'open' | 'half-open' = 'closed';

  constructor(
    private threshold = 5,
    private resetTimeout = 30000,
  ) {}

  async execute<T>(fn: () => Promise<T>, fallback?: () => T): Promise<T> {
    if (this.state === 'open') {
      if (Date.now() - this.lastFailure > this.resetTimeout) {
        this.state = 'half-open';
      } else {
        if (fallback) return fallback();
        throw new Error('Circuit breaker is open');
      }
    }

    try {
      const result = await fn();
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      if (fallback) return fallback();
      throw error;
    }
  }

  private onSuccess() { this.failures = 0; this.state = 'closed'; }
  private onFailure() {
    this.failures++;
    this.lastFailure = Date.now();
    if (this.failures >= this.threshold) this.state = 'open';
  }
}

// Usage
const breaker = new CircuitBreaker(5, 30000);
const result = await breaker.execute(
  () => externalApi.call(),
  () => cachedResult, // fallback
);

resilience4j (Java/Spring)

@CircuitBreaker(name = "paymentService", fallbackMethod = "paymentFallback")
@Retry(name = "paymentService")
@TimeLimiter(name = "paymentService")
public CompletableFuture<Payment> processPayment(Order order) {
    return CompletableFuture.supplyAsync(() -> paymentClient.charge(order));
}

public CompletableFuture<Payment> paymentFallback(Order order, Throwable t) {
    return CompletableFuture.completedFuture(Payment.pending(order.getId()));
}

# application.yml
resilience4j:
  circuitbreaker:
    instances:
      paymentService:
        slidingWindowSize: 10
        failureRateThreshold: 50
        waitDurationInOpenState: 30s
  retry:
    instances:
      paymentService:
        maxAttempts: 3
        waitDuration: 1s
        exponentialBackoffMultiplier: 2

Timeout Pattern

async function withTimeout<T>(fn: () => Promise<T>, ms: number): Promise<T> {
  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), ms);
  try {
    return await fn();
  } finally {
    clearTimeout(timer);
  }
}

Pattern Selection

| Pattern | Use When | |---------|----------| | Retry | Transient failures (network, 5xx, 429) | | Circuit breaker | Repeated failures to same dependency | | Timeout | Preventing hung connections | | Bulkhead | Isolating resource pools per dependency | | Fallback | Degraded but available service |

Anti-Patterns

| Anti-Pattern | Fix | |--------------|-----| | Retrying non-idempotent operations | Only retry idempotent requests (GET, PUT with same key) | | No jitter on retry delay | Add random jitter to prevent thundering herd | | Circuit breaker per request (not per service) | Share breaker instance across requests to same service | | No fallback for open circuit | Provide cached/degraded response | | Retrying 400 errors | Only retry transient errors (429, 5xx, network) |

Production Checklist

• [ ] Retry with exponential backoff + jitter
• [ ] Circuit breaker on all external service calls
• [ ] Timeouts on all outbound requests
• [ ] Fallback responses for critical paths
• [ ] Health check endpoints
• [ ] Monitoring: circuit breaker state, retry counts