ML Batch Processing Pattern

Classification

• Domain: Computer Science, AI/ML
• Category: ML System Design Patterns
• Novelty: 6/10 (established pattern with modern evolution)
• Practitioner Evidence: 10/10 (Google, industry standard)

Mental Model

Batch processing decouples prediction from real-time requests by pre-computing predictions on scheduled intervals. Like meal prep for the week instead of cooking each meal on-demand—you process predictions in bulk during off-peak hours, store results, and serve them instantly when requested.

When to Use

• Predictions needed for all users/items at regular intervals (daily recommendations, weekly reports)
• Training data arrives in batches rather than continuously
• Cost optimization prioritized over real-time freshness (batch = cheaper compute)
• Predictions can tolerate staleness (hours/days old acceptable)
• High-throughput scenarios where latency isn't critical

Core Framework

1. Schedule Determination

Identify prediction cadence based on business requirements

• Daily batch: Nightly recommendation refresh for morning users
• Hourly batch: Stock predictions updating each trading hour
• Weekly batch: Monthly subscription churn predictions
• Event-triggered: Batch after data warehouse ETL completion

2. Data Ingestion Setup

Configure batch data pipeline from sources to ML system

• Extract from data warehouse/data lake (BigQuery, Snowflake, S3)
• Apply feature transformations matching training pipeline
• Validate schema consistency with model expectations
• Handle missing values using same imputation as training

3. Distributed Processing Architecture

Parallelize prediction computation across infrastructure

• Use MapReduce/Spark for horizontal scaling across datasets
• Partition data by entity (user_id, product_id) for independent processing
• Configure batch size based on memory constraints (1K-100K records/batch)
• Implement checkpointing for fault tolerance on long-running jobs

4. Model Serving Configuration

Deploy model in batch-optimized inference mode

• Load model once per batch job (avoid reload overhead)
• Use batch prediction APIs (TensorFlow batch_predict, PyTorch batch inference)
• Enable GPU batching for deep learning models (32-512 samples/batch)
• Leverage model compilation (TensorRT, ONNX) for throughput optimization

5. Prediction Storage Design

Store pre-computed predictions for fast lookup

• Key-value store for individual lookups (Redis, DynamoDB: user_id → prediction)
• Columnar storage for analytics (Parquet, BigQuery: all predictions for analysis)
• Include metadata (model_version, prediction_timestamp, confidence_score)
• Set TTL based on batch frequency (1.5x batch interval for overlap)

6. Keyed Predictions Pattern

Enable distributed batch prediction with result matching

• Attach unique keys to input records (primary keys, composite keys)
• Preserve keys through prediction pipeline (input → features → predictions)
• Join predictions back to original entities using keys
• Handle missing predictions (timeouts, errors) with fallback logic

7. Monitoring & Alerting

Track batch job health and prediction quality

• Job completion metrics (duration, throughput, failure rate)
• Data quality checks (null rate, distribution shifts, schema violations)
• Model performance monitoring (prediction distribution, confidence intervals)
• Alerting on batch failures or stale predictions (SLA breaches)

Practical Application

E-commerce Recommendation System

Problem: Generate personalized product recommendations for 10M users Batch Solution:

1. Nightly job extracts user behavior (purchases, views, clicks) from data warehouse
2. Spark cluster processes 10M users in parallel (10K users/partition × 1K partitions)
3. Recommendation model generates top-100 products per user (batch size: 256 users)
4. Predictions stored in Redis with 36-hour TTL (user_id → [product_ids + scores])
5. Web app reads pre-computed recommendations in <5ms (vs. 200ms real-time inference)

Credit Card Fraud Detection (Batch Component)

Problem: Update fraud risk scores for all accounts daily Batch Solution:

1. Daily batch (3am) processes all 50M accounts using last 30 days transactions
2. Feature engineering pipeline computes aggregates (transaction velocity, geography patterns)
3. XGBoost model scores all accounts (1M accounts/minute on 100-node cluster)
4. Risk scores stored in Aurora DB (account_id, risk_score, score_date)
5. Real-time transactions query batch scores + apply real-time rules for final decision

Edge Cases & Nuances

Cold Start Problem: New users/items without predictions

• Fallback to popularity-based or demographic-based defaults
• Trigger on-demand prediction for high-value new entities
• Include new entities in next batch cycle with minimal features

Prediction Staleness: Batch predictions lag reality

• Hybrid approach: batch for stable predictions + real-time updates for high-velocity features
• Monitor staleness impact on business metrics (click-through rate decay over time)
• Decrease batch interval if staleness hurts performance (daily → hourly)

Batch Job Failures: Incomplete or failed batch runs

• Implement idempotent batch jobs (can safely re-run without duplicates)
• Use transactional writes to prediction store (all-or-nothing semantics)
• Maintain previous batch predictions as fallback until new batch succeeds

Cost vs. Freshness Tradeoff: More frequent batches = higher cost

• Profile actual prediction change rate (how often do top-10 recommendations shift?)
• A/B test batch frequencies to measure impact on engagement metrics
• Use event-triggered batches for critical updates (product catalog changes)

Anti-Patterns

Batch for Latency-Critical Applications: Using batch for fraud detection that must block transactions in real-time Over-Engineering Batch Infrastructure: Building distributed system for 10K records processable on single machine Ignoring Data Freshness Requirements: Daily batches for inventory predictions when stock changes hourly No Fallback Strategy: System breaks when batch job fails with no stale predictions

Trade-offs

Batch vs. Online Inference:

• Batch: Lower cost (bulk processing), higher latency (stale predictions), simpler ops (scheduled jobs)
• Online: Higher cost (per-request compute), lower latency (fresh predictions), complex ops (SLA-driven)

Batch Frequency:

• More frequent (hourly): Fresher predictions, higher compute cost, more operational complexity
• Less frequent (daily): Stale predictions, lower cost, simpler ops, higher storage requirements

Distributed vs. Single-Node:

• Distributed: Scales to billions of records, complex infrastructure, slower for small datasets
• Single-node: Simple, fast for <10M records, memory/compute constraints, no fault tolerance

Related Frameworks

• Streaming ML Pattern: Continuous prediction updates from streaming data (complements batch)
• Online Learning Pattern: Incremental model updates as new data arrives (batch retraining alternative)
• Lambda Architecture: Batch layer + speed layer for hybrid batch/streaming systems
• Feature Store Pattern: Centralized feature computation for batch and online consistency

Practitioner Sources

• Google ML Design Patterns (Lakshmanan et al.): Batch Serving pattern (#17), Keyed Predictions pattern
• ML System Design: Batch vs. online prediction serving tradeoffs, architecture patterns
• Apache Spark MLlib: Distributed batch prediction at scale, best practices
• AWS SageMaker Batch Transform: Managed batch inference service, cost optimization