Electricity Forecasting Framework

Overview

This skill provides end-to-end support for electricity load/demand forecasting projects, from data preprocessing to model deployment. It covers traditional statistical methods, modern machine learning approaches, and state-of-the-art deep learning architectures.

Quick Start

1. Define Your Forecasting Task

| Horizon | Type | Typical Use | |---------|------|-------------| | 1-48 hours | Short-term (STLF) | Grid operations, unit commitment | | 1 week - 1 month | Medium-term | Maintenance scheduling, fuel planning | | 1-12 months | Long-term (LTLF) | Capacity planning, infrastructure investment |

2. Prepare Your Data

# Run the data preparation script
python scripts/prepare_data.py --input raw_load.csv --output processed/

Required data columns:

• timestamp: Datetime index (hourly or sub-hourly)
• load: Target variable (MW or kWh)
• temperature: Weather feature (°C)
• Optional: humidity, wind_speed, solar_radiation, holiday_flag

3. Select Your Model

See references/model-selection.md for detailed guidance.

Quick recommendation:

• Baseline: Start with persistence or seasonal-naive
• Production STLF: Use XGBoost or LightGBM with weather features
• Research/SOTA: Try Temporal Fusion Transformer (TFT) or iTransformer

4. Train and Evaluate

python scripts/train_model.py --model xgboost --data processed/ --horizon 24

Key metrics to track:

• MAPE (%): Mean Absolute Percentage Error - business interpretability
• RMSE (MW): Root Mean Square Error - penalizes large errors
• MAE (MW): Mean Absolute Error - robust to outliers
• Coverage (%): Prediction interval coverage probability

Core Workflows

Data Preprocessing

1. Load raw data with proper datetime parsing
2. Handle missing values: Forward-fill for short gaps, interpolate for longer
3. Feature engineering:
   • Temporal: hour, day_of_week, month, is_weekend, is_holiday
   • Lag features: load_t-1, load_t-24, load_t-168 (weekly)
   • Rolling stats: rolling_mean_24h, rolling_std_7d
   • Weather: temperature, humidity, apparent_temperature
4. Normalization: RobustScaler or MinMaxScaler for deep learning models

See references/feature-engineering.md for complete feature list.

Model Training

# Example training workflow
from electricity_forecasting import ForecastPipeline

pipeline = ForecastPipeline(
    model_type="xgboost",
    horizon=24,
    lookback=168  # 1 week of history
)

pipeline.fit(train_data, val_data)
predictions, uncertainty = pipeline.predict(test_data)
metrics = pipeline.evaluate(predictions, actuals)

Hyperparameter Tuning

Use scripts/hyperparameter_search.py for automated tuning:

python scripts/hyperparameter_search.py \
  --model lightgbm \
  --data processed/ \
  --n-trials 50 \
  --study-name stlf-tuning

Uncertainty Quantification

For risk-aware decision making:

• Quantile Regression: Predict multiple quantiles (0.1, 0.5, 0.9)
• Conformal Prediction: Distribution-free uncertainty bounds
• Ensemble Methods: Model disagreement as uncertainty proxy
• Monte Carlo Dropout: For neural networks

See references/uncertainty.md for implementation details.

Model Reference

Statistical Models

| Model | Best For | Pros | Cons | |-------|----------|------|------| | ARIMA | Stable series | Interpretable, fast | Assumes linearity | | SARIMA | Strong seasonality | Captures daily/weekly patterns | Manual parameter tuning | | Prophet | Multiple seasonalities | Handles holidays well | Less accurate for STLF | | TBATS | Complex seasonality | Automatic parameter selection | Slower training |

Machine Learning Models

| Model | Best For | Pros | Cons | |-------|----------|------|------| | XGBoost | Production STLF | Fast, accurate, handles missing | No native uncertainty | | LightGBM | Large datasets | Faster than XGBoost, memory efficient | Sensitive to hyperparameters | | Random Forest | Baseline ML | Robust, easy to tune | Lower accuracy than boosting | | CatBoost | Categorical features | Handles categoricals natively | Slower training |

Deep Learning Models

| Model | Best For | Pros | Cons | |-------|----------|------|------| | LSTM | Sequential patterns | Captures long-term dependencies | Slow training, hard to tune | | GRU | Similar to LSTM | Faster convergence | Similar limitations | | Transformer | Long sequences | Parallel training, attention | Data-hungry, complex | | TFT | Multi-horizon | Interpretable attention, uncertainty | Complex implementation | | N-BEATS | Pure deep learning | Strong baseline, interpretable | Less flexible than TFT | | iTransformer | SOTA performance | Inverted transformer architecture | Recent, less battle-tested |

See references/deep-learning-models.md for architecture details and PyTorch implementations.

Evaluation Best Practices

Time Series Cross-Validation

Never use random k-fold! Use expanding or sliding window:

# Expanding window CV
from sklearn.model_selection import TimeSeriesSplit

tscv = TimeSeriesSplit(n_splits=5, test_size=168)  # 1 week test
for train_idx, test_idx in tscv.split(data):
    train, test = data[train_idx], data[test_idx]
    # Train and evaluate

Backtesting Framework

python scripts/backtest.py \
  --model xgboost \
  --data processed/ \
  --cv-splits 5 \
  --horizon 24 \
  --metrics mape,rmse,mae

Benchmark Comparison

Always compare against:

1. Persistence: load_t = load_t-1
2. Seasonal Naive: load_t = load_t-24 (for hourly data)
3. Weekly Naive: load_t = load_t-168

Deployment

Production Pipeline

1. Model serialization: Save with joblib or ONNX
2. Feature pipeline: Ensure identical preprocessing at inference
3. Scheduling: Cron or Airflow for automated forecasts
4. Monitoring: Track forecast drift and retrain triggers

See references/deployment.md for MLOps patterns.

Real-time Inference

from electricity_forecasting import DeploymentModel

model = DeploymentModel.load("models/xgboost-stlf.joblib")
features = prepare_features(latest_data)
prediction = model.predict(features, return_uncertainty=True)

Common Pitfalls

1. Data leakage: Ensure no future information in features
2. Holiday handling: Special days need explicit modeling
3. Temperature nonlinearity: Use heating/cooling degree days
4. Concept drift: Retrain quarterly or when MAPE degrades >20%
5. Peak prediction: Models often under-predict peaks - consider quantile loss

Resources

• Feature Engineering Guide
• Model Selection Guide
• Deep Learning Architectures
• Uncertainty Quantification
• Deployment Patterns
• Datasets Reference

Scripts

| Script | Purpose | |--------|---------| | scripts/prepare_data.py | Data cleaning and feature engineering | | scripts/train_model.py | Model training with validation | | scripts/hyperparameter_search.py | Automated hyperparameter optimization | | scripts/backtest.py | Time series cross-validation | | scripts/evaluate.py | Comprehensive metric calculation | | scripts/deploy_model.py | Export model for production |

Example Usage

# Complete workflow example
# 1. Prepare data
python scripts/prepare_data.py --input data/load_2024.csv --output data/processed/

# 2. Train model
python scripts/train_model.py --model lightgbm --data data/processed/ --horizon 48

# 3. Hyperparameter tuning
python scripts/hyperparameter_search.py --model lightgbm --data data/processed/ --n-trials 100

# 4. Backtest
python scripts/backtest.py --model lightgbm-best --data data/processed/ --cv-splits 5

# 5. Deploy
python scripts/deploy_model.py --model lightgbm-best --output models/production/