markov-regime-features - Research Notes

Experiment Overview

| Item | Details | |------|---------| | Date | 2025-12-13 | | Goal | Debug why Markov features (indices 19-24) appeared constant in observation heatmaps while other features showed high variability | | Environment | Python 3.10, PyTorch 2.0+, alpaca_trading package | | Status | Success |

Context

When visualizing RL observation features as a heatmap, the Markov regime features (volatility and trend probabilities) appeared as constant horizontal bands with values ~0.33, while all other features showed expected variability over time. This made the Markov features useless for regime detection.

Root Cause

The standalone build_inference_observation() function defaults to uniform priors when markov_vol_probs and markov_trend_probs are not provided:

# From inference_obs_builder.py:391-401
if include_markov:
    if markov_vol_probs is not None:
        obs[:, feat_idx:feat_idx+3] = markov_vol_probs
    else:
        obs[:, feat_idx:feat_idx+3] = [0.33, 0.34, 0.33]  # Uniform prior

The 6 Markov features are: | Index | Feature | Description | |-------|---------|-------------| | 19 | vol_prob_low | P(low volatility regime) | | 20 | vol_prob_medium | P(medium volatility regime) | | 21 | vol_prob_high | P(high volatility regime) | | 22 | trend_prob_down | P(downtrend regime) | | 23 | trend_prob_neutral | P(neutral regime) | | 24 | trend_prob_up | P(uptrend regime) |

Verified Workflow

Solution: Use InferenceObservationBuilder class

The InferenceObservationBuilder class maintains Markov state and updates probabilities based on price history:

from alpaca_trading.gpu.inference_obs_builder import InferenceObservationBuilder

# Create stateful builder with GPU Markov system
obs_builder = InferenceObservationBuilder(window=100, use_gpu_markov=True)

# Build observation - Markov states are updated from price history
obs = obs_builder.build(
    prices=prices,
    high=high,
    low=low,
)

# Access current regime estimates
print(f"Volatility: {obs_builder.vol_probs}")  # e.g., [0.15, 0.60, 0.25]
print(f"Trend: {obs_builder.trend_probs}")      # e.g., [0.20, 0.30, 0.50]

Visualizing Regime Evolution

To see how regimes change over time, build observations at multiple points:

eval_points = list(range(150, n_bars, 10))
vol_history = {'low': [], 'med': [], 'high': []}

for end_idx in eval_points:
    obs_builder.build(prices=prices[:end_idx], high=high[:end_idx], low=low[:end_idx])
    vol_history['low'].append(obs_builder.vol_probs[0])
    vol_history['med'].append(obs_builder.vol_probs[1])
    vol_history['high'].append(obs_builder.vol_probs[2])

plt.plot(eval_points, vol_history['high'], 'r-', label='P(High Vol)')
plt.plot(eval_points, vol_history['low'], 'g-', label='P(Low Vol)')

Failed Attempts (Critical)

| Attempt | Why it Failed | Lesson Learned | |---------|---------------|----------------| | Using build_inference_observation() directly | No Markov state passed, defaults to uniform [0.33, 0.34, 0.33] | Use InferenceObservationBuilder class for stateful Markov tracking | | Setting include_markov=False | Removes features entirely, breaks model compatibility | Keep Markov features, just ensure proper initialization | | Manual Markov probability calculation | Complex HMM implementation needed | Use GPUMarkovSystem via use_gpu_markov=True |

Final Parameters

# Correct usage for dynamic Markov features
obs_builder = InferenceObservationBuilder(
    window=100,           # Must match training window
    use_gpu_markov=True   # Enable GPU-accelerated HMM
)

# For live trading - maintain single builder instance across bars
# For backtesting - create new builder per symbol, maintain across timesteps

Key Insights

• The standalone function is for single-shot inference where Markov state is passed externally
• The builder class is for stateful inference where Markov state evolves with price data
• GPU Markov system uses last 20 returns to estimate regime probabilities
• Uniform priors [0.33, 0.34, 0.33] indicate Markov system not receiving updates
• In heatmaps, constant horizontal bands at indices 19-24 are a red flag for this issue

References

• alpaca_trading/gpu/inference_obs_builder.py - Feature building implementation
• alpaca_trading/gpu/harmonized_gpu.py - GPUMarkovSystem class
• notebooks/develop_branch_testing.ipynb - Visualization examples