Evaluating ML Models

Use this skill for rigorously assessing model performance, comparing alternatives, diagnosing issues, and optimizing hyperparameters.

When to use this skill

Model training complete — need systematic performance assessment
Comparing multiple models/algorithms — statistical model comparison
Diagnosing overfitting/underfitting — bias-variance analysis
Hyperparameter tuning — finding optimal configurations
Selecting appropriate metrics — matching metrics to business objectives
Experiment tracking — reproducible experimentation
Production readiness check — validation before deployment

When NOT to use this skill

Feature engineering and preprocessing → use engineering-ml-features
Exploratory data analysis → use analyzing-data
Building interactive data apps → use @building-data-apps
Notebook setup and workflows → use @working-in-notebooks

Quick tool selection

| Task | Default choice | Notes | |---|---|---| | Cross-validation | sklearn.model_selection | Standard CV, stratified, time series, grouped | | Classification metrics | sklearn.metrics | Accuracy, precision, recall, F1, ROC-AUC, PR-AUC | | Regression metrics | sklearn.metrics | MAE, RMSE, R², MAPE | | Hyperparameter tuning | Optuna | Bayesian optimization with pruning | | Distributed tuning | Ray Tune | Large-scale distributed search | | Experiment tracking | MLflow | Open-source, model registry | | Cloud experiment tracking | Weights & Biases | Collaboration-focused | | Model comparison | scipy.stats | Paired t-test, McNemar's test |

Evaluation workflow

1. Choose cross-validation strategy

Match CV to your data characteristics:

| Data Type | CV Strategy | Implementation | |---|---|---| | Standard tabular | K-Fold | KFold(n_splits=5, shuffle=True) | | Classification (imbalanced) | Stratified K-Fold | StratifiedKFold(n_splits=5) | | Time series | Time Series Split | TimeSeriesSplit(n_splits=5) | | Grouped/ clustered | Group K-Fold | GroupKFold(n_splits=5) |

2. Select appropriate metrics

Classification:

Balanced: Accuracy
Imbalanced: F1, Precision-Recall, ROC-AUC
Cost-sensitive: Custom business metrics

Regression:

Scale-independent: MAPE, R²
Error magnitude: MAE (robust), RMSE (penalizes large errors)

3. Analyze performance

Cross-validation mean ± std (estimate variance)
Validation curves (bias-variance tradeoff)
Learning curves (data sufficiency)
Error analysis by segment

4. Hyperparameter optimization

Use Bayesian optimization for efficient search:

import optuna

def objective(trial):
    params = {
        'n_estimators': trial.suggest_int('n_estimators', 10, 200),
        'max_depth': trial.suggest_int('max_depth', 2, 32, log=True)
    }
    model = RandomForestClassifier(**params)
    return cross_val_score(model, X, y, cv=5).mean()

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)

5. Track and compare experiments

Log everything for reproducibility:

Hyperparameters
Metrics (CV and test)
Model artifacts
Dataset versions

Core implementation rules

1. Always use proper validation

from sklearn.model_selection import cross_val_score, StratifiedKFold

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X, y, cv=cv, scoring='roc_auc')
print(f"CV AUC: {scores.mean():.3f} (+/- {scores.std() * 2:.3f})")

2. Match metrics to problem type

# Classification with imbalance
from sklearn.metrics import classification_report, roc_auc_score

print(classification_report(y_true, y_pred))
y_proba = model.predict_proba(X_test)[:, 1]
print(f"ROC-AUC: {roc_auc_score(y_test, y_proba):.3f}")

# Regression
from sklearn.metrics import mean_absolute_error, root_mean_squared_error

print(f"MAE: {mean_absolute_error(y_true, y_pred):.3f}")
print(f"RMSE: {root_mean_squared_error(y_true, y_pred):.3f}")

3. Validate on hold-out test set

# Final evaluation on untouched test set
# Never optimize hyperparameters on test data!
y_pred = best_model.predict(X_test)
test_score = accuracy_score(y_test, y_pred)
print(f"Test accuracy: {test_score:.3f}")

4. Analyze errors systematically

# Error by segment
errors = y_pred != y_true
error_df = X_test[errors].copy()
error_df['true'] = y_true[errors]
error_df['pred'] = y_pred[errors]

# Analyze patterns
print(error_df.groupby('category').size())

Common anti-patterns

| Anti-pattern | Solution | |---|---| | ❌ Single train/test split without CV | Use stratified k-fold for robust estimates | | ❌ Optimizing accuracy on imbalanced data | Use F1, PR-AUC, or class-balanced metrics | | ❌ Data leakage in preprocessing | Fit preprocessors on train only, use pipelines | | ❌ Not checking calibration for probabilities | Use sklearn.calibration.CalibratedClassifierCV | | ❌ Ignoring inference speed/memory | Profile prediction time, consider model size | | ❌ No error analysis | Segment errors by features to find patterns | | ❌ Overfitting validation set | Keep final test set completely untouched | | ❌ Not tracking random seeds | Set random_state everywhere for reproducibility |

Progressive disclosure

Detailed reference guides for specific topics:

references/cross-validation.md — CV strategies for different data types (K-Fold, Stratified, Time Series, Group K-Fold)
references/metrics-guide.md — Choosing and interpreting classification and regression metrics
references/hyperparameter-tuning.md — Optuna and Ray Tune optimization patterns
references/experiment-tracking.md — MLflow and Weights & Biases setup

Related skills

@engineering-ml-features — Upstream feature engineering before evaluation
@orchestrating-data-pipelines — Production model deployment
@assuring-data-pipelines — Model monitoring in production

evaluating-ml-models