Distributed Training

Skill Profile

(Select at least one profile to enable specific modules)

[ ] DevOps
[x] Backend
[ ] Frontend
[ ] AI-RAG
[ ] Security Critical

Overview

This skill covers when and how to scale training beyond a single process: data-parallel vs model-parallel tradeoffs, Ray Train/Tune patterns, and minimum operational guardrails (reproducibility, artifact storage, failure recovery). It's designed to align with Cerebra's CerebraML direction (MLflow-backed experiment tracking + artifact storage).

Why This Matters

Scalability: Train larger models and datasets
Speed: Reduce training time significantly
Resource Utilization: Efficiently use multiple GPUs/nodes
Reproducibility: Track all training artifacts
Cost Optimization: Better resource utilization

Core Concepts & Rules

1. Core Principles

Follow established patterns and conventions
Maintain consistency across codebase
Document decisions and trade-offs

2. Implementation Guidelines

Start with the simplest viable solution
Iterate based on feedback and requirements
Test thoroughly before deployment

Inputs / Outputs / Contracts

Skill Composition

Depends on: None
Compatible with: None
Conflicts with: None
Related Skills: None

Quick Start / Implementation Example

Review requirements and constraints
Set up development environment
Implement core functionality following patterns
Write tests for critical paths
Run tests and fix issues
Document any deviations or decisions

# Example implementation following best practices
def example_function():
    # Your implementation here
    pass

Assumptions

Sufficient compute resources available
Network connectivity between nodes
Shared storage accessible from all nodes
MLflow server is configured

Compatibility

Python: Full support
PyTorch: Full support
TensorFlow: Full support
Ray: Full support
MLflow: Full support

Test Scenario Matrix (QA Strategy)

Technical Guardrails & Security Threat Model

1. Security & Privacy (Threat Model)

Top Threats: Injection attacks, authentication bypass, data exposure

[ ] Data Handling: Sanitize all user inputs to prevent Injection attacks. Never log raw PII
[ ] Secrets Management: No hardcoded API keys. Use Env Vars/Secrets Manager
[ ] Authorization: Validate user permissions before state changes

2. Performance & Resources

[ ] Execution Efficiency: Consider time complexity for algorithms
[ ] Memory Management: Use streams/pagination for large data
[ ] Resource Cleanup: Close DB connections/file handlers in finally blocks

3. Architecture & Scalability

[ ] Design Pattern: Follow SOLID principles, use Dependency Injection
[ ] Modularity: Decouple logic from UI/Frameworks

4. Observability & Reliability

[ ] Logging Standards: Structured JSON, include trace IDs request_id
[ ] Metrics: Track error_rate, latency, queue_depth
[ ] Error Handling: Standardized error codes, no bare except
[ ] Observability Artifacts:
- Log Fields: timestamp, level, message, request_id
- Metrics: request_count, error_count, response_time
- Dashboards/Alerts: High Error Rate > 5%

Agent Directives & Error Recovery

(ข้อกำหนดสำหรับ AI Agent ในการคิดและแก้ปัญหาเมื่อเกิดข้อผิดพลาด)

Thinking Process: Analyze root cause before fixing. Do not brute-force.
Fallback Strategy: Stop after 3 failed test attempts. Output root cause and ask for human intervention/clarification.
Self-Review: Check against Guardrails & Anti-patterns before finalizing.
Output Constraints: Output ONLY the modified code block. Do not explain unless asked.

Definition of Done (DoD) Checklist

[ ] Tests passed + coverage met
[ ] Lint/Typecheck passed
[ ] Logging/Metrics/Trace implemented
[ ] Security checks passed
[ ] Documentation/Changelog updated
[ ] Accessibility/Performance requirements met (if frontend)

Anti-patterns / Pitfalls

⛔ Don't: Log PII, catch-all exception, N+1 queries
⚠️ Watch out for: Common symptoms and quick fixes
💡 Instead: Use proper error handling, pagination, and logging

Reference Links & Examples

Internal documentation and examples
Official documentation and best practices
Community resources and discussions

Versioning & Changelog

Version: 1.0.0
Changelog:
- 2026-02-22: Initial version with complete template structure