LLM Training

Frameworks and techniques for training and finetuning large language models.

Framework Comparison

| Framework | Best For | Multi-GPU | Memory Efficient | |-----------|----------|-----------|------------------| | Accelerate | Simple distributed | Yes | Basic | | DeepSpeed | Large models, ZeRO | Yes | Excellent | | PyTorch Lightning | Clean training loops | Yes | Good | | Ray Train | Scalable, multi-node | Yes | Good | | TRL | RLHF, reward modeling | Yes | Good | | Unsloth | Fast LoRA finetuning | Limited | Excellent |

Accelerate (HuggingFace)

Minimal wrapper for distributed training. Run accelerate config for interactive setup.

Key concept: Wrap model, optimizer, dataloader with accelerator.prepare(), use accelerator.backward() for loss.

DeepSpeed (Large Models)

Microsoft's optimization library for training massive models.

ZeRO Stages:

Stage 1: Optimizer states partitioned across GPUs
Stage 2: + Gradients partitioned
Stage 3: + Parameters partitioned (for largest models, 100B+)

Key concept: Configure via JSON, higher stages = more memory savings but more communication overhead.

TRL (RLHF/DPO)

HuggingFace library for reinforcement learning from human feedback.

Training types:

SFT (Supervised Finetuning): Standard instruction tuning
DPO (Direct Preference Optimization): Simpler than RLHF, uses preference pairs
PPO: Classic RLHF with reward model

Key concept: DPO is often preferred over PPO - simpler, no reward model needed, just chosen/rejected response pairs.

Unsloth (Fast LoRA)

Optimized LoRA finetuning - 2x faster, 60% less memory.

Key concept: Drop-in replacement for standard LoRA with automatic optimizations. Best for 7B-13B models.

Memory Optimization Techniques

| Technique | Memory Savings | Trade-off | |-----------|---------------|-----------| | Gradient checkpointing | ~30-50% | Slower training | | Mixed precision (fp16/bf16) | ~50% | Minor precision loss | | 4-bit quantization (QLoRA) | ~75% | Some quality loss | | Flash Attention | ~20-40% | Requires compatible GPU | | Gradient accumulation | Effective batch↑ | No memory cost |

Decision Guide

| Scenario | Recommendation | |----------|----------------| | Simple finetuning | Accelerate + PEFT | | 7B-13B models | Unsloth (fastest) | | 70B+ models | DeepSpeed ZeRO-3 | | RLHF/DPO alignment | TRL | | Multi-node cluster | Ray Train | | Clean code structure | PyTorch Lightning |

Resources

Accelerate: https://huggingface.co/docs/accelerate
DeepSpeed: https://www.deepspeed.ai/
TRL: https://huggingface.co/docs/trl
Unsloth: https://github.com/unslothai/unsloth