Kubernetes AI Expert

Expert in deploying AI/ML workloads on Kubernetes with GPU scheduling, model serving frameworks, and MLOps patterns.

GPU Workload Scheduling

NVIDIA GPU Operator

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm install gpu-operator nvidia/gpu-operator

GPU Resource Requests

| Resource | Description | |----------|-------------| | nvidia.com/gpu: N | Request N GPUs | | nvidia.com/mig-3g.40gb: 1 | MIG slice | | Node selector | nvidia.com/gpu.product | | Toleration | nvidia.com/gpu |

Full manifests: resources/manifests.yaml

Model Serving Frameworks

Framework Comparison

| Framework | Best For | GPU Support | Scaling | |-----------|----------|-------------|---------| | vLLM | High-throughput LLMs | Excellent | HPA/KEDA | | Triton | Multi-model serving | Excellent | HPA | | TGI | HuggingFace models | Good | HPA |

vLLM Deployment

Key configurations:

--tensor-parallel-size - Multi-GPU inference
--max-model-len - Context window
--gpu-memory-utilization - Memory efficiency

Triton Inference Server

Multi-model serving from S3/GCS
HTTP (8000), gRPC (8001), Metrics (8002)
Model polling for dynamic updates

Text Generation Inference (TGI)

HuggingFace native
Quantization support (bitsandbytes-nf4)
Simple deployment

Deployment manifests: resources/manifests.yaml

Helm Chart Pattern

# values.yaml structure
inference:
  enabled: true
  replicas: 2
  framework: "vllm"  # vllm, tgi, triton
  resources:
    limits:
      nvidia.com/gpu: 1
  autoscaling:
    enabled: true
    minReplicas: 1
    maxReplicas: 10

vectorDB:
  enabled: true
  type: "qdrant"

monitoring:
  enabled: true

Auto-Scaling

Horizontal Pod Autoscaler (HPA)

Scale on:

GPU utilization (DCGM_FI_DEV_GPU_UTIL)
Inference queue length
Custom metrics

KEDA Event-Driven Scaling

Scale on:

Prometheus metrics
Message queue depth (RabbitMQ, SQS)
Custom external metrics

HPA/KEDA configs: resources/manifests.yaml

Networking

Ingress Configuration

Rate limiting (nginx annotations)
TLS with cert-manager
Large body size for AI payloads
Extended timeouts (300s+)

Network Policies

Restrict pod-to-pod communication
Allow only gateway → inference
Permit DNS egress

Monitoring

Key Metrics

| Metric | Source | Purpose | |--------|--------|---------| | GPU Utilization | DCGM Exporter | Scaling | | Inference Latency | Prometheus | SLO | | Tokens/Second | Custom | Throughput | | Queue Length | App metrics | Scaling |

Setup

# Install DCGM Exporter
helm install dcgm-exporter nvidia/dcgm-exporter

# ServiceMonitor for Prometheus
# See resources/manifests.yaml

Managed Kubernetes

AWS EKS

Instance types: g5.2xlarge, p4d.24xlarge
AMI: AL2_x86_64_GPU
GPU taints for isolation

Azure AKS

VM sizes: Standard_NC*, Standard_ND*
A100 support via NC24ads_A100_v4

OCI OKE

Shapes: BM.GPU.A100-v2.8, VM.GPU.A10
GPU node pools with taints

Terraform examples: ../terraform-iac/resources/modules.tf

Best Practices

Resource Management

Always set GPU limits = requests
Use node selectors for GPU types
Implement tolerations for GPU taints
PVC for model caching

High Availability

Multiple replicas across zones
Pod disruption budgets
Readiness/liveness probes

Cost Optimization

Spot instances for dev/test
Auto-scaling to zero when idle
Right-size GPU instances

Resources

Deploy AI workloads at scale with GPU-optimized Kubernetes.