Local Model Deployment Advisor

Use this skill to turn a device profile and deployment requirements into a practical local LLM deployment recommendation. It supports NVIDIA GPUs, Apple Silicon unified memory, and CPU-only systems.

Inputs to collect

1. Run python scripts/detect_hardware.py to obtain CPU, RAM, GPU, VRAM, and installed runtime commands.
2. Ask for missing deployment requirements only when they materially affect the result: use case (chat, code, multilingual, or reasoning), concurrent requests, average/maximum input tokens, maximum output tokens, target TTFT, target output speed, and preferred runtime.
3. Use the YAML catalogues in data/ as the source of model architecture, formats, licenses, quantizations, runtimes, hardware constraints, and workload profiles.

Recommendation procedure

1. Reserve at least 10–20% of accelerator memory for runtime overhead and operating-system use.
2. Estimate weights, KV cache, and overhead with scripts/estimate_memory.py. For GQA/MQA models, use the catalogue's KV head count and head dimension; never approximate KV cache from hidden size when architecture data exists.
3. Use scripts/recommend_models.py with a workload profile to hard-filter unsupported runtime, platform, format, license, and memory combinations. Present its minimum, balanced, and quality-first plans, including rejection reasons.
4. Select a runtime: Ollama for the simplest single-user setup; llama.cpp for CPU/Apple Silicon and GGUF; vLLM or SGLang for NVIDIA multi-user serving; Transformers for custom Python integration.
5. State the exact model, quantization, expected resource use, limitations, launch command, and verification steps. Do not claim measured performance unless benchmark_model.py has been run. Save its local history and pass --calibration-history to a later recommendation when matching hardware/workload observations exist.
6. Create a repeatable deployment directory with scripts/generate_deployment.py, then use scripts/preflight_deployment.py before starting it. The preflight checks the selected GPU's free VRAM, disk capacity, local port, Docker daemon, driver/CUDA visibility, immutable model revision, and (when requested) registry access and Docker GPU passthrough.
7. For Ollama, run scripts/calibrate_ollama.py after the model is loaded. It records /api/ps quantization, VRAM usage, and context length against the generated lock-file prediction. For vLLM, benchmark with --metrics-url .../metrics; only enable --enable-per-request-metrics after evaluating its CPU overhead.

Commands

python scripts/detect_hardware.py --output hardware.json
python scripts/recommend_models.py --vram-gb 24 --runtime vllm --task chat --workload team-api
python scripts/generate_deployment.py --model qwen2.5-7b-instruct --runtime vllm --workload team-api --output ./deployment
python scripts/preflight_deployment.py --model qwen2.5-7b-instruct --runtime vllm --workload team-api --gpu-device-id 0 --output ./deployment
python scripts/benchmark_model.py --url http://localhost:8000/v1/chat/completions --model Qwen/Qwen2.5-7B-Instruct --catalog-model qwen2.5-7b-instruct --runtime vllm --metrics-url http://localhost:8000/metrics
python scripts/calibrate_ollama.py --model qwen2.5:7b

Guardrails

• Treat estimates as planning values, not a guarantee. Driver, backend, context length, concurrency, and model architecture affect actual memory use.
• Treat TTFT and output-speed targets as benchmark acceptance criteria until the target hardware has been measured; do not present them as predicted performance.
• Prefer a smaller quantized model with headroom over an oversized model that risks out-of-memory failures.
• Require a network-enabled model download only when the user explicitly asks to install or start the deployment.
• Generate Docker deployments with 127.0.0.1 binding and explicit GPU device IDs by default. Use 0.0.0.0 only when remote access is intentional and separately secured.
• Keep credentials in local environment variables such as HF_TOKEN; never place them in a report, lock file, catalogue, or command history. Run --verify-docker-gpu and --check-model-access only when the user authorizes their image pull or registry request.
• Do not infer cold-start time from a warmed server. benchmark_model.py measures actual startup only when the user explicitly supplies --startup-command; otherwise it labels the first request as a cold-request sample. Keep calibration history local and hardware-scoped.