Local LLM Setup for Agents

Stand up local inference (Ollama, llama.cpp, LM Studio, vLLM) and point an agent at it, sized to the hardware that is actually there. Read the machine and the failure logs before recommending anything.

When to Use

The user wants a model running on their own machine for an agent ("run models locally", "set up Ollama", "which local model fits my laptop").
A local model download, install, or generation is failing or stalling and needs a diagnosis.
A model has to be sized against specific hardware before anything is downloaded.
An agent has to be pointed at a self-hosted OpenAI-compatible endpoint.
A local model answers in prose or code blocks instead of calling tools, or ignores the conversation — see Step 6 before blaming the model.

Not for picking a hosted/cloud model (for the user's hosted ModelScope models, see the modelscope-inference skill), and not for the bundled llama.cpp runtime's own UI details beyond identifying that the user is on that path.

Step 0 — Respect the permission posture

Installs and system changes belong to the user unless they say otherwise. Read-only inspection is always fair game: hardware queries, $HERMES_HOME/logs/errors.log, config.yaml, connectivity probes, source greps. Then hand over exact commands and say plainly what you did NOT do.

"Help me find out which model is good for my laptop" plus "teach me, don't do it yourself" means diagnose and instruct — do not install, pull, or write config. Close by stating the distinction explicitly, so the user knows nothing was executed behind their back.

State it once, in a line, not as recurring preamble. When the user asks several things ("why did it fail, which model, how do I install it"), answer all of them in one organized answer and stop — re-explaining an earlier finding, or re-asserting that nothing was done, reads as stalling and is the fastest way to lose them.

Lead with the takeaway. Open with the problem and the solution in a few plain lines the user can act on; measurements, tables and history come after, and only the numbers that change a decision. This user asks for exactly that and it is standing, not situational — an answer that arrives as a wall of tables before the point reads as noise, and they will interrupt mid-session to ask for it. Keep narration of what you did out of the reply entirely.

Step 1 — Measure the machine

nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv
powershell.exe -NoProfile -Command "Get-CimInstance Win32_ComputerSystem | Select-Object @{n='RAM_GB';e={[math]::Round($_.TotalPhysicalMemory/1GB,1)}} | Format-List"
powershell.exe -NoProfile -Command "Get-CimInstance Win32_Processor | Select-Object Name,NumberOfCores | Format-List"

Take VRAM from nvidia-smi, never from Win32_VideoController.AdapterRAM — it is a 32-bit counter that reports 4096 MB for an 8 GB card and will undersize every recommendation.
Ignore integrated GPUs (Intel Arc, Radeon) in the sizing math; their "VRAM" is shared system RAM.
Check free disk. A local model is 5–25 GB each.

Step 2 — Identify which "local" path is in play

Two unrelated paths run under the word "local", and confusing them misdiagnoses everything:

| Path | What runs | Weights come from | |---|---|---| | Hermes desktop Settings → Providers → Local Models (onboarding: "Run models locally") | Hermes' own bundled llama.cpp runtime; it downloads the build and fits memory itself | llama.cpp build from GitHub releases, GGUF weights from HuggingFace | | An external server the user installs (Ollama, LM Studio, vLLM) | that server, on its own port | that server's own registry |

Confirm which one before theorizing: curl -s http://localhost:11434/api/tags answers the Ollama question, and local_runtime.enabled in config.yaml says whether the bundled runtime was ever touched.

A user who asks for Ollama while clicking through the bundled-runtime screen is on the wrong path, not making a mistake — say so early.

Step 3 — Choose a model that fits

Budget rule: installed size ≤ VRAM − ~1.5 GB (desktop/display overhead plus room for the KV cache). On 8 GB VRAM that is ≈5.5–6 GB maximum for a fully-GPU model.

Hard filter: the model must support tool calling. An agent edits files and runs commands through tool calls; a chat-only model cannot act, however smart it is. Filter the registry by the tools capability rather than reading parameter counts.

Prefer a small dense model fully in VRAM over a large MoE that spills to RAM — low active parameters help generation but do nothing for prompt processing.

See references/local-model-sizing.md for VRAM tiers, how to read a registry tag page, and pull-time math.

Step 4 — Install the backend and pull

Ollama on Windows: installer from ollama.com/download/windows, or winget install Ollama.Ollama. It installs a service on port 11434.

ollama --version
curl http://localhost:11434/api/tags      # → {"models":[]}
ollama pull <tag>
ollama list

Own weights instead of the registry: put FROM ./model.gguf in a Modelfile, then ollama create <name> -f Modelfile.

When the direct download host is blocked or intercepted (HuggingFace, GitHub releases), do not stall on it — switch source. The backend's own registry is a different route from HuggingFace and is often healthy on the same machine; domestic mirrors (hf-mirror.com, modelscope.cn) serve the same GGUFs for import by Modelfile. Confirm reachability per host, then recommend the route that already works.

Quote the download cost before recommending a pull, and do the cheap one first: at ~6 MB/s a 23 GB model is about an hour, a 5 GB model about fifteen minutes.

Step 5 — Wire it into the agent

Keep the cloud model as default and add local as a switchable alias:

hermes config set providers.ollama.base_url http://localhost:11434/v1
hermes config set model.aliases.local ollama/<tag>

then /model local in a chat, /model <cloud-model> to switch back.

Or make local the default for everything:

hermes config set model.provider custom
hermes config set model.base_url http://localhost:11434/v1
hermes config set model.default <tag>

In the desktop app the same wiring is Settings → Providers → Custom Endpoints — NOT the "Local Models" tab, which is the bundled llama.cpp runtime and a different code path. Create an endpoint with Name, Base URL http://localhost:11434/v1, API key left empty, API mode Auto-detect, Discover models on, and press the Test button before saving: it probes the server, lists the models it finds, auto-fills Model, and rewrites the Base URL to whatever actually served /v1/models (typing the bare root instead of /v1 404s every later request). The panel's Context length field is where the served window gets pinned per endpoint.

The bare provider name ollama is the LOCAL endpoint and maps to custom; ollama-cloud is the hosted API-key service. Use ollama for a server the user installed.
The window the agent gets is the one the server serves, and on the /v1 (OpenAI-compatible) route Ollama ignores a num_ctx in the request body — so model.ollama_num_ctx cannot widen it. Set the window on the server, then tell the agent the same number scoped to that endpoint (see Step 6) — not through the global model.context_length, which would cap a cloud provider the user still runs.
Set these with hermes config set — never hand-edit config.yaml.
Offer to keep a cloud fallback for hard tasks (hermes fallback add).

Step 6 — Align the context window (the tool-calling trap)

Keep exactly one definition of the endpoint in config. A leftover legacy custom_providers: entry is normalized before the modern providers: dict and deduped on (name, base_url, model), so it silently claims the route and the entry carrying your context_length is discarded — the value then resolves to None and the agent falls back to an unusable advertised window.

Symptom: the model replies with a shell or Python snippet in a code block, or narrates what it would do, instead of emitting a tool call. Nothing executes. It is not the model, not the chat template, and not the install — a hand-made request with a small prompt usually returns a perfectly good tool_calls payload from the same server.

Mechanism: the agent sends a large fixed preamble on every call (system prompt, skills index, every tool schema — commonly 15–25k tokens). When the server's runtime window is smaller, it truncates from the front and the tool schemas are the first thing to go. Ollama picks its default window from a VRAM tier (4k / 32k / 256k), so a small GPU lands on 4k.

Fix the server, then mirror it on the agent:

| Setting | Where | Value | |---|---|---| | OLLAMA_CONTEXT_LENGTH | server env var | the agent's floor (64k) or higher — 64k is a minimum, not a recommendation. Size it to the model's advertised maximum and measure the cost rather than assuming it; local-llm-serving carries the timing probe | | OLLAMA_KV_CACHE_TYPE | server env var | q8_0 — halves the cache | | OLLAMA_FLASH_ATTENTION | server env var | 1 — required by a quantized cache | | OLLAMA_NUM_PARALLEL | server env var | 1 — num_ctx is split across parallel slots | | context_length under the endpoint | config: providers.<key>.context_length, or per model at providers.<key>.models.<tag>.context_length | the same number the server serves — preferred: it is keyed to this endpoint's base URL and resolved before any probe | | model.context_length | hermes config set | the same number, but global — it caps every provider, cloud ones included. Use it only when this local server is the sole provider in play |

Env vars are read at launch: restart the whole server (on Windows quit both the tray app and the ollama process), never just re-run a command in an open shell — setx does not reach a running process, so a correct change looks like it did nothing.

Tool Call Failures: Detected Model Hijack

When Hermes starts, the gateway may auto-load a detected model (e.g., deepseek-flash) instead of your configured local endpoint. This causes tool call failures because the logs show a different model than what should be calling tools.

Symptoms:

Logs show "Model context warmed: deepseek-flash -> 1000000 tokens" but your config has base_url: http://localhost:11434/v1
Tool calls silently fail or return text responses instead of executing
The model name in logs doesn't match your Ollama server's models

Diagnosis:

Check gateway logs for "Model context warmed" lines showing the detected model
Look in Hermes desktop app: Settings → Providers → check if an alternate model is listed
Inspect .env file for overriding env vars like:
- DEFAULT_MODEL=deepseek-*
- PROVIDER=default or empty string (defaults to hosted API)

Resolution:

# Remove overrides from .env, then restart gateway:
hermes gateway --provider custom restart
# in Hermes desktop app: /restart

# Verify in logs:
grep "Model context warmed" ~/.hermes/logs/gateway.log | tail -5
# Should show your local model, not a detected one like deepseek-flash

# If detected model hijack persists, re-create the endpoint explicitly:
hermes model add local-localhost-11434 --url=http://localhost:11434/v1
hermes config set tool.default-model <your-local-model-name>

Prevention tips:

Keep .env minimal — only secrets and URLs, not model names
Explicitly set tool.default-model in config.yaml (not just model under each provider block)
Avoid having DeepSeek or hosted API keys in the config that could auto-populate the provider

Pitfalls

Judge a local model for agent work by prompt processing, not generation rate. MoE activations shrink per-token generation cost, but every expert is exercised while reading the prompt — so with weights spilled to system RAM the prefill runs at CPU/RAM bandwidth. An agent turn is many sequential calls that each re-send the growing history, so one request becomes minutes of silence.
On a VRAM-tight machine a large model also costs context, not just speed. Weights fill VRAM, the KV cache has nowhere left, so a long-context model ends up running at a short effective window — the worst of both. Prefer a small model fully in VRAM with a real window.
Measure the split instead of arguing about it. ollama ps during generation prints the CPU/GPU share; a majority-CPU line is the verdict.
Benchmark with a full agent task (multi-step, tools loaded), never a one-shot chat prompt. A single small prompt never triggers prefill of the agent's fixed preamble, so it flatters exactly the models that will disappoint in daily use.
Verify second opinions from another model instead of repeating them. Check its size, VRAM and tag claims against the registry tag page and state corrected numbers; be explicit about which part of its reasoning holds and which does not.
Never recommend from a parameter count or a family name — run the tag sizes.
An empty ollama ps proves nothing. The table lists only models currently in memory, and a model is unloaded after its idle timeout or a server restart. Send one request, then read it. Concluding "no context is configured" from an idle server sends the whole diagnosis down the wrong path.
A model's advertised context is not the served context. The registry page and the model metadata report the training maximum; /api/ps reports what the server actually hands out (read that JSON, not the ollama ps table — its PROCESSOR cell contains a space, so a column-index parse in a script reads the neighbouring field). Only the served value decides whether the tool schemas survive.
Restart the server after changing any OLLAMA_* env var. They are read at process launch, and neither setx nor the environment-variables dialog reaches a process that is already running — the change appears to do nothing when it is in fact correct.