Run LLMs Locally with Harbor

Harbor is a containerized LLM toolkit. This skill guides through setup and usage of local LLM infrastructure.

Initial Setup Workflow

Check prerequisites
Install Harbor CLI
Start default services (Ollama + Open WebUI)
Pull a model
Verify the setup

Step 1: Check Prerequisites

docker --version        # Need Docker 20.10+
docker compose version  # Need Docker Compose 2.23.1+
git --version

If Docker missing:

Linux: Install via official Docker repo
macOS: Install Docker Desktop
Windows: Install Docker Desktop + WSL2 (run all commands in WSL2)

If Docker Compose too old:

# Ubuntu/Debian
sudo apt-get update && sudo apt-get install docker-compose-plugin

Linux permission fix (if docker commands fail):

sudo usermod -aG docker $USER
# Log out and back in

Step 2: Install Harbor

# Check if installed
harbor --version

# Install if missing
curl https://av.codes/get-harbor.sh | bash
source ~/.bashrc  # or ~/.zshrc

Verify:

harbor doctor  # All checks should pass

Step 3: Start Harbor

harbor up

First run downloads images (~5-10 min). Wait for healthy output:

✔ Container harbor.ollama  Healthy
✔ Container harbor.webui   Healthy

Open UI: harbor open

First launch requires creating a local admin account.

Step 4: Pull a Model

# Recommended small model
harbor pull qwen3:4b

# Verify
harbor ollama list

Step 5: Verify

harbor open - opens UI
Select pulled model from dropdown
Send test message

Core Commands

| Command | Purpose | |---------|---------| | harbor up | Start services | | harbor down | Stop all services | | harbor ps | Show running services | | harbor logs [service] | Tail logs | | harbor open [service] | Open in browser | | harbor url [service] | Print service URL | | harbor pull <model> | Download model | | harbor restart | Restart stack |

Model Management

Ollama Models

harbor ollama list          # List cached models
harbor ollama pull <model>  # Pull model
harbor ollama rm <model>    # Remove model
harbor ollama run <model>   # Interactive chat
harbor ollama ctx 8192      # Set context length

Pull Sources

# Ollama registry
harbor pull qwen3:4b
harbor pull llama3.2:3b
harbor pull gemma3:4b

# HuggingFace via Ollama
harbor pull hf.co/bartowski/gemma-2-2b-it-GGUF:Q4_K_M

# llama.cpp from HuggingFace (GGUF)
harbor pull microsoft/Phi-3.5-mini-instruct-gguf:Q4_K_M

HuggingFace Tools

harbor hf scan-cache           # Show cache status
harbor hf token <token>        # Set token for gated models
harbor hf download user/repo   # Download model
harbor hf find gguf gemma      # Search HF in browser
harbor hf path user/repo       # Find local cache path

Adding Services

Start Additional Services

# Add to current stack
harbor up searxng        # Web search
harbor up tts            # Text-to-speech
harbor up comfyui        # Image generation
harbor up llamacpp       # Alternative backend

# Multiple services
harbor up searxng tts llamacpp

# Skip defaults, only specified services
harbor up --no-defaults llamacpp

Configure Defaults

harbor defaults           # Show default services
harbor defaults add tts   # Add to defaults
harbor defaults rm webui  # Remove from defaults

Available Backends

| Backend | Command | Best For | |---------|---------|----------| | Ollama | harbor ollama model | Ease of use | | llama.cpp | harbor llamacpp model <url> | GGUF models | | vLLM | harbor vllm model user/repo | Production inference | | TGI | harbor tgi model user/repo | HuggingFace models | | SGLang | harbor sglang model user/repo | Fast inference |

Backend Configuration Examples

# llama.cpp
harbor llamacpp model https://huggingface.co/user/repo/blob/main/model.gguf
harbor llamacpp args '-c 4096'

# vLLM
harbor vllm model google/gemma-2-2b-it
harbor vllm args '--max-model-len 4096'

# TGI
harbor tgi model meta-llama/Llama-3.2-3B-Instruct
harbor tgi quant awq

Configuration

Config Commands

harbor config ls                     # List all config
harbor config ls | grep OLLAMA       # Filter config
harbor config get webui.host.port    # Get value
harbor config set webui.name "My AI" # Set value

Service-Specific Environment

harbor env <service>                 # List env vars
harbor env <service> KEY value       # Set env var

Profiles (Save/Load Configurations)

harbor profile ls                    # List profiles
harbor profile save mysetup          # Save current config
harbor profile use mysetup           # Load profile
harbor profile use <url>             # Load from URL
harbor profile rm mysetup            # Delete profile

Cache Locations

harbor config ls | grep CACHE        # Show cache paths
harbor size                          # Show disk usage
harbor hf cache /path/to/cache       # Change HF cache

Network & Access

URLs

harbor url ollama           # Local URL
harbor url --lan ollama     # LAN URL
harbor url -i ollama        # Docker internal URL
harbor qr webui             # QR code for URL

Tunnels (Internet Access)

harbor tunnel webui         # Create temporary tunnel
harbor tunnel down          # Stop tunnels
harbor tunnels add webui    # Auto-tunnel on startup
harbor tunnels ls           # List auto-tunnels

Troubleshooting

Diagnostics

harbor doctor               # System check
harbor ps                   # Container status
harbor logs                 # All logs
harbor logs -n 1000 ollama  # Extended logs

Container Access

harbor shell ollama         # Interactive shell
harbor exec ollama <cmd>    # Run command
harbor run litellm --help   # One-off command

Common Fixes

Services won't start:

harbor down
harbor up

Permission errors on volumes (Linux):

harbor fixfs

GPU not detected: Install NVIDIA Container Toolkit, then restart Docker.

Reset config:

harbor config reset

Advanced Features

Aliases

harbor alias set myenv 'code $(harbor home)/.env'
harbor run myenv
harbor alias ls
harbor alias rm myenv

History

harbor history             # Interactive history
harbor history ls          # List history
harbor history clear       # Clear history

AI Help

harbor how to filter logs?   # AI-powered help (requires ollama running)

File Search

harbor find .gguf            # Find GGUF files in caches
harbor find bartowski        # Find files by name

Eject Config

harbor eject > docker-compose.yml  # Export standalone config

GPU Monitoring

harbor top                   # nvtop for GPU usage

Service Quick Reference

| Service | Handle | CLI | Purpose | |---------|--------|-----|---------| | Open WebUI | webui | harbor webui | Chat UI | | Ollama | ollama | harbor ollama | LLM backend | | llama.cpp | llamacpp | harbor llamacpp | GGUF backend | | vLLM | vllm | harbor vllm | Production backend | | SearXNG | searxng | - | Web search | | ComfyUI | comfyui | harbor comfyui | Image generation | | n8n | n8n | - | Workflow automation | | LiteLLM | litellm | harbor litellm | API proxy | | Aider | aider | harbor aider | AI coding | | Jupyter | jupyter | harbor jupyter | Notebooks |