Run AI Models Locally on Your Hardware

Step 1: Install llama.cpp

llama.cpp is the engine that runs GGUF-format models efficiently on consumer hardware. Install it via Homebrew (macOS) or build from source:

# macOS — easiest method
brew install llama.cpp

# Verify the install
llama-cli --version

If you prefer building from source (for latest features or Linux):

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
make -j4

# Optional: enable Metal GPU acceleration on macOS
GGML_METAL=1 make -j4

Metal acceleration (Apple Silicon) significantly improves inference speed. If you're on a Mac, use the Metal build.

Step 2: Download a GGUF model

GGUF is the model format llama.cpp uses. The best place to find GGUF models is HuggingFace. Here are our recommendations for different hardware levels:

Download using huggingface-cli (install with pip install huggingface_hub) or download directly from the HuggingFace website. Store models in a dedicated directory:

# Create a models directory
mkdir -p ~/.hermes/models

# Example: download Mistral 7B (TheBloke maintains high-quality GGUF quants)
huggingface-cli download TheBloke/Mistral-7B-Instruct-v0.2-GGUF \
  mistral-7b-instruct-v0.2.Q4_K_M.gguf \
  --local-dir ~/.hermes/models

# Or for Llama 3.1 8B
huggingface-cli download bartowski/Meta-Llama-3.1-8B-Instruct-GGUF \
  Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  --local-dir ~/.hermes/models

Step 3: Configure Hermes to use the local endpoint

Hermes Agent connects to llama.cpp through a local HTTP server. First, start the llama.cpp server with your model:

# Start the local inference server
llama-server \
  --model ~/.hermes/models/mistral-7b-instruct-v0.2.Q4_K_M.gguf \
  --host 127.0.0.1 \
  --port 8080 \
  --ctx-size 8192 \
  --n-gpu-layers 99

The --n-gpu-layers 99 flag offloads as many layers as possible to the GPU (Metal on Apple Silicon). Adjust this number based on your hardware — try 99 first and lower it if you run out of memory.

Next, configure Hermes to use the local endpoint. Run the setup wizard and choose "local" when prompted:

hermes setup

Or edit ~/.hermes/config.yaml directly:

provider: local
model: mistral-7b
endpoint: http://127.0.0.1:8080/v1

No API key is needed for local models — the endpoint is on your machine.

Step 4: Test performance and adjust

Start an interactive Hermes session to test your local model:

hermes
> What model are you and where are you running?

The first response may be slow as the model warms up. Subsequent responses should be faster. If performance is unacceptable:

• Try a smaller quant: Q4_K_M is a good balance. Q3_K_M uses less RAM but slightly lower quality. Q5_K_M uses more RAM but higher quality.
• Reduce context size: Lower --ctx-size to 4096 or 2048. Less context = faster and less RAM.
• Adjust GPU layers: On Apple Silicon, try --n-gpu-layers 99. If you have a dedicated GPU, experiment with the number.
• Use a smaller model: Drop from 8B to 7B or 3B parameters if speed is critical.
• Close other applications: Local models compete for RAM. Close browser tabs and other heavy apps before running inference.

Privacy benefits and tradeoffs

Running models locally is the most private way to use AI — but it comes with tradeoffs. Here's an honest assessment:

Benefits

• Nothing leaves your machine: Prompts, responses, and data stay on your hardware. No network calls to external APIs.
• No API keys to manage: No credentials to leak, rotate, or pay for. The model is yours.
• No usage tracking: Cloud providers log every API call. With local models, you control all logging.
• Works offline: Once the model is downloaded, you don't need internet access. Useful for travel, remote sites, or air-gapped environments.
• No rate limits: Run as many prompts as you want, as fast as your hardware allows. No tiered pricing or quota anxiety.

Tradeoffs

• Slower: Even with GPU acceleration, a 7B local model is slower than cloud APIs. Expect 10-30 tokens/second vs. 50-100+ from cloud providers.
• Less capable: A 7B or 8B local model is noticeably less capable than GPT-4o or Claude Sonnet 4 — especially on complex reasoning, long-form writing, and nuanced instruction following.
• RAM-hungry: Models consume significant memory. An 8 GB machine running a 7B model has little RAM left for other applications.
• No vision (usually): Most GGUF models are text-only. Vision-capable local models exist but require more setup and more RAM.
• Maintenance: You manage the llama.cpp server process — starting, stopping, and updating it. It's not a set-and-forget cloud service.
• No tool calling (limited): Smaller local models may not reliably output structured tool calls. Test your model's tool-use capabilities before relying on it for complex agent workflows.

Canadian data residency and PIPEDA

For Canadian organizations handling personal information under PIPEDA, local models offer a compelling data residency story:

• All processing happens on Canadian soil (your office) — no cross-border data transfer concerns
• No third-party processor to vet or include in your privacy impact assessment
• Full control over data retention and deletion

However, note that running a local model doesn't automatically make you PIPEDA-compliant. You still need appropriate safeguards, access controls, and data handling policies. The model simply removes the third-party API provider from the equation.

Next steps

• Set up cloud providers alongside your local model — use local for sensitive work, cloud for complex reasoning
• Schedule automations that run against your local model overnight
• Secure your environment — even without API keys, your .env and config files should be protected
• Explore self-hosting on a dedicated Linux server with more RAM and GPU power