# Local Inference Stack — Docker Compose Kit

```
as_of: 2026-07-12
tier: T3 (authored)
layer: Infrastructure
type: launch artifact — downloadable kit
```

A complete, copy-paste-runnable local AI stack: **Ollama** (model server) + **Open WebUI** (chat interface) + **Qdrant** (vector store for RAG). Runs on Linux, macOS, and Windows via Docker Desktop or WSL2. Everything stays on your machine.

**The only genuinely user-specific values** (everything else works as-is):

1. `WEBUI_SECRET_KEY` in `.env` — a random string you generate once (step 3 of the quickstart shows the one-liner).
2. The GPU block — uncomment it only if you have an NVIDIA GPU with the container toolkit installed. Everything runs on CPU without it, just slower.

---

## docker-compose.yml

Save this as `docker-compose.yml` in an empty directory (e.g. `~/local-ai/`).

```yaml
name: local-ai

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama_models:/root/.ollama
    environment:
      # Keep models in VRAM/RAM for 1h after last use (avoids reload lag)
      - OLLAMA_KEEP_ALIVE=${OLLAMA_KEEP_ALIVE:-1h}
      # Allow one chat model + one embedding model resident at once (RAG-friendly)
      - OLLAMA_MAX_LOADED_MODELS=${OLLAMA_MAX_LOADED_MODELS:-2}
      - OLLAMA_NUM_PARALLEL=${OLLAMA_NUM_PARALLEL:-1}
    healthcheck:
      test: ["CMD", "ollama", "--version"]
      interval: 30s
      timeout: 5s
      retries: 3
    # ---- NVIDIA GPU passthrough ----
    # Uncomment the block below if you have an NVIDIA GPU AND the
    # NVIDIA Container Toolkit installed (see GPU notes). With the block
    # commented out, Ollama runs on CPU — fully functional, just slower.
    #
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: all
    #           capabilities: [gpu]

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    depends_on:
      - ollama
    ports:
      - "${WEBUI_PORT:-3000}:8080"
    volumes:
      - open_webui_data:/app/backend/data
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - WEBUI_SECRET_KEY=${WEBUI_SECRET_KEY}
      # First account created becomes admin; signups stay open until you
      # disable them in Admin Settings (or set this to "false" after setup).
      - ENABLE_SIGNUP=${ENABLE_SIGNUP:-true}
      # Point Open WebUI's built-in RAG at the Qdrant container
      - VECTOR_DB=qdrant
      - QDRANT_URI=http://qdrant:6333

  qdrant:
    image: qdrant/qdrant:latest
    container_name: qdrant
    restart: unless-stopped
    ports:
      - "6333:6333"   # REST API + dashboard
      - "6334:6334"   # gRPC
    volumes:
      - qdrant_data:/qdrant/storage

volumes:
  ollama_models:
  open_webui_data:
  qdrant_data:
```

## .env template

Save this as `.env` in the same directory. Only `WEBUI_SECRET_KEY` must be changed.

```bash
# --- required ---
# Session-signing key for Open WebUI. Generate with:
#   openssl rand -hex 32
WEBUI_SECRET_KEY=change-me-run-openssl-rand-hex-32

# --- optional (defaults shown) ---
WEBUI_PORT=3000
ENABLE_SIGNUP=true
OLLAMA_KEEP_ALIVE=1h
OLLAMA_MAX_LOADED_MODELS=2
OLLAMA_NUM_PARALLEL=1
```

---

## GPU notes

**NVIDIA (Linux or Windows/WSL2).** Install the NVIDIA Container Toolkit (`nvidia-ctk`), run `sudo nvidia-ctk runtime configure --runtime=docker`, restart Docker, then uncomment the `deploy:` block in the compose file. Verify with `docker exec ollama nvidia-smi` — your GPU should be listed. Windows users: use Docker Desktop with the WSL2 backend and a current NVIDIA driver on the Windows side; no driver install inside WSL.

**Apple Silicon.** Docker containers cannot reach the Metal GPU. For GPU speed on a Mac, run Ollama natively (`brew install ollama`) and keep only `open-webui` and `qdrant` in Docker, changing `OLLAMA_BASE_URL` to `http://host.docker.internal:11434`. The all-Docker version still works on macOS — CPU-only.

**AMD.** Use the `ollama/ollama:rocm` image tag and add `devices: [/dev/kfd, /dev/dri]` to the ollama service instead of the NVIDIA block. Linux only; support varies by GPU generation.

**CPU fallback.** Do nothing — the file as shipped is CPU-only. A 7–8B model at 4-bit quantization is usable on a modern 8-core CPU with 16 GB RAM; expect roughly 5–15 tokens/second.

## Sizing rule of thumb

Model RAM/VRAM need at Q4 quantization is roughly `parameters × 0.6 GB per billion`, plus 1–2 GB overhead. So: 8B → ~6 GB, 14B → ~10 GB, 32B → ~21 GB, 70B → ~43 GB. If a model does not fit in VRAM, Ollama splits it across GPU and CPU — it works, but speed drops sharply.

---

## Quickstart (10 steps)

1. **Install Docker.** Docker Desktop (macOS/Windows) or Docker Engine + Compose plugin (Linux). Verify: `docker compose version` prints a version 2.x number.
2. **Create the stack directory** and save the two files above into it: `mkdir ~/local-ai && cd ~/local-ai`, then create `docker-compose.yml` and `.env`.
3. **Generate the secret key:** run `openssl rand -hex 32` and paste the output as the value of `WEBUI_SECRET_KEY` in `.env`.
4. **(GPU only)** Install the NVIDIA Container Toolkit and uncomment the `deploy:` block in `docker-compose.yml`. Skip this step for CPU.
5. **Start the stack:** `docker compose up -d`. First run downloads ~4 GB of images. Check all three are healthy: `docker compose ps`.
6. **Pull a chat model:** `docker exec ollama ollama pull llama3.1:8b` (a safe default that fits in 8 GB). More capable if you have the memory: `qwen2.5-coder:32b` or `llama3.3:70b`.
7. **Pull an embedding model** (needed for RAG/document chat): `docker exec ollama ollama pull nomic-embed-text`.
8. **Open the UI:** browse to `http://localhost:3000`. The first account you create is the admin account — create it now, before anyone else on your network can.
9. **Test chat:** pick your model from the dropdown, send a message. First response takes 10–60 seconds while the model loads; after that it's fast. Test RAG by uploading a PDF via the `+` button and asking about it.
10. **Lock it down and maintain it.** In Admin Settings, disable open signups (or set `ENABLE_SIGNUP=false` in `.env` and `docker compose up -d` again). Update images later with `docker compose pull && docker compose up -d`. All data (models, chats, vectors) survives restarts in the three named volumes; back them up with `docker run --rm -v local-ai_open_webui_data:/data -v $(pwd):/backup alpine tar czf /backup/webui-backup.tar.gz /data`.

## Sanity checks

- Ollama API up: `curl http://localhost:11434/api/version`
- Models on disk: `docker exec ollama ollama list`
- Qdrant up: `curl http://localhost:6333/healthz` (dashboard at `http://localhost:6333/dashboard`)
- What's loaded in memory: `docker exec ollama ollama ps`

## What this stack does not do

No authentication on the Ollama port (11434) or Qdrant port (6333) — anyone on your LAN can reach them. If that matters, remove those `ports:` mappings and let only Open WebUI (which has auth) talk to them over the internal Docker network. It also does not serve the public internet; do not port-forward any of this without putting real auth in front of it.