Run local LLMs with Ollama and Open WebUI in Docker

Update (October 2026): the follow-up puts this same local Ollama behind Kong AI Gateway 2.x on Konnect, with a provider and a model instead of plugin config: From plugin to entities: local Ollama behind Kong AI Gateway 2.x.

I wanted a model on my own laptop that I could call from scripts and from a chat window, with no API key and no prompts leaving the machine. Ollama runs the model; Open WebUI gives it a browser front end. Docker Compose wires the two together in about twenty lines.

This post is the setup I use, with pinned versions, plus two things I learned the slow way: on a Mac, Docker means CPU inference, and once you have a local model it takes one plugin to put a gateway in front of it.

Versions used: Ollama 0.40.11 · Open WebUI v0.12.4 · Kong Gateway 3.9.4 · Apple Silicon Mac with OrbStack

What you need

• Docker with Compose v2 (Docker Desktop, OrbStack or Docker Engine).
• RAM for the model you pick. A 1B model runs in about 2 GB; 8B models want 8 GB or more free.
• Disk space. The two images are several GB together, and each model adds 1 to 5 GB.
• An internet connection for the first image and model pulls. After that, inference is fully local.
• jq (optional) to tidy the curl output.

1. The Compose file

mkdir ollama-chat && cd ollama-chat

Save this as compose.yaml:

services:
  ollama:
    image: ollama/ollama:0.40.11
    restart: unless-stopped
    ports:
      - "127.0.0.1:11434:11434"
    volumes:
      - ollama:/root/.ollama

  open-webui:
    image: ghcr.io/open-webui/open-webui:v0.12.4
    restart: unless-stopped
    depends_on:
      - ollama
    ports:
      - "127.0.0.1:3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - open-webui:/app/backend/data

volumes:
  ollama: {}
  open-webui: {}

A few choices in there are deliberate:

• Pinned tags, not latest or main. Both projects release often. A pinned tag means the setup that worked today still works next month, and you upgrade on purpose.
• Ports bound to 127.0.0.1. Ollama’s API has no authentication. Publishing 11434:11434 exposes it on every interface of your machine, including on café Wi-Fi.
• No version: key. Compose v2 ignores it and prints a warning.
• Named volumes. Models live in the ollama volume and chats in open-webui, so docker compose down keeps both.

2. Start it and pull a model

docker compose up -d
docker compose exec ollama ollama pull llama3.2:1b

llama3.2:1b is small enough to answer in seconds rather than minutes on CPU, which makes it a good first check. Swap in llama3.1:8b or anything else from the Ollama library once the plumbing works.

If a pull from ghcr.io times out, that is the registry, not your setup: run docker compose pull again.

3. Call the API

curl -s http://127.0.0.1:11434/api/chat -d '{
  "model": "llama3.2:1b",
  "messages": [{"role": "user", "content": "Name three HTTP methods."}],
  "stream": false
}' | jq -r '.message.content'

Here are three HTTP methods:

1. GET
2. POST
3. PUT

"stream": false returns one JSON object. Leave it out and you get one JSON line per chunk, which is what a chat UI wants and what a script usually doesn’t.

4. Chat in the browser

Open http://localhost:3000. The first account you create becomes the admin of this Open WebUI instance; it is stored in the local volume. Pick llama3.2:1b from the model list and chat.

What to expect on a Mac: CPU, not GPU

This surprised me the first time. Ask Ollama where the model is loaded:

docker compose exec ollama ollama ps

NAME           ID              SIZE      PROCESSOR    CONTEXT
llama3.2:1b    baf6a787fdff    1.8 GB    100% CPU     4096

Docker on macOS runs containers in a Linux VM, and that VM has no access to the Apple GPU. Ollama’s Docker docs cover NVIDIA, AMD (the rocm tag) and Vulkan on Linux hosts; Metal acceleration comes with the native macOS app. On my Mac the 1B model generated roughly one token per second, with other containers competing for the same VM.

So pick by what you need:

Put a gateway in front of it

A local model is also a cheap way to try LLM traffic controls without paying per token. Kong Gateway’s AI Proxy plugin accepts an OpenAI-style request, translates it for Ollama and translates the answer back. The config below runs on open-source Kong 3.9 in DB-less mode, on the same Docker network as the Compose project. Save it as kong.yaml:

_format_version: "3.0"
services:
  - name: local-llm
    url: http://ollama:11434
    routes:
      - name: chat
        paths: ["/chat"]
    plugins:
      - name: ai-proxy
        config:
          route_type: llm/v1/chat
          model:
            provider: llama2
            name: llama3.2:1b
            options:
              llama2_format: ollama
              upstream_url: http://ollama:11434/api/chat

docker run -d --name kong --network ollama-chat_default \
  -p 127.0.0.1:8000:8000 \
  -e KONG_DATABASE=off -e KONG_DECLARATIVE_CONFIG=/kong.yaml \
  -v "$PWD/kong.yaml:/kong.yaml:ro" kong:3.9

curl -s http://127.0.0.1:8000/chat -H 'content-type: application/json' \
  -d '{"messages":[{"role":"user","content":"Name three HTTP methods."}]}'

Compose names the network <folder>_default, so ollama-chat_default if you followed step 1. The response comes back in OpenAI chat-completion shape, with token usage filled in (trimmed here):

{
  "object": "chat.completion",
  "model": "llama3.2:1b",
  "choices": [{"index": 0, "finish_reason": "stop",
    "message": {"role": "assistant", "content": "Here are three HTTP methods:\n\n1. GET\n2. POST\n3. PUT"}}],
  "usage": {"prompt_tokens": 30, "completion_tokens": 18, "total_tokens": 48}
}

That is the useful part: your client code speaks one format, and the model behind it can be local today and hosted tomorrow. Rate limits, auth and logging go on the same route. Kong’s docs have the Llama 2 / Ollama-format example used here, and a dedicated Ollama provider example for releases that support it.

Clean up

docker rm -f kong          # if you tried the gateway
docker compose down        # keeps models and chats
docker compose down -v     # also deletes the volumes

Summary

• Pin the image tags and bind ports to 127.0.0.1; Ollama’s API is unauthenticated.
• You need the internet for pulls, not for inference.
• On a Mac, Docker means CPU. For speed, run Ollama natively and keep only Open WebUI in a container.
• One AI Proxy plugin turns the local model into an OpenAI-compatible endpoint, a cheap place to practise gateway policies before they touch a paid provider.