Ollama Quickstart: Run Local LLMs on Your Budget Build (2026)

You want a large language model that runs on your hardware, on your network, with no API bill and no data leaving the house. Ollama is the fastest way to get there: one command installs it, one more pulls a model, and you have a working local LLM in about five minutes. This guide assumes a budget, VRAM-first build (like the one we spec out here) and walks you from empty machine to a model you can actually query.

What is Ollama?

Ollama is a lightweight runtime that downloads, manages, and serves large language models locally. It ships with the model files in GGUF-style quantized form, handles the inference engine for you, and exposes each model both through a simple CLI and through an HTTP API. That API is the key: it speaks the same protocol as OpenAI, so any tool that can talk to OpenAI can talk to your local box instead.

The hardware you actually need

For inference, VRAM capacity matters far more than raw speed. A model that does not fit in memory gets swapped to system RAM and collapses to a few tokens per second. So pick your model to fit your card, not the other way around. Approximate VRAM needed for Q4-quantized models:

If you are building from scratch for this purpose, a used 24GB RTX 3090 is the budget king: it runs 32B models at genuinely usable speed and can stretch to 70B with partial offload. The full reasoning (and why VRAM beats CUDA cores) is in the VRAM-first build guide.

Install Ollama

Linux & macOS

curl -fsSL https://ollama.com/install.sh | sh

The script detects your GPU (NVIDIA CUDA or AMD ROCm) and installs the Ollama server, which starts automatically in the background. Verify it is running:

ollama --version

Windows

Download the installer from ollama.com/download, or via Winget:

winget install Ollama.Ollama

Ollama runs in the system tray. On Windows, WSL2 also works well if you prefer a Linux toolchain.

Pull and run your first model

Start small to confirm the whole stack works before you pull something big. A 7B–8B model is the right first target: fast download, runs on almost anything, and good enough to judge whether your setup is healthy.

ollama run qwen2.5:7b

The first run downloads the model (a few GB), then drops you into an interactive prompt. Type a message, press Enter, and you should get a streamed response. When it works, exit with /bye or Ctrl+D.

The OpenAI-compatible API (the important part)

Ollama serves an HTTP API at http://localhost:11434, and under /v1 it speaks the OpenAI chat-completions protocol. This is what lets you point any OpenAI-compatible tool — OpenClaw, your own scripts, other agents — at your local model instead of a cloud API. Test it directly:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen2.5:7b","messages":[{"role":"user","content":"Say hi in one line."}]}'

If that returns a JSON completion, your local model is now a drop-in replacement for any hosted endpoint. Set the base URL to http://localhost:11434/v1 in whatever tool you are wiring up; Ollama does not require an API key, so any non-empty placeholder works.

Useful commands & config

# list models on this machine
ollama list

# remove a model to free space
ollama rm qwen2.5:7b

# start the server manually (if not auto-starting)
ollama serve

# run a specific model non-interactively
ollama run llama3.1:8b "Summarize this: ..."

# show where models are stored
OLLAMA_MODELS  # env var; default ~/.ollama/models

Two environment variables matter most. OLLAMA_HOST controls the bind address (default 127.0.0.1, i.e. local-only — leave it that way unless you have a reason to expose it). OLLAMA_MODELS lets you point model storage at a fast SSD if your default disk is slow.

Troubleshooting

• "model not found" / download stalls — the first pull can be several GB. Confirm you have the free space, and retry; Ollama resumes partial downloads.
• Very slow after the first token — the model is larger than your VRAM and is swapping to RAM. Drop to a smaller size tag (e.g. :7b instead of :32b) or a lower quant.
• CUDA not detected on Linux &mdash> Ollama needs the NVIDIA driver, not just CUDA libs. Run nvidia-smi; if it errors, fix the driver first, then restart Ollama.
• "address already in use" on 11434 — another Ollama (or tool) is bound to the port. Find and stop it, or set OLLAMA_HOST=127.0.0.1:11435.
• API returns nothing from another machine — by design Ollama binds to localhost. That is not a bug; see the security note below before exposing it.

Security & privacy

The whole point of self-hosting is that your data stays local, and Ollama respects that by default: it listens on 127.0.0.1 only. Keep it there. If you genuinely need another machine on your LAN to call the model, bind to a specific interface and put authentication in front of it — do not expose port 11434 to the internet, because Ollama has no built-in auth and anyone who reaches it can run your (paid-for) GPU.

FAQ

• Do I need a NVIDIA card? No. AMD (ROCm) and Apple Silicon work; Intel Arc is supported in recent builds. But NVIDIA has the smoothest experience.
• How big are these models? A 7B–8B Q4 model is ~5GB; a 32B is ~20GB. Check your free disk before pulling.
• Can I run more than one model? Yes — pull several and switch with ollama run <name>. Only the active one holds VRAM at a time.
• Is this free? The runtime is open source (MIT) and the models are free to download. Your only cost is the electricity and the hardware you already own.

Some links above are affiliate links. If you buy through them, ghettoboxai.com may earn a commission at no extra cost to you.