# The Ultimate Offline Local AI Handbook 2026

No cloud. No API keys. No rate limits. No token billing. This handbook is deliberately limited to the 15 requested models and local runtimes.

## Verification policy
Official Ollama pages are the source of truth for pull/run commands. If a requested name does not match an official tag, the handbook says so instead of fabricating a command. Hardware estimates are practical planning ranges, not guarantees; measure tokens/sec on your own machine.

## Model index

| Tier | Model | Ollama status |
|---|---|---|
| S+ | [Qwen3-Coder 30B](models/qwen3-coder-30b/README.md) | Official Ollama release verified |
| S+ | [Qwen2.5-Coder 32B](models/qwen25-coder-32b/README.md) | Official Ollama release verified |
| S+ | [Devstral 24B](models/devstral-24b/README.md) | Official Ollama release verified |
| S+ | [Codestral 22B](models/codestral-22b/README.md) | Official Ollama release verified |
| S | [Gemma 4 27B](models/gemma4-27b/README.md) | Official Ollama family verified; exact requested 27B tag is not listed. Use gemma4:26b. |
| S | [DeepSeek Coder V2 Lite](models/deepseek-coder-v2-lite/README.md) | Official Ollama release verified |
| S | [Qwen3-Coder 14B](models/qwen3-coder-14b/README.md) | Official local release pending |
| S | [DeepSeek-R1 Distill Qwen 32B](models/deepseek-r1-distill-qwen-32b/README.md) | Official Ollama release verified |
| A+ | [Llama 3.3 70B](models/llama33-70b/README.md) | Official Ollama release verified |
| A+ | [Mistral Small 24B](models/mistral-small-24b/README.md) | Official Ollama release verified |
| A+ | [Qwen2.5 14B Instruct](models/qwen25-14b-instruct/README.md) | Official Ollama release verified |
| A+ | [DeepSeek-R1 Distill Llama 8B](models/deepseek-r1-distill-llama-8b/README.md) | Official Ollama release verified |
| A | [Llama 3.1 8B Instruct](models/llama31-8b/README.md) | Official Ollama release verified |
| A | [Gemma 2 9B](models/gemma2-9b/README.md) | Official Ollama release verified |
| A | [Phi-4](models/phi4-14b/README.md) | Official Ollama release verified |

## Installation: Windows, Linux, macOS

1. Install Ollama from the official installer for your operating system and confirm `ollama --version`.
2. Start the local service; Ollama listens on `http://localhost:11434`.
3. For an officially verified tag, run the model folder command. For pending models, use only the upstream Hugging Face card and a verified GGUF/llama.cpp conversion.
4. Keep models on a fast SSD and set a context length your RAM/VRAM can sustain.

### Windows
Use PowerShell. Install the NVIDIA driver before Ollama if using CUDA. Store models on a dedicated NTFS SSD using Ollama's supported model-location setting; avoid syncing model blobs to cloud drives.

### Linux
Use the official Ollama installer or package, verify `systemctl`/service status, and install the matching NVIDIA or ROCm driver if applicable. For llama.cpp, build with the backend for your GPU and test with a small prompt before loading a large context.

### macOS
Use the Ollama macOS app or CLI. Apple Silicon uses Metal and unified memory; leave memory for the OS and editor. MLX is an alternative for models with a verified MLX conversion.

## Hardware matrix

| System memory | Best fit | Practical recommendation |
|---|---|---|
| 16 GB RAM | 8-14B Q4 | Llama 3.1 8B, Gemma 2 9B, Phi-4, Qwen2.5 14B with short context |
| 32 GB RAM | 14-32B Q4 | Devstral, Codestral, Mistral Small, Qwen2.5-Coder 32B; use GPU offload where available |
| 64 GB RAM | 32-70B Q4/Q5 | DeepSeek-R1 Qwen 32B, Llama 3.3 70B with CPU offload or large GPU |
| 128 GB RAM | 70B Q5/Q6 | Llama 3.3 70B and large-context workflows with room for tooling |

### GPU guide

| GPU | Recommended local use |
|---|---|
| RTX 3050 | 8B Q4; partial offload for 14B |
| RTX 3060 | 8-14B Q4/Q5; partial offload for 22-24B |
| RTX 4060 | 8-14B Q4; efficient laptop/desktop assistant |
| RTX 4070 | 14-24B Q4 with careful context |
| RTX 4080 | 22-32B Q4/Q5; strong coding workstation |
| RTX 4090 | 24-32B Q4/Q5, Devstral, Qwen3-Coder 30B; 70B requires offload or multiple GPUs |

## Troubleshooting

- **Model not found:** run `ollama list`, then use the exact official tag in the model folder.
- **Out of memory:** reduce `num_ctx`, choose Q4, close GPU-heavy apps, or use CPU offload.
- **Slow responses:** check GPU layers, thermal throttling, background indexing, and SSD health.
- **Bad tool calls:** enforce a JSON schema in the host application and keep temperature low.
- **Context truncation:** lower prompt size or raise context only after memory testing.
- **Windows CUDA issue:** update the NVIDIA driver and restart Ollama; confirm the GPU is visible in Task Manager.
- **macOS memory pressure:** choose a smaller quant, reduce context, and leave 6-10GB unified memory free.
- **Privacy check:** disconnect network if required, disable web/search tools, and inspect the host app's logs for outbound calls.

## Benchmark protocol

Run the same prompt set locally: 10 code-generation tasks, 10 repair tasks, 5 tool-call JSON tasks, and 5 reasoning tasks. Record model tag, quant, context, GPU, tokens/sec, first-token latency, pass rate, and peak memory. Never compare results across different prompts or sampling settings.

## Sources

- Ollama Library: https://ollama.com/library
- llama.cpp: https://github.com/ggml-org/llama.cpp
- vLLM supported models: https://docs.vllm.ai/en/latest/models/supported_models.html
- MLX: https://github.com/ml-explore/mlx