· 7 min read
Our local LLM setup
Why and how we run models locally: Ollama, GPU memory, wiring local models into the dev stack and tests, and what it cost over six months.
Why we run models locally
Cloud LLM APIs work well until they don't: rate limits in the middle of a sprint, usage costs that balloon while you iterate, latency spikes at peak hours and having to send proprietary code to a third party. We run local models for development, testing and prototyping, and use cloud APIs for production inference. That lets us iterate quickly without the meter running.
The economics are simple. We were spending roughly $1,200/month on API calls during development. A one-time $800 GPU upgrade cut about 80% of that, so the setup paid for itself in under a month.
Ollama
Ollama handles the painful parts of running models locally: model formats, quantization, GPU memory management. Install it, pull a model, and you have an OpenAI-compatible API running on localhost.
# Install Ollama (macOS/Linux)
curl -fsSL https://ollama.com/install.sh | sh
# Pull models for different use cases
ollama pull llama3.1:8b # Fast general purpose, ~4.7GB
ollama pull codellama:13b # Code generation, ~7.4GB
ollama pull deepseek-coder-v2:16b # Best code quality, ~8.9GB
ollama pull nomic-embed-text # Embeddings for RAG, ~274MB
# Verify everything is running
ollama list
curl http://localhost:11434/api/tagsGPU configuration and memory management
The biggest factor in local LLM performance is whether the model fits in GPU memory. With partial offloading, where some layers run on the GPU and the rest on the CPU, performance falls off a cliff and the model feels unusable.
Memory requirements by model size
| Model size | Quantization | VRAM required | Tokens/sec (RTX 4070) | Tokens/sec (M2 Pro) |
|---|---|---|---|---|
| 7-8B | Q4_K_M | ~4.5 GB | ~65 | ~25 |
| 13B | Q4_K_M | ~7.5 GB | ~35 | ~14 |
| 13B | Q8_0 | ~13 GB | ~30 | ~11 |
| 34B | Q4_K_M | ~19 GB | ~12 | ~5 |
| 70B | Q4_K_M | ~38 GB | N/A | N/A |
For development work such as autocomplete, code review and test generation, we target 30+ tokens per second. Below that, the latency breaks flow. In practice that makes 13B models our ceiling on a single consumer GPU.
Tuning Ollama for development
Ollama's defaults are conservative. On a dedicated development machine we override several of them.
# Set environment variables before starting Ollama
# On Linux, add these to /etc/systemd/system/ollama.service.d/override.conf
# On macOS, use launchctl setenv
# Keep models loaded in memory longer (default is 5 minutes)
export OLLAMA_KEEP_ALIVE="30m"
# Allow parallel requests (useful for editor + terminal simultaneously)
export OLLAMA_NUM_PARALLEL=2
# Restrict to specific GPU if you have multiple
export CUDA_VISIBLE_DEVICES=0
# Increase context window for larger code files
# Note: larger context = more VRAM usage
export OLLAMA_MAX_LOADED_MODELS=2Integrating with the dev stack
A local model is only useful if it's wired into the tools you already use. We integrate it at three levels: editor, CLI and application.
Editor
Most AI-powered editors accept a custom endpoint. Point one at Ollama's OpenAI-compatible API and you get local inference with the same UX.
{
"continue.models": [
{
"title": "Local CodeLlama",
"provider": "ollama",
"model": "codellama:13b",
"apiBase": "http://localhost:11434"
},
{
"title": "Local Llama 3.1",
"provider": "ollama",
"model": "llama3.1:8b",
"apiBase": "http://localhost:11434"
}
]
}Application code
For applications that use the OpenAI SDK, switching to a local model means changing two lines. We toggle between local and cloud with an environment variable.
import OpenAI from "openai";
const isLocal = process.env.AI_PROVIDER === "local";
export const ai = new OpenAI({
baseURL: isLocal
? "http://localhost:11434/v1"
: "https://api.openai.com/v1",
apiKey: isLocal ? "ollama" : process.env.OPENAI_API_KEY!,
});
// Usage is identical regardless of provider
export async function generateSummary(content: string): Promise<string> {
const response = await ai.chat.completions.create({
model: isLocal ? "llama3.1:8b" : "gpt-4o-mini",
messages: [
{
role: "system",
content: "Summarize the following content in 2-3 sentences.",
},
{ role: "user", content },
],
temperature: 0.3,
max_tokens: 200,
});
return response.choices[0].message.content ?? "";
}Testing against local models
Before promoting any prompt change to production, we run a small evaluation suite against local models. It catches regressions without burning API credits.
import { describe, it, expect } from "vitest";
import { ai } from "@/lib/ai-client";
const MODEL = process.env.AI_MODEL ?? "llama3.1:8b";
describe("prompt evaluation suite", () => {
it("should extract structured data from unstructured text", async () => {
const response = await ai.chat.completions.create({
model: MODEL,
messages: [
{
role: "system",
content: `Extract contact info as JSON: { "name": string, "email": string, "company": string | null }`,
},
{
role: "user",
content: "Hi, I'm Sarah Chen from Acme Corp. Reach me at sarah@acme.co",
},
],
temperature: 0,
response_format: { type: "json_object" },
});
const parsed = JSON.parse(response.choices[0].message.content ?? "{}");
expect(parsed).toHaveProperty("name");
expect(parsed).toHaveProperty("email");
expect(parsed.email).toMatch(/@/);
}, 30_000);
it("should respect output length constraints", async () => {
const response = await ai.chat.completions.create({
model: MODEL,
messages: [
{
role: "user",
content: "Explain TCP/IP in exactly one sentence.",
},
],
temperature: 0,
max_tokens: 100,
});
const text = response.choices[0].message.content ?? "";
const sentences = text.split(/[.!?]+/).filter(Boolean);
expect(sentences.length).toBeLessThanOrEqual(2);
}, 30_000);
});A local RAG pipeline
One of the most useful things we run locally is a RAG pipeline over our internal documentation. We index our codebase, architecture docs and runbooks, then query them in plain language during development.
import { readdir, readFile } from "fs/promises";
import { join } from "path";
const OLLAMA_BASE = "http://localhost:11434";
async function getEmbedding(text: string): Promise<number[]> {
const response = await fetch(`${OLLAMA_BASE}/api/embeddings`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
model: "nomic-embed-text",
prompt: text,
}),
});
const data = await response.json();
return data.embedding;
}
async function indexDirectory(dir: string) {
const files = await readdir(dir, { recursive: true });
const documents: Array<{ path: string; content: string; embedding: number[] }> = [];
for (const file of files) {
if (!file.endsWith(".md") && !file.endsWith(".mdx")) continue;
const content = await readFile(join(dir, file), "utf-8");
const chunks = splitIntoChunks(content, 500);
for (const chunk of chunks) {
const embedding = await getEmbedding(chunk);
documents.push({ path: file, content: chunk, embedding });
}
}
return documents;
}
function splitIntoChunks(text: string, maxTokens: number): string[] {
const paragraphs = text.split(/\n\n+/);
const chunks: string[] = [];
let current = "";
for (const para of paragraphs) {
if ((current + para).length > maxTokens * 4) {
if (current) chunks.push(current.trim());
current = para;
} else {
current += "\n\n" + para;
}
}
if (current.trim()) chunks.push(current.trim());
return chunks;
}Our workflow
After months of tweaking, our local LLM workflow looks like this:
Pull models on machine setup
Run the setup script once per machine. Models are stored in ~/.ollama/models and shared across projects. We pin model names in a .ollama-models file in each repo so everyone runs the same versions.
Start Ollama alongside your dev server
Ollama runs as a background service, with no containers or orchestration. It starts with the OS and listens on port 11434.
Develop and test locally, ship on the cloud
Set AI_PROVIDER=local in .env.development and all AI features route to Ollama. When a feature is ready, run the eval suite against the production model to check the prompts still work, then deploy.
Costs after six months
| Category | Cloud only (before) | Hybrid local/cloud (after) |
|---|---|---|
| Monthly API costs | $1,200 | $280 |
| Development iteration speed | Bounded by rate limits | Unbounded |
| Offline development | Impossible | Fully functional |
| Data privacy concerns | Required legal review | None for dev data |
| Hardware investment | $0 | $800 one-time |
| Break-even point | n/a | 1 month |
Limitations
Local models are not a substitute for frontier cloud models, and we use the two for different work. Local is good at autocomplete, boilerplate generation, test writing, documentation drafts, embeddings and RAG retrieval. Cloud is better at complex reasoning, multi-step planning, novel architecture suggestions and long-context analysis. Neither replaces human code review, architecture decisions or security audits.
Written by the MiraiTech team in Tokyo
If something here is wrong, or you are working on a similar problem, write to us at contact@mirai-tech.net.