Case study · Tools
Offline coding assistant
An offline coding assistant for VS Code. It runs small open-weight models on the laptop itself, so it costs nothing per request and works without a network connection.
- Year
- 2025
- Duration
- 2.5 months
- Type
- Tools
- Status
- In production

Results
- 85% pass@1 on HumanEval
- 120 tokens per second
- Under 8 GB of RAM
The problem
Cloud coding assistants send proprietary code to outside servers, which breaks compliance rules for clients in healthcare, finance and defense. Large teams also pay $500+ a month in API costs.
What we built
We built an offline coding assistant that runs three small language models (Qwen 2.5, Llama 3.3 and Mistral Small) on the developer's laptop through Ollama. It benchmarks the models as it runs and sends each task to the best fit: Qwen for fast completions, Llama for harder reasoning, Mistral when memory is tight. There are no API costs, and no data leaves the machine.
- Model comparison and routing
- Live benchmarks of Qwen 2.5 (72B), Llama 3.3 (70B) and Mistral Small (24B) on speed, accuracy and memory use. Each task goes to the model that suits it best.
- VS Code extension
- An LSP-based extension for inline completions, code explanations, refactoring suggestions and unit test generation, all inside the editor.
- Pull requests from the editor
- Highlight buggy code and describe the fix. The assistant rewrites the code, writes tests and opens a PR without leaving VS Code.
- Fully local inference
- All inference runs locally through Ollama with CUDA or AVX2 acceleration. There are no network requests, telemetry or API keys, and code never leaves the developer's machine.
How it works
Everything runs on the developer's machine. A router picks the model, and the editor talks to it over LSP.
- 01
Input
VS Code extension sends the code context and the user's instruction over LSP
- 02
Model selection
Task classifier picks a model based on complexity, speed and available RAM
- 03
Inference
Ollama runs the chosen model (Qwen, Llama or Mistral) with CUDA or WebGPU acceleration
- 04
Post-processing
AST check confirms the generated code is syntactically valid before it is shown
- 05
Delivery
Results stream back to VS Code as inline completions, diff views or a new PR
Screens
Decisions
- Ollama 0.3.8AI / ML
- We used Ollama instead of llama.cpp directly for its model management and hot-swapping. Switching between Qwen, Llama and Mistral takes under 2 seconds, compared with 30+ seconds when loading models by hand.
- TauriFrontend
- For the desktop wrapper, Tauri gave us an 8MB binary where Electron needed 80MB (10x smaller), plus native WebView2 rendering. Next to a running LLM, every MB of RAM counts.
- WebGPUInfrastructure
- A fallback for machines without CUDA. It is 40% slower than CUDA, but it gives GPU acceleration on AMD and Intel GPUs that Ollama's default CUDA path does not support.
- VS Code LSPFrontend
- We built on the Language Server Protocol instead of VS Code's own extension API, so the same extension works unmodified in Cursor, Neovim and any other LSP-compatible editor.
What went wrong
Running out of RAM
- Problem
- A 70B parameter model running next to VS Code and a browser used 24GB+ of RAM. On 16GB developer machines that meant swap thrashing, and the tool was unusable on standard hardware.
- Fix
- The assistant checks free memory and picks a quantization level to match: Q4_K_M (4-bit) on 16GB machines, Q5_K_M on 32GB and Q8 on 64GB+. On long contexts, the KV cache is pruned aggressively.
- Result
- It runs well on 16GB MacBooks, with the model using under 8GB. Quantization costs 3% on HumanEval.
Slow completions
- Problem
- Completions first took 800ms+ because of model cold starts and tokenization overhead. Cloud copilots respond in 200-300ms, so ours felt sluggish.
- Fix
- We added speculative decoding. A small draft model (Mistral 7B) runs continuously and proposes likely completions, and the larger model checks them in a single forward pass, accepting 60-70% of the draft tokens.
- Result
- Median completion latency fell from 800ms to 280ms. In blind testing, users said it felt "as fast as GitHub Copilot".
Team and stack
Hiroshi built the VS Code extension and the Tauri desktop app. Kenta built model comparison and routing on top of Ollama. Hiroki built the performance benchmarking suite.
- Ollama
- PyTorch
- Qwen 2.5
- Llama 3.3
- Mistral
- Tauri
- WebGPU
- VS Code LSP