Skip to content
MiraiTech
All work

Case study · Tools

Offline coding assistant

An offline coding assistant for VS Code. It runs small open-weight models on the laptop itself, so it costs nothing per request and works without a network connection.

Year
2025
Duration
2.5 months
Type
Tools
Status
In production
Offline coding assistant, main screen

Results

  • 85% pass@1 on HumanEval
  • 120 tokens per second
  • Under 8 GB of RAM

The problem

Cloud coding assistants send proprietary code to outside servers, which breaks compliance rules for clients in healthcare, finance and defense. Large teams also pay $500+ a month in API costs.

What we built

We built an offline coding assistant that runs three small language models (Qwen 2.5, Llama 3.3 and Mistral Small) on the developer's laptop through Ollama. It benchmarks the models as it runs and sends each task to the best fit: Qwen for fast completions, Llama for harder reasoning, Mistral when memory is tight. There are no API costs, and no data leaves the machine.

Model comparison and routing
Live benchmarks of Qwen 2.5 (72B), Llama 3.3 (70B) and Mistral Small (24B) on speed, accuracy and memory use. Each task goes to the model that suits it best.
VS Code extension
An LSP-based extension for inline completions, code explanations, refactoring suggestions and unit test generation, all inside the editor.
Pull requests from the editor
Highlight buggy code and describe the fix. The assistant rewrites the code, writes tests and opens a PR without leaving VS Code.
Fully local inference
All inference runs locally through Ollama with CUDA or AVX2 acceleration. There are no network requests, telemetry or API keys, and code never leaves the developer's machine.

How it works

Everything runs on the developer's machine. A router picks the model, and the editor talks to it over LSP.

  1. 01

    Input

    VS Code extension sends the code context and the user's instruction over LSP

  2. 02

    Model selection

    Task classifier picks a model based on complexity, speed and available RAM

  3. 03

    Inference

    Ollama runs the chosen model (Qwen, Llama or Mistral) with CUDA or WebGPU acceleration

  4. 04

    Post-processing

    AST check confirms the generated code is syntactically valid before it is shown

  5. 05

    Delivery

    Results stream back to VS Code as inline completions, diff views or a new PR

Screens

Offline coding assistant, screen 1
1 / 2

Decisions

Ollama 0.3.8AI / ML
We used Ollama instead of llama.cpp directly for its model management and hot-swapping. Switching between Qwen, Llama and Mistral takes under 2 seconds, compared with 30+ seconds when loading models by hand.
TauriFrontend
For the desktop wrapper, Tauri gave us an 8MB binary where Electron needed 80MB (10x smaller), plus native WebView2 rendering. Next to a running LLM, every MB of RAM counts.
WebGPUInfrastructure
A fallback for machines without CUDA. It is 40% slower than CUDA, but it gives GPU acceleration on AMD and Intel GPUs that Ollama's default CUDA path does not support.
VS Code LSPFrontend
We built on the Language Server Protocol instead of VS Code's own extension API, so the same extension works unmodified in Cursor, Neovim and any other LSP-compatible editor.

What went wrong

Running out of RAM

Problem
A 70B parameter model running next to VS Code and a browser used 24GB+ of RAM. On 16GB developer machines that meant swap thrashing, and the tool was unusable on standard hardware.
Fix
The assistant checks free memory and picks a quantization level to match: Q4_K_M (4-bit) on 16GB machines, Q5_K_M on 32GB and Q8 on 64GB+. On long contexts, the KV cache is pruned aggressively.
Result
It runs well on 16GB MacBooks, with the model using under 8GB. Quantization costs 3% on HumanEval.

Slow completions

Problem
Completions first took 800ms+ because of model cold starts and tokenization overhead. Cloud copilots respond in 200-300ms, so ours felt sluggish.
Fix
We added speculative decoding. A small draft model (Mistral 7B) runs continuously and proposes likely completions, and the larger model checks them in a single forward pass, accepting 60-70% of the draft tokens.
Result
Median completion latency fell from 800ms to 280ms. In blind testing, users said it felt "as fast as GitHub Copilot".

Team and stack

Hiroshi built the VS Code extension and the Tauri desktop app. Kenta built model comparison and routing on top of Ollama. Hiroki built the performance benchmarking suite.

  • Ollama
  • PyTorch
  • Qwen 2.5
  • Llama 3.3
  • Mistral
  • Tauri
  • WebGPU
  • VS Code LSP