Experiment · LLM
Local LLM code reviewer
A VS Code extension that reviews diffs with models served by Ollama on the laptop. It points out likely bugs, suggests changes and leaves inline comments without sending code anywhere.
- Type
- LLM
- Status
- Live
- Models
- Qwen 2.5 72B, Llama 3.3 70B, CodeLlama
Results
- 85% bug detection rate
- 2.9 s average response
- No API cost
On the 200 test PRs: 85% bug detection recall, a 16% false positive rate, 2.9s average response time and no API cost. In a survey of 12 developers, the tool scored 4.1/5 for usefulness. Three engineers adopted it as their default pre-commit review step.
What we tested
We wanted to know whether open-weight models running locally could review code about as well as cloud APIs, without the source ever leaving the machine. The reviewer is a VS Code extension backed by Ollama. It parses diffs into Tree-sitter ASTs, gives the model structured context and shows the review as inline comments.
Method
We compared three local models (Qwen 2.5 72B Q4, Llama 3.3 70B Q5 and CodeLlama 34B) with GPT-4o on 200 real pull requests in Python, TypeScript and Go. A senior engineer reviewed each PR independently to set the ground truth. We measured bug detection recall, false positive rate, suggestion quality (rated 1-5 by the engineer) and inference latency. To build the context, Tree-sitter extracted each changed function with its surrounding scope and imports, up to 4,096 tokens.
Findings
- 01
Qwen 2.5 72B came close to GPT-4o
On the 200-PR benchmark, Qwen 2.5 72B reached 85% bug detection recall and GPT-4o reached 89%. That is a smaller gap than we expected for a model running locally on an RTX 4090.
- 02
AST context halved false positives
Moving from raw diff prompts to function context extracted with Tree-sitter cut the false positive rate from 34% to 16%, because the model could see type signatures and imports.
- 03
Context mattered more than quantization
Going from Q4 to Q8 changed recall by only 2%. Better context construction improved it by 12%. Engineering time is better spent on the prompt context than on model precision.
- 04
Under 3 seconds, fast enough to use while coding
Average response time was 2.9s on an RTX 4090. That is fast enough to run a review while writing code, as well as when a PR is opened.
How it is built
A review starts when a developer saves a file or stages a diff. The extension finds the changed hunks and sends them to the local FastAPI server, which uses Tree-sitter to extract the full function around each change. That context is formatted into a review prompt and sent to Ollama. The response is parsed into issues (line number, severity, suggestion) and shown as inline diagnostics with a one-click "Apply Fix" code action.
Ollama serves the quantized models with GPU offloading. Tree-sitter parses source files into ASTs so the right context can be extracted. A local FastAPI server connects the VS Code extension to the model server. The extension uses LSP for inline diagnostics and code actions.
- Ollama
- Python
- AST
- FastAPI
- Tree-sitter
Problems we hit
Large diffs overflowed the context
PRs touching 10+ files went over the 4,096-token context budget. We rank the changed functions by complexity, review the top k and add a note that more issues may exist.
Wrong line numbers
The local models sometimes attached a suggestion to the wrong line. We added a post-processing step that checks each suggested line against the actual diff and drops annotations that do not match.