Skip to content
MiraiTech
All work

Case study · Tools

Pull request review agents

Three agents work through a GitHub pull request in turn. The planner writes a checklist, the analyzer looks for bugs with help from Semgrep and CodeQL, and the fixer proposes changes with unit tests.

Year
2025
Duration
2.5 months
Type
Tools
Status
In production
Pull request review agents, main screen

Results

  • 87% bug catch rate
  • 92% test coverage
  • 7x faster than manual review

The problem

Senior engineers spend 8-12 hours a week reviewing PRs, and the average review takes 2-3 days to come back. Tired reviewers let serious bugs through, and junior developers wait days for feedback, which slows down how fast they learn.

What we built

We built a review system with 3 agents. A Planner decides what to check based on the PR's scope. An Analyzer looks for security vulnerabilities, performance problems and logic bugs using Claude plus static analysis tools. A Fixer writes corrected code with unit tests. Results are posted as GitHub PR comments, and some issues get a fix PR.

Three review agents
PlannerAgent writes a review checklist for the PR. AnalyzerAgent reports issues with severity and confidence scores. FixerAgent writes corrected code with unit tests. A LangGraph supervisor coordinates them.
Security and static analysis
Claude's review runs alongside Semgrep (OWASP patterns) and CodeQL (data flow analysis), which together catch both pattern-based and logic-based vulnerabilities.
Fix PRs
For high-confidence issues, the system opens a fix PR with the corrected code, unit tests (92% minimum coverage) and a commit message that explains the change. It can be merged in one click.
Review history
Every review is stored in Neon Postgres with its outcome: whether the suggestion was accepted and whether the fix was correct. We use that history to improve the reviews over time.

How it works

Three agents review each PR with help from static analysis, and fixes are generated for high-confidence issues.

  1. 01

    PR webhook

    GitHub webhook fires on PR open or update → fetch diff, file tree and commit history

  2. 02

    Planning

    PlannerAgent reads the PR scope → review checklist with focus areas and skip rules

  3. 03

    Analysis

    AnalyzerAgent runs Claude, Semgrep and CodeQL in parallel → ranked issue list with severity and confidence

  4. 04

    Fix generation

    FixerAgent fixes high-confidence issues → writes unit tests → validates with pytest (92% minimum coverage)

  5. 05

    GitHub output

    PR comments with inline annotations, fix PRs and an updated GitHub Checks status

Screens

Pull request review agents, screen 1
1 / 2

Decisions

LangGraph supervisorAI / ML
Different PRs need different reviews, so we used a supervisor graph rather than a fixed sequence. A CSS-only PR skips security analysis, while a database migration gets extra scrutiny. The supervisor picks the route for each PR.
Semgrep + CodeQLBackend
The two tools catch different classes of vulnerability. Semgrep is good at pattern matching (SQL injection, XSS). CodeQL's data flow analysis finds injection chains that cross function boundaries.
Claude 3.5 SonnetAI / ML
Claude was better at understanding what the code was meant to do. GPT-4o produced more false positives (32% against Claude's 11%), and that much noise trains developers to ignore real issues.
GitHub Checks APIInfrastructure
We report through GitHub Checks so results sit with the rest of CI. Each review appears as a check run with line-level annotations, where developers already look.

What went wrong

Too many false positives

Problem
Early versions flagged 40+ issues per PR, and 35% of them were false positives. Developers started ignoring every suggestion, which defeated the point of automated review.
Fix
We calibrated confidence from historical feedback. Each suggestion gets a confidence score trained on what developers accepted and rejected. Only suggestions above 75% confidence are shown by default, and the rest sit in an expandable "lower confidence" section.
Result
The false positive rate fell from 35% to 8%. Developers engaged with 71% of suggestions, up from 23%.

Large PRs and the context window

Problem
PRs with 50+ changed files did not fit in Claude's context window. Truncating them made the analyzer miss cross-file issues, which are also the bugs people find hardest to catch.
Fix
Analysis now runs in two passes. The first reviews each file on its own. The second finds cross-file dependencies and reviews only the connected groups of files together, using AST-based dependency resolution to decide which files interact.
Result
Cross-file bug detection improved by 45%. Large PRs (100+ files) used to time out and now finish review in 90 seconds.

Team and stack

Hiroshi designed the review pipeline and built the GitHub integration. Kenta built the 3 LangGraph agents and wired in Semgrep and CodeQL. Hiroki built the review history in Neon Postgres and the feedback data behind the confidence scores.

  • LangGraph
  • Claude 3.5 Sonnet
  • Semgrep
  • CodeQL
  • GitHub API
  • Pytest
  • Neon Postgres