Case study · Tools
Pull request review agents
Three agents work through a GitHub pull request in turn. The planner writes a checklist, the analyzer looks for bugs with help from Semgrep and CodeQL, and the fixer proposes changes with unit tests.
- Year
- 2025
- Duration
- 2.5 months
- Type
- Tools
- Status
- In production

Results
- 87% bug catch rate
- 92% test coverage
- 7x faster than manual review
The problem
Senior engineers spend 8-12 hours a week reviewing PRs, and the average review takes 2-3 days to come back. Tired reviewers let serious bugs through, and junior developers wait days for feedback, which slows down how fast they learn.
What we built
We built a review system with 3 agents. A Planner decides what to check based on the PR's scope. An Analyzer looks for security vulnerabilities, performance problems and logic bugs using Claude plus static analysis tools. A Fixer writes corrected code with unit tests. Results are posted as GitHub PR comments, and some issues get a fix PR.
- Three review agents
- PlannerAgent writes a review checklist for the PR. AnalyzerAgent reports issues with severity and confidence scores. FixerAgent writes corrected code with unit tests. A LangGraph supervisor coordinates them.
- Security and static analysis
- Claude's review runs alongside Semgrep (OWASP patterns) and CodeQL (data flow analysis), which together catch both pattern-based and logic-based vulnerabilities.
- Fix PRs
- For high-confidence issues, the system opens a fix PR with the corrected code, unit tests (92% minimum coverage) and a commit message that explains the change. It can be merged in one click.
- Review history
- Every review is stored in Neon Postgres with its outcome: whether the suggestion was accepted and whether the fix was correct. We use that history to improve the reviews over time.
How it works
Three agents review each PR with help from static analysis, and fixes are generated for high-confidence issues.
- 01
PR webhook
GitHub webhook fires on PR open or update → fetch diff, file tree and commit history
- 02
Planning
PlannerAgent reads the PR scope → review checklist with focus areas and skip rules
- 03
Analysis
AnalyzerAgent runs Claude, Semgrep and CodeQL in parallel → ranked issue list with severity and confidence
- 04
Fix generation
FixerAgent fixes high-confidence issues → writes unit tests → validates with pytest (92% minimum coverage)
- 05
GitHub output
PR comments with inline annotations, fix PRs and an updated GitHub Checks status
Screens
Decisions
- LangGraph supervisorAI / ML
- Different PRs need different reviews, so we used a supervisor graph rather than a fixed sequence. A CSS-only PR skips security analysis, while a database migration gets extra scrutiny. The supervisor picks the route for each PR.
- Semgrep + CodeQLBackend
- The two tools catch different classes of vulnerability. Semgrep is good at pattern matching (SQL injection, XSS). CodeQL's data flow analysis finds injection chains that cross function boundaries.
- Claude 3.5 SonnetAI / ML
- Claude was better at understanding what the code was meant to do. GPT-4o produced more false positives (32% against Claude's 11%), and that much noise trains developers to ignore real issues.
- GitHub Checks APIInfrastructure
- We report through GitHub Checks so results sit with the rest of CI. Each review appears as a check run with line-level annotations, where developers already look.
What went wrong
Too many false positives
- Problem
- Early versions flagged 40+ issues per PR, and 35% of them were false positives. Developers started ignoring every suggestion, which defeated the point of automated review.
- Fix
- We calibrated confidence from historical feedback. Each suggestion gets a confidence score trained on what developers accepted and rejected. Only suggestions above 75% confidence are shown by default, and the rest sit in an expandable "lower confidence" section.
- Result
- The false positive rate fell from 35% to 8%. Developers engaged with 71% of suggestions, up from 23%.
Large PRs and the context window
- Problem
- PRs with 50+ changed files did not fit in Claude's context window. Truncating them made the analyzer miss cross-file issues, which are also the bugs people find hardest to catch.
- Fix
- Analysis now runs in two passes. The first reviews each file on its own. The second finds cross-file dependencies and reviews only the connected groups of files together, using AST-based dependency resolution to decide which files interact.
- Result
- Cross-file bug detection improved by 45%. Large PRs (100+ files) used to time out and now finish review in 90 seconds.
Team and stack
Hiroshi designed the review pipeline and built the GitHub integration. Kenta built the 3 LangGraph agents and wired in Semgrep and CodeQL. Hiroki built the review history in Neon Postgres and the feedback data behind the confidence scores.
- LangGraph
- Claude 3.5 Sonnet
- Semgrep
- CodeQL
- GitHub API
- Pytest
- Neon Postgres