LLM Pull Request Code Review Assistant

Project overview

We built an AI code review assistant for a software company whose engineering teams merge a steady stream of pull requests across many repositories. The assistant reads each diff in the context of the surrounding code, posts inline comments on security weaknesses, style violations and suspicious logic, and summarises the change for the human reviewer.

The engagement covered discovery with the platform team, prompt and retrieval design, the Python review service, a TypeScript integration layer and dashboard, and a gradual roll-out repository by repository. It is part of our wider generative AI development practice, where we put language models inside existing engineering workflows rather than beside them.

The client and the challenge

Senior engineers were the bottleneck. Every pull request waited for one of a small group of reviewers, who spent much of their time pointing out the same issues: unescaped input, secrets in config files, missing error handling and departures from the house style guide. Important design feedback got less attention because routine comments crowded it out.

The client wanted automated first-pass review that understood their own conventions, did not leak proprietary source code, kept noise low enough that developers would not simply ignore it, and never approved or merged anything on its own.

Solution architecture & delivery

A webhook from the Git host fires when a pull request is opened or updated. The TypeScript integration service fetches the diff, the changed files and the relevant parts of the repository, then queues a review job. The Python review service splits large diffs into logical hunks so each model call sees a focused change plus the functions it touches, instead of one oversized prompt.

Each hunk goes through three passes. Deterministic linters and a secret scanner run first, because rules that can be checked mechanically should not cost a model call. Next, an OpenAI model receives the hunk, the team's style guide and a short set of retrieved examples of past accepted comments, and returns structured findings with a category, severity, line reference and suggested fix. A final filtering step drops duplicates, merges findings on the same lines and suppresses anything below the repository's severity threshold.

Findings are posted as inline comments with a clear label that they come from the assistant, plus a summary comment describing what the change does and which areas deserve human attention. Developers can react to a comment to mark it helpful or wrong; that feedback is stored and used to tune prompts and the retrieved examples, following the evaluation approach in our guide to AI agent testing and evaluation.

For confidentiality, only the diff and the minimum surrounding context leave the client's network, requests go through an account configured not to retain data for training, and repositories marked sensitive can be excluded entirely. The assistant has read and comment permissions only; merge rights stay with people, a pattern we describe in tool calling with human approval.

Key features we built

• Automatic review on every opened or updated pull request
• Inline comments for security, style, error handling and logic issues
• Plain-language summary of each change for the human reviewer
• Repository-specific style guides and severity thresholds
• Linter and secret-scanner pass before any model call
• Helpful/wrong feedback on each comment used to tune prompts
• Dashboard of findings by repository, category and team
• Command-line tool to run the same review locally before pushing

How we delivered it

1. Discovery. Interviews with reviewers and a study of historical review comments to find the recurring issue types worth automating and the ones that needed human judgement.
2. Evaluation set. A labelled set of past pull requests with known issues, used to compare prompt designs and context strategies before anything touched live repositories.
3. Build. Two-week sprints delivering the webhook integration, review service, filtering rules and dashboard, with a shadow mode that generated comments privately for comparison.
4. Pilot. Live comments switched on for a few volunteer repositories, with weekly sessions to review false positives and adjust thresholds.
5. Roll-out and support. Onboarding of remaining repositories, model version updates and prompt maintenance under a support agreement.

Technology choices

• Python — hosts the review pipeline, diff chunking, linter orchestration, filtering logic and the offline evaluation harness.
• TypeScript — used for the Git host webhook integration, the command-line tool and the review dashboard, sharing typed finding schemas across all three.
• OpenAI — the language models that read code hunks and return structured findings, called with strict output schemas so results can be validated before posting.

Outcomes

• Routine style and security comments arrive automatically, so senior reviewers spend their time on design and architecture
• Developers get feedback on an open pull request before a human reviewer picks it up
• Secrets and unsafe input handling are flagged consistently across every repository
• Review standards are written down as style guides the assistant applies, rather than living in individual reviewers' heads
• Engineering leads can see which issue types recur by team and target training accordingly

FAQ