Replacing a self-hosted PR review bot with Claude Code
Cut PR review latency from ~20 minutes to under 3, decommissioned the GPU box, and gave engineers back the time they were spending babysitting AI infrastructure.
Context
The original pipeline had been built when running an open-weights model locally was the only practical option for a team that didn't want code leaving their infrastructure. Two years later, the constraints had shifted — vendor APIs were defensible, the open-weights gap had widened, and the GPU box had become a maintenance tax that nobody on the team was excited to own.
Problem
Three concrete pain points. First, the open-weights model couldn't reason across files in a multi-file PR — most real review value is exactly there. Second, latency depended on whoever was hitting the inference box at the same time, which meant reviews that landed 20 minutes after a PR was opened (too late). Third, the prompt scaffolding lived in a Python service, a Slack bot wrapper, and a config repo — adding a single rule meant touching three places.
Approach
The key choice was using Claude Code as an agent rather than a single API call — this lets the review reason across the diff, related files, and the PR description in one pass, which is what the previous setup couldn't do. Prompt scaffolding consolidated to a single YAML config in the repo. Review rubrics were tuned per directory (the API code gets stricter checks than the data-pipeline code). Findings post as inline PR comments because Slack DMs were getting ignored.
Outcome
Reviews now land within ~3 minutes of PR open consistently. The most concrete quality improvement was cross-file reasoning — the new setup catches issues like 'this change to the validator breaks the assumption in handler.go:42', which the previous setup had no way of noticing. The Ollama box is decommissioned. Engineers extend the rubric directly via PRs to the YAML config.
What I learned
Two takeaways worth surfacing. First, the rubric matters more than the model — a thoughtful rubric on a mid-tier model beats a vague rubric on the best available model, every time. Second, posting back as PR comments (not DMs) was the single biggest behavioural change: the review feels like part of the PR, not a notification to ignore.