If the agent isn't using my diagnostics library, I could introspect and update AGENTS.md, but there's still a delta, and it'll drift. Instead, I write a pre-commit hook that checks the git diff for direct calls to observability modules that bypass my diagnostics module. Boom: fail the commit, inject a prompt telling the agent to use the diagnostics library. All of a sudden it's ten out of ten.
— Geoffrey Huntley, nobody should type code by hand anymore. here's what's left.
I keep hearing complaints about code review. About volume, about quality, about trying to take responsibility for code produced by colleagues who have abdicated their responsibility for thinking and are uncritically hitting enter.
But how much of that is just grumpiness? Frustration? “AI slop” is the new “I didn’t like it.” Code review was always something I paid attention to as a manager, because when there was conflict on a team, it was the place most likely to show up. Code review is subjective work presented with the illusion of objectivity; everything unsaid can play out in it.
In the software factory model, no one writes the code and no one reviews the code. These go together: once the code is generated, human review is the bottleneck, so you validate some other way, through guardrails and automation. Most teams aren’t building from scratch, though. Akin to the challenge of self-driving cars, they are on mixed roads, humans and machines sharing the same lanes, which is the harder case (I loved this podcast on the topic).
So I wanted to see whether I could reverse engineer code review: some kind of machine gate a PR has to pass before it’s allowed to request actual human time and attention. Much of my career has been in open source, the write-code-in-public kind, so I picked a team with an open source repo I knew well enough to have context. I deliberately didn’t pick the highest-performing team I’d worked with: this one had mixed levels, engagement, and AI adoption, and I was curious how that played out in review.
What I did
First, I constrained Claude to be read-only on GitHub. The last thing I wanted was to leave a trail of inadvertent and likely weird AI-generated comments.
Then I had Claude (Opus) find the strongest code reviewers on the team, seeded with three names I asked it to confirm or deny. Its reasoning tracked with what I knew. It also flagged some reviewers who were thorough but inclined to nitpick, which made me add a style guide to the scope.
An aside on style guides: I started my software engineering career at Google, where the style guide was canon. I had readability in two languages, Objective-C and Java (basically, I jumped through a set of hoops to prove I could deliver high-quality reviews according to the style guide). I don’t really subscribe to any religions, in software development or otherwise, but one thing I took from that period of my career is that the true benefit of a style guide is that we do not discuss it, and spend the time on the things that matter instead.
Claude (Sonnet) built a corpus of 4,176 review comments and summaries by twelve reviewers, on over 1,700 PRs going back about 20 months. A Fable agent per reviewer pulled out the patterns in what they flagged, with examples: ~300 of them, which another agent compiled into 67 themes. Then I had it turn the themes into checks. Finally it went through every PR merged since the start of September (~500) to work out which checks the version the first reviewer saw would have failed, and whether review caught it. As on many teams, there were already AI review bots on the repo, so “review” in everything that follows means humans and bots together, unless noted otherwise.
This took real time, most of it reading what came back. It’s the kind of data I would have loved to have when I was running large orgs, and now it’s feasible.
What the machine recommended
What came out was a tidy recommendation: 57 checks:
11 style guide rules
8 test requirements, either tests for classes of issue or required test types for certain kinds of work
26 CI and static analysis checks (lint rules and scripts that run on every push)
12 instructions for a project-specific review skill that an AI reviewer could run, the judgement calls that don’t reduce to a regex.
I’ll call those style, tests, CI and judgement from here on. It also set aside 16 norms that authors had pushed back on at least once, sometimes for good reasons, and said not to enforce those mechanically at all.
Run against the last six weeks, more than half of PRs would have failed at least one check, with ~700 issues between them. 71% of those were never raised in review, by human or bot, and most of those merged without anyone mentioning them. Review raised 75% of the judgement issues, for example around metrics and concurrency, 32% of the style issues, 14% of the missing tests and 10% of the CI-level problems. Count only humans and it’s 43%, 22%, 9% and 5%.
(Some caveats. The CI checks were simulated by agents rather than actually run. The agents could see the review history, which probably flatters the judgement numbers: the checks were mined from these reviewers’ comments, so “review raised it” is partly “the check describes what the reviewer said”. A second pass re-checked every eighth PR and agreed with about nine in ten of the findings it was confident about. And only GitHub is visible, which is obviously incomplete.)
On paper, that’s the case for a gate. Write the bar down, have an AI teammate check every PR against it, and only then ask for human time.
Very nice. But what does it say about code review?
What it taught me about code review
It’s inconsistent. 92% of PRs had exactly one human reviewer (6% had none), so whatever that person missed went through. Individual reviewers raised anywhere from none to 38% of the problems in the PRs they reviewed, with a median around 20%. And there were 65 cases where someone flagged a problem on one PR, then let the same problem through on another. Reading the examples, my impression was that the miss was often a side change rather than the point of the PR. That makes sense, and with more and bigger PRs, it’s a prime reason to move this work to automated checks.
Some of that is just asking people to do a machine’s job. PR review was never the way to catch mechanical things. How do you hold test coverage in your head? That’s why teams that cared about these things always instrumented them. Looking back, readability at Google was kind of weird: a linter installed in my head. Test coverage requirements would probably have been more impactful.
The misses don’t obviously cost anything. I had Claude trace the 400+ unflagged issues forward: did a bug fix, hotfix or revert later touch the same lines? Then it compared that against other new code in the same PRs, adjusted for size. About the same. 20 of the 345 it could trace were followed by a fix, against about 17 expected. Reading the 20, only about half fixed anything the check plausibly described. The clearest cases were missing tests, which fits my hunch about coverage, but on small numbers to YMMV.
That’s a lower bound: each PR had between two and 42 days of follow-up. It also only measures bugs, not the slower cost of architectural mismatches that make the next change harder to test, or dead code someone reads around for years. But it was enough to shake my intuition that the misses mattered.
So what is review doing? Enforcing things the team hasn’t necessarily agreed to. Some of what reviewers raised was really shared: everybody flagged leftover dead and debug code. A lot of it wasn’t. Dependency injection, the architecture rule that came up most, was raised by only half the reviewers. Asking for missing tests was something seven of the twelve did over the 20 months, but in the six weeks I audited, review asked for tests only about 14% of the time, mostly humans. And in 16 places a reviewer’s norm got pushed back on, and in a few of them reviewers wanted opposite things: for instance, whether a feature flag should be read at runtime or once at launch.
These are all reasonable stances to take, and it’s possible they were situationally dependent in ways that I don’t have insight into. But from this data, and for an individual onboarding, it would be hard to distinguish a team norm from one person’s preference. It’s muddier still when, as here, some of what read as a reviewer’s judgement was actually AI output transferred via C+P.
That’s where the conflict comes from, and where bias gets in. Google studied pushback in its own code review and found the amount authors get varies with their gender, race and age, at a cost of more than 1,000 engineer hours a day, borne by engineers who aren’t white men. It is very easy to shred a PR if the person who wrote it seems like someone whose competence is in question, whether that’s down to what we politely call “unconscious bias”, because they were annoying in the meeting yesterday, or because the perception is they have handed their judgement to the machine.
So would I actually implement this?
There’s a real cost. An AI review on every PR costs tokens, and new lint rules and required tests add CI minutes to every push. The tokens are largely a more effective use of the AI review that’s already in place, at about a dollar a PR. Implementing everything would add somewhere between 6% and 38% to the CI minutes the repo already uses, depending on how carefully the expensive checks are scoped. Then there’s time. A gate adds 5 to 30 minutes before a PR goes green, a few percent of the time a PR already waits for its first human review. The bigger cost is that 55% of PRs would fail a check on their first push, so more than half would go round a fix-and-rerun cycle before a human was even asked to look. Against a bug rate that, so far, doesn’t move, that’s hard to justify on bug prevention alone. The case for it is consistency, which I think has value of its own, but has to be balanced against the cost.
When individual enforcement becomes team enforcement, the team needs to agree. The cheap checks, which are also the lowest leverage, are the ones nobody disputes. The highest-leverage ones, like dependency injection, or what counts as properly testing a feature flag, are the ones that need an expensive team discussion first. Skipping that discussion would turn one reviewer’s preference into a CI failure. But the value of that discussion could easily lower than what it would cost to have it.
The biggest value of code review isn’t in the repo. The most valuable thing review does is ask whether we’re building the right thing in the right way. Nearly half of what review raised here didn’t map to any check: product and behaviour questions specific to that PR. Ideally, those come up before review, because review is the last gate before user impact. But they’re context-specific, and the data to answer them isn’t in the codebase. What they point to is structure: who gets to decide what. What can an IC decide about their own project? What needs a broader group? Google’s readability and owner split was a version of this: style was settled once, for everyone, and ownership said whose call the rest was. Without that structure, those decisions get made in code review, by whoever happens to be reviewing.
So, would I implement it? The uncontested checks, yes: done efficiently, they add only a percent or two to CI. The rest depends on decisions the team hasn’t made yet, and depending on the team those discussions could be expensive. Right now code review is where those decisions get made, one PR at a time, by whoever is reviewing. Even if that worked, more or less, when humans wrote the PRs, I don’t think it works in an agentic world.
Anthropic detailed the activity of its A.I. agents in a blog post on Friday, without naming the targeted websites. But two sources with knowledge of the incidents said Anthropic’s A.I. agents had submitted 20 visa applications through a form available on the State Department’s website. All the applications were incomplete and were not processed, they said.
— The New York Times, Anthropic Agents Tried to Fill Out Visa Forms on State Dept. Website
Context matters when you’re working with AI. Every codebase has its own house rules: how components are structured, which scripts to run, and which conventions apply to different frameworks and file types.
Copilot can help you make changes across a project, but it works best when it knows those rules, too. Much of that knowledge can live inside the repository, so Copilot can pick it up automatically instead of requiring you to explain the same standards, scripts, and workflows every time you start a new task.
In this installment of our GitHub Copilot app for Beginners series, we’ll look at four ways to tailor how Copilot works with your project:
Custom instructions communicate the conventions Copilot should follow.
Agent skills package repeatable procedures.
Custom agents bring specialized guidance to larger tasks.
Model Context Protocol (MCP) servers connect Copilot to external tools and data sources, expanding what it can do.
Let’s look at how each one works and how they can work together.
Start with custom instructions
Suppose you ask Copilot to add a custom page size to an application. Making the change is only part of the job. The new code should also follow the patterns your team already uses.
Repository custom instructions give Copilot that baseline context. Add a .github/copilot-instructions.md file to define guidance that applies across the repository, such as your preferred architecture, naming conventions, or expectations for testing and validation.
You can also create path-specific instruction files inside .github/instructions. These files use an applyTo field to tell Copilot when the guidance is relevant.
When Copilot works on a matching file, it can use both the repository-wide instructions and the more specific guidance. That lets you keep the broad rules in one place without forcing every file type into the same mold.
Package repeatable procedures as agent skills
Custom instructions are useful for standards that should shape many tasks. But some guidance only matters when Copilot is performing a particular procedure, such as running quality checks, preparing a release, or debugging a failed workflow.
That’s where agent skills come in. A skill is a folder containing instructions and, optionally, scripts and other resources that Copilot can load when they’re relevant. Project-level skills can live in .github/skills, with each skill in its own subfolder.
For example, a .github/skills/quality-checks folder could contain a SKILL.md file that explains:
When Copilot should use the skill.
Which scripts run the project’s tests and linting.
What each script checks.
How to troubleshoot common failures.
The description in SKILL.md helps Copilot determine when the skill applies. After updating the interface, Copilot can load the quality-checks skill and follow the project’s established process for testing and linting instead of guessing which commands to run.
Use a custom agent for broader, specialized work
Some tasks are larger than a single procedure. An accessibility review, for example, may require Copilot to inspect components, templates, styles, and interactions across multiple files.
Custom agents let you create specialized versions of Copilot for work like this. A repository-level agent profile lives in .github/agents and can define the agent’s purpose, instructions, tools, and relevant project context.
Use a custom agent when a recurring type of work benefits from a dedicated role or perspective, especially when the task may span multiple files and draw on several tools or sources of context.
Connect Copilot to external tools with MCP servers
Instructions, skills, and agents help Copilot understand how to work within your project. MCP servers expand what it can access and do beyond the repository.
MCP servers connect Copilot to external tools, data sources, and services. A documentation server can give it access to framework guidance, while a server such as Playwright can let it open and interact with your application in a browser.
You can browse available MCP servers in the Copilot app or add a custom one. Once Playwright is configured, you can ask: Use the Playwright MCP server to test the functionality manually.
Copilot can then use Playwright to open the application in a browser and interact with it much as you would. That gives it a way to verify not only that the code passes automated checks, but also that the updated interface behaves as expected.
Choose the right type of customizations
These customizations solve different problems, but they become more useful when you combine them:
Custom instructions define the conventions Copilot should follow across the repository or for particular files.
Agent skills package repeatable procedures that Copilot can load when they’re relevant.
Custom agents provide focused expertise and guidance for broader tasks.
MCP servers connect Copilot to external tools, services, and data sources, expanding what Copilot can do.
Take this with you
Giving Copilot useful project context starts with four questions:
What project convention do I keep explaining?
What multi-step procedure do I repeat?
What complex task could benefit from a dedicated custom agent?
What outside tool or source does Copilot need?
Not sure where to begin? The community has shared ready-made instructions, skills, agents, and more through Awesome Copilot. Find one that addresses a workflow you already repeat, review what it contains, and adapt it to your repository’s tools and standards.
Start small: choose one convention you keep explaining, add it to .github/copilot-instructions.md, and see how Copilot applies it to your next task.
Log in
