Christ is my all
2708 stories
·
3 followers

Speeding up code review with `small-change`

1 Share

Imprint’s average engineer is merging 4x more PRs than 2 years ago. Yay. Aggregated behind the average is something even more interesting: we have two teams that are merging 10x their PRs over that period. One engineer is merging 25x the PRs per week of our average engineer two years ago, and a number of others are closing in on that number. That’s pretty neat, and the slope of the line suggests our average in a year is going to look much closer to today’s p90 than today’s p50. Even more importantly, we aren’t just shipping more, our quality has improved: incidents are down 42% half-over-half, and critical incidents are down 78% half-over-half. I think we’ve always known the quality/velocity tradeoff wasn’t a particularly useful construct, but that’s undeniable today.

As we’ve walked this path, we–like everyone else progressing down it–have run into the problem of pull request review. The p50 pull request review took about 33 minutes to complete 2 years ago, but now takes us about 77 minutes. The slope here correlates tightly with the number of pull requests getting merged. Tightly enough that addressing it became one of my priorities for the week.

We solved it! Just kidding, we’ve absolutely not solved it, but I think our current approach is an interesting one, which we call the small-change.


The work discussed here was shaped and done by many folks at Imprint including Dan, Joy, Daniel, Lenny, Taylor, Ted, and Annie, among many others.

Why review pull requests?

There are four reasons we review pull requests:

  1. Knowledge sharing between author and reviewer – software engineering is generally learned via apprenticeship, and reviews are a primary mechanism of experienced engineers teaching others what good software looks like. We believe the current volume means that this knowledge sharing will have to move towards operating on clusters of pull requests, rather than individual pull requests
  2. Preventing errors – review has always been an expensive time to catch errors, and we believe that effective CICD along with harness verification techniques like poteto’s pstack are more effective mechanisms for preventing errors than review
  3. Preventing malicious behavior – my experience is that even careful code review will generally miss competent malicious behavior, although fortunately most malicious behavior is incompetent. We believe both sorts of behavior are best identified using a combination of fixed rules (modifying these specific objects/classes/files requires review), anti-abuse mechanisms that we’ve refined in our LLM-based member support experiences, auditing, and audit log inspection
  4. Compliance obligations in our documented controls – these can be changed, but today they require that an independent human reviews changes in a subset of our repositories

In the modern development paradigm, we think there is a better solution for each of these problems. But, we also want to make these changes very cautiously.

Small steps with small-change

We believe we move fast by finding narrow, contained ways to validate new practices, so to iterate our way into an effective solution, we’ve developed the concept of a small-change. Here’s how it works:

  1. When a pull request is marked ready, CICD determines whether it is eligible to be a small-change. This is represented as a passing Github check. There is also a label for visibility, but it is the check itself, managed by the build and not human modifiable, that determines eligibility. (The label is used for initial filtering, but the check is still independently verified.)
  2. Something is a small-change if it meets all of these criteria:
    1. It is less than 1,000 lines of code
    2. There are no p1 or p2 findings from our ai-code-review automated review bot
    3. The author has “experience” in the service or application, defined as 10+ prior merged pull requests into that service or application’s directories. For changes working across multiple services or applications, the author must have experience in all of them to qualify
    4. Impacts fewer than N distinct services or applications. N is 1 on our frontend monorepo, and N is 3 in our backend monorepo, where typical changes will modify both a gRPC service and a common HTTP gateway
    5. Shared libraries and utilities files are not eligible for small-change
    6. Each repository has a .small_change file which allows per-path customization while we continue to tune the behaviors. For example, a service dealing with money movement can be excluded
  3. Our Slack automation tags every reference to a pull request with reacji labeling the state of that pull request, for example if it has failing tests, if it has been approved but tests are still running, or if it is ready to merge. We added a small-change reacji that indicates which of those pull requests meet the small-change criteria
  4. In our internal skills repository, we created two skills that have been distributed across harnesses:
    1. review-small-change which takes a list of repositories and reviews all open, non-draft, small-changes, including providing approval if they meet the review’s criteria (all checks pass including small-change check, and not touching on sensitive topics such as payments and security).

      Importantly, this skill has only two options: approve or ignore. It never comments on the PR to avoid the proliferation of “meat proxy review” which has become a bottleneck-pretending-to-be-a-solution on some teams

    2. review-pr-cluster which reviews batches of pull requests to identify common themes that came up, and suggest opportunities for discussion and adding verifications

  5. We set the social expectation that any error caused by a small-change is a failure in our system, and has to be solved at the system level: improvements to ai-code-review, to our verification mechanisms, to our CICD, to our harness, etc.

This took about two days of infrequent attention to implement and rollout, although I imagine we will be tuning the knobs here for a long time. Maybe forever.

Meat proxy review

I mentioned this in 4.1 above, but meat proxy review has become an interesting bottleneck for many teams. It’s a particularly difficult bottleneck because it initially looks like a solution, but it makes things worse. The pattern is roughly:

  1. Team is overwhelmed with reviews
  2. Team decides to review pull requests with their local harness
  3. Pull requests get a large number of comments, some of which make sense, and some of which don’t. It’s unclear which is which without study, and you don’t know which comments reflect the reviewer’s point of view. So you end up having to do substantial work to disprove comments that the reviewer may not even agree with
  4. Those teams then start using harnesses to respond to the harness-generated comments, making the entire review process an illusion that doesn’t facilitate learning, and most importantly could be done without the involvement of the other team members

In every case that meat proxy review is performed, the team would get better outcomes by investing into the centralized review mechanism, the harness’ loop for addressing the centralized review’s feedback, and simply stamping approval.

fleet: centrally orchestrated harness

I’ll briefly mention our other approach to reducing code review volume, which anchors on our centrally-orchestrated harness, fleet. (Which is similar to Stripe’s Minions although not following the one-shot approach in particular). We rely on fleet to implement and verify software in a highly controlled harness running in the cloud, with independent evaluation mechanisms to identify the correct number of independent reviewers for any given change. That number can be as low as one (e.g. fleet writes, one human reviews) for changes evaluated as safe.

Results / next steps / reception

I’ll be honest, it’s too early to give a useful result here. From early results, about 10% of changes qualify as a small-change. Those are now eligible for very fast approval, and generally get it unless they trip a flag, generally due to touching a security or money adjacent topic.

While 10% is a small percentage, the core concept here is working, and ultimately will succeed or fail based on our ability to expand that percentage over the next few months in tandem with expanding our verification mechanisms to be increasingly comprehensive. This is ultimately going to fail or succeed depending on our ability to improve our verifications, which does require more difficult sorts of techniques to accomplish in low scale-complexity, high domain-complexity software than in the high scale-complexity, low domain-complexity products where many techniques like incremental rollouts are exceptionally effective. (Don’t get me wrong, they still work, they just provide less coverage when most users have different configuration and behavior.)

Folks have not been uniformly excited about this, it’s a substantial change from how we’ve learned to be good engineers, and it’s uncomfortable to reevaluate whether a long-held best practice has become merely dogma. The most energizing part of the current cycle is that we get to reevaluate these practices we grew up on every week; the most painful part of the current cycle is that we’re likely to fall behind if we ever stop reevaluating them.

Read the whole story
rtreborb
55 minutes ago
reply
San Antonio, TX
Share this story
Delete

What Is the Most Important Part of a Church Service?

1 Share
What Is the Most Important Part of a Church Service?

Whatever else is true of Reformed folk, I’d say this is indisputable: We love our preaching. Many of our greatest heroes are preachers, our church services tend to be formed around the sermon, many of our favorite books had their genesis in the pulpit, and our conferences are mostly made up of a succession of sermons based upon a common theme. As I said: We love our preaching.

I would never disparage such a commitment, of course, but I would say this: If there is a negative consequence to the Reformed emphasis on preaching, it could be that some people interpret it to mean that the rest of a church service is not very important, or that the rest of a service is important not in its own right, but only because of its connection to the preaching.

But to ask whether one part of a church service is more important than another is like asking whether the most important ingredient in bread is the flour, the yeast, or the water. It’s a nonsense question because if you take away any of those ingredients, you don’t have bread. If you eat only the flour but skip the water, you may have eaten something, but it’s not bread. If you knead together yeast and water, you have created some substance, but it’s not dough and if you bake it, it won’t emerge as a well-formed loaf. You need it all to make something that is genuine, that is nourishing, that you can describe as bread.

I have often been struck by how many people arrive late to church. Week after week, many people miss the call to worship, as if it’s not all that important to be called away from the cares and concerns of the other six days of life and to be called specifically to a time of corporate worship. Many people miss the opening song, as if we don’t benefit from having our hearts tuned to praise our God. Many people miss the confession of sin, as if there is no great benefit to hearing a pastor lead the entire church in confessing their transgressions, and they miss the assurance of pardon, as if there is no special benefit to hearing a pastor assure the congregation from God’s Word that their sins have truly been pardoned, set aside as far as the east is from the west. Many people roll in only after all of this has taken place—on time for the preaching, but too late for so much else.

But the fact is, if you prioritize arriving for the sermon but miss the prayer or singing, you haven’t had a full experience of worshipping God. If you don’t put in the effort to arrive on time, and so consistently miss the first few minutes, you are depriving yourself of elements that have been integrated into the service because they are of such benefit to Christians. Corporate worship cannot be less than a sermon, but it must always be far more. And like the ingredients in bread, though they can be described separately, they need to be experienced together, because each is a necessary part of the whole.

So to those who are habitually late, perhaps because they simply don’t put the effort into being on time—consider the opening elements of a service, and what you are missing. To those who are habitually late because they dislike the songs or see little benefit in the first few minutes, know that you are depriving yourself of an opportunity to worship the Lord with his people—to hear from him, to speak to him, and to experience his blessings. And to all of us, even those who are always present from the call to worship to the benediction, let’s regard each element as essential, each a necessary part of distinctly Christian worship through which we express our praise to God while he lavishes his grace upon us.

Read the whole story
rtreborb
1 day ago
reply
San Antonio, TX
Share this story
Delete

Reddit to Remove RSS

1 Share

Jeff Johnson (June):

This new RSS rate limiting was perhaps foreshadowed by an r/modnews post a couple of weeks ago[…]

judy-funnie (September, via Jeff Johnson):

Because RSS is now a common surface for large-scale scraping and automated abuse, we'll stop supporting RSS feeds on November 13, 2026 while preserving moderation workflows that depend on it through supported alternatives.

[…]

To close this gap, in the next few months, we'll further limit access [to Old Reddit] for logged-in users to only those who have used Old Reddit in the last six months (which covers the vast majority of all Old Reddit users).

Ana Maria Constantin:

Public API access ends in March 2027, Reddit said in a separate post for developers. It will stop accepting new API requests on 31 October. From 12 January, it will start cutting off apps that have not registered.

Jeff Johnson:

Since I use RSS to follow Reddit, I intend to discontinue my Reddit use on November 13, unless Reddit backtracks on their user-hostile decision to drop RSS.

[…]

My biggest regret in leaving Reddit is that I‘ll have to shut down my product support subreddit.

[…]

The most ridiculous part of Reddit’s announcement was the suggestion that subreddit moderators replace RSS feeds with Discord notifications. What perversity, insanity to recommend a competing social network as the only practical way to use yours! I’m also skeptical of Reddit’s public excuse for dropping RSS: bot scraping. As I noted back in June, Reddit is already rate-limiting RSS feeds. If the current limits are insufficient, why not impose stricter limits?

John Brayton:

I am disappointed because I know that many Unread customers use Reddit’s RSS feeds to keep up with subreddits.

Sarah Perez:

Moderators have already expressed worry about how this shift would impact their workflow. When Reddit hinted earlier that it was preparing for the end of RSS on its site, many responded by complaining that Reddit would become unmanageable for them. Meanwhile, many users stressed that RSS was how they enjoyed accessing Reddit’s content.

Previously:

Update (2026-10-07): John Brayton:

I overlooked the fact that Feedly uses Reddit’s API to retrieve articles from subreddits. The folks at Feedly expect this functionality to continue working after Reddit removes their feeds.

Read the whole story
rtreborb
3 days ago
reply
San Antonio, TX
Share this story
Delete

Surveillance Denied

1 Share


Read the whole story
rtreborb
5 days ago
reply
San Antonio, TX
Share this story
Delete

Apple Built a Photo-Realistic Functional Original iPod for Apple Music’s 404 Page

1 Comment and 2 Shares

It seems like this amazing Easter egg only launched yesterday, to celebrate the original iPod’s 25th anniversary. Exquisitely well done. More like this, please.

Link: music.apple.com/404

Read the whole story
rtreborb
5 days ago
reply
San Antonio, TX
Share this story
Delete
1 public comment
Belfong
8 days ago
reply
I miss the simplicity of such interface.
malaysia

Drew Estate Adds Eight New Feral Flying Pigs, Undercrown El Tigre Flying Pig

1 Share

For those who thought last month's release of a half-dozen new perfecto cigars wasn't enough, Drew Estate has upped the ante for October. T52 Feral Flying Pig. Check. Undercrown 10 Feral Flying Pig. Check. Box-pressed Flying Pig. Also check.

The post Drew Estate Adds Eight New Feral Flying Pigs, Undercrown El Tigre Flying Pig appeared first on halfwheel.

Read the whole story
rtreborb
5 days ago
reply
San Antonio, TX
Share this story
Delete
Next Page of Stories