Agents at work - The build pipeline

We have all heard scary stories about disasters that happened "because of AI". And if they can do those things, who's to say they haven't introduced something in our code that we're happily unaware of until it explodes?

Those are reasonable worries, but they often result in the wrong "fix": even stricter gating on the pull requests.

A fairly large part of engineers I have discussed AI adoption with these last few months experience being constrained in two specific areas:

• Throughput (PRs merged and deployed) has increased somewhat, but not nearly as much as anticipated. The reason is that although AI usage enables faster production, code review is the new bottleneck.

• Companies have begun imposing token budgets on employees because token-spend has dramatically increased

Those two areas are related, and often point to the same underlying issue: companies have mostly adopted AI in the production layer, but not many have adopted it elsewhere. I'll be focusing on the first of the two constraints. Let's start by looking at how software is traditionally built, and how AI has changed that.

The build pipeline

Most companies have been using some version of the Agile Software Development Lifecycle for the past couple of decades. With AI, this has been disrupted, and employees are now experimenting with using AI in different ways at different stages in the lifecycle:

• Designers can now iterate on and experiment with their ideas and design with prototypes in a way they could only dream of before, without needing access to a developer to work with. Which still meant days of work for something they can now do alone in hours.

• Likewise, developers are now able to get a lot done with both faster iterations on coding and with access to design resources, without needing a designer.

• And, finally, a product owner can test out their ideas for new features in their product, and even get early user feedback on their ideas, without needing the assistance of neither their designer nor their developers

So the speedup in ideas, creativity, prototyping etc has been supercharged with AI. Most companies are not quite there yet, though, but that's mostly down to access and training. And to a certain degree: resistance.

However, even in the companies where they have come far on the above, the actual build pipeline hasn't moved that much yet. Designers still do much or most of the final design work like before (which, if we're honest, yields better results in most cases).

Developers, however, are increasingly speeding up parts of their work, having LLMs producing much more of their code than before. But from this point, the pipeline has seen much less change. In Norway, it's still broadly considered mandatory to read and understand the code you submit in a pull request. Even reckless not to. And many companies require it.

This creates a chokepoint at the review phase, with PRs piling up and developers becoming bored by reading code rather than writing it. Some even considering reducing their use of AI as a "solution" to this problem.

This is an AI leadership responsibility and is caused by lack of direction.

The AI-pilled build pipeline

I have a fairly elaborate build pipeline that I use when building non-trivial things. At the moment, it looks like this:

^{My current build pipeline, but I am constantly experimenting and revising it.}

This article is already quite lengthy, so I won't go into details here (maybe in a future article, let me know if that's something you would like to see), but I'll briefly talk about the main differences from the agile process we're used to.

The specification

The specification is the most important artefact of the whole pipeline because it carries not just instructions on what to build, but also why. It captures the intent behind the build, which is important later in the pipeline.

Review + repair phases

Phases 2, 3 and 4 are production phases. Each are completely dependent on the previous phases being correct to be able to produce value, so between each of them is a review step with a possible bounded repair step if serious issues have been surfaced by the review.

In the agile methodology, these phases don't exist. But that doesn't mean the work wasn't reviewed or repaired before! It was, constantly. Since people were doing the work, continuously assessing the work, discussing it with the PO, the designer or maybe even a stakeholder was common. With agents doing the building, that work becomes standardised steps instead.

Product evaluation

This is the central phase if we want to remove the pull request chokepoint. The product is now complete from the perspective of the build pipeline. Also agent-controlled like the other phases. It employs Gherkin acceptance criteria, and I am working on learning how/if AgentRx constraint specifications can improve it.

This eval layer is fundamentally a confidence layer, asking: "How confident am I that what I built is what I wanted to build?"

Different builds have different risk profiles, depending on what you're building. So you must determine your risk tolerance for each build.

Conclusion

To get the full benefit from using AI in our work, we must dare to challenge the way we work. But even though there has been a lot of focus on how "coding is solved", and LLMs are improving with increasing velocity, that's not the part of our processes we should be focusing on.

AI may have solved "coding" to a large degree, and the best models are able to produce better code than most developers. But that doesn't mean we can just sit back and watch the show. AI has not solved engineering. And LLMs make mistakes, so it's our job to evaluate the risk, assess the likelihood of it happening, consider the blast radius if it does, and determine what our tolerance is in each case.

This article was first published on LinkedIn. Part 2 in this series covers AI-enabled systems, and what it means for governance when we introduce non-deterministic elements as central components in an otherwise deterministic system.