From Babysitting AI to Developing the Factory

So we let the agents take a step in the spec-driven workflow. What do we do meanwhile? We can’t idle. And we have tokens to max out. So we spawn a new agent to work on a new thing. But then the original agent comes back with a question or a wall of markdown for us to review. Pretty soon the agent threads pile on and on. And you, the human, spend more and more time context switching and less and less time in the flow zone. Which also leads to burnout.

Joshua Kerievsky’s seminal article on #techsafety talks about common injuries to software makers. The way we currently make stuff with AI is prone to many of these injuries e.g. fractured flow, merge misery, brain hernia.

How can we shift AI coding from being a cause of injury to makers, to being a remedy to some of these common injuries? That’s what I’m exploring in my coaching conversations. And this shift requires engineers and their leaders to shift their perspective. From making software features to making software factories.

My advice to the frustrated burned-out engineer was to start looking at AI prompts and loops as source code in their own right. And to take on the challenge of turning this into something they enjoy working with, where they feel that Claude works for them, not vice versa. It’s about regaining our human agency using agentic AI.

An agent makes a decision we don’t like, and we find it in the code review or the spec review. One option is to override it: include this requirement in the spec, include this intent, use this pattern in the code. Change this, change that.

What we need to look at is the system that made this decision. The software factory needs to be designed for the limitations and capabilities of the coding agents, with the right guardrails.

When a rule isn’t having the intended effect, treat it as a bug. Create an evaluation or a test that proves it, then work on it through test-driven development. Improve the skills, improve the context, improve the system that creates the code, and then rerun and see if it’s better.

If “humans make the decisions” means humans decide how to code something, we’re back to humans reviewing all the code. There are also decisions about what’s important: choosing the feature, the architecture, the outcome, the system behavior we want to see. In my conversation with Shay Mandel at Next Insurance, he put it this way: “We are not the initial reviewer. We are the last reviewer.”

The human is reviewing work that agents deem “Done”. And the expectation is that over time, these reviews come up with fewer and fewer findings. How? By fixing the factory so that it can find and fix the finding itself, or ideally build quality in from the get-go.

Let’s say you’re reviewing a draft PR. Instead of asking an agent to fix the findings, work with it to fix the specification of the factory (whether that’s agents.md, constitution or skill files), and then use the draft PR and the finding as a “specification by example” of what it should be able to find itself as part of its workflow, as well as consider when it is specifying and planning to avoid in the first place.

You can then either move on to the next thing you’re working on and see if it does better, or ask the agent to repeat the whole workflow using its new factory version and evaluate whether it’s indeed building quality in.

Once you get to the point where what the agentic workflow considers good quality meets your own criteria, you can start trusting it more and more. This is the point where human in the loop can start to shift towards human on the loop. Where you can start turning off some of the lights in the software factory and let the agents do their thing, while the humans get more capacity to amplify the loop, improve the factory, and innovate at a higher level.