Why Your AI Effort Has Activity But Not Impact

Is it AI theater? Are you seeing AI output? Or are you seeing AI impact? I want to walk you through what I’m seeing in most organizations I talk to, and what should be the goal with AI.

The goal isn’t just to do activity with AI. The goal isn’t even to create output in a certain area of the organization. The goal is to achieve AI impact. That means we focus on the real constraint across the business, not local optimums in the area where it’s easy for AI to make a difference.

The first stage for organizations with AI ambitions is typically to start using AI. They plan a bit, they look at it, but the first step where there’s real stuff happening is activity. And it’s good. You give people, at least some people in the organization, access to AI tools. You’re piloting some stuff. You’re maybe running AI literacy training, running hackathons, experiments.

There’s a lot of motion. You could present all that activity and say we’re doing AI, but you’re not necessarily seeing any value from it yet. There are probably some spots where there’s value, but it’s not scalable. And that’s a good start. Don’t get me wrong.

In a lot of organizations I see, the activity plateaus at this point. You start to celebrate the activity, to enjoy that activity. It’s easier to measure how many people are using AI tools, how many have been through training, how many participate in things you organize. So you show that, and check the box that we’re now doing AI.

That is especially a comfortable position when the reason you were doing AI was FOMO, fear of missing out. Whether it’s the board or the leaders, you feel like we should be doing these things. It’s the thing to do these days. So we’re doing something.

But the longer you stay in this AI theater, the more you’ll probably start facing questions. How much is this costing us? What’s the return on that investment? What are we getting from it? There’s a lot of activity around AI, but it’s not really moving the needle.

Another thing happens too. This AI activity is not helping people in the trenches understand the vision for how AI will really affect work at the company. Combine that with the fear and uncertainty that is rampant in the industry, and people might actually sabotage it. Or at least refuse to participate in using AI to become more effective. They’re afraid it will hit them, that they are literally training their replacement.

So those are the problems with AI theater. It’s not really providing any value, and at some point it might create real risk in what people are thinking. Over time there’s less and less support for AI improvements.

We are seeing improvement in specific areas. Those AI experiments are turning into deliverables. There are more skills, there are applications, there are groups in the organization seeing higher throughput.

The classic example is software engineering. AI coding agents and harnesses were trained on open-source code, so their improvement in coding is much faster than in other areas. And engineering teams are typically early adopters, which is not a huge surprise. So we’re seeing them deliver more. We’re seeing organizations where the engineers are not doing manual coding anymore. They’re coding everything through the agents. In some cases they’re actually asked to only code through agents.

We’re seeing that start to happen across the industry. The result, in many cases, is higher throughput, more output due to AI. And that’s great. But in many cases that output doesn’t turn into impact. We’re still not seeing any change to, let’s say, the revenue per person. We’re not seeing better buying into our products. We’re not seeing efficiencies hitting the wider organization. The reason is that we might be facing a bottleneck.

There are a couple of reasons. One is engineering might not be the bottleneck. It might not have been the bottleneck to begin with. So maybe we’re building things, but that output is sitting in queues. It’s waiting for somebody to do something with it, maybe deploy it, maybe use it.

Maybe on the other end there’s an upstream bottleneck. Our throughput at feeding this very hungry AI-augmented engineering organization is not good enough. It’s like feeding a set of teenagers that do sports all day. You spend all of your time going to Costco or Sam’s Club, buying tons of protein and tons of food, and it’s never enough.

That’s what it feels like in these organizations. The engineering teams work on things. They create more stuff. A lot of the time that actually makes life harder for everybody else. It’s a local optimization of the engineering organization, but it’s neutral or even detrimental to the overall throughput.

So what we really want is the next level, which is AI impact. It’s not about delivering more software necessarily. It’s about two things.

One is that even inside our product group, it has to be about impact and not output. We improve our ability to feed this engineering team with really valuable product ideas. We discover those ideas and fail fast. And when we have a good idea, we build it, validate that it’s actually working, and tweak it all the way to real change for the customers that eventually impacts our business. That’s one path to AI impact.

The other path goes way beyond the product engineering team or the product lifecycle. This is what you might call an AI-native organization. There might be other areas which are the bottleneck. It might not be the product anymore. It might be the customer success factory. It might be that people aren’t aware of our product, or that we have a leaky bucket of customers: they come in, they start to use, they don’t stick with the product.

That might be a product opportunity. Maybe we can improve the product to improve retention. Or it might be the processes we use to monitor what our customers are doing. We learn from that, meet them where they are, and improve the whole ecosystem around these customers. Our bottleneck might be our pace of onboarding partners, if that’s our business model. It really depends on our key business processes. And what’s our inflection point?

So what do we actually need to do to scale AI impact? We use AI to the point where we attack these bottlenecks. We improve end-to-end flow in our key business processes: building our product, revenue, the customer success factory, making something physical, whatever it is. We focus on the real constraint across the business, not local optimums in the area where it’s easy for AI to make a difference.

And we subordinate other groups. We might tell our engineers: listen, your throughput is already amazing. You’re doing great things with AI. How about, instead of improving your output, which doesn’t really help us at this point, we leverage some of your expertise on other areas in the business.

That’s a hard process for many organizations. It’s culturally against how many organizations work. It cuts across silos. But it’s also a deja vu of what we’ve been doing inside the product development pipeline for many years, and in manufacturing, when there’s a bottleneck. The theory of constraints talks about subordinating the system to it. People find ways to work outside their comfort zone, on other areas where they might make the most impact.

This article is arranged from the solo episode where I walked through the activity, output, and impact levels, and why the constraint matters more than the easiest AI win. Listen on the episode page or on Spotify.

To sum up, the goal isn’t just to do activity with AI. The goal isn’t even to create output in a certain area of the organization. The goal is to achieve AI impact.

And that’s why what I’m talking about these days is how we notice that we’re focused on activity or output, and how we leverage AI to deliver impact, especially at our organization’s constraint.