AI Activity, Output, or Impact? How to Escape AI Theater

Help me think through how this episode applies to my situation. Start by asking what I am trying to change. Separate the episode's claims from your suggestions, and say when the notes do not support a claim. Use the transcript to find passages, then check the audio before quoting. Transcript: https://yuvalyeret.com/podcast/episodes/ai-activity-output-or-impact-how-to-escape-ai-theater/transcript.md ## Published episode notes Most AI programs start by measuring activity: licenses, prompts, token usage, and pilots. Some move on to output: more code, content, or analysis. Neither tells you whether the business moved. In this solo episode, I explain the three levels of AI adoption: activity, output, and impact. The common trap is local productivity. One team can produce more while end-to-end delivery gets slower because the real constraint is somewhere else. The practical test is simple. Name the business outcome that is stuck, identify the constraint holding it back, and ask whether AI is moving that constraint. If you cannot answer that, you are probably measuring AI theater. Chapters 00:00 From AI activity to impact 03:59 Why more output can make flow worse 07:24 Use AI at the business constraint Read more Why Your AI Effort Has Activity But Not Impact More on moving from AI activity to impact Follow Yuval on LinkedIn ## Transcript This transcript was edited from automatic speech recognition for readability. Speaker turns may be wrong or absent, and it may contain recognition errors. Check the audio before quoting or attributing a passage. Episode: https://yuvalyeret.com/scaling-ai-podcast/ai-activity-output-or-impact-how-to-escape-ai-theater/ ## Where the AI effort really is Where's your AI? Is it AI theater? Are you seeing AI output? Or are you seeing AI impact? Welcome to Scaling AI from Activity to Impact, also known as Scaling with Agility. Today, what I want to walk you through is what I'm seeing in most organizations I talk to, and what the goal with AI should be. The first stage for organizations that have AI ambitions is typically to start using AI. They plan a bit and look at it, but the first step where there's real stuff happening is activity. And that's good. You give people, at least some people in the organization, access to AI tools. You're piloting some stuff. You're maybe running AI literacy training, hackathons, experiments. There's a lot of motion. You could present a lot of activity and say, "We're doing AI," but you're not necessarily seeing value from it yet. There are probably some spots where there's value, but it's not at scale. And that's a good start. Don't get me wrong. What happens in a lot of organizations is that at this point the activity plateaus. You start to celebrate the activity and enjoy that activity. It's easier to measure how many people are using AI tools, how many people have been through training, how many people participated in things you organized, and then show that and check the box that you're now doing AI. That is an especially comfortable position when the reason you were doing AI was FOMO, fear of missing out. Whether it's at the board level or among leaders, you feel like, "We should be doing these things. It's the thing to do these days, so we're doing something." But the longer you stay in AI theater, where there's a lot of activity around AI but it's not really moving the needle, the more you'll probably start facing questions like: How much is this costing us? What's the return on that investment? What are we getting from it? Another thing that happens is that this AI activity is not helping people in the trenches understand the vision for how AI will really affect work at the organization, at the company. And that, combined with the overall fear and uncertainty that is rampant in the industry around what's going on with AI, leads to the point where people might actually sabotage, or at least refuse to participate in, using AI to become more effective. They're afraid it will hit them. They're afraid they are literally training their replacement. ## Why activity becomes a comfortable trap So those are the problems with AI theater. It's not really providing value, and at some point it might actually create risks in how people think about AI in the organization. Over time, there's less and less support for AI improvements. The next level is AI output, or what we might call the AI factory. We are seeing improvement in specific areas. Those AI experiments are turning into deliverables. There are applications. There are groups in the organization that are seeing higher throughput. The classic example is software engineering, especially because AI coding agents and harnesses were trained on open source code. They're faster at coding. Their improvement in coding is much faster than in other areas. And engineering teams are typically early adopters for technology, which is not a huge surprise. So we're seeing them deliver more. We're seeing organizations where the engineers are not doing manual coding anymore. They're coding everything through the agents. In some cases, they're actually asked to only code through agents. We're seeing that starting to happen across the industry. And the result is, in many cases, higher throughput, more output due to AI. And that's great, but in many cases, that output doesn't turn into impact. We're still not seeing any change to, let's say, the revenue per person in the organization. Or we're not seeing better uptake of our products. We're not seeing efficiencies hitting the wider organization. And the reason is that we might be facing a bottleneck. There are a couple of reasons. One is that engineering might not be the bottleneck. ## When output still does not create impact It might not have been the bottleneck to begin with. So maybe we're building things, but those things we build, that output, are sitting in queues waiting for somebody to do something with them. Maybe to deploy them. Maybe to use them. Maybe, on the other hand, there's an upstream bottleneck where our speed, our throughput at feeding this very hungry AI-augmented or AI-assisted engineering organization, is not good enough. It's like feeding a set of teenagers who do sports all day. You spend all of your time going to Costco or Sam's Club and buying tons of protein and tons of food to feed them, and it's never enough. So that's what it feels like in these organizations. What happens at this point is that all these engineering teams will work on things. They will create more stuff. A lot of the time, what that actually does is make life harder for everybody else. So it's a local optimization of the engineering organization, but it's neutral or even detrimental to the overall throughput of the organization. So what we really want to achieve is the next level, which is AI impact. It's not about delivering more software necessarily. It's about two things. One is that even inside our product group, we need to make sure it's about impact and not output. We need to improve our ability to feed this engineering team with really valuable product ideas, discover those valuable ideas, fail fast, and when we have a good idea, build it. We also need to validate that it's actually working and tweak it until it creates real change for customers and eventually impacts our business. So that's one path to AI impact. The other aspect of AI impact, and what you might call an AI-native organization, goes way beyond the product engineering team or the product lifecycle. There might be other areas in the organization that are the bottleneck. ## Follow the constraint, not the easiest AI win The bottleneck might not be the product anymore. The bottleneck might be elsewhere in the customer's success factory. It might be that people aren't aware of our product or that we have a leaky bucket of customers. They come in, they start to use, they don't stick with the product. And that might be a product opportunity. Maybe we can improve the product to improve retention. Or it might be the processes that we use to monitor what our customers are doing, to learn from that, to meet them where they are, and to improve the whole ecosystem around these customers. Our bottleneck might be our pace of onboarding partners, if that's our business model. It really depends on our key business processes and our inflection point. What do we actually need to do right now to scale AI impact? We use AI to attack these bottlenecks. We improve end-to-end flow in the organization, in our key business processes, whether that's building our product, revenue, the customer success factory, making something physical, or whatever it is. We focus on the real constraint across the business, not local optima in the areas where it's easy for AI to make a difference. We subordinate other groups. We might tell our engineers, "Listen, your capacity, your throughput, is already amazing. You're doing great things with AI. How about instead of improving your output, which doesn't really help us at this point, "we leverage some of your expertise to work on other areas in the business?" That's a hard process for many organizations. It's culturally against how many organizations work. It cuts across silos. It asks people to work on areas they're not used to. ## The practical goal: AI impact But it also connects to some of the things we've been doing inside the product development pipeline for many years. And in manufacturing, when there's a bottleneck, the theory of constraints talks about subordinating the system to it and people finding ways to work outside their comfort zone on other areas where they might make the most impact. So to sum up, the goal that we're talking about here isn't just to do activity with AI. The goal isn't even to create output in a certain area of the organization. The goal is to achieve AI impact. And that's why a lot of what I'm talking about these days, here in the podcast and in general, is how do we notice that we're focused on activity or output, and how do we leverage AI to deliver impact, especially at our organization's constraints? ## Source boundary These are the published show notes from the podcast feed. They are a starting point for discussion, not a verbatim record of the conversation. The transcript is machine-generated and may contain errors or unlabeled speakers. Check the audio before quoting anyone.