AI Isn’t Failing — Our Operating Systems Are

The stat everyone is quoting is that about 95% of AI projects fail — meaning people are spending money on AI and not seeing the value afterwards. That is understandable. It is very new technology and it is certainly not well understood yet. But the conclusion most people draw from it, that the models are immature or the data isn’t clean, is the wrong one.

I worked through a different explanation with Dave West, CEO of Scrum.org, on the Scrum.org Community Podcast. I think AI is one example where organizations without the right operating system, without the right approach, are struggling to get the value out of it.

When Dave brought up those numbers, I had déjà vu, because we both know what the CHAOS report says about the success rates of IT projects — success from the perspective of do they complete on budget, on time, do they deliver the scope. Later on, after we improved that and created better feature factories, we started to look at whether they were delivering outcomes. Is there evidence that those successful projects, that minority of successful projects, are actually moving the needle for the business?

We’ve gone a long way towards creating more value for our businesses. I don’t think we’re anywhere close to the nirvana of being customer-centric, outcome-focused, empirical, evidence-based product organizations in IT or in technology in most organizations. But when you start to look beyond the people who have been exposed to these ideas, it’s like the stone age.

Dave pointed out where the AI spend is actually happening, which sharpens the problem:

A lot of AI adoption is being done outside traditional IT organizations. It’s being done in sales, marketing, legal. And they’re taking this technology and saying, “can I build something that quickly reviews contracts?” And they build something and it works, and then they realize it doesn’t work quite as well as they thought, and then they’re like, oh, that was unsuccessful.

It’s a similar challenge: there’s this new technology, there’s this new opportunity, and it’s very tempting to spend your time on LLM geekery, making sure the data is clean and talking about legal aspects all the time. But very few people talk about why are we doing this. What’s the business purpose for this?

Even the disciplined organizations are running AI adoption as a project — and we’ve learned what the problems are with treating complex strategic initiatives as projects. We’ve learned that focusing on scope, even if you deliver that scope, doesn’t necessarily deliver the value.

Dave put the structural version of it well:

EOS and most operating systems being used by organizations would treat AI like they would treat “we need to build a new building” or “we need to run an event.” It literally is a defined, scoped sort of like it’s a project. When your operating system has this concept of projects within it as the mechanism for driving change, that isn’t necessarily the most successful orientation for AI.

Yes — but I’d argue it’s not just AI. Whenever I look at the challenges companies are really facing, the strategic challenges, what keeps their CEO and senior management team up at night — tackling those challenges, developing their organization, working on growth — whether it’s a mom-and-pop shop, a scaleup, a mid-market company or an enterprise, those challenges are complex. Regardless of whether they are product development challenges or company development challenges.

There’s a lot that is unknown, a lot that is uncertain. They’re bets. They’re bets on: if we build this new standard operating procedure, if we build this new process, if we give people ChatGPT, if we do this, if we do that — will people want to use it? Is it something that is worth our investment? Is it even feasible to achieve?

A lot of the time our focus is too much on “is it possible to achieve this,” or even “let’s just assume it is, let’s build a plan and deploy it.” We’re not thinking about the human aspects. We’re not thinking about our organization as an organism, as a living thing. We’re not really making the assumption that we don’t know — that we need to balance championing something with being skeptics about it.

And it is not in the interest of leadership teams to be skeptics about their own strategic initiatives. That is the uncomfortable part.

How I backed into company-level operating systems: in some of the organizations I work with, Dyno Therapeutics was one example, we realized that in order to really move the needle on the strategic growth the organization needs, you have to go beyond just the product organization. You improve the product organization, you level up your agility, and you see the constraint is elsewhere. It’s in marketing, or bringing in the right partners, or making the onboarding experience work well.

So we saw opportunities to leverage the same ideas we use to accelerate and create better outcomes inside the technology organization — in people operations, in finance, in marketing, in partner success, and in the interaction between these different groups, because a lot of strategic value comes from the interaction.

Think about what a mid-market company or a scaleup faces. There’s running the operation, running the customer factory, running marketing, bringing people through the funnel, fulfillment, serving them — whatever it is that you do. It can be a professional service, a dental clinic, an architectural firm. It can be a software-as-a-service company that onboards and activates people, where you need to make sure they use the product, that they’re happily using it, that they don’t churn out the other end, that they refer other people, that the flywheel is spinning.

That’s running the operation. But you constantly need to think about how to spin the flywheel faster. You need to continue developing the organization — developing a product that serves your external customers better, and also developing the company so that people inside it can do a better job serving those customers and operating that customer factory.

The product metaphor works because once you start to see these key business processes as products, and you have people who own their effectiveness at creating outcomes, you start to think about how to develop them.

Dave named the practical benefit, and did it from experience rather than theory:

As a small business when I was at Tasktop, making decisions about where we invested — do we invest in a help desk, do we instead invest in more software engineers — was actually really hard. At that moment of crisis we made a decision and hired somebody or spent some money. It was very not strategic. The benefit of approaching this from a product paradigm is you get that strategicness, because you’ve decided where you’re putting your money and then you can review how that’s going.

As Dave talked through decomposing a business into products, there was a siren going on in my head, because of the anti-pattern I see often when people go into this.

It’s tempting to take your organization and say: you’re the VP Sales, sales is a product. You’re the VP Marketing, marketing is a product. You’re the VP Product, product is a product. You’re the VP People, that’s a product.

But what we saw at Dyno is that the product is not those functions. The product is the onboarding experience, and separately the overall employee experience — what they call the aviator experience — and the partner experience, and capsule development. That often cuts across functions.

The jury is still out on whether “products” is even the right language for these organizations. There are advantages and disadvantages, and the alternative is to speak about value streams, and the difference between operational value streams — how the organization creates value directly — and what enables it to create value.

I do see an advantage in product language, and it’s not because we talk about product in Scrum. The main advantage is that when I talk to leaders of mid-market firms, there’s a trend: in order to scale, to better fulfil the needs of your customers, you need to productize. That’s a language that is starting to resonate. Be intentional about the balance between the bespoke services you provide and productized services. Even if you’re not a product company, productize your services.

Take me as an example. I’m a service provider — an agile coach, a management consultant, whatever you want to call it. You can consume my services by having a conversation with me and we’ll figure out the best way to support your organization. Or I could create a productized service: there’s an SKU, an agility roadmap, I help you figure out how to find gold with AI.

Going through that switch from bespoke into a productized service is a product development exercise, although it might not have any product in it. You see a lot of legal firms, accountants, financial advisors and doctors switching models — from a visit to a concierge approach or a membership — even though there’s no product around that. They need to develop their organization to a point where that works. That is complex. That’s the sort of stuff we’re talking about, and that’s why I think product could be a useful metaphor.

Dave made a point about his own AI usage that connects directly to this:

The most successful uses personally of AI that I’ve had are where I bound the problem. I provide a consistent context, I pick the right model, the AI engine knows the boundaries. The solution that comes out is very different when it hasn’t got those boundaries.

I think about it as context development. The additional information you provide the AI engine when you’re working with it on anything is crucial. I spend more time developing the context for my AI usage than the actual prompts, and it has improved my personal AI effectiveness dramatically.

Look at how I do it. I create a space — a ChatGPT project, or a Gemini Gem — and the instruction prompt I use there, I develop using something like Scrum. I don’t run a full Scrum process, but I stop myself, say after a day of using that prompt, and I ask myself as well as the AI engine: how are we doing? Are we getting good results? How could we improve this prompt based on the conversations we’ve had? Let’s look at how often I had constructive feedback for you, or even non-constructive feedback for you, because I’m not as nice with AI as I should be with the future overlords.

And we improve the prompt. We create another version of it. It’s not under source control right now — I should probably do that better — but I have a version 4.5 of my personal advisory board Gem that I use.

The more you think about your products, your customer factory, how you’re creating value in your organization, where the constraints are, where the bottlenecks are, what the strategic focus is, the better results AI will give you. And the better results the people developing AI capabilities for the organization will be able to deliver.

Where I see people finding gold using AI — whether it’s a small organization, an individual, or an enterprise — is when they use product techniques, whether they call it product or not, to think through AI strategy and execute on AI attempts with the mindset of: what’s the problem we want to solve here, what is our strategy, where do we want to play with AI inside the organization.

The challenge in the past was where do we hire another person. The challenge right now is: we’re not going to hire additional people, but where do we want to hire AI?

Is it marketing? Is it the fact that we’re leaking people? Is it the overall customer experience, which involves both the product and customer success? Is it onboarding and activating people — we get MQLs, but those people aren’t really using our product? Maybe it’s something on the product side. Maybe it’s something else that relates to onboarding customers.

Whatever it is, we want to focus AI on our strategic challenges, on the things we believe will move the needle or will affect the bottom line if we improve them. Things like OKRs provide a really good mechanism for grounding that focus. And if you’re using a product model, those OKRs work within the context of the products and within the context of a broader business strategy.

This article is based on my crossover conversation with Dave West, CEO of Scrum.org, on the Scrum.org Community Podcast. We go further on the operating-system question, the product-versus-value-stream debate, and the Toyota production-system-versus-design-system analogy.

If this sounds familiar, take the AI Traction Scorecard. It is 11 questions, about five minutes, and it will help you see whether your AI effort is creating traction, mostly producing activity, or getting stuck in theater. Use it to find the next constraint to inspect before the next pilot turns into another status update.

Whatever it is, focus AI on your strategic challenges — on the things you believe will move the needle, or will affect the bottom line if you improve them.