# Beyond Token Caps w/ Tomer Elias: How Enterprises Actually Measure AI Impact: transcript Automatic transcript of the published podcast audio. Recognition errors are possible. Speakers are unlabeled; do not attribute a passage to Yuval or a guest without checking the audio. Episode: https://yuvalyeret.com/scaling-ai-podcast/beyond-token-caps-w-tomer-elias-how-enterprises-actually-measure-ai-impact/ Source RSS GUID: 056c8877-b280-4cad-ac85-9b1b29504332 Source: published Riverside RSS audio enclosure Transcription: faster-whisper base.en, English ## Transcript [00:00:00] Welcome to the Scaling AI from Activity to Impact podcast. [00:00:05] I'm Yvall Yerit, and today with me is Tomer Elias. [00:00:09] I met Tomer at a private product leadership community, [00:00:14] and it was pretty obvious very quickly [00:00:18] that we share an interest area in exploring [00:00:22] what are people doing out there to evolve beyond token maxing [00:00:28] and AI activity and starting to get some real outcomes [00:00:33] and some real impact. [00:00:34] So I thought let's invite Tomer here, [00:00:37] and let's compare notes. [00:00:40] What I'm seeing, what you are seeing Tomer. [00:00:43] So welcome to the podcast for people [00:00:45] that aren't familiar with you. [00:00:47] Would you share a bit about your background [00:00:50] and what brings you to be interested in this challenge? [00:00:54] Sure, so it's great to be here. [00:00:56] My background is more than 15 years of product executive. [00:01:00] I've led multiple startups from 0 to 1, [00:01:03] from 1 to 100 in unicorns and IPO. [00:01:05] My focus was always around AI and data. [00:01:10] That's something that I found myself the most passionate about. [00:01:17] I was part of the first AI lab in Israel, [00:01:20] built a cybersecurity unicorn at Big AD, [00:01:24] and I also was part of a comedy that helped define [00:01:28] what would be an energetic identity, right? [00:01:31] The best practice, the protocol together with OpenAI, AWS, [00:01:36] CloudFlare, and recently I started to see [00:01:39] how everyone are talking about tokens and cost of tokens, [00:01:42] and I tried to figure out how enterprises today adopt AI [00:01:47] and what's their maturity level, started to talk about it [00:01:50] within the community that we are at, [00:01:52] but also reached out to friends of mine and people on LinkedIn [00:01:56] to get them to share a little bit more on the stage that they are at. [00:02:01] And cool, that's very cool. [00:02:02] So I mean, I guess the first question, not the main, [00:02:06] but maybe the first question that I would ask you [00:02:08] since you've been around AI, [00:02:12] I'm curious to hear how different is this age of gen AI [00:02:19] and what's going on around it from, let's call it, [00:02:23] classic traditional machine learning style. [00:02:26] I would start with ambiguity, you know, machine learning [00:02:29] or other algorithms are mostly definitive. [00:02:33] When it comes to a lot of labs or the use of authentic tools, [00:02:36] the outcomes are never the same. [00:02:38] You need to put up a lot of effort to make it produce the same outcome [00:02:43] or same level of quality of outcome across workflows [00:02:48] and use cases. [00:02:49] Sounds like people. [00:02:51] Like, yeah, I mean, if you look at my background, [00:02:55] like a lot of my work for people that aren't familiar is in managing, [00:03:00] it's called a human operating system. [00:03:02] So my background initially was in computer operating systems, [00:03:06] Linux, kernel stuff, networking, routing. [00:03:11] But then as from becoming an engineering leader of organizations [00:03:16] building software operating systems, [00:03:19] I eventually started to focus more on the human operating system [00:03:23] and culture hacking. [00:03:26] And, you know, I realized humans are pretty unpredictable as well. [00:03:31] And there's a lot of ambiguity when leading human systems [00:03:35] and you need to create certain guardrails to still get value [00:03:39] out of human systems, but you also need to provide agency. [00:03:43] And there's an interesting, ICN, actually an interesting corollary [00:03:51] similarity between what works, what I've seen work [00:03:56] to unleash the potential of people versus what works to unleash [00:04:01] the potential of agents and what guardrails do you need? [00:04:05] Like, you could give a developer four months to work on something. [00:04:09] You would waste a lot of money to give them a developer [00:04:13] team, you know, a product team, you know, multiple months, [00:04:17] you're wasting a lot of dollars and you're not sure [00:04:20] that you're getting something. [00:04:21] So there's a lot of similarities between that and it's quite similar. [00:04:25] But in terms of the waste or the cost itself, with humans, [00:04:30] you have something that is definitive, you have time and money [00:04:33] and manpower with AI, it's not definitive. [00:04:36] It could be that because of an error or a loop or something [00:04:40] that was not really configured correctly. [00:04:42] Or because people don't have the site, AI can cost you way more [00:04:47] in very short period of time if you don't have the right guardrails [00:04:51] and observability. [00:04:52] Yeah. [00:04:53] Yeah. [00:04:54] So part of the conversation, and I was chatting to somebody else [00:04:59] in the space a couple of weeks ago, and we were talking about [00:05:05] this rogue AI, you know, agents that are cost endless tokens. [00:05:12] And I think part of that conversation is, you know, [00:05:16] what's the model that you even use? [00:05:19] I mean, it's a choice to give your agents unlimited budget [00:05:24] or to give your humans unlimited tokens in a time period. [00:05:29] Like, I don't work at an enterprise. [00:05:31] I'm an individual. [00:05:33] And I know that my, the way I work is I work through the subscriptions, [00:05:37] right? [00:05:39] So if I start, if I give cloud code or codecs or tragic, [00:05:45] if you work as it's called as of today, a goal that is complex, [00:05:50] what's the worst case scenario that it would hit its window [00:05:54] and say, okay, I'm going to rest for a bit. [00:05:57] So I think it's a choice that organizations have or the people [00:06:02] that get, you know, an unbounded budget. [00:06:06] Is that how you're thinking about it or how, how, what are people doing about this? [00:06:11] Yeah, so I'll share a little bit from the conversation that I had with some enterprises. [00:06:16] First of all, the challenge is to define those, the cap itself, right? [00:06:21] Would you say 100 a month for an employee or 300 or 1000? [00:06:27] How do you make sure you do not really block productivity [00:06:31] or block real product development by putting the cap to low? [00:06:36] Also, Priceline just say that AI puts the credit card at the hand of the employees, [00:06:42] but without the oversight and training cost can spiral. [00:06:45] But high usage doesn't really mean a bad thing. [00:06:49] It's not good or bad. [00:06:51] The question is what's the impact that you get out of that? [00:06:54] And that's something that is really difficult to quantify today. [00:06:57] And it really resonates with what you say, because you can put the guardrail, [00:07:01] but if it's 100, 300 or 1000, when you don't have a way to really evaluate what the outcome, [00:07:10] whether it's wasteful or impactful to organization, [00:07:14] it doesn't really matter what's a cap. [00:07:17] So I'm going to push, I mean, yes, but I'm going to push back on this for a second. [00:07:21] Again, bringing it back to people. [00:07:25] Yes, but I can tell you that I've been working to help organizations, [00:07:30] figure out what are the outcomes of what we're doing and measuring outcomes [00:07:35] and closing the loop on outcomes for almost a decade before AI, [00:07:40] because you have a lot of KPIs or KPIs, but you have a lot of teams [00:07:46] that are focused on vanity metrics. [00:07:49] Like if you look at most engineering organizations, [00:07:52] most even product managers, they shy away from measuring outcomes and impact, [00:07:58] because it's harder to commit to outcome and impact. [00:08:01] So, I mean, it's a, I think it's a big opportunity that AI is shining a light on, [00:08:11] but it's not necessarily a really new challenge. [00:08:14] So, and the fact that it's not new doesn't mean that we have a great solution, [00:08:19] but we've learned a lot of things along the way about what's the bottleneck [00:08:24] to really focusing on outcomes. [00:08:27] And in some of my conversations and the work that I'm doing with organizations, [00:08:32] I see how hard it is for leaders to actually focus on outcomes. [00:08:39] Like I have this tool that I use, I've quoted with AI that I connect to an organization's [00:08:48] GRA instance, okay, or whatever they're using, get up linear ADO there. [00:08:55] And what it does is it analyzes the work that people are focused on, [00:09:03] whether it's the goals, OKRs, features, GRA, FX, stories, whatever it is. [00:09:10] And it tells an organization, it gives them visibility to how much of the work is managed [00:09:17] as let's do this thing as an activity, how much of the work is managed as output, [00:09:22] let's build this feature, let's do this screen, let's do this button, [00:09:26] let's add this integration, whatever, how much work is managed as outcomes? [00:09:31] Like let's enable this type of customer to do a certain thing, [00:09:35] whether it's an internal customer and internal persona or a real end user or customer. [00:09:42] And how much is impact? [00:09:44] Like things that really move the needle from this perspective, [00:09:46] which is not an ideal way to manage the work that's a side conversation. [00:09:50] You want to guess, when I look at the typical organization, [00:09:56] what's the percentage of work that is managed in each one of these categories? [00:10:03] I would say about 20% outcome. [00:10:08] Yeah, that's a good organization. [00:10:11] Yeah, yeah, and at this point, I'm not really surprised. [00:10:17] It's both hard to think about your customers, but there's also, [00:10:23] like the fact that you're willing to put it on paper or on record, [00:10:28] that this is the outcome you're focused on. [00:10:32] I'll be able to measure it at the end. [00:10:34] I'll be able to see whether you're actually moving the needle [00:10:38] and that requires a maturity of the organization that a lot of organizations are struggling with that. [00:10:43] So you see, I do, so I think that both of us agree, [00:10:53] once you implement AI in your organization, [00:10:55] it bubbles up or self-surface your DNA and the organizational culture [00:11:01] that didn't change for a while and now needs to change if you really want to push impact with AI. [00:11:07] It just made people be more structured when it comes to the implementation process [00:11:14] and the evaluation process of those investments. [00:11:17] My perspective coming from an industrial engineer, [00:11:20] so I look at enterprises and companies as factories at the end of the day. [00:11:25] Either the outcome, it can be a service, it can be a product, it can be a software. [00:11:30] But when factories started, humans were on the line, right? [00:11:35] They had to do things in order to deliver the product. [00:11:38] And then it was man and machine, right? [00:11:41] It was humans and machines and later on, it became only machines. [00:11:45] Along those ways, you had to add definitions of what's impact, right? [00:11:50] What's good looks like? What's wasteful? [00:11:52] How much money you put on that line and what's the outcome that you get and whether it's valuable? [00:11:59] The same should be when we invite more digital twins or digital helpers [00:12:06] to be part of our agentic repositories or organizational workflows. [00:12:11] But the situation coming back to LMs is that LMs are not like a machine [00:12:18] because the outcome cannot really be trusted. [00:12:22] It's not like the same every time that the LMs would do one workflow, [00:12:28] the output would not be the same. [00:12:31] So in that perspective, you need better observability and better control [00:12:35] on the actions and the activity of the agent. [00:12:39] And you need also better monitoring on what's out of your guardrails [00:12:44] and in terms of costing, in terms of abuse, in terms of waste, and in terms of impact. [00:12:49] And without having that definition, and that's by the way, [00:12:52] something that I do see also already happening in enterprises that are not tech forward. [00:12:57] Some enterprise that I chat with, I won't say their names, [00:13:02] companies that are not, doesn't really develop software, already started to define OKRs [00:13:08] and KPIs for each business unit. [00:13:11] And today, they ask during performance review, they ask the employees whether they utilized AI [00:13:17] in order to achieve their KPIs and OKRs. [00:13:19] And later on, when you'll see more agents over there, [00:13:22] they will try to tie those back to agent outcome or agent impact. [00:13:29] So you see how those building blocks are stacking along for that future. [00:13:34] So I 100% agree with the factory metaphor. [00:13:39] And I think applying some of the techniques from the factory continuous improvement movement, [00:13:48] Goldrat, you know, lean is what we need to do here. [00:13:54] I, and I think it applies at multiple levels. [00:13:58] So one level is that I'm seeing as useful is if people that are deploying AI in the organization, [00:14:08] those people with those OKRs and KPIs are looking at their business processes [00:14:15] using the factory lens and they show up as industrial engineers [00:14:20] or continuous improvement engineers or professionals. [00:14:23] That's the right mindset to find the AI use case that would really move the needle [00:14:28] because like Goldrat said, this, you know, any improvement away from the bottleneck [00:14:33] or from the constraint is meaningless and any improvement directly at the constraint [00:14:37] is, you know, a real multiplier. [00:14:40] So that's the thing you want to focus on and that's the way to drive AI usage effectively [00:14:47] from activity to real impact. [00:14:50] But if you go back to the, let's call it the design factor, [00:14:54] where it's the organization that is building these capabilities that we're trying to. [00:14:58] I mean, I see a couple of levels of what it will need a bucket for experimental, right? [00:15:04] If you think about it, you have a bucket for production, like what we do with AI [00:15:09] to move things to production to product. [00:15:12] But you need to have a bucket for experiments, you know, to build things, new things, right? [00:15:16] Yeah, but I think even that is a design factory. [00:15:21] The design mindset still applies. [00:15:24] Bank driven still applies. [00:15:25] You still want to think about intent. [00:15:27] So you give some agency to figure things out. [00:15:29] And then you want to tighten the design. [00:15:32] Correct. [00:15:32] You'll explore, converge, build, all of that still applies even when you're experimenting [00:15:38] but I guess there are a couple of different guardrails that are important. [00:15:45] You mentioned observability and I'd love to go into that. [00:15:48] What I see in a lot of organizations is the first thing that they want, [00:15:53] that they tackle tightening is they are building what we ask it to build. [00:16:00] And that is the thing that we, is the thing that it built operating to the specification [00:16:08] that we aligned on. [00:16:11] And that's one level of the challenge. [00:16:13] I would say building something that meets the specification using, you know, reasonable [00:16:21] budget, you know, time is not that much of an issue. [00:16:25] And I would say observability at that level is not that. [00:16:29] I mean, it's not that big of a challenge. [00:16:32] I mean, it is a challenge to make sure that you do regression testing and all the things that, you know, [00:16:37] a good solid software factory should be doing. [00:16:40] And if they're not, AI is amplifying the problem. [00:16:44] Yeah, it also can help them deal with that. [00:16:49] But you still see organizations that the developers are focused on AI coding [00:16:56] and nobody is looking at the downstream bottleneck and improving that because it's people that [00:17:02] aren't as comfortable with, you know, with leveraging AI. [00:17:07] But it takes time. [00:17:09] Yeah. [00:17:10] Organizations solve that and scale the AI from activity to good output. [00:17:14] Now it's the real challenge of observability. [00:17:17] Like, how do you observe whether they actually move the needle the way you wanted it to? [00:17:24] And the first challenge is that it's often not even defined. [00:17:29] What did you want to achieve? [00:17:32] And that's correct. [00:17:33] It's not necessarily defined in a measurable way. [00:17:36] And you cannot always attribute everything to AI, right? [00:17:40] You start with, let's say, small steps towards it. [00:17:45] It's really hard to say, hey, your AR are tripled just because you implemented AI, right? [00:17:51] And maybe that it was tripled because of economical changes and maybe you have now great sales [00:17:58] team. [00:17:59] It's not just because of AI. [00:18:01] But the question is whether you can attribute first. [00:18:04] Like, do you have a way to attribute every kind of change in your organization and tied back [00:18:10] to an agentic workflow or AI process? [00:18:14] You start with that, right? [00:18:17] Maybe to factories, right? [00:18:19] You need to trace back to the cause to understand whether it was part of or impactful at all. [00:18:27] So I would say start with that. [00:18:30] Start with realizing whether your agentic code output something that was at least embedded [00:18:38] into your product eventually or was it just a wasteful session or multiple sessions that were [00:18:44] wasteful, even not experimental. [00:18:46] I think that taking those baby steps would help eventually with measuring impact. [00:18:51] And that's my kind of advice to everyone while trying to implement AI. [00:18:56] So let's maybe take a use case and work through it. [00:19:00] So a common one that I see all over is we're having a lot of meetings in the organization [00:19:07] and we believe that if we aggregate the recordings from all of these meetings and add that to [00:19:15] context, it will be useful. [00:19:19] So if I take the approach that you talked about, the first level would be to measure how many [00:19:29] people in the organization or maybe it's one person or one team that took on this challenge. [00:19:34] Okay? [00:19:35] And you can look, you need to be able to say these sessions that used these amount of tokens [00:19:44] were used to work on this challenge, on this OKR goal, whatever structure you used to manage [00:19:52] that. [00:19:53] And then to be able to say at each point in time, okay, so far we've had five people work on this [00:20:00] with 10 sessions on it, we've wasted $1,000 in tokens on it and so far there's no output. [00:20:10] For example, so far it's only games. [00:20:13] Nobody checked in and EPR. [00:20:15] Nothing is running as a capability in the organization. [00:20:19] How long would you allow the experiment to happen? [00:20:22] That's a good question. [00:20:24] I mean, I think of course the answer is it depends. [00:20:28] I'm a consultant. [00:20:29] The answer is it depends. [00:20:31] I think the important conversation is up from talking about key criteria. [00:20:35] So when you manage your portfolio of experiments, you'd say, okay, [00:20:39] this is one of our top priority things. [00:20:41] We want to run on it. [00:20:43] But like we do in VC funding, you get pre-seed money. [00:20:47] You get a couple of weeks with a certain token budget. [00:20:51] Maybe initially, by the way, I would say you don't get a token budget. [00:20:55] You just use your subscription. [00:20:57] Yeah. [00:20:58] Try to run with it. [00:20:59] Try to run with it. [00:21:00] If you come back and show promising results, hopefully, you know, I wouldn't consider the fact [00:21:08] that there's nothing to show for us promising results. [00:21:11] So probably there needs to be some sort of output. [00:21:16] It can be an MVP. [00:21:17] It can be a proof of concept, whatever. [00:21:19] But you come back with it and you show this as promise. [00:21:22] There's an updated mini lightweight business case. [00:21:26] Maybe even AI looks at that business case and says, okay. [00:21:29] Now, even if we're not sure that this will eventually work, [00:21:34] it's still worthwhile to explore that option. [00:21:36] That's a real options approach and innovation accounting. [00:21:40] Now, it makes sense to continue to work on this, maybe. [00:21:44] Maybe not. [00:21:45] Maybe we kill it at that point. [00:21:46] And there's probably a good conversation to have in each organization around [00:21:51] build care that that's a decision. [00:21:53] Take place. [00:21:54] Yeah. [00:21:55] What do we empower individuals to decide? [00:21:59] Maybe it's a process of like today, a lot of organizations have a situation [00:22:04] where you run out of tokens. [00:22:05] You automatically can ask and get the tokens, which is not a bad approach. [00:22:10] But I can see a reality where when you ask for tokens, you say, [00:22:15] this is why I want these tokens. [00:22:17] This is what I'm working on. [00:22:18] This is the evidence that I have so far. [00:22:20] And I can automatically say, you know, and cut away. [00:22:23] You know, this is interesting. [00:22:24] Go ahead. [00:22:25] We don't need to go and talk to anybody. [00:22:27] You know, I'm just helping you think through this. [00:22:30] And if you think it's worthwhile, go ahead and do it. [00:22:35] And go ahead. [00:22:36] Maybe I should take you even back. [00:22:39] Lots of conversation starts with whether I should buy or build it, right? [00:22:44] That build mode drove everyone to say, I'll just use cloud and build it myself. [00:22:49] Without really thinking about the experimental stage, you know, [00:22:53] how much time will it take you to ramp up to do something that is not part of your core business, right? [00:22:59] It's not what you're supposed to do as a business to build your own tools [00:23:03] because your manpower are not built to do that. [00:23:06] Your DNA is not built to do that as well. [00:23:09] And whether you have the right capacity to also not just build, [00:23:13] but iteratively support and maintain and make it work in scale. [00:23:18] And you know, those things are, let's say, I wouldn't say ideas, [00:23:22] but best practices that most wouldn't think about in advance. [00:23:26] But once you decide to go that route, [00:23:29] you need to understand how much time it will take to ramp up, [00:23:32] how much time it will take to trial and make it happen, [00:23:35] and whether the outcome is better than a product that you could have bought it for. [00:23:40] So you need to look at desirability. [00:23:42] Would people even use it? [00:23:44] And until you know, don't worry too much about build versus buy. [00:23:49] Once you align on desirability, there is the question of visibility and viability. [00:23:55] And there you do need to think about total cost of ownership. [00:23:58] I 100% agree. [00:23:59] And I would also say that if you look at all of this spec-driven frameworks, [00:24:04] harnesses that I looked at, they don't ask those questions. [00:24:08] You try to engineer the tool. [00:24:10] That created harnesses because they want to engineer. [00:24:14] They're not asking questions around before you do anything. [00:24:19] Maybe even before the spec, for sure after the spec, [00:24:24] is there an open source for this? [00:24:26] Is there a subscription service for this? [00:24:29] Why are we doing this ourselves? [00:24:32] So what I do see in tech-forward companies, those that I spoke with, [00:24:36] some of them already defined a role for each BU, [00:24:41] a person that's supposed to evaluate those things before they start rebuilding. [00:24:46] Think about a person, like today we have a go-to-market engineer. [00:24:50] They build agent tools in order to drive go-to-market business using automation. [00:24:56] It doesn't have to be agents or tokens, et cetera. [00:24:59] It can be also taking things with ZAPR and tying them back altogether [00:25:04] to make things run smoothly or faster or more efficiently. [00:25:08] I believe that in the future, you'll see more business engineers. [00:25:11] Within business units, you'll have business engineers that their projects [00:25:17] and problems will be tied to what we want to automate [00:25:22] and make things run smoothly. [00:25:24] They will decide whether to build or buy based on their overall perspective, [00:25:28] what's painful, what's really will move them middle, [00:25:31] and what will allow us to amplify it also across the company [00:25:35] and not just build a point solution for now that maybe later will cost us more [00:25:40] because we need to maintain it. [00:25:42] So let's take it to the next level and assume we're really AI-native. [00:25:46] What I see happening is that each one of these business engineers [00:25:50] has a digital twin agent that is aware of what's the current tech stack, [00:25:59] whether it's through configuration management, assets, mapping. [00:26:03] It has access to a map of what you currently have. [00:26:06] Maybe you even have the way to do this internally, [00:26:10] that the person that is starting to look at it with their AI [00:26:13] is not thinking about. [00:26:15] I've coached a go-to-market engineer where AI told them to do something [00:26:20] and they said, why didn't it suggest to go to do it in Slack [00:26:24] that is connected to Salesforce? [00:26:26] Does it know that you have Slack that is connected to Salesforce? [00:26:30] The context is missing. [00:26:31] It doesn't have the context. [00:26:32] So we built on the fly a skill of mapping the ecosystem. [00:26:37] We told go look at Slack, map everything that you know. [00:26:40] It found out that there are Salesforce and there's a lot of stuff. [00:26:43] It's not to have it grill you for a second, [00:26:46] but understand the ecosystem and try to dump the considerations, [00:26:51] to map the considerations that the business engineer will take into account, [00:26:56] so that eventually what you want when it comes to agency [00:27:00] is everybody that is working on a iOS case. [00:27:03] And at some point, agents that are trying to find effective use cases [00:27:09] they can go to this agent and ask it. [00:27:12] It doesn't make sense to do this. [00:27:14] Do we have a way to do it right now? [00:27:16] If we don't, you know, is there something off the shelf that would do it well? [00:27:20] Like what are the options? [00:27:23] I like what I don't want to see is bottlenecks [00:27:29] that are because we're relying on single point of failure people. [00:27:35] I don't want to see people that become bottlenecks [00:27:38] and need to be in each one of these loops. [00:27:41] And now in order to really make it to you build, [00:27:45] get yourself outside of this loop while still managing the decision-making. [00:27:50] That for me is the question of scaling. [00:27:53] So actually now in order to really make it work, [00:27:56] getting back to the problem that AI surface all the, let's say, [00:28:00] structural, cultural challenge of an organization, [00:28:03] making sure that you have the right knowledge base [00:28:06] that is updated, curated, tagged, classified, et cetera. [00:28:11] That's a huge problem. [00:28:13] Enterprises, it will be very hard for them to build something that you mentioned, [00:28:18] like them mapping the stack and putting the context in the stack [00:28:21] based on departments and different be used. [00:28:23] Smaller companies will be able to close the gap and maybe make some order [00:28:27] and make sure they have the right updated knowledge base [00:28:31] and their AI will get the context that it will be fresh and correct. [00:28:35] But the challenge, it was always data. [00:28:38] Looking back like five years ago when I was part of Big AD, [00:28:42] looking back at how organizations manage data in structured, unstructured, cloud, [00:28:49] all of that allowed me to understand that enterprises will, [00:28:53] they will take more time to really adopt AI properly. [00:28:58] It's not just the guardrail and governance, it's also to me to make sure [00:29:02] your infrastructure, your data infrastructure is ready for that. [00:29:06] And did you see anyone trying to solve that better? [00:29:10] I haven't talked to anybody that's solving this at the enterprise scale. [00:29:15] I'm really curious, like, I don't think any of the people that worked at HPSCMDB [00:29:21] when I coached them on how to do agile 15 years ago is still there. [00:29:26] But I'd be curious to see here what the CMDB and asset management players are doing about this. [00:29:33] I can imagine them and service now and these different players having a perspective on this. [00:29:39] Maybe the other area, all the way downstream in the process that I'm curious what you're seeing is [00:29:48] observability. [00:29:50] So observability, not whether the solution works, but whether it's, or not whether the solution can work, [00:29:58] not regression testing and, you know, and evolves. [00:30:04] But are people actually using it? [00:30:07] Is it useful? [00:30:10] Is it in production creating results that are within boundaries? [00:30:16] All of that closing the loop on, it's not necessarily impact. [00:30:21] I agree with you. [00:30:22] We don't need to be at the point where we can attribute, you know, collecting all of the Zoom [00:30:28] or Google need transcripts and putting them somewhere. [00:30:31] It would be hard to attribute that directly to revenue growth or profitability. [00:30:37] But if we made an assumption that, I don't know, will make better decisions [00:30:44] if we have access to this or there would be. [00:30:48] It can take your meetings. [00:30:50] It can save time. [00:30:51] I don't know. [00:30:52] There needs to be an hypothesis about what's the value? [00:30:56] Why are we doing that thing? [00:30:58] How do we, how do we close the loop on, are we really getting that value? [00:31:05] So for example, if we collected all of those transcripts for everybody, we can observe that [00:31:12] we've collected them. [00:31:13] That's a good start. [00:31:14] That's better. [00:31:15] That's better than nothing. [00:31:16] And we can see that somebody is using them. [00:31:19] Exactly. [00:31:20] So we need to be able to see the limitary that people are accessing this. [00:31:25] And that's a good start. [00:31:26] Is it driving decisions? [00:31:28] That's a next level. [00:31:30] That's exactly the next level. [00:31:31] The next step would be to tie it to a business objective in your organization and see whether [00:31:37] it helped somewhat. [00:31:40] Either by making it faster, shortening the time, less people need to be involved, faster [00:31:46] maybe decision making. [00:31:47] I saw some examples of agents that help with an approval process that took, let's say month [00:31:54] within an organization and now with agents that come with all the needed context for every, [00:32:00] even if there's person at the end of the day that needs to approve, they have all the information [00:32:05] they need and approve the process. [00:32:07] Exactly. [00:32:08] It's a full key. [00:32:09] They don't need to go back and forth. [00:32:10] It's like the... [00:32:11] Yeah. [00:32:12] The world there in Goldrat's example, you always want to see the white light from the world they're [00:32:19] welding rather than going back and forth. [00:32:21] That's what you want to measure, right? [00:32:23] Exactly. [00:32:24] So we are in an early maturity stage. [00:32:27] That's what you see at enterprises. [00:32:30] Smaller companies are adopting it faster. [00:32:32] And that's why you hear different startups that are saying that they consume a lot of tokens [00:32:37] and they are burning tokens. [00:32:39] It's not a bad thing. [00:32:40] You just need to make sure that you burn them efficiently. [00:32:43] You know, your people are doing, they're not abusing it, of course, but they are using the [00:32:48] harness correctly in a way that wouldn't cost you more money. [00:32:52] But eventually, high usage can drive faster product development, can drive faster decision-making, [00:33:00] can streamline your business objectives, could be more customers, bigger pipeline or building [00:33:07] a new pipeline. [00:33:08] It really depends on your kind of initiatives and use cases and what you evaluate that would [00:33:13] be more impactful for you. [00:33:15] Yeah. [00:33:16] So, all right. [00:33:17] So maybe let's pause for a second. [00:33:22] And that's one good advice that you're offering for people out there, which is, it's okay [00:33:29] to experiment, it's okay to see usage. [00:33:34] I would say, while that's going on, have some default guardrails. [00:33:39] It's okay to extend if there's outcome, but it's safer to start with some guardrails. [00:33:49] And I would say be careful with how you navigate the conversation around what people do with [00:33:57] their subscription. [00:33:59] So it's the tricky, goal deluxe of encouraging people to try and experiment, but it's not [00:34:07] about use the tokens at all costs or whatever. [00:34:12] It's all about psychological safety, I would say, eventually it depends on the DNA of the [00:34:16] organization. [00:34:17] Exactly. [00:34:18] If it's a healthy DNA, people would feel safe to experiment. [00:34:21] Even if it's a toxic culture, you'd get toxic token usage and activity theater and people [00:34:29] that are afraid to improve their throughput because they're thinking that they're cutting [00:34:36] the branch that they're sitting on. [00:34:38] But let's assume you're an organization and you're at that point. [00:34:45] You're seeing people using, they're starting to use more and more, what would be the first [00:34:51] next step that you would suggest for these organic data streams? [00:34:56] So it starts with education, making sure people are educated enough to use that. [00:35:03] And instead of telling them, hey, just use it, you need to help them as a manager to define [00:35:08] use cases or OKRs that you would like them to try and achieve using those tools. [00:35:14] And similarly to how you do quarterly reviews, you would like to do maybe a monthly review [00:35:22] on the process itself and give them helping hand in that kind of process. [00:35:28] So let's go. [00:35:29] I agree with you. [00:35:30] Let's do maybe an example of what such an OKR that a manager gives their team might look [00:35:38] like. [00:35:39] At what altitude do you think that would be? [00:35:42] So an easy thing would be product, right? [00:35:46] When you see a scrum team and you ask scrum team to build that capability into the product, [00:35:52] you would like to see them using AI in order to do that, right? [00:35:56] And evaluating how much pieces of code was written by AI, right? [00:36:01] Evaluating how much of those pieces of code passed the review, right? [00:36:05] Went from PR and QA and really is part of the product. [00:36:10] And eventually you would like to also monitor whether there are regressions testing that found [00:36:15] things that were not supposed to happen because of the use of AI. [00:36:19] Having those loop and having sure that there is a feedback and sharing that feedback with [00:36:25] whoever initiative the usage of those coding agents will learn and be educated about what [00:36:32] should happen next in order to improve it, that's the gold I think standard here for a [00:36:37] start. [00:36:40] A later one, a lot of one for the product in this. [00:36:44] Yeah. [00:36:45] I worked with Siemens in Israel many years ago and one of the OKRs, we didn't call them [00:36:51] OKRs then, but one of the OKRs that the VP gave the organization was, I want to see experiments. [00:37:01] I want to see compound engineering, you would call it today. [00:37:05] I want to see us changing the way we work all the time. [00:37:11] So I don't want to just see experiments, I want to see failed experiments. [00:37:16] I want to see that we're challenging our comfort zone and, you know, we're trying different things [00:37:21] in how we work with AI all the time. [00:37:25] That would be another interesting, interesting example. [00:37:28] And you can start with a Slack channel or a town hall call when everyone shares, but when [00:37:34] you scale and have lots of people, you need a kind of a system or a software that will [00:37:39] help you monitor that in real time. [00:37:41] Understand where the organization moved from experiment or what's the bucket on experiment [00:37:48] and what's the bucket on real production value. [00:37:51] And you would like to kind of make sure it has kind of breaks and, you know, so let's be [00:37:58] clear what I'm talking about is experiments in how we work. [00:38:02] Yeah. [00:38:03] And if you think about your Cloud MD file or your cadence and the file, there's the story [00:38:10] about the Lean Sensei that came to a manager and saw the workstation and said, you're cheating [00:38:18] the company out of money, you're, you know, you're not doing your job. [00:38:25] Because what they saw is that the standard work to comment, which describes the standard [00:38:31] operating procedure for that whole process, it was printed on Fox paper for this reason. [00:38:38] And they saw that it was yellow. [00:38:40] You could barely see. [00:38:42] And that indicates sort of like I would look at your Cloud MD file and see that it's from [00:38:47] a year ago. [00:38:49] No chance that the Cloud MD file from a year ago reflects, you know, compound engineering [00:38:55] and continuous improvement. [00:38:57] There's no chance that an agent that is configured with guidance from a year ago is the most effective [00:39:05] way to do things today. [00:39:07] Yeah. [00:39:08] So that's something else that you can measure. [00:39:10] And you know, that you can do with how many PRs do you have on your preferences files? [00:39:16] Are your skills evolving? [00:39:18] Are you creating new skills? [00:39:19] Are people using these things? [00:39:20] Yeah. [00:39:22] Metal level, metal loops that I think organizations. [00:39:27] We need to start to look at. [00:39:30] In order to really do that, we didn't mention that there is a need also for some security [00:39:34] tools, right? [00:39:35] Not just pure observability like data dog, but really having the ability to define what [00:39:40] MCPs are safe, what skills are safe, whether they're integrations that are unsafe for [00:39:45] organization, DLP, you know, making sure the data is not leaked from highly sensitive [00:39:50] location into your agents, your coding, et cetera. [00:39:54] Those are things that are still being built. [00:39:56] I see startups that are still in stealth that are building towards that. [00:40:01] And that's great. [00:40:02] It will allow more organizations to adopt AI. [00:40:05] I do see that even without those guardrails and definitions are enterprise that I talk with [00:40:10] that told me that they are starting with sandbox environments, you know, in order to allow [00:40:15] people to experiment and have kind of safe zone without corrupted kind of outcomes. [00:40:21] So again, still early maturity phase, we are talking about real future here. [00:40:27] The reason that we don't see all those challenges yet is because there are not yet those security [00:40:33] guardrails as well implemented and working seamlessly in the organization. [00:40:38] Yeah. [00:40:39] I mean, there are a lot of missing pieces, both on the technical stack side, as well as [00:40:45] from a human fluency perspective, like most people in most organizations are barely making [00:40:51] their first steps out of chat mode. [00:40:55] Like I think it would be interesting to see now that both flawed and chat GPT integrate [00:41:02] co-work or work mode closer to chat, I think I'm guessing. [00:41:08] They just are doing is they're trying to drive adoption of a gentic mode by making it [00:41:16] closer to what people are doing, and it will be very interesting to see will that move the [00:41:20] needle and drive more people, especially out of the coding engineering organization to explore [00:41:28] a grant interfaces more. [00:41:31] Two things that really happen this week is that cloud desktop to co-work and match that [00:41:36] into the chat, and also allow the people to continue conversation also on the mobile, [00:41:43] even when your computer is off. [00:41:47] You will not be able to have the ability to control files in your desktop when your computer [00:41:52] is off, but it's still allowing you to do much more than before. [00:41:57] I do see companies that are intentionally trying to shift from driving cloud within just the [00:42:05] development side to business, and that's the most interesting thing. [00:42:09] You don't see a lot of waste, you don't see a lot of tokens that are being burned because [00:42:14] it's so early in maturity, but you do see some exciting experiment and some outcomes that [00:42:21] are not being shared online. [00:42:24] There's a lot of slope outside. [00:42:27] You see some kind of nuggets of success in organizations that are trying to drive business [00:42:32] people to use AI encoding, even to develop their own workflows and automations. [00:42:39] Yeah, and I go back to your comment about build versus buy, and I think there's another [00:42:46] variant of that, which is when you look at the people outside of product and engineering, [00:42:54] it's not clear to them who's going to build. [00:42:58] If you look at the typical organization, I mean, I talked to a cheese product technology [00:43:05] officer for a very successful Israeli company a couple of months ago, and he said, my organization, [00:43:14] like, we're on top of this shit. [00:43:18] We do spec driven, it's a gen, we're 10x, all good, and I asked him, okay, so what's going [00:43:24] on beyond your organization? [00:43:28] And he said, you know what, I don't know. [00:43:30] I don't know. [00:43:31] And I talked to some other people elsewhere in the organization and, you know, they're still [00:43:37] thinking about do we use glean, do we use this, do we do that, it's not clear who's going [00:43:44] to work on making closing the books faster, integrating young and the other tools and Sales [00:43:52] course beyond what the tools provide out of the box, which is great, which is probably [00:43:56] a great starting point. [00:43:58] What do you do beyond that? [00:43:59] Who is going to tackle that? [00:44:00] So is it go to market engineers? [00:44:03] What does that look like? [00:44:05] What about organizations that don't have, you know, the tech talent inside their go to market [00:44:11] organization to do this? [00:44:12] Don't need to hire. [00:44:13] It is coming from the vendors, is that the right approach, is it just consultants? [00:44:20] It's a mix. [00:44:21] Oh, the one certainly on how that's going to evolve. [00:44:25] Eventually it's a mix that puts lots of opportunity outside for people in advisory as well as other [00:44:31] people that would like to be more product managers, because you see how people try, how [00:44:37] organization try to nurture and make people ascend to become product managers in those [00:44:43] business units. [00:44:44] So together with a go to market engineer or business engineer, you'll see products. [00:44:49] Are you seeing that dynamic? [00:44:52] You do see some. [00:44:53] It's not yet there. [00:44:54] You do see some that are trying. [00:44:57] Not everyone are real product managers, eventually you can say, Hey, I'll give you the title and [00:45:02] you have maybe some knowledge about my organization, but eventually you need to have the right kind [00:45:07] of a set of skills to do that. [00:45:09] But you see more and more the need of product managers and engineers in other BUNIs units [00:45:15] in order to drive that adoption, because it's not just the build mode, you don't just [00:45:20] build those agents or build those tools for the organization, whether it's an application [00:45:25] or you need a way to also maintain it and put it in production, you know, in business production [00:45:30] to allow others to use. [00:45:32] And in order to do that, you want to ask, you know, your best seller to do that and build [00:45:38] that. [00:45:39] You may need your advice. [00:45:40] I would say even if it's buy mode. [00:45:43] If it's buy mode, you want somebody with the product hat, as well as an engineering hat [00:45:49] to think through what is it that we're buying? [00:45:53] Does it make sense? [00:45:54] How it fits our ecosystem? [00:45:55] How it fits our flows? [00:45:56] What are the outcomes that we're focusing on, closing the measurement loop that this is [00:46:01] actually driving value rather than just we spend some money on a vendor? [00:46:06] I mean, five. [00:46:09] As long as it's not seat based and it's a usage based, you'll see more and more organization [00:46:15] taking that stuff, concept of whether I should pay now or just use tokens for that. [00:46:22] Yeah. [00:46:23] So, I mean, I think it's an interesting dilemma. [00:46:26] You talked about growing product managers or identifying and staffing product managers [00:46:33] and engineers outside of the product and engineering organization, outside of IT. [00:46:39] One of the things I'm trying to figure out is, okay, that's one option. [00:46:43] Another option is not necessarily staffing or defining a product management role, but [00:46:54] teaching product skills and engineering, meaning, let's call it workable, serviceable product [00:47:01] and engineering skills to people in the business. [00:47:04] I think there's an interesting opportunity that AI can play in that. [00:47:09] So, could you teach now that people are going out of jet into work mode? [00:47:17] Could you teach work mode to apply some of these techniques, to teach people to think [00:47:22] like product managers, to think about engineering aspects, could an AI agent help people find [00:47:30] where the value is, help you manage your consideration and investment in a way that maximizes [00:47:38] value realization from AI. [00:47:40] That's like getting back, getting back to context, that AI agent will need a good context and [00:47:45] you'll need to build a rug that represents where it should look for and make sure it's [00:47:50] updated, again, not a year old skills needed, product skills, this is fascinating. [00:47:58] Let's keep talking about this as we each look at the market from different perspectives, [00:48:03] but I think there's a lot of alignment on this. [00:48:06] So thank you for staying with us, dear listener. [00:48:11] Thank you, Tomer, for coming and see you next time on Skilling AI from Activity to Impact.