Developers of the System, Not the Code: Inside Next Insurance's Agentic DLC w/ Shay Mandel

Help me think through how this episode applies to my situation. Start by asking what I am trying to change. Separate the episode's claims from your suggestions, and say when the notes do not support a claim. Use the transcript to find passages, then check the audio before quoting. Transcript: https://yuvalyeret.com/podcast/episodes/developers-of-the-system-not-the-code-inside-next-insurances-agentic-dlc-w-shay-mandel/transcript.md ## Published episode notes "We are now developers, not of the code. We are developers of the system." Most AI coding rollouts end up with faster typing and the same delivery system. Shay Mandel's team at Ergo-Next Insurance went after the system instead. They broke the whole product development lifecycle into skills an agent runs , problem definition, PRD, engineering review, technical design, implementation, testing, code review , and told everyone to stop writing the artifacts and start fixing the thing that writes them. 00:00 Welcome, and why this conversation 00:31 Shay Mandel, Ergo-Next Insurance, and 200 people building product 03:46 What you should get out of this conversation 05:14 Breaking the development lifecycle into skills 06:37 The engineering review skill that runs before engineering reviews 08:38 How the engineers actually feel about it 10:40 Developers of the system, not of the code 11:11 Auto-improvement loops: pointing an agent at a KPI 14:21 Where humans still decide, and where they don't 21:21 Digital twins of your best insurance experts 22:20 Ask-engineering and ask-product agents across time zones 24:21 What changed in how the work is managed, and what didn't 29:24 Epic-level Kanban and flow metrics for human and agent handoffs 30:39 Advice for leaders starting the journey 31:32 Setting expectations: 60% to 95%, and why the last 5% needs a human 33:29 When you tell a PM to open a terminal, you lose them 35:24 Avoiding skill sprawl: one shared brain, not 200 37:44 Beyond product and engineering: actuaries, finance, HR "They are not supposed to write anything on their own anymore. They just need to fix the skills or fix the context." - Shay Mandel "The initial version will probably be sixty or seventy percent accurate, and we'll get to ninety or ninety-five. The extra five percent is why we need a human in the loop." - Shay Mandel "You need to think from a product perspective, even about your AI harness and your AI capabilities." - Yuval Yeret Links and resources Shay on LinkedIn: https://www.linkedin.com/in/shaymandel/ Product Leaders AI meetup: https://luma.com/ProductLeaders.ai Yuval's Linkedin – https://www.linkedin.com/in/yuvalyeret/ Scaling AI: From Activity to Impact , yuvalyeret.com/insights Don't Redesign Your Process Yet. Change Who Writes the Artifacts. – https://yuvalyeret.com/blog/change-who-writes-the-artifacts-before-your-process How Next Insurance Broke Its Whole Lifecycle Into Agent Skills – https://yuvalyeret.com/blog/how-next-insurance-broke-its-lifecycle-into-agent-skills ## Transcript Riverside's published transcript for this episode. Speaker labels are machine-generated; check the audio before quoting. Episode: https://yuvalyeret.com/scaling-ai-podcast/developers-of-the-system-not-the-code-inside-next-insurances-agentic-dlc-w-shay-mandel/ Source RSS GUID: 432a1ffe-9ae5-4258-aad6-5b2e118eeada Source: Riverside podcast:transcript URL ## Transcript Yuval Yeret: Welcome to Scaling AI from Activity to Impact. I'm Yuval and today with me is Shai Mandel. Shai is leading product at Next Insurance, and we've had an interesting chat about his work on getting an agentic development lifecycle up in the air, and I thought let's continue the chat over here. So the listeners can benefit. Hi Shay! welcome to the show. Shay Mandel: Hi, thank you. Thank you for having me. Excited to be here. Yuval Yeret: So for people who aren't familiar with you or with Next, can you give a quick context of who you are, what you're doing, what's the context that you're working Shay Mandel: Yeah, so I'm I'm kind of throughout my career I was always between the kind of on the boundary between engineering and product and I'm a developer at heart, but I also like to see you know big systems, how we impact, how we impact customers, how we improve processes. So it's both from the engineering management side and on the product management side. I joined Next more than seven years ago. We were quite a small company, but we grew like almost a hundred percent every year in terms of revenue and na and last year we were acquired. We are doing small business insurance. So think about like I don't know, technician that comes to your house, he needs to have a general liability and maybe if he has an employee he needs to have workers comp we provide all these kind of coverages digitally. You can just go online and create a policy, no need for an agent. We can also work with agents, but that's like our kind of preferred way. and we grew very fast because it's a it's a very big market and a lot of demand in the US, and and doing it digitally is something that we're unique. Other companies do not provide it yet because it's a very sophisticated kind of line. Very each profession is different, and so on. Yuval Yeret: Any discount for Israeli podcast hosts that run small consulting businesses? Shay Mandel: Yeah, I I think we don't do any any discounts. It's all all AI and and Yuval Yeret: It's all this price. Shay Mandel: logic and yeah, that is pricing it in the right way. We are trying to tailor the price very much to the the exact need. And usually our policies are pretty priced very well. Yuval Yeret: Cool. Just as a side side note taking to myself to to consider it for next year. Shay Mandel: Yeah. Yuval Yeret: So so Shay Mandel: Yeah. Yuval Yeret: how many people are building products or are in product and engineering at Next these days? Shay Mandel: overall I think it's w we are kind of separated the functions into a few functions. Engineering obviously, product management, handling the kind of the user experience and you know different marketing channels and so on. And we have insurance product management. They own the development of the specific you know coverages, limits, all of that within the insurance, and for each specific business type, what needs to be the right coverage and so on. over all these three groups, around two hundred, most of it is like about, you know, engineering is more than half, but then, you know, product, insurance product and design altogether it's about another half. Yuval Yeret: Okay, cool. That that separation between the business product manager or the business product product manager and the technical product manager is a familiar one. I work quite a bit with a big futures exchange and they have a similar similar situation where where they have this interesting split. Cool. So Maybe be before we dive deeper, let's let's write something that I'm trying to introduce to the podcast. For the listener that's tuning into the conversation, what do you think what would we hope is the impact that today's episode will have for them? What will they learn? What will they be able to do, maybe? Shay Mandel: Yeah, so I I think w we won't get too technical, but I think we can talk about kind of, you know, what's kind of a nice goal for a company that really wants to adopt the agentic development lifecycle. so some ideas over there and you know what what kind of a vision that we have. Some of the struggles getting there, a lot of learnings, a lot of things that you know you need you kind of caveats and Just maybe even setting expectation of how long will it take. And then yeah, maybe I can talk a little bit about, you know, how do we get and w where we are now and how do we get there. Yuval Yeret: Okay. maybe w we can also talk about the results. What what are you seeing from a measurable perspective? What's the impact of this agentic lifecycle? Cool. All right. Sounds good. So let's let's dive in. So When you look at let let's maybe talk about current state. What does the current state of development lifecycle look like for the majority Shay Mandel: Yeah. So Yuval Yeret: of the organization, let's say? Lo or what do you Shay Mandel: Yeah, so I I can't say it's the majority for everyone yet. We try to focus first on you know a few niches and and make sure that we solve it over there. But it does affect most of the people. currently what we did we define we kind of broke down the existing processes. We didn't change the processes much, but we said, you know, in order to create good requirements, there are a set of steps you need to do, and we created a skill for each one of them. So like describing the problem. Then analyzing the problem, getting some data, maybe doing some w user research, creating wireframes, all of these are things that you usually do. And then eventually you articulate everything in in one PRD where you define the solution. So we broke down these into skills. Each one is a skill of its own. There is like an Uber skill that kind of helps you route. So it could tell you you say I want to do this PRD, it it can tell you, you know, what in which stage are you, and then it will direct you. And then it helps you, it's interactive, collects all the it asks you like, okay, you want to define the problem, let's define it. what how do you measure it? What do you expect the impact to be, and so on, and it kind of interviews you and eventually it creates the a section in the PRD which is describing the problem. And then it asks you, you know, do you have data or do you want me to get data to to kind of give them more context? And do you want to set the goal and so on? So that's on on the kind of the product side. And then we have we know that usually we hand off to engineering and then there are a lot of reviews and questions and there is kind of a lot of feedback. We created a skill for that, so we call it PRD review by engineering. And this is a skill that goes and looks at everything and it makes sure that you know for every domain and so on, the the requirements actually answer the questions or the details that we need. And provides feedback for the PM and also suggestions on how to fix it and Yuval Yeret: So it preempts in a sense the the actual engineering review in the hope Shay Mandel: Yes. Yuval Yeret: that when the engineering looks at it or when the engineering bots get it, there there's less rework. Okay. Shay Mandel: Yes, exactly. So it tries to do this as much as automatic as possible. But eventually after this stage, yes, it's ready for let's say engineering review and real by a human. We assume that there are humans in the loop, we want them still to look at it until we get the confidence. yeah, and then it can take it to the next kind of phases, which is in engineering, you know, doing the implementation plan. Technical design, implementation, testing, review, all of this, all skills that we have and supposed to be automated. And and they are automated. The Currently, we have this in most areas, most domains. We mapped all the domains. Each domain created their own context to explain and to make it a little bit more accurate and more also more efficient in how to do the review and the tech design and the implementation and the testing and everything. We are now in a stage that you know we start running PRDs through this process. And we have the people in the engineering. And also in products, they are not supposed to write anything on their own anymore. They just need to fix the skills or fix the context. So everything that is written will be written in a better quality. so we are doing this in maybe three squads right now. Yuval Yeret: Mm-hmm. Shay Mandel: This is the mode operations, but the others are also they adopt they are adopting it pretty fast. They understand the value, they understand the efficiency. So almost all engineers are contributing to this and also PMs are contributing. Yuval Yeret: How do the engineers feel about it? Do they Shay Mandel: Okay. Yuval Yeret: do they find enjoyment in this new role that they that they have to Shay Mandel: I think you know it's a it's an evolution and I think as as any any time that you adopt something new you see the same adoption curve, right? So there are the early adopters that jump on it and they really like it and they you know move forward. There are the late adopters and there are the luggage and and you know, it's the same in any population. I think most of them are now in in the stage that they say, okay, I understand that I need to do this and I also understand the benefit. And yeah, let's do it. So most of them are already in this state of mind, which is good. And so they they are pretty happy about this. You know, there is always some frustration where you write and you think everything is written well, and then eventually the result is the quality is not what you expect. No maybe sometimes not enough detail, sometimes too much detail, too long to read. Yuval Yeret: Mm-hmm. Shay Mandel: so we are, you know, tuning, I think, and it takes time to tune. But I think overall it's it's it's already saving us time. Yuval Yeret: But but but I think an important piece that I like in that I want to emphasize in what you said was that people's role right now is it's clear that the model is not gonna be perfect. But you're trying not just to fix the model's outputs or even to avoid fixing the model's outputs, but instead fixing the system. the inputs into the the model, whether that's the context or the skills, and then see that the model is able to create the right solution. So that's essentially the e everybody is developing this system that is the genetic development lifecycle, which is the right mindset. Shay Mandel: Yeah. And we definitely we talked with the old engineers and also with product and we explained everyone. You need to change the state of mind. Cl this is how we said it. We need a new state of mind. We are now developers, not of the code, we are developers of the system. And Yuval Yeret: Mm-hmm. Shay Mandel: yeah, and I I think they are adopting this. So they are working on it and Yuval Yeret: I love it. I love it. Okay. So so where were we in the life cycle? The there was coding work, right? Shay Mandel: Yeah, so so we defined all the skills. Yeah. We defined all the skills both on the product side and on the engineering side. we are tuning this. We are also having some teams that are starting what we call the loops, which is kind of the auto improvement loop. So we take a business problem and we say this is a problem that kind of repeats itself and we want it to be continuously improving. So we it should go from underst like monitoring what we have right now, deriving from that what's like the new feature, and then going through all the features. There are humans in the loop, but this is like a process that should be like humans should mostly review, maybe tune and push you to the next step. Yuval Yeret: So you're giving an agent essentially the agency to come up with features. With features Shay Mandel: Yes. Yuval Yeret: that will improve the the system's behavior, the product behavior for the users. Shay Mandel: Yeah. And today we look at it as feature suggestions and there is a human that decides which of these are we actually going to pursue immediately. And yeah, and and then we are moving forward. Yuval Yeret: And then the genetic lifecycle runs through it end to end and deploys and and measures in real world. Shay Mandel: Yeah. And then we are Yeah. And then creates a new list of suggested features based on the new Yuval Yeret: Cool. Alright. Shay Mandel: So that's the Yuval Yeret: So so where would you say is currently the bottleneck for this? Like what's what's limiting your ability to Create better products, more products, to penetrate new markets. Where is it currently? Shay Mandel: I I think that you know, it's First of all, there are a lot of details. Okay, it's it sounds very pretty and so on. You know, just getting one PRD to to go through all these steps. Each step needs its own attention and tuning and so on. We are now in in this process. We are looking ahead on doing this loop, so we are starting to do this. Again, a lot of details. How do you monitor? What do you monitor? When? Some things are, you know, changing fast, some ch things change To a long time, especially in insurance, for example, you know, you you introduce a new product, you start getting claims only after a year. You see if it's you know too many claims, too little claims, should we adjust something? so we are working on you know tuning this, understanding what exactly what we need. still wherever there is kind of we still have a lot of Like we we make sure that people are taking the decisions, not the agents. The agents are executing, they can suggest the next steps and so on, but humans are deciding on what is exactly the the right Yuval Yeret: So so that's an interesting dilemma because that could mean I mean at the extreme, I could say that how to code something is a decision. And if I want humans to make those decisions, then I'm back all the way to humans at least reviewing all the code. Okay. there could also be a decision of what's important, like just choosing what's the feature, what's the architecture, what's the outcome that we want to see, the the leading indicators, what's the system behavior that that we wanna see. So where where do you think is the right altitude for the decisions moving forward? Shay Mandel: Yeah. So I think when you execute there are lot of decisions to make, but we are trying to let this the the machine do this. But the there are important kind of decision junctions that you need to w we still need we still think there is a place for human judgment. So deciding on the priorities and like selecting the features is one. Then most of the work on creating the PRD might be automatic, but eventually like We expect the human to read this is what we want to do and approve this. That's on the on the product and the business side. Engineering, like plan creating the technical design and the plan. We let the s the agent do this, but we want a human review before we starts coding. So that's the inter in intervention points. And then event then they can go and do all the you know development and and improving the development, making sure all the tests, all of this is can happen automatically. there will be a PR, we run an automatic PR review by an agent. And once he says it's done, like the review is is okay after you know maybe multiple iterations or fixing stuff. automatically then we still want a human to look at it before it goes to production. Now I'm talking about kind of important changes. Maybe in the future we can see this for small features. Like the features I talked about are actually changes in our insurance product that are you know it's it's it's in it's kind of you know making a Yuval Yeret: Yeah. Shay Mandel: a mistake there can be very harmful and it might take a long time to see the the output. If we're talking about you know selecting a button colour between one option the on the other or the other. Yuval Yeret: Something in the user experience is less Shay Mandel: Yeah, but it's not major. It's not like changing the overall experience. It's about like deciding if the the button should be on the top or on the bottom or both sides and color this or that. And we can test it and we'll usually test it through A B test. Then it's okay for the agent to go all the way, create the two variants of the test, once it is in production, and eventually come back with the result of the the A B test and say, you know, this one is is the one that is winning. And let's switch to it. Today we still want to review it and say, okay, switch manually, but maybe in the future, you know, we we can push this automatically because if it's very clear that one is solution is is better than the other, then we let it go. we're not there yet today. We are still we need to build our own confidence in the system before we. Yuval Yeret: So at the moment you are still reviewing the PRs, the code changes. Shay Mandel: Yeah, we are not the initial reviewer. We are the last reviewer. So there are multiple iterations of automatic. Yuval Yeret: Yeah, the last one. So hopefully it's zero, you have zero zero feedback. That's the goal, right? Shay Mandel: Yes. Yuval Yeret: And if you have feedback, then going back to your previous comment, the engineer should go in and change the guidelines, the context, whatever, so Shay Mandel: Yes. Yuval Yeret: that you know, next time the reviewer or even the designer goes in and does the Shay Mandel: Yes. Yuval Yeret: does the work better the first time. Shay Mandel: Yes. Yuval Yeret: Okay, cool. So you actually w answered one of the the questions I had for you, which is what how much agency does the AI really have? And I think the example of you give it a KPI, essentially, you give it an area and you say, improve this area, improve our ability to I don't know, convert people that start to look for business insurance or something along those lines is an example of we don't even give it features. We tell it to focus on outcomes. We we really run a loop around a goal to to improve an outcome. Any any interesting insights that you've learned from trying to create these loops that you can share? Shay Mandel: Yes, I think you know we we definitely go this way. For example, yes, we have a funnel, improve the conversion in the funnel. this is the metric, you know, maybe you know, from start to co to quote or to purchase. and that's kind of common, you know, every f everyone has a funnel and so on. our funnel is unique in terms of like the questions we ask and so on, but he has ability to make these changes. I think this is based like you know, not everyone can do it, but yeah, it's based on the fact that we formalized everything there. So the questionnaire and all of that is very formal and it's very we moved it to be almost no code in most of the areas, so it's e it's fairly easy to change. and then when we run the loop, what we see is you know it has a lot of ideas. Some of them are unrelated or you know, i it doesn't have all the Of the experience that our subject matter experts have. And in some cases it's It's like different things that you know are hard to do. So some of the changes in insurance product you need to file, you need to get approval from the state, the department of insurance, and so on. So it's not easy to just make a change. In some cases it does. and in some cases it's it can tell like you know, all the competitors are doing X, you should do it the same. And we say, Okay, but we are digital and they are not, and we are we don't want to be the same. Okay, Yuval Yeret: Yeah. Shay Mandel: we still want to be differentiated in some. some sense. So these are all things that Yuval Yeret: I'll do it. Shay Mandel: you know we can get a lot of recommendations for features, but we definitely see that you know we still need the the human eye to take a look and to make the right decisions. Yuval Yeret: And there's still a lot of context to teach it, it seems. Like Shay Mandel: Yes, but I think when we try to teach like you can teach the context, you c we try to build the context of like this is our strategy, these are whatever marketing channels and kind of give him the I kind of try and to look at it as a junior let's say PM when it creates the PRD and whatever training I'm going, I'm taking the PMs so the the human PM so understanding what's the company, what's the goals, what's the To write this in some way in a in a kind of a context file or you know background and y we have multiple files of this nature. there is still stuff that you know we need to there is like some of it we when we understand okay, if we just knew this, then we add it to the context, right? And we say, okay, this is our strategy and it's going against our strategy. Okay, let's articulate this in the strategy MD file. Yuval Yeret: I mean w one thing that Shay Mandel: I I'm optimistic that Yuval Yeret: comes to mind and I don't know if it if it makes sense is Like a junior PM is probably not gonna be an insurance expert. But if you If you think about the most senior insurance expert that you have on staff, can you try have you tr have you tried systemizing their their knowledge? Making a digital twin for them that is available for for the for the agents. And and another similar idea is like let's take the ideas from the world's best growth hackers. Like the world best books and resources on growth hacking and bring them into what this growth hacker agent is a knows. Shay Mandel: Yes. So first of all, yes, w we are taking, you know, as I said, like the state of mind is that everyone should improve the the brain that we have in the company. So yes, the the best insurance experts, they are helping us write this this brain. and yeah, and we are also looking at, you know, asking Claude like how would you solve this and let's build this into the the the brain. And yes, we also what we did is we created these agents that enable you to ask different functions in the company. So you can have ask insurance expert and ask engineer. So if if I'm a product manager and I'm kind of I want to do a new feature, but I'm not sure what types of whatever different statuses we have in the system for this user behavior. I can just say I want to ask engineering. what are these statuses? And it will look in the code and everything and all the context that we have and will bring me back the the answer. So instead of me talking with an engineer face to face, which is a big problem in our company because we are in a very different time zones, we said we need to solve this somehow. So I can just ask a question. So maybe this is background for me to understand before even I start talking about like what's the problem, what's the solution I need. I just maybe exploration and so on. I can do this. And the same goes for engineer, for example, when he has a dilemma, like he sees the PRD, everything is done, it went through all the reviews, but still when he goes to implement, there is some question. And he says, What will the PM do? If the PM would sit the next table in the same time zone, that would be nice to just, you know, ask him. Yuval Yeret: Mm-hmm. Shay Mandel: But since this is not the case, he can ask the bot. And the bot will give him, you know, what the the PM said like this is how I make decisions usually in if I have like specific guidelines and things. So this is what we are we we are in the in the process of building this. And we obviously Yuval Yeret: And and I assume that the agents themselves will have access to this as well. So this growth Shay Mandel: Yes, probably. Yuval Yeret: hacker agent will have access to the insurance expert when they have an idea, then they assume, you know, can we even do this? They will go out and ask the insurance expert. The agent. Shay Mandel: Yes, of course. It all sits on the on top of the same brain. Yeah. Yuval Yeret: Cool. All right. So Maybe changing gears a bit, one last maybe last topic is how has all of this changed? How you manage the work? Like how you plan, how you execute, what what does the day to day work management look like? Where where is all of this managed? How is all of this managed? Shay Mandel: So I I think on purpose, we didn't say let's jump to the end and let's change everything, all the processes and everything. We are open-minded and saying, you know, if these things change, what can we improve, what can we solve, like and save like steps in the process and so on. But and and there was some thinking about like, okay, let's let's give the PM the ability to push features to production. And we said, Okay, we'll get there someday, but for now, because we want to tune each part of the process to make sure that we get confidence and it's tuned and it's doing well, we are not changing the kind of the roles and all the talk about like, you know, overlapping roles and so on, we said on purpose. Let's make Yuval Yeret: Okay, so there are still product roles, engineering roles, there is still the end to end life cycle of a feature, let's say, that we talked about. But Shay Mandel: We keep this the same. Yuval Yeret: In classic engineering product relationship there's releases, planning releases, roadmap Planning sprints, b those kind of things. Do you do any of that these days? Shay Mandel: So yes, so far maybe we're a little bit traditional. We know that you know we are we know that someday we may want to change this, but for now we continue with the same processes until we see that we can really either expedite them really a lot or you know completely remove them. But for now. We just take I think what we did is we didn't change the process. We still have a quarterly planning and we have sprint planning and we have roadmaps and a list of features that we want to deliver in a in a quarter. I think two things that we changed is one is we we have more we think we have more capacity. Yuval Yeret: Mm. Shay Mandel: We believe we have more capacity and we can do things let's say faster. And this can be in two kind of ways. One is do more. So if we want to tune, let's say whatever, X professions in a quarter, now we can say maybe we can do them expo the same X in a month or something like that. And the goal to is to get to to this to be in a loop and to do be auto completely automatic. But right now we see that you know, since we are also optimizing the system as we go and so on, it takes the time. It's Still, we we do see success already that we can do some of these very much faster in in a big way. so that's one way accelerating, like doing more of the same, and the second is doing higher quality in a sh in the same time. So instead of doing okay, that's the MVP, that's phase one, let's make it as simple as possible. We can, in some cases, say, okay, we can dream bigger, we can do maybe. changes that are little bit bigger. We still go with the phase one, phase two, MVP and so on, but maybe the phase one is is a bit it's a bit bigger or it's a bigger change. It's something that is more more bold that we can move forward. Yuval Yeret: Okay. Are you still like I'm guessing in days past you managed things like stories, whether it's in Jira, Azure DevOps, Linear or whatever, are you still managing these sort of things or is it mostly the feature at this point and the stories are just flowing summary? Shay Mandel: Yeah, so so with all these flows, there is a big question about like where do you store the kind of the the the data of the of the process and the current status and all of that. We do use Jira Yuval Yeret: Mm-hmm. Shay Mandel: and Confluence for that. and yes, Jira lists the task. So if I create a implementation plan, eventually it will create tickets. And the goal is that the the agent will take the tickets and will do them. Yuval Yeret: Mm-hmm. Shay Mandel: But if they have comments and so on, we found this as a at least for now as a good place to write whatever they do. So if they walk inter Yuval Yeret: Mm-hmm. Shay Mandel: asynchronously in the background, they can still write the output as comments on the Jira and say, okay, I did this. This is what I'm planning to do. This is what I've done. This is how I reviewed it, and so on, and then you can have you can track it and you can see what happened. And it also enables the like the agents, the different agents to take a look and to read the context and to understand where we are. So yeah, we still use Geo and Yuval Yeret: So so as a le as leaders of this process, what's the dashboard that you look at to see what features are flowing where in this life cycle? Shay Mandel: So we think we need to build this kind of dashboard. Right now we look at the epic levels and we see the completion of the epics. We we are starting to think about like okay, how do I see this? Everything that is going on and assuming that much more will go on, what's the right way to visualize this and to understand what projects we're working on, in which stage they are and so on. Yuval Yeret: One of the things that I've seen for many years, and I think it applies applies well here as well, is an epic level Kanman board. So in first of all, an epic level statuses that reflect well all of the touch points between agents and humans. So all of the If you think about the PRD, the agent finished writing a PRD, it's now waiting for human review. That's a distinct state change, right? So all of these state changes, especially when it's a handoff between humans and agents, show them separately and then use a Kanman board to reflect where where things are, where things are waiting for the humans. I I think that might be useful out of the box. Like it's so easy to create. dashboards these days, but since you're already sitting on top of Jira, which I think makes sense, I think it's something worthwhile doing. And then you also get the benefit of flow metrics that show you telemetry of how long it takes you in each one of these steps and you can start to calculate flow efficiency and how much time things are are really waiting. Shay Mandel: Yeah, yeah. So we started this, we have like initial dashboard for understanding the status of each epic, but yeah, everything you said it's good ideas. Yeah. Yuval Yeret: Cool. Awesome, so what What maybe are one or two things to avoid for leaders that are starting this journey and one or two things to make sure that they that they do Shay Mandel: Yeah. So I I think It's it's been a journey we're in this for maybe three or four months. It's moving very quickly. People expect it to move very quickly. I think the management especially, you know, with all the hype and everything, they they are expecting it to v move very, very quick. and I think in the reality, you know, from ID to POC, like always, it's very quick. From ID to production level, you know, write this in a way that will pass all the reviews and so on, takes time. I think it the first thing to do is to set expectations and to like to give the demo but also say, you know, it will take time until we get to this level and to be like it will be whatever. We we talk about it kind of as grades, so we say, you know, the initial version will probably be sixty percent accurate or seventy. And we'll probably get to ninety percent or ninety five percent. and the extra five percent is why we need a human in the loop, because we think, you know, and and it As we know, you know, the last percentages are always the hardest to to fine-tune. So we set expectations this way. We did a lot of training in terms of like the mindset we talked about, explaining. First, before even we talked with the engineers about this, we we defined a little bit the process, like how do we want to do this? where is all this information stored? Is it a repository? Is it wiki and so on? So we there were quite a few decisions there. We decided to go with a repo. So we'll have the version control and we have everything in one place and people can contribute but people can also review and so on. So GitHub provides us a lot of mechanism to do that. So I think you should first of all set the expectations, then define the processes and then train. And I think people understand that they want to jump on it, so let them do it. And we said kind of consciously that okay let people run, let them understand also the you know the problems, the the challenges and so on, but they will be part of the process and they can also contribute back. So w we said, you know, we have a weekly meeting where we collect the learnings and we understand what people are doing in different places, and and we are trying to consolidate this and to cross pollinate across the organization. So I think this is a kind of a good practice. And I think the the some kind of learning, I don't know if it's like you know, stop doing, start doing, whatever, but it's be kind of conscious to the fact that you have different users with different different levels of technical ability, let's call it. So for the engineers, there is no issue to install many different things and play with them and try and so on. For product managers And insurance experts, they live in a different place. When you tell them open a terminal and just write these commands, you lost them. Okay? Especially if Yuval Yeret: Mm-hmm. Shay Mandel: you are trying to let them do it on their own and so on. So we really wanted to make this and part of it cloud can do for you. In some cases, at least with us, there are different set of permissions for each user. So engineers usually can do some admin or root access on Yuval Yeret: Okay, admin, yeah. Shay Mandel: their machines and the others don't and it makes sense. We just need to make sure that you know the the skills that we write Yuval Yeret: Mm. Shay Mandel: work for everyone and the expectation is set in the right way and create very clear kind of getting started guides that are maybe in some cases different for the different audience or with some different kind of levels of the the guide. So yeah, and this is Yuval Yeret: One of the things I've seen in other organizations that I that I work with and talk to is there's this pattern that I see that the harness is created by engineering. spec driven harness that is very engineering oriented, you might say, both in what it focuses on and what it doesn't focus on, but also how it's structured. And y you see when you give it When you give it to anybody that's not an engineer, or even not everybody that's not the engineer that wrote it, and you know, people don't really know what to do with it. They don't even understand what's the the benefit of it. So it's you need to think from a product perspective, even about your AI harness and your AI capabilities. You need to make sure that it's something that people are easy to activate and easy to retain and and on in all these aspects, especially when we start to move beyond engineering. Shay Mandel: Yes. And what I kind of I you know talked with other companies and so on, what I saw and what I was kind of worried about is some companies they have a lot of skills because everyone builds its own skill. They don't use the generic skill and then you when you look at the statistics, you know, every skill is used by maybe one or two users. And then, you know, a lot of what you build and all these brains they are not really utilized. So I'm I'm very focused on not getting them and trying to let everyone use the same thing. The way we we are doing this is kind of you can call it like you know eating our own dog food. We are we are working with the so we have a squad that I lead for the AI what we call AI enablement. So we build kind of the you know the infrastructure and the the guidelines and the methodologies and so on. And the business squads that are using this and we have our own we have one of our people over there to make sure that you know we build the right infrastructure for them and also that they utilize it and as well that they can contribute. Yuval Yeret: So you have a forward deployed engineer. Shay Mandel: Yes. And also part of the as I said before, part of their definition of work for this quad is not to deliver the feature, it's to build a system that can deliver more features in the future. So and they shifted, Yuval Yeret: Mm-hmm. Shay Mandel: they also shifted some of their OKRs with approval of management and so on. That you know, okay, it's okay that the first quarter will be slower because we understand it will accelerate the next quarters. And this quarter is the focus on building the machine. And I think this is very important and and and we are also making sure that they don't build a machine, like you said, just for their squad. No, it should be a machine that is built in a way that can be extendable by other teams and also be used by other squads. So yeah, there is a lot of investment in that, but I think it it's it's important because it's I don't want it to be that you know every squad, every engineer has their own way of work. I don't th I think it's won't be productive and I think it will be very hard to maintain. Yuval Yeret: So let's assume product, insurance products, engineering, operations is all agentic with the humans spending most of their time applying taste and judgment and key decision junctions and optimizing this system. Where i if you look at the rest of the organization, what's your vision? What's the organization's vision for what being AI first, AI native means beyond this building and improving the products, the insurance products and Shay Mandel: Yeah, so I think eventually, you know, it should be in all areas and it can contribute in every place. and I think there is like, you know, basically they can just use Claude or ChGPT or whatever, the vanilla version. What's what's the added value that we have as a company to provide, let's say, to the finance team and I think we do have it because the like once we build this then the connectivity between the pieces and the ability to keep the organizational context can contribute a lot. So for example we didn't talk about the actuaries but actuaries also you know the they are part of the companies they are more sophisticated business people but still we we can build them tools and we can help them do this in a more efficient way. So the one the the connectors that we have to the data that we use now to let's say define the hypothesis and how much we will improve in the funnel, they can use the same thing in order to look at the claims and con and connect it to the funnel and see like, you know, how much of this these questions that they want to ask the customer how much they influence the the the overall Yuval Yeret: That's all. Shay Mandel: conversion and will it be beneficial or not. So everything this is Definitely connected. We also like provide them more tools to look at like you know what competitors are doing because the product needs to see it and the insurance product needs to see it, and also the actuary probably needs to borrow ideas from you know from what others in the in industry are doing. They are doing it today, they have ways to do it, you know, by looking at other websites and different places and things. We can automate this, and this can be beneficial for everyone. And all the way to let's say we talked about finance. Finance can collect these you know, funnel metrics and say, okay, based on that, I can say, you know, what's the my prediction for the budget for next year, for example. How much can we improve the the funnel based on this? How much can we improve the top line and the bottom line and so on? So it should go almost every place, I hope every place, even the people team can probably, you know, learn from it like HR and and you know, use some tools that we can connect and and automate some of their work. Yuval Yeret: Yeah. Shay Mandel: They also have a lot of operation to do. Yuval Yeret: Mien one of the interesting opportunities I think is We look at the product and engineering like a factory, right? Like a pipeline. And we understand that there's a value stream, there is a flow, there's a set of stages, we create skills for each one of the stages, and we work on optimizing this system, you know, with the intent that it becomes more and more agentic. I guess the the the question is what other factory lines. are material to next insurance business. Like I don't know, claims for sessing. Is that a major one that you know takes a lot of people, a lot of energy, hiring sometimes, I I I don't know, closing the the books each month, whatever it is, and applying the same concept that you talked about For the product and engineering applying in that space is an interesting opportunity. I guess maybe it's a topic for another time, but the mindset that you need, the that engineering mindset of we're constantly tuning the process is a mindset that's somewhat more difficult to find in other business processes, in my experience. So a applying AI outside of engineering requires bringing an engineering mindset. to the rest of the the organization, whether it's by applying engineering capacity to it or by, you know, building engineering skills or mindset elsewhere. Shay Mandel: Yeah. And and we're doing both. I we we talked mostly about like, you know, the agentic the development lifecycle. We already implemented this like in in the support and the claims and so on and like the customer facing, we did this already, you know, a year ago that we implemented AI over there and we continue to improve and yes, we we have product managers that we deploy there and a team that supports these different teams and they are applying the engineering mindset and they are always thinking about how to improve that. Yuval Yeret: So you're saying in a sense the product and engineering is actually behind some of the rest of the organization when it comes to leveraging Shay Mandel: Yes. Yuval Yeret: agentic. Makes sense in in this case. Cool. Awesome. This is all fascinating, Shy. I think we've achieved our intended impact or outcome for for this episode. I think our listeners have a lot of interesting ideas to reflect on on how does an agentic life cycle look like in practice? What are some things to pay attention to, what are some things to avoid. if anybody wants to Connect, see what you're doing, over at the next insurance. Is there a place you would like to invite people to take a look at? Shay Mandel: Yeah, so first of all we have a meetup that we are kind of opening to the to the to the industry. It's local in in the Bay Area. but it's also on Zoom, so everyone in the relevant time zone can join us. It's on Luma, so luma.com slash product leaders dot ai. And everybody can use join this. Yuval Yeret: Okay. Shay Mandel: And yeah, and you can follow me on LinkedIn and trying to post from time to time. Also feel free to, you know, to just connect with me on LinkedIn and ask questions and I'm always happy to to chat and to learn and you know, every this I think this is all new for everyone in the world and we are all learning from each other, so it's great to chat and to understand and to to learn from each other. Yuval Yeret: Awesome. So thank you for being here and sharing, Shy. It's been fascinating for for me at least. see you on the next episode of Scaling AI from activity to impact. Shay Mandel: Thank you. Thank you for having me. Bye bye. Yuval Yeret: Mm-hmm. ## Source boundary These are the published show notes from the podcast feed. They are a starting point for discussion, not a verbatim record of the conversation. The transcript is machine-generated and may contain errors or unlabeled speakers. Check the audio before quoting anyone.