“We trained 1359 people!”
“we transformed 127 teams”
Enough with the vanity metrics.
Ever wonder how your change initiative is REALLY doing?
Think of your organization as an internal market for the change.
Consider Pirate Metrics (AARRR) to see how your change “Product” is doing in this market.
Acquisition (or awareness) – are people discovering our change? How effective are we at acquiring new interested internal customers for the change?
Activation – Are these people taking the actions we want them to? Are they starting the change process?
Retention – Are our activated changers continuing to engage with the change?
Revenue – Are we making money from user activity?
What does this mean in the world of change? Revenue stands for “Value Extraction” which should be measured in terms of change outcomes (not just activity).
Referral – Do changers like the change enough to tell others about it? How often do we get inbound requests to change from people that weren’t on our radar yet?
If you want to go the extra mile you could ask changers the Sean Ellis PMF survey – “how would you feel if you could no longer use this new changed way of working?” and measure the percent who answer “very disappointed.” (an interesting sidebar discussion is whether the PMF threshold of 40% applies for internal change as well. Or should we aim higher? lower? WDYT?)
How about an Example
Let’s say we are trying to get people to change the names of their work artifacts to reflect outcomes. We are focused on a change audience of everyone in a Product discipline in the organization.
Acquisition (or awareness) – How many Product people are aware of this concept? we could use internal analytics to see how many of these people opened this new guidance, showed up for an info session, etc.
Activation – How many of the Product people started naming items using an outcome orientation? This is a bit harder to measure at scale. One technique we could use is to include a gauge for each work artifact where the product person indicates whether the name is outcome oriented in their view.
Retention – Do these Product people continue to rename items? to name new items using an outcome oriented name?
“Revenue” – Are we seeing positive outcomes from these new names? Here is where we would have to be clear with our outcome hypothesis from the get go. As part of the change plan we will lay that out and consider it before actually going through the change.
Referral – Do we start to see more and more Product people showing awareness, getting interested, even without us reaching out? Can we measure somehow when people forward the “playbook” to others?
As you can see, the concepts apply nicely, even if the telemetry to measure change might require some creativity.
One other thing you can try is to consider AARRR metrics as leading indicators when you’re designing the change / intervention. While “Revenue”/Outcome is what you’re really aiming for, the AARRR funnel can help lead to outcome while avoiding vanity metrics.
So …
Think about a change/intervention you’re currently advocating for in your organization
What would the Pirate Metrics tell you?
Would you dare asking the PMF survey question?