A pilot can look healthier every month and still leave the room with nothing it can act on. The number on the slide went up. Each function can explain why that rise belongs to them. Ask what would have made the number go down, and the explanation changes owner.
Eric Ries, author of The Lean Startup, set the trap out on 23 December 2009 in Why vanity metrics are dangerous. A vanity metric is the number that looks like progress and cannot tell you what to do next. His test for a team is not the meeting with a formal agenda. It is the last few decisions that were actually hard: a crisis, a fight over what to build next, a review of something that failed. How much data that could change a mind was in the room. If the answer is a gross total, you are about to pay for a chart.
Why a hit is a poor receipt
His example of the worst kind is hits. It counts a technical process, not a person. It is a gross number, so one hit each from a million people is treated as the same event as a million hits from one person. Most people in the room cannot say what a hit even is. An image, a page and a script file can all qualify. And the number has no cause inside it. A million hits this month does not say what produced them, whether they were worth the same or how you would get more of the ones that matter. Each of those questions needs a smaller number. He says that smaller number is the point, and that it is more work, which is why the gross one survives.
The excuse for keeping the gross number is that the direction is enough. If hits are higher than last month, that is progress, and the hard questions can wait. He says the direction is not a safe proxy. The moment you steer by it, a pattern sets in. When the number rises, people attribute the rise to whatever they were working on. When it falls, they blame someone else. He says he has watched this. Over time each person lives in a private reality. The realities drift, and the team cannot agree on the next move.
How a department learns the wrong lesson
A functional organisation makes the drift worse. If the engineers are all on one effort, marketing on another and the same is true of testing and operations, each department grows its own account of what the number means. The prioritisation meeting then has a familiar shape. Each group is sure the people down the hall want to repeat a kind of project that has already failed. The chart will not settle it. A line over time, with arrows for the launches you remember, shows that two things happened near each other. It does not show that one caused the other.
A fall that was the plan
He describes a company that had grown for many quarters and then had two down quarters, just after raising money. They judged themselves on gross revenue against target. When the revenue fell they looked harder, and found that one group of customers was leaving while another was arriving. The customers on the way out were worth more in the short term. The new ones were where the later growth sat. The decline was the strategy working, and the size of the swap had surprised them.
The board meetings went badly. New charts appeared, with the strategy changes and the customer behaviour split by segment. The argument was reasonable. The board still could not know whether it was hearing insight or a polished excuse. The insight was right, which is easy to say afterwards. Several of the executives who presented it were no longer there to be proved right. His point is that the mess was avoidable. If the goals had been set on numbers that could be acted on, the change of strategy could have been rolled out segment by segment, in controlled trials, and the trials could have been used to predict the effect before the whole book of revenue moved.
He is plain about the limit. Numbers you can act on do not guarantee a good decision. They put facts in the room at the time of the decision. Without them, he says, the team is rolling dice. That is the standard I would take into a proposal. Not whether the room will leave with a dashboard. Whether the number on that dashboard could have changed one of last quarter's hard calls, and whether a trial could have been run on part of the customer base before the whole offer moved.
What the agenda gives away
Read the proposed measure before you book. Sign-ups, page views, a count of pilots, a total of messages sent and a slide called engagement are the gross number in office clothes. They can rise because one enthusiastic account did the same thing many times, or because a technical event was counted as a person. They do not say which customer, which behaviour or which change in the offer produced them. If the provider cannot rewrite the measure as a rate for a defined customer, with a cause the team can turn on and off, the day will rehearse the private realities Ries describes.
Ask whose work the rise will be credited to, and whose work a fall will be blamed on. If both answers are whoever is presenting, you have the attribution habit, and a workshop will not cure it by drawing a better chart. The cure he describes is a comparison that holds the rest still. One segment, or one cohort, sees the change. Another does not. The difference is the evidence. A before-and-after line for the whole business, with arrows for the things the team is proud of, is the chart he says everyone already knows is unpersuasive.
The decision the number has to face
Bring a decision the team is about to make, not a case the provider likes. Name the number that will be in the room when that decision is made, and write down what movement would stop the idea, what movement would earn another test and what movement would mean you still do not know. If every movement can be explained as someone else's doing, the number cannot do the job. A metric that only ever confirms the plan is a gross number pointed inward.
The people who can fund or stop the idea have to agree that line before the result exists. Afterwards, a disappointing cut will be redescribed as a different segment, a different week or a different definition of the customer. That is the board meeting he describes, where the finer chart arrives only once the gross number has already failed. Agree the cut while nobody needs it as a defence. If the sponsor will not name a result that ends the work, you do not have a metric. You have a story that will be updated.
Where a total still fools a careful team
A regulated team is not exempt from the gross number. A count of pilots completed, a count of staff who logged in and a count of risks reviewed are all hits by another name. They can rise while the customer behaviour you meant to change stays put. The trial he wants is still available. You do not have to move the whole book. One cohort, one office, one product line, held against another that did not get the change. The learning is the difference. A programme that reports only the total has chosen the number that lets every department keep its own account.
Watch the sentence the room uses when the number falls. If the sentence names a cause you could run again, you are close to what he calls a metric you can act on. If the sentence names a market, a season or another department, you are back in the private reality. Write that sentence in the session, against last quarter's actual fall or last quarter's actual rise, before anyone designs a new slide.
What a later call should show
Do not score the day by whether people can define a vanity metric. That is a quiz on his vocabulary. Score it on the next hard decision. Was the number per customer, or a total. Did it name a cause. Was there a comparison that held something still. Could the result have stopped the work, and did anyone with the authority to stop it accept that before the result arrived. If the total went up and every function claimed it, the old habit travelled and the metric did not.
A good decision is not guaranteed, which is his line and a useful one to keep. You are not buying foresight. You are buying a room in which a disappointing comparison can be read as a fact, rather than as an attack on the people who ran the trial. If the culture still treats a down segment as a failure of the presenter, no dashboard will be allowed to say the awkward thing. The workshop has to practise that reading on a number the sponsor already cares about, with the sponsor present.
What you are willing to stop
Name the offer, the customer and the movement that ends it. Name the comparison and who is in the group that does not get the change. If the proposal has no such group, the day will produce a chart with arrows, and the arrows will be whatever the team was working on. That is the pattern he watched. It feels like learning. It is each function editing the same total.
Collective Campus is building high-performance culture and teams for the AI world. The lean startup and product sprint is for a team that can bring a live bet and a number that is allowed to end it. What the bet itself should put under test is in what a lean startup workshop should test. A pilot with no end is the subject of pilot purgatory. Funding that arrives before anyone has shown they want the offer is in the budget that cleared too early.
What to bring
Bring the number you already steer by, the decision it is supposed to inform and the name of the person who can stop the work. If that number is a total, and that person will not agree a result that ends the idea, you do not need a facilitator. You need a different number.
