In the third week of the 12W program, we made an unusual decision: to stop tracking a KPI that had been considered core. This decision wasn't born out of laziness—after two weeks of observation and reflection, we realized that metric was misleading the team's direction. The experience forced me to reconsider a fundamental question: when we set KPIs, are we actually measuring meaningful progress, or are we simply feeding the psychological need to look like we're doing something?
According to Harvard Business Review research, roughly 67% of organizations use KPIs to measure performance, but a significant share of those metrics eventually devolve into window dressing—failing to actually drive better decisions. That figure mirrored the dilemma we hit in week three. At the time, we had set a "daily new user registrations" KPI, targeting 50 new signups per day. The numbers looked solid in the first two weeks, and team morale was high—but we gradually noticed a problem: those new users barely engaged with the product afterward. They behaved like one-off traffic, vanishing the moment they signed up. The number on the metric kept climbing, but the product's core value delivery was standing still.
That realization pushed us back to square one to re-examine the logic behind our KPI design. The root issue was that we'd confused measurable behavior with meaningful behavior. Daily new user count is an easy number to track—it delivers instant positive feedback and makes the team feel good. But this metric overlooks the far more critical variables: user quality and retention. When a KPI can't distinguish between 50 good users and 50 bad users, it loses its value as a basis for decision-making. In the early stages of a startup, with limited resources, scattering attention across multiple surface-level metrics is often the main reason teams run themselves ragged without seeing real results.
From this experience, we drew three important lessons. First, a KPI should answer a business hypothesis—not satisfy a psychological need. Part of why we originally set that metric was to quickly see proof of "growth" and reinforce team confidence. But a healthy metric should ask: "If this number improves, is the product actually getting better?"—not "Does this number look good?" Second, tracking frequency and decision frequency need to match. We were tracking new users daily, but the team only made product adjustments weekly. That frequency mismatch caused the metric to lag behind what was actually happening. Finally, every KPI should come with a "kill clause" that clearly defines the conditions under which the metric is no longer worth tracking.
Based on these reflections, we made a concrete adjustment right away: we replaced "daily new user registrations" with a compound metric called "share of new users among weekly active users." This new metric accounts for both user growth and user quality, and we shifted the tracking cadence from daily to weekly, syncing the data with our decision-making rhythm. The adjustment wasn't complicated—any team could complete the relevant event tracking and dashboard updates within a day. What mattered was that this change pulled the team's attention back to delivering product value, rather than getting lost in surface-level number games.
Conclusion: KPIs Are Tools, Not Goals
The most dangerous thing in a startup isn't making mistakes—it's running efficiently in the wrong direction. A dropped KPI doesn't represent failure; it represents the team's courage to acknowledge that existing assumptions need to be revised. Real progress isn't about how many metrics you track; it's about your willingness to regularly review whether those metrics are still serving your business goals. For the next KPI you're about to set, ask yourself one question: if this number improved dramatically, but the product itself didn't get better, would I still track it? If the answer is "no," then it never should have made it onto your metrics list in the first place.
"What gets measured gets improved—but only measuring the right things leads to real progress." — Peter Drucker