TPS Reports to KPI Reports: Unlocking Metrics that Matter
#managing_others I hate being presented to
Like many of you, I've experienced the frustration of writing status reports. I’m not referring to the targeted memos we craft to provide context to our colleagues, but rather the regular reports we're expected to write to 'management,' whether that's our direct manager or the ambiguous managers in a matrixed organization.
Reflecting on the movie Office Space in 1999, with its comical take on status reports called TPS reports, I found solace in the fact that I wasn’t alone in questioning the value of reporting for reporting's sake. Over the years and decades, my steadfast stance on reports and later presentations has remained unaltered, even as I’ve held managerial roles for nearly 25 years. Most status reports and presentations I’ve encountered, except those specific to GAAP accounting, were authored by teams measuring too many things and reporting on only the metrics that fit their narrative that everything is okay. We're all too familiar with these metrics, where everything is green despite the project's true nature being in the red. Kate Vitasek even coined a term for this phenomenon, which she aptly named the “Watermelon Metric!”
An alternative version of status reports, which often appear as large showcase meetings, celebrate people’s work - but rarely discuss the business impact of the work. How do we as business leaders show the work in a meaningful way without resorting to showing off what just doesn’t matter?
Moving Beyond TPS Reports
Luckily, as I’ve advanced in my career, I find myself in a privileged position where I can dictate the content my teams generate that helps me understand how we’re doing as an organization so that my teams do not feel like they are generating TPS reports. In this article, I discuss how teams I work with have graduated from TPS reports with watermelon metrics to reporting on KPIs using metrics that matter.
The Importance of Key Performance Indicators (KPIs)
Over the years, I’ve learned that the key to meaningful reporting lies in focusing on Key Performance Indicators (KPIs) that truly reflect the health and progress of our projects. Unlike TPS reports, which often drown in superfluous details, KPI reports hone in on one or two critical metrics that the organization decides up front which provide a clear and honest picture of where the project stands. For instance, instead of boasting about ‘website traffic’ that can be easily manipulated and doesn't directly correlate with success, an e-commerce business might prioritize metrics like ‘conversion rates’ and ‘customer lifetime value,’ which offer a more comprehensive and actionable view of performance. By concentrating on KPIs that matter, we can make informed decisions, identify areas for improvement, and ultimately drive better outcomes for the organization.
Standardized Dashboards: The Future of Reporting
When organizations regularly report on the same metrics, they don’t need to create presentations. Instead, they can develop a standardized dashboard that provides real-time insights into key performance indicators, eliminating the need for redundant status reports. This approach saves time and ensures that everyone has access to the same data, fostering a more collaborative and transparent environment. Teams can focus on analyzing the data and making strategic decisions rather than compiling reports that often serve more as bureaucratic exercises than tools for real progress. By embracing KPIs aligned with the organization’s goals and immune to manipulation, we can create a culture where metrics truly matter and drive the success of our projects and the organization as a whole.
Identifying Solid Metrics
If it were only that easy! As it turns out, figuring out which KPIs to measure is the real art, and it’s a subject that I’m still evolving my thinking on despite my decades of experience. When I evaluate a metric we’re considering to help us run the business, I try to understand the mathematical properties of the metric, the subjective qualities of the metric, and if the metric is situationally relevant.
Mathematical properties of metrics
To start, it’s important to understand that metrics are measures of behaviors within a system, whether that’s a business, an industry, or a project. They are the culmination of numerous interactions and activities happening within that system. Because these interactions are complex, a metric’s value can vary even if the underlying system remains unchanged.
For instance, consider a business that experiences seasonal fluctuations. Just because more money is made week over week doesn’t necessarily indicate improvement. Take the travel industry: bookings naturally increase as summer approaches. An uptick in revenue leading up to summer doesn’t mean you’ve fundamentally improved your business—it’s just the result of seasonal demand.
Given these nuances, metrics can exhibit a range of values under very similar circumstances, a concept known as variance. To explain variance in simple terms: imagine you’re throwing darts at a target. Even if you aim at the exact same spot every time, the darts will land in slightly different places due to minor differences in each throw. Similarly, in business, your metrics can fluctuate due to small, everyday variations in operations.
Understanding seasonality and variance is therefore critical. Seasonality refers to predictable fluctuations that occur at specific times of the year, while variance represents the natural spread of data points around a central value. By recognizing these patterns, you can differentiate between normal fluctuations and genuine changes in performance. This awareness allows you to interpret your metrics accurately and make informed decisions that truly reflect your business’s health and progress. Generally speaking you want to actively correct your metrics for seasonality while choosing metrics that have low variance.
Subjective Qualities of Metrics
Beyond the mathematical properties, I’ve noticed over the years that some metrics are inherently better than others. Since a metric often represents the score in the game we’re playing, it’s demoralizing for the team to operate with a broken scoreboard. Through experience, I’ve found that good metrics tend to have these four subjective qualities: they are Sensitive, Actionable, Guardrailed, and Aligned (SAGA).
Sensitive
A sensitive metric is one that the team can actually influence through their activities. For example, measuring the time to recover from a site outage is sensitive because the team’s actions directly impact it. In contrast, goaling individuals on improving the stock price isn’t sensitive, as many factors influencing the stock price are beyond their control.
In an e-commerce setting, a sensitive metric could be the ‘average response time of customer support tickets.’
For a B2B SaaS platform, measuring the ‘time to resolve customer-reported issues’ can be sensitive.
A media streaming company might track the ‘buffering time per stream session’ as a sensitive metric.
For a hotel booking site, ‘time to confirm a booking’ is a metric that teams can directly influence.
Actionable
The metric needs to be actionable, meaning the team can take specific actions to improve it. Goaling teams on revenue isn’t directly actionable since a company can be making more or less money for various reasons, many of which could be independent of the actual team or colleague.
In an e-commerce context, ‘average number of clicks before a customer adds a product to their shopping cart’ is actionable.
For a B2B SaaS platform, an actionable metric could be ‘average response time to customer queries.’
A media streaming company might focus on the ‘number of recommendations viewed before selecting a show’ as an actionable metric.
For a hotel booking site, ‘conversion rate from search to booking’ is directly actionable.
Guardrailed
A good metric needs to have defined guardrails to ensure it measures meaningful success and not just superficial improvements. For example, a bug that tricks users to a product listing page may generate more product listing views, but it will not benefit the business in a meaningful way.
In e-commerce, ‘cart abandonment rate’ needs to have guardrails ensuring the metric reflects genuine customer interest and not just browsing activity.
For a B2B SaaS platform, ‘user engagement’ should be measured in a way that reflects actual usage and not just logins.
A media streaming company might use ‘average viewing duration,’ ensuring it measures engagement and not just users leaving the stream running.
For a hotel booking site, ‘customer satisfaction score’ should be guarded against inflated feedback from limited-time promotions.
Aligned
The metric needs to be aligned with the organization’s goals, such as saving money, making money, or saving time. This alignment is crucial because it ties directly to the long-term success of the company, which is ultimately reflected in revenue.
In e-commerce, ‘repeat purchase rate’ aligns with generating more revenue.
For a B2B SaaS platform, ‘customer renewal rate’ aligns with sustained revenue growth.
A media streaming company might focus on ‘subscription renewal rate,’ which directly ties to recurring revenue.
For a hotel booking site, ‘booking completion rate’ aligns with both saving time and generating revenue.
The SAGA Framework
By combining these qualities, we can propose a group of metrics that meet all four criteria of the SAGA framework (Sensitive, Actionable, Guardrailed, Aligned). Here are some examples of metrics from various industries that satisfy these conditions:
E-commerce Company
Metric: Conversion Rate from Product View to Purchase
Sensitive: The team can directly influence this metric through activities like improving product descriptions, optimizing website design, and enhancing user experience.
Actionable: Specific actions such as A/B testing of product pages and refining the checkout process can be taken to improve this metric.
Guardrailed: This metric needs to have guardrails to ensure it reflects genuine interest and not just temporary boosts from marketing campaigns or discounts.
Aligned: Higher conversion rates directly contribute to increased revenue, aligning with the goal of making money.
B2B SaaS Platform
Metric: Customer Renewal Rate
Sensitive: The team can influence this metric by improving product features, providing excellent customer support, and ensuring the software meets clients’ needs.
Actionable: Teams can take actions such as conducting customer feedback sessions, implementing requested features, and providing training to improve renewal rates.
Guardrailed: Guardrails should be in place to ensure renewals are genuine and not influenced by short-term incentives or discounts.
Aligned: A higher renewal rate ensures sustained revenue and long-term customer relationships, aligning with the goal of making money.
Media Streaming Company
Metric: Subscription Renewal Rate
Sensitive: The team can influence this metric by curating content, improving the user interface, and enhancing streaming quality.
Actionable: Actions like introducing new features, running personalized marketing campaigns, and offering exclusive content can improve this metric.
Guardrailed: Guardrails ensure that renewals are based on content quality and user satisfaction rather than temporary promotions.
Aligned: A higher subscription renewal rate leads to consistent revenue, supporting the goal of making money.
Hotel Booking Site
Metric: Booking Completion Rate
Sensitive: The team can impact this metric by optimizing the booking process, improving site speed, and providing clear pricing and availability information.
Actionable: Specific actions include streamlining the booking process, reducing steps, and offering customer support during booking.
Guardrailed: Guardrails are necessary to ensure the completion rate reflects actual bookings and not just attempts or abandoned bookings due to technical issues.
Aligned: A higher booking completion rate directly results in increased revenue, aligning with the goal of making money.
By adopting the SAGA framework, organizations can ensure their metrics not only reflect true performance but also drive meaningful improvements and strategic decisions. This leads to the success of their projects and the organization as a whole, creating a culture where metrics truly matter and foster long-term growth and sustainability.
Situational Relevance of Metrics
Regarding understanding customer behavior and driving business growth, Dave McClure’s pirate metrics framework, known as AARRR, is particularly insightful. AARRR stands for Acquisition, Activation, Retention, Referral, and Revenue. These metrics help track the customer lifecycle from the initial point of contact through to becoming loyal advocates. Acquisition measures how many customers are introduced to your product, while Activation gauges how many of those customers have a first successful experience. Retention looks at how many customers continue to use your product over time, and Referral tracks how many customers are recommending your product to others. Revenue, however, is often a grab bag of metrics related to revenue generation that may not be directly product-related, such as price changes or one-time promotions. Despite this, understanding Revenue within the AARRR framework remains crucial, as it ultimately measures the financial return of your customer lifecycle strategies.
For most businesses, situational relevance is grounded in identifying and addressing key constraints. This is where the Theory of Constraints (TOC) comes into play. TOC posits that every system, including businesses, has at least one constraint that limits its performance toward achieving its goal. In the context of AARRR, a business is typically constrained at some point within the customer lifecycle—whether it’s Acquisition, Activation, Retention, Referral, or Revenue. Identifying this constraint is vital because it reveals where efforts should be focused for maximum impact. For example, if a company struggles with customer retention, improving the activation experience won’t significantly boost overall performance. Instead, targeted strategies to enhance retention will yield better results. If this sounds familiar, it’s probably because you read this article I wrote last August.
Deciding on a metric focusing on the key constraint is the most efficient way to drive the business forward. By applying the Theory of Constraints to the AARRR framework, businesses can prioritize the most critical areas for improvement. This targeted approach ensures that resources are directed where they will have the most substantial effect, allowing for quicker and more impactful changes. For instance, if retention is identified as the primary constraint, focusing on metrics such as ‘churn rate’ or ‘repeat usage’ can provide actionable insights. By improving these metrics, the business can enhance its overall performance with minimal effort, as resolving the constraint often leads to a cascade of positive effects throughout the lifecycle. This strategy aligns with the SAGA framework, ensuring that metrics are relevant and drive meaningful progress towards the company’s goals.
Transforming Business with Metrics That Matter
The journey from TPS reports to KPIs is not just about changing what we measure; it's about transforming how we understand and drive our business.
During the first two years of my tenure at Upwork, our business results were anemic. We had a number of reasons for this, but one of the reasons was that we had a tendency at the time to reward product launches over business results. In an internal memo, I wrote to the staff at the time, I made a simple observation that nobody remembers rocket launches - they only remember that one time when someone landed on the moon. OKRs, god bless them, tend to reward activity over business results the way most individuals and teams define them. I remember feeling frustrated because we all wanted to win and we were working hard, but somehow it felt like the game was slipping through our fingers.
The reason is because terminal goals like OKRs require leadership or in some cases yourself to define what those goals are once they are done. Terminal goals aren’t focused on the work, but instead are focused on outcomes that may or may not be realistic. KPIs on the other hand, create focus around how the work is done so that you can grow your outcomes over time. It is in this way, we move away from finite games to a game we can keep playing indefinitely.
It was around this time that I decided to impose the 4 Disciplines of Execution framework on the organization and, perhaps critically, implement monthly operations reviews against business metrics that imperative teams were to report against monthly. By shifting our focus from what we were launching to what business metrics we were trying to influence, we changed the games for our teams and they were motivated to play!
Unfortunately, the transition wasn’t easy. In fact, it took more than six months to determine whether we were even measuring the right things. But when it finally got dialed in, the business started to improve and the rest as they say was history.