Navigating Costly Data Science Errors: The Truth Behind KPIs
Written on
Understanding Misleading KPIs
When faced with declining sales, the Russian country manager of a beauty company inquired if we could collaborate to enhance their performance. My business partner Darya and I promptly initiated a pilot project that revamped supply chain delivery algorithms in two locations. Remarkably, within just a few days, we achieved over a 10% sales increase—an unprecedented result in my experience.
Despite my strong belief in metrics-driven management, this experience serves as a cautionary tale about the hidden and costly pitfalls associated with data-driven strategies. In this case, the company’s key performance indicators (KPIs) were being measured poorly, leading to an enlightening conversation that highlighted potentially expensive data science errors.
Instead of celebrating the significant improvement, the supply chain manager’s initial response was skepticism: “These results can't be right! My KPIs haven’t changed, so the results must be incorrect.” When I probed about the KPIs in question, it became evident that although the manager was knowledgeable and well-intentioned, there was a critical blind spot within the company’s management practices.
Investigating KPI Accuracy
I wondered if the company truly had 97% shelf availability. To verify, I visited several locations and captured images of the shelves, revealing stark discrepancies between the actual stock and the optimistic figures portrayed by the KPI dashboard. Management was convinced about their reported 97% availability and failed to recognize that sales enhancements could stem from improved supply chain demand forecasting.
The fundamental question arises: If the KPIs were static, how could sales performance improve? This reflects a common mistake in data-driven management—relying on incorrect data to answer the right questions or vice versa.
Identifying the Root Causes
Managing by metrics is a complex endeavor, especially amidst the surge of machine learning and artificial intelligence. In fact, the abundance of data often complicates matters further.
- When and How to Measure KPIs? Determining shelf availability systematically is challenging. Even obtaining an accurate inventory count can be fraught with inaccuracies. Additionally, one must consider whether the products are displayed or stored away.
- Frequency of Measurement: Is frequent (e.g., weekly) reporting, which may be inaccurate, preferable to less frequent (monthly or quarterly) assessments that are more thorough but costly?
- Purpose of Measurement: Every managerial decision should originate from the intended impact of the KPI. For instance, is the focus on product availability in a store or its actual display on the shelf? Understanding the rationale behind each KPI is crucial.
When I queried management about their KPI measurement approach, they replied, “Our measurement aligns with corporate standards.” While this is a valid response, it often results in competent managers engaging in activities that add little value.
Transitioning to Business Science
I have previously stated that traditional data science is obsolete. This scenario exemplifies why!
Simply measuring, analyzing, and reporting numbers is insufficient unless it begins with a clear purpose. I take pride in demonstrating the value of data in this instance, but I firmly believe that most of the success stemmed from a shift in management perspective.
- From asking how to enhance forecast accuracy
- To questioning how to boost sales despite inaccurate figures
Shifting focus to value-driven inquiries is the essence of Business Science.
The accuracy of forecasts becomes less relevant if the primary objective does not prioritize value creation. Inaccurate KPI assessments can distort the perceived impact.
Addressing Management Errors
To optimize demand with the highest predictive accuracy and business impact, an autonomous data science model must tackle three sources of errors concurrently:
- Usage Error: Employing the correct model for the wrong application, such as relying on budget figures for supply chain decisions.
- Input Error: Implementing a valid model but using incorrect inputs, such as unexpected promotions or macroeconomic shocks.
- Residual Error: Predicting one outcome and observing another, which is often regarded as the primary error type.
It’s essential to confront each error source simultaneously:
- Usage Error: Customize forecasts for each specific scenario.
- Input Error: Use autonomous test-and-learn methods for adjustments.
- Residual Error: Develop meta-models to identify the best combination of models.
To achieve success in Business Science, it must be designed to be more autonomous than merely scientific. The best defense against input error is self-learning; sometimes, prediction errors can yield profitable outcomes, thus requiring a focus on profit rather than accuracy.
The New Paradigm of Management
An effective management strategy requires directly targeting profit enhancement. Demand modeling necessitates intricate approaches, such as meta-modeling, which may not always be transparent. Conversely, supply models are always fully explainable.
Optimal supply entails establishing clear strategic goals, such as maximizing economic profit while considering the costs associated with unsold goods. This creates a new alliance between humans and machines.
On one side, there’s advanced demand modeling operating automatically. On the other, classic, fully-explainable supply-side models ensure optimal results at all times.
This dual approach addresses several concerns:
- Senior managers gain sales and profit insights that are easier to interpret than complex forecasting details.
- Operational managers retain control over strategies, objectives, and constraints, continuing to set the parameters as they have always done.
- Customer purchasing behavior, rather than potentially flawed KPI data, drives self-learning, allowing for automatic corrections of any errors, even those arising from dirty data.
This redefined management by numbers represents a revolution in autonomy.
Happy KPIs!
PS: For more insights into Business Science, get free access to your monthly summary, including new software and university-level learning.
The video titled "The 10 Biggest Mistakes Companies Make with KPIs" provides valuable insights into common pitfalls in KPI management and how to avoid them.