One of the first steps when embarking on a real-world machine learning (AI) project is to consider metrics and Key Performance Indicators (KPIs) for evaluating a model's performance.

However, the selection of these metrics must be approached with great care, as, in practice, it is possible to end up with multiple metrics that may be contradictory.

I am currently working on a ChatBot for customer support automation, with the primary KPI being cost reduction. An obvious secondary metric is the reduction in the number of generated customer support tickets. Consider an ML model that impedes users from creating customer support tickets; this would result in a significant decrease in ticket numbers, optimizing the secondary metric and reducing costs.

However, it is evident that this approach would conflict with another crucial metric: user satisfaction. On the other hand, user satisfaction could be improved by involving real-world agents early in the chat process. This, however, contradicts the initial goal of cost reduction.

In reality, it becomes a matter of carefully weighing multiple, sometimes conflicting, metrics to optimize for the desired outcomes.

Image taken from: https://sketchplanations.com/goodharts-law