For executives considering AI adoption, the question is whether the technology can reduce costs or time to revenue sufficiently to justify the investment. Yet the benchmarks used to evaluate AI tell us little about its performance in actual business decisions. Concordance with decisions already made by your organization offers a more direct way to assess whether AI can deliver that value.
In the modern AI toolkit, benchmarks have become the gold standard for evaluating whether a model performs better than its competitors or previous versions. But improvements in benchmark performance have not necessarily correlated with business outcomes:
- They target problems that may not be related to the ones your organization cares about.
- Subject to Goodhart's law, models are heavily optimized for benchmarks, potentially at the expense of performance that businesses care about.
Rather than trying to solve the shortcomings of benchmarks by creating even more benchmarks, a better approach may be to directly measure how closely AI outputs agree with decisions already made by your organization, and whether improving either or both could deliver greater business value.
The problem with a single right answer
Many benchmarks measure how frequently a model provides a predetermined answer. But in business decisions, there may be several equally acceptable answers. Moreover, an answer that is right at one point in time can be horribly wrong at another.
A demand letter to a large past-due customer who is currently engaged in high-value negotiations may be an appropriate response to the overdue payment, but probably a strategically bad one.
Whether a part or customer belongs to a specific market, whether to maintain a specific safety stock, which logistics decision to make, or even whom to hire are rarely decisions where the answer is strictly yes or no. Several potential options may be reasonable.
Measuring agreement with concordance
One way to address this complexity is to measure agreement. A formal method used for decades is concordance, which measures the degree of agreement between two or more raters. In the case of AI, its outputs can be compared with decisions made independently by qualified business stakeholders, using agreement on values, labels, or rankings (for example, of different versions of generated text).
For the examples above:
- Market classification: Does the AI classify a customer or part into the same market as a human expert?
- Safety stock: Does the AI's proposed inventory level agree with the level selected by an experienced planner or a planning system?
- Logistics: When several supplier options are possible, does the AI reasonably rank the suppliers?
- Customer communications: Is a generated email acceptable?
Other benefits of measuring agreement
Measuring the degree of agreement rather than just a pass/fail rate allows the company to understand why decisions differed, whether those differences would affect business performance, and what improvements to data, models, and context could potentially deliver greater value for the money spent.
Agreement does not, by itself, prove that a decision is correct. But when combined with evaluation of outcomes, your organization gains visibility not only into AI performance but, even more valuably, into the decision-making processes of the company itself.
Disagreements may identify weaknesses in the AI system, but they may also reveal inconsistencies or opportunities for improvement in existing business processes.
What this changes for organizations
Raw benchmark evaluations can obscure how model performance translates into business outcomes, along with the hoped-for ROI.
It also allows you, not the model providers, to define what constitutes acceptable performance.