How to evaluate an AI use case in a business workflow

Most organizations are now past the stage of asking whether AI is real and into the harder stage of deciding where to point it. The failure rate at this stage is high, and the failures share a shape: a pilot demonstrates something impressive, everyone is encouraged, and then the project stalls somewhere between the demonstration and production without anybody being able to say precisely why.

The reason is usually that the evaluation asked the wrong question. "Can a model do this?" is answerable in an afternoon and tells you very little. The questions that determine whether a use case will reach production are about the workflow, the data, and the cost of being wrong — and all of them can be answered before a line of code is written.

The first question: what does a wrong answer cost?

A language model is fluent whether or not it is correct. This is the defining property of the technology and every serious design decision follows from it. A model does not fail the way software fails, with an error and a stack trace. It fails by producing something plausible, well-formatted and wrong, in a tone of complete confidence.

So the first question about any use case is what happens when that occurs. Not whether it will occur — it will — but what it costs when it does.

If a wrong answer is caught immediately by the person receiving it, at trivial cost, the use case is a good candidate. Drafting a reply that a human will read carefully before sending; suggesting a classification that an operator confirms in a second; retrieving three passages that a reviewer will read anyway. In each case the model is proposing and a human is disposing, and a bad proposal costs a moment.

If a wrong answer is expensive, hard to detect, or acted upon before anyone checks, be very careful. An incorrect diagnosis that sends an engineer down a three-hour investigation. A misclassified document that routes a legal notice into the wrong queue for a fortnight. A summarisation that quietly omits the clause that mattered. These are not disqualifying, but they change what has to be built: the system now needs to be able to refuse, and refusing well is most of the engineering.

And if a wrong answer is invisible — if nothing downstream would ever reveal the error — stop. There is no safe version of that system, because there is no feedback loop by which anyone could ever learn it was wrong.

The second question: where is the data, and where is it allowed to go?

This is the question that kills more enterprise AI projects than any technical limitation, and it is usually asked far too late.

The data that makes a model useful is, with grim regularity, the data an organization is not permitted to send anywhere. Support logs contain hostnames, internal addresses, session identifiers, and often credentials that were never meant to be written down. Case files contain personal records. Contracts contain terms nobody wants sitting in a third party's corpus. Clinical, financial and legal data arrive with their own constraints attached.

A pilot that quietly ignores this — because it uses synthetic data, or because it is "only a pilot" — will produce an encouraging result and then die at the security review. Everybody involved will be surprised, and nobody should be.

Ask early: what data does this need to be genuinely useful, where does that data live, and what is it permitted to touch? If the honest answer is that the useful data cannot leave the network, that is not the end of the project. It is a constraint, and it is far better to design for it from the first day than to build a system that assumes a hosted API and then attempt to retrofit a perimeter around it. Retrofitting produces a boundary with exceptions in it, and exceptions in a boundary are the whole problem.

The third question: can the answer be checked?

An answer nobody can verify is an answer nobody should act on.

The practical form of this question is whether every claim the system makes can be traced back to something that produced it — a source document, a specific log line, a record, a passage — and whether the person reviewing it can follow that trail in seconds rather than minutes. A system that says "the contract permits termination on 30 days' notice" is far less useful than one that says the same thing and shows you the clause.

Provenance is not a nice-to-have. It changes what the reviewer is doing. Without it they are asked to trust a conclusion, which they cannot reasonably do, so they either check everything themselves, in which case the system has saved nothing, or they stop checking, in which case it is a matter of time. With it, they are checking working, which is a fast, natural task that people do well.

A related and stricter requirement: the citation must actually resolve. It should point at the stored source text, not at the model's paraphrase of it. Models will produce citations that look right and are not, and a system that does not verify them is simply moving the fabrication one level down.

The fourth question: who decides whether the answer is good enough?

This is the one that separates systems that reach production from systems that do not.

The intuitive design is to ask the model how confident it is and route on the answer. It is also the single most dangerous thing you can do, because a model's self-reported confidence correlates with how confident its prose sounds, not with whether it is correct. It is not a probability. Routing on it means the system will be most assertive precisely when it is most fluently wrong.

The alternative is to compute the decision deterministically, from the evidence actually retrieved rather than from the model's opinion of itself. How much corroborating evidence was found? Did independent sources agree? Did any cited passage fail to resolve? Was a similar case found, and how similar was it really? These are quantities, they can be measured without the model's participation, and a threshold over them is a decision the organization can reason about and tune.

The rule that follows is worth stating plainly: no evidence, no commit. An answer with nothing resolvable behind it should score zero regardless of its content and regardless of its tone.

A system built this way will defer more often than a naive one. That is the trade, and it is the right one. In an operational setting, an agent that hands a case back and says why is useful. An agent that is confidently wrong once will cost more credibility than it earns in a hundred correct answers, and it will be switched off.

The fifth question: what does success look like, numerically?

"It seems to work well" is not an evaluation, and a demonstration is not evidence.

Before building, establish what is being measured and what the current baseline is. How long does this task take a person today? What is their error rate? How often is a case escalated, and how often is that escalation correct? Without those numbers there is no way to know whether the system helped, and every subsequent conversation about it will be conducted on impressions.

Then build an evaluation set: real cases, with known correct answers, held back from development. Not many are needed — a few dozen genuinely representative cases will tell you more than a thousand synthetic ones — but they must be real, they must include the hard cases, and they must be assembled before anybody has seen the system's output, or the selection will be quietly biased toward what it does well.

Measure two things separately. Accuracy when the system commits to an answer, and the rate at which it commits at all. A system that is right 95 per cent of the time on the third of cases it is willing to answer may be far more valuable than one that is right 80 per cent of the time on everything — and the second is much easier to build, which is why so many pilots look better than the systems they become.

The sixth question: does the workflow have a place to put the output?

A surprising number of AI projects produce something genuinely good that nobody uses, because no thought was given to where it lands.

If the output arrives in a separate tool that an operator must remember to open, it will not be opened. If it appears in the queue they already work, in the format they already read, attached to the case they are already looking at, it will be used. This is not a detail. It is frequently the difference between a project that shows value and one that does not, and it is decided by integration work that nobody finds interesting.

Putting it together

The use cases that reach production tend to share a profile. The task is one a person currently does by reading and judging. A wrong answer is cheap and immediately visible. The output can be checked against a source. There is a human in the loop who is going to look at it anyway. And the result lands where the work already happens.

The ones that stall share the opposite profile: the value depends on the system acting autonomously, the data cannot be moved, the answer cannot be verified, and nobody established a baseline, so nobody can prove it helped.

None of this requires a decision about which model to use, and that decision is far less consequential than the industry's discourse implies. Almost every hard problem in applied enterprise AI is a problem about the boundary drawn around the model, the evidence demanded of it, and the human left in the loop.