Build the evaluation set before the AI demo

An AI demonstration selects the inputs the team knows will work. An evaluation set preserves the inputs the product must handle, including the ones that are incomplete, ambiguous, confidential, adversarial, or impossible to answer. Building that set before the demo changes the conversation from “this looks intelligent” to “this behaviour is useful and its limits are known.”

An evaluation set does not need thousands of examples to begin. It needs coverage of the jobs, sources, users, consequences, and common failure classes that define the first release. Twenty carefully selected cases from real work can be more informative than hundreds of synthetic questions that repeat the happy path.

Describe the expected properties, not one perfect sentence

Generative answers can be correct without matching an exact reference. A case should therefore record required facts, permitted sources, prohibited claims, acceptable uncertainty, format requirements, and whether the system should answer, ask, refuse, or escalate. For extraction, expected fields may be exact; for support or research, source use and factual properties matter more than wording.

Include negative cases deliberately. The source may be missing, outdated, contradictory, or outside the user’s permissions. The request may attempt prompt injection or ask for an action the tool is not allowed to perform. A system that performs well only when all inputs are cooperative is not ready for production.

Evaluate the pipeline in parts

For RAG, measure whether the correct evidence was retrieved before grading the answer. For tool use, record whether the right tool was selected, whether arguments passed validation, and whether confirmation was respected. For document extraction, distinguish field accuracy from document classification and review routing.

Some properties can be checked with code, some with model-based grading, and some require a domain specialist. Model graders are useful for scale but should themselves be calibrated against human decisions. A single aggregate score hides the difference between harmless style variation and a confidential data disclosure.

On an NDA-protected workflow, the polished demonstration handled complete documents well but concealed the difficult inputs operators saw in routine work: missing fields, conflicting values, and sources that should not be combined. We turned those examples into separate checks for retrieval, extraction, and the final decision. That made failures diagnosable and stopped prompt changes from appearing to fix problems that belonged earlier in the pipeline.

Make the set part of change control

Store evaluation cases with versions and rerun them when the model, prompt, retrieval strategy, source parser, permissions, or tools change. Review failures by class rather than tuning against one embarrassing example. New production incidents and user corrections should add cases, so the set becomes a record of what the product has learned.

The evaluation set is not a final quality certificate. It is a repeatable argument about readiness. Creating it before the demo keeps model selection, prompt work, and stakeholder expectations tied to the actual job rather than the most persuasive conversation the team can stage.