AI Engineering · How-to

How to test an AI automation before it reaches production

'It looked good in the demo' isn't a test. Here's the evaluation process we use to prove an LLM automation works, and to keep it working when prompts, models and data change.

By Max Chuchmai3 min read

On this page
  1. Step 1: Build the test set first
  2. Step 2: Decide what “good” means
  3. Step 3: Grade with code where you can
  4. Step 4: Run it on every change
  5. Step 5: Keep measuring in production
  6. A note on cost and speed
  7. What to ask your AI vendor or team

Every AI project has a moment where the demo works beautifully. Someone tries five examples, they all look great, and the team wants to ship. Then real data arrives: messy emails, unusual documents, edge cases nobody thought of. Quality drops, and nobody can say by how much.

Evaluation, or “evals”, is how you replace that guesswork with numbers. It’s the single habit that most separates AI systems that last from those that get quietly switched off.

Step 1: Build the test set first

Before writing a prompt, collect real examples of the task with the correct answer for each. For most business automations, 50 to 200 examples is a solid start.

Make sure the set includes:

  • Typical cases, in the proportions you expect in production.
  • Hard cases: ambiguous inputs, missing information, unusual formats, other languages.
  • Cases where the right answer is “I don’t know” or “send to a human”. A system that always answers confidently is dangerous.
  • Past failures, added every time something goes wrong in production.

The people who do the task today should label the examples. Their disagreements are useful too: if two experts give different answers, the task definition needs work before any AI does.

Step 2: Decide what “good” means

Write down, specifically, what makes an output correct. Vague criteria like “a good summary” can’t be measured. Specific ones can:

  • The extracted invoice total matches the document exactly.
  • The classification is one of the six allowed categories.
  • Every claim in the email draft is supported by the research brief.
  • The reply contains no pricing that isn’t on the approved price list.

Step 3: Grade with code where you can

Use the cheapest reliable grader for each criterion:

  1. Code checks for anything exact: matching values, valid formats, allowed categories, required fields present, length limits. These are fast, free and never inconsistent.
  2. Model-graded checks for judgements code can’t make, such as “is this summary faithful to the source?” or “is the tone appropriate?”. Give the grading model a clear rubric and ask for a short reason with every score.
  3. Human review for a sample, to make sure the model grader agrees with your experts. If it doesn’t, fix the rubric before trusting it.

Track a few clear numbers per run: accuracy on each criterion, the share of cases correctly sent to a human, cost per task and response time.

Step 4: Run it on every change

Your test set becomes a regression suite. Run it whenever anything changes:

  • A prompt edit, however small.
  • A new model or model version. Newer isn’t always better for your specific task.
  • A new tool, data source or retrieval setting.
  • A change in the format of incoming data.

Compare the results with the previous run before you release. If accuracy on any criterion drops, you’ll know before your customers do. We treat this exactly like automated tests in normal software: no green suite, no release.

Step 5: Keep measuring in production

A test set describes the past. Production data drifts. Keep watching with:

  • Sampled review. A small share of live outputs gets a human check every week.
  • Feedback signals. Thumbs up or down, edits, and rejections from the people who use the output. Our article on human-in-the-loop design explains how to capture them.
  • Alerts for changes in the inputs (new document types, new languages) and in the outputs (more “send to human” decisions, lower confidence, higher cost).

Every confirmed production failure goes back into the test set. Over time the suite becomes a precise description of what your automation must handle.

A note on cost and speed

Evals also make cost decisions easy. With a solid test set you can try a smaller, cheaper model and see exactly what you’d lose in quality, if anything. Many tasks run perfectly well on a smaller model, and routing only the hard cases to a larger one can cut running costs significantly. Without evals, that decision is a guess.

What to ask your AI vendor or team

  • Can you show me the test set and the latest results?
  • What happens to quality when we change the model?
  • How do you know when quality drops in production?
  • How are production failures fed back into testing?

If the answers are vague, the system isn’t ready to run your business process.


Every AI automation we ship comes with a test suite and production monitoring. See our AI and automation services.

Written by

Max Chuchmai

Founder & Managing Partner, CoderStein. System architecture, AI automation, and GTM engineering.

Follow on LinkedIn

Have a process you'd like AI to take over?

Book a 30-minute call. We'll tell you honestly whether it should be a workflow, an agent, or left alone.

Scope an automation