# LLM evaluation: a practical 7-step workflow > Set up LLM evaluation with a golden dataset, useful metrics, CI gates, regression tests, and production feedback. A practical workflow for engineers. Say you ship a support agent after trying a handful of questions in a playground. It answers those beautifully, so you merge a shorter prompt and move on. Then a customer asks about an exception to the refund policy, and the agent confidently promises something your backend won't allow. That's a hypothetical failure, but it's exactly what your LLM evaluation suite should catch before release. A demo gives you examples of success; a release gate needs evidence across the situations you actually support. Start with saved inputs, explicit expectations, and a command that fails when behavior gets worse. Build from there. ## What is LLM evaluation? Evaluation means running your application against defined cases and checking whether its behavior meets your requirements. If you're asking, "How do I evaluate LLM outputs?", start by writing down what makes an answer acceptable for one concrete task. Metrics make those expectations measurable. Accuracy checks correctness; relevance checks whether the response addresses the request; groundedness checks whether claims have support in supplied evidence; toxicity checks for abusive or harmful content. Pick the checks your application needs, then define how each one passes or fails. A **golden dataset** is a versioned collection of inputs, context, and expected outcomes. Those outcomes can be facts, allowed actions, or required refusals; they don't need to be exact reference sentences. Use code for deterministic checks. Use an **LLM-as-judge** to apply a written rubric to qualities that need interpretation, and calibrate it against human review of the same examples. Keep a judge's verdict inspectable, with reasons and supporting evidence. ## Why AI agent testing goes beyond single-call evaluation A single-call test can inspect a response alongside its input. An agent test also needs the trajectory: retrieved evidence, tool calls, arguments, retries, delegation, and changes to persistent state. Imagine an agent that says a booking failed after successfully creating it. Scoring only its final message misses the side effect; retrying the task might create another booking. Your test should inspect the booking store and verify the recovery path. For AI agent testing, evaluate intermediate boundaries and the final outcome. A wrong identifier selected early can affect later calls even when each tool executes correctly. Account for nondeterminism explicitly: repeat important cases and retain every result. Also test permission boundaries with the same rigor as task completion; the [AI agent security checklist](/blog/ai-agent-security-checklist) covers approval gates, isolation, and shutdown behavior. ## How to set up LLM evaluation for your application: a 7-step workflow ### 1. Define success criteria before choosing a metric Pick one user workflow and write its acceptance criteria in plain language. For the hypothetical refund agent, success means checking eligibility against the supplied policy, applying only an authorized refund, and reporting the recorded outcome accurately. Separate task success from hard constraints: a correct explanation doesn't compensate for refunding the wrong account. Write both as assertions, with the evidence each assertion needs. Agree with the workflow's owner on what should block release before anyone sees the candidate's scores; don't negotiate the threshold after a disappointing run. ### 2. Build a small golden dataset you understand I'd start with 20–50 carefully reviewed, production-like examples rather than 500 unreviewed synthetic ones. Include routine successes, ambiguous requests, missing information, prohibited actions, and tool failures; choose cases that exercise different decisions. Use sanitized real requests where available, with permission to reuse them, and deliberately authored fixtures where coverage is missing. Store each case with an ID, input, relevant context, initial state, and expected outcomes. Here's a hypothetical fixture: ```yaml id: refund-outside-policy input: 'Refund this order.' context: return_window_days: 30 order_age_days: 45 expected: refund_created: false explains_ineligibility: true ``` Keep a held-out set for release decisions so prompt tuning doesn't become memorizing the examples you inspect daily. For a concrete example of separating tasks from their validation, see [evaluating coding agents on real tasks](/blog/coding-agent-benchmark-real-prs). ### 3. Match metrics to actual failure modes Turn each failure mode into the cheapest reliable check. Validate structured fields with code, compare recorded state with expected state, and verify tool calls against your [agent guardrails and execution policies](/blog/ai-agent-guardrails). Save model judges for questions that require interpretation, such as whether an explanation accurately reflects a policy exception. Give each judge the request, relevant evidence, response, and a narrow rubric with explicit pass/fail conditions. Compare its decisions with human labels before trusting it as a gate. Report task completion, groundedness, and permission violations separately; a blended average can conceal the exact failure that matters most. ### 4. Make evaluation a CI command Create a runner that loads fixtures, resets test state, invokes the application, executes assertions, and writes a machine-readable report. Run it against isolated services with test credentials and intercepted external writes. A failed hard constraint should produce a nonzero exit code and a trace linked from the pull request. Run a fast suite on pull requests and a broader, repeated suite before release or on a schedule. Set cost and duration budgets, and distinguish infrastructure failures from failed assertions. Don't quietly retry away a behavioral failure until the pipeline turns green; preserve the original attempt and classify the failure. ### 5. Regression-test every prompt and model change Treat a prompt tweak as a code change: compare the candidate with the current version on identical fixtures and starting state. Version the model identifier, prompt, retrieval configuration, tool schemas, and judge rubric with each run, and review case-level regressions alongside aggregate results. For agents running on a runtime such as [Octomind](https://octomind.run), put automated end-to-end tests around the configured workflow, including tool effects. The testing layer should repeat those checks across changes and expose the failing trajectory, so evaluation becomes a maintained regression suite. Block critical regressions even when the average improves. Require a reviewed explanation for accepted tradeoffs, and keep the previous configuration available for rollback. ### 6. Monitor the behavior you actually ship Log trace IDs, configuration versions, tool calls, results, latency, and failures with appropriate redaction and access controls. Sample successful-looking outputs as well as explicit errors: an incorrect answer can still arrive in a perfectly valid response. Assign someone to review samples against the same rubric used before release. Watch for shifts in request types, retrieved sources, tool errors, and judged failure rates. Investigate changes by cohort and configuration before calling them drift; a changed judge can move scores without any application change. Give each alert an owner and a concrete response, such as pausing an affected workflow. ### 7. Turn production failures into permanent tests When an incident or review reveals a miss, preserve the smallest sanitized case that reproduces it. Add the initial state and tool responses needed to expose the failure, then confirm the current version fails for the expected reason. Fix the behavior and verify that the new version passes without breaking existing cases. Add related variations to a separate challenge set so one fix doesn't merely memorize one input. Keep development cases and held-out evaluation cases distinct. Review stale fixtures when product policy changes, recording why their expectations changed; a suite should reflect the current contract without erasing failure history. ## FAQ ### What is LLM evaluation? LLM evaluations measure whether an application's outputs and actions meet defined requirements across a representative set of cases. They combine deterministic assertions, model-based judgments, and human review where needed. The useful result is evidence about specific failures and tradeoffs, with enough context to decide whether a change should ship. ### What is LLM testing? LLM testing applies software-testing discipline to applications that generate probabilistic outputs. You define fixtures and expected behavior, automate checks, and investigate regressions. Tests can inspect response structure, factual support, tool use, and resulting state; repeated runs help reveal variation that a single successful example won't expose. ### How do you evaluate AI agents? AI agent testing checks task completion, intermediate decisions, tool arguments, and persistent effects across an entire run. Use controlled environments, representative tasks, and explicit permission boundaries. Repeat consequential cases, simulate tool failures, and inspect traces so a plausible final answer can't hide an unauthorized action or incomplete task. ## Make the next release earn its place Your next model upgrade should arrive with a comparison report: which tasks improved, which cases regressed, and which boundaries still held. Your next production failure should leave behind a reproducible fixture with an owner. Start Monday with one workflow and enough cases to challenge it. When the next prompt change lands, will your team have evidence to ship it—or just another convincing demo?