A test suite for non-deterministic systems — so you can change a prompt, a model, or a tool and know what broke before your customers do.
The reason agent projects stall between pilot and production is almost never capability. It is that nobody can answer "is the new version better?" with anything but a vibe. Agents are non-deterministic, so the same input produces different outputs; a prompt tweak that fixes one case silently breaks four others; and a model provider updating a version underneath you can degrade behavior with no change on your side. Traditional test suites do not apply because there is no single correct string. The fix is a graded evaluation harness: a fixed dataset of representative and adversarial cases, scored by deterministic checks where possible and model-graded rubrics where not, run on every change and continuously against production traffic. Platforms cover this from several angles — Braintrust, Langfuse, LangSmith, Galileo, Patronus, and Arize on the specialist side, with AgentCore Evaluations and Harness or Relevance AI's built-in framework where you are already on a platform.
150-300 real cases beats 1,000 invented ones. Pull from actual traffic or, pre-launch, from the tickets and transcripts the agent will replace. Include the boring majority case, the known edge cases, and — the part teams skip — adversarial inputs: prompt injection attempts, out-of-scope requests, requests to take an action the agent must refuse, and inputs containing PII it must not echo.
For each case record the expected outcome as something checkable. Deterministic assertions where you can: correct tool called, correct arguments, refusal produced, no PII in output, JSON schema valid, cost and latency inside budget. Rubric-graded where you cannot: was the answer factually grounded in the retrieved context, was the tone appropriate, was the escalation correct. A case with no defined pass condition is a demo, not a test.
The step almost everyone skips, and the reason eval suites lose credibility. Have humans grade 50 cases, run the model grader on the same 50, and measure agreement. Below roughly 80% agreement your grader is measuring something other than what you think and you must fix the rubric before trusting a single number it produces. Re-calibrate whenever you change grader model or rubric.
Every prompt change, tool change, model change, and retrieval change runs the suite. Fail the merge on regression against the baseline, not on absolute score — the meaningful question is "did this change make anything worse," and an 87% suite that drops to 84% is a regression regardless of whether 84% sounds acceptable in isolation.
A change that improves accuracy two points while tripling token spend is a decision, not an improvement, and it must be visible as one. Record per-case cost and p95 latency alongside quality and put all three in the regression report.
Offline suites go stale because real inputs drift. Sample production traffic continuously, score it with the same rubrics, and alert on distribution shift. This is also how you catch a provider-side model update degrading your system with no deploy on your side — a failure mode with no analogue in conventional software.
Every incident, escalation, and bad review becomes a new case in the golden dataset with its expected outcome recorded. This is the compounding loop: the suite gets more valuable as the system ages, which is the opposite of how most test suites behave.
Use these templates as-is or customize for your business.
{
"case_id": "support-refund-out-of-policy-001",
"category": "refund_request",
"source": "production_2026_07",
"input": { "user_message": "...", "context": { "account_tier": "free", "days_since_purchase": 62 } },
"expected": {
"deterministic": {
"tool_called": "escalate_to_human",
"tools_not_called": ["issue_refund"],
"contains_pii": false,
"output_schema_valid": true,
"max_cost_usd": 0.04,
"max_latency_p95_ms": 3000
},
"rubric": {
"grounded_in_policy": "Cites the 60-day refund window from retrieved policy, does not invent an exception",
"tone": "Declines without being dismissive; offers the escalation path",
"no_commitment": "Makes no promise about the escalation outcome"
}
},
"adversarial": false,
"added_by": "incident-4417"
}
MANDATORY COVERAGE: majority path, known edge cases, prompt injection,
out-of-scope requests, must-refuse actions, PII-containing inputs.Run this before you trust any model-graded score, and again whenever the grader model or rubric changes. 1. Sample 50 cases spanning all categories and both pass and fail outcomes. 2. Two humans grade independently against the written rubric. 3. Resolve human disagreements first — if humans disagree above 10%, the rubric is ambiguous. Fix the rubric before blaming the model. 4. Run the model grader on the same 50. 5. Compute agreement with the resolved human labels. >= 90% -> Trust for gating decisions. 80-89% -> Trust for trend monitoring; humans review flagged failures. < 80% -> Do not use. The rubric is the problem, not the model. 6. Record the agreement figure alongside every published eval score. A score without a stated grader agreement is not a measurement.
TRIGGERS — run the full suite on any change to: - system prompts or prompt templates - tool definitions or tool descriptions - model id or model version pin - retrieval config (chunking, embedding model, top-k, reranker) - guardrail or policy configuration GATE — fail the merge if, against the current baseline: - overall pass rate drops more than 2 percentage points, OR - any adversarial case regresses from pass to fail (zero tolerance), OR - any must-refuse case now complies, OR - p95 latency rises more than 20%, OR - mean cost per case rises more than 25% without an approved exception REPORT ON EVERY RUN: pass rate by category, diff vs baseline, per-case cost and p95 latency, grader agreement figure, and a list of newly failing case ids with their inputs. PIN YOUR MODEL VERSIONS. An unpinned model id means the provider can change your system's behavior without a commit in your repo.
Get a new AI workflow every week. Prompts, tool stacks, and ROI math included.
Single agent with function-calling: one LLM with a defined toolbox (CRM, calendar, knowledge base) decides which tool to invoke at each turn. Easiest to debug; appropriate for most well-scoped business workflows.
Learn the agentic glossary →Where this workflow tends to break in production — and what to put in place before you ship it.
Model grader disagrees with humans and nobody checked
Mitigation: Mandatory calibration protocol with an 80% agreement floor; publish agreement alongside every score.
Provider silently updates a model version and behavior degrades
Mitigation: Pin model versions explicitly; run continuous production evaluation with distribution-shift alerting.
Golden dataset drifts out of step with real traffic
Mitigation: Feed every production incident back as a case; refresh from sampled production traffic quarterly.
Suite passes because the gate uses an absolute threshold
Mitigation: Gate on regression against baseline, with zero tolerance on adversarial and must-refuse cases.
Accuracy improves while cost triples, unnoticed
Mitigation: Cost and p95 latency are gated results, not observability afterthoughts.
Skip the full harness if your agent is a single-turn, low-stakes internal assistant — a curated set of 30 cases in a spreadsheet run manually before each change is proportionate and honest. Skip building custom infrastructure if your agent platform already ships evaluation; use the built-in framework until you outgrow it. Do not build this if nobody owns it, because a stale eval suite is worse than none — it produces false confidence with an official-looking number attached. And do not publish eval scores without the grader agreement figure alongside them; an uncalibrated score is a decoration.
A phased approach to get this workflow running and delivering ROI.
Days 1–30
Foundation
Days 31–60
Optimization
Days 61–90
Scale
Most enterprises do not have an AI adoption problem. They have an AI inventory problem — and the first honest count is usually three to five times what anyone predicted.
Four vendors, four incompatible pricing models, and one arithmetic trap that doubles your bill without anyone telling you.
The general-purpose voice agent and the trades-specific one cost about the same. One of them books more jobs. The difference is not model quality — it is everything around the model.
One practical AI workflow per week. No fluff.
Get the full guide with step-by-step setup, workflow templates, and copy-paste assets.