Evals
A judge that can't be contaminated, and a CI gate that fails closed.
In short: an automatic quality check that runs before your changes go live and blocks them if the agent got worse. Like tests, but for judgement rather than code — because an agent that gets worse usually doesn't error, it just quietly starts being wrong.
Agent behaviour regresses silently. A prompt tweak that helps one case quietly breaks three others, and nothing errors. Evals turn that into a build failure.
Running them
agentfast eval
Runs every case in evals/cases.yaml, scores it, prints a per-metric breakdown, and exits
non-zero if anything is below threshold — so it's a CI step with no further wiring:
- name: refund_damaged_item
run:
prompt: "I'd like a refund for order ord_1001 — the item arrived damaged."
expect:
contains: ["refund", "5-10 business days"]
tools_used: ["kb_search", "order_lookup", "refund_request"]
grounded: true
Two kinds of scorer
Deterministic — containment, tool use, grounding. Cheap, fast, no model call, no flakiness. Most assertions belong here.
LLM judge — for the things a string match can't see: is this answer actually helpful, is the tone right, did it address what was asked.
The judge runs in a clean room
This is the part that matters. The judge executes in a fresh context with no workspace access: it sees the input and the output, and nothing else. Not the agent's reasoning, not its scratchpad, not the run's history.
A judge that can see the agent's reasoning gets talked into a pass. The agent explains why its answer is correct, and the judge — being the same kind of model — finds that explanation persuasive. Contaminated judges grade generously and drift toward always passing, which is worse than no eval at all because it looks like a green build.
The judge also fails closed. If it errors, times out, or returns something unparseable, the case fails. A broken judge must never produce a green build.
Thresholds
evals:
cases: evals/cases.yaml
judge_model: claude-sonnet-4-5
thresholds:
containment: 0.8
tool_use: 0.9
grounding: 0.7
Set these where you actually want the build to break. Thresholds nobody believes get ignored, then disabled, then deleted.
In CI
- name: Agent quality gate
run: agentfast eval
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
agentfast new writes real cases for the template you scaffolded, so the gate works from the
first commit. Add yours next to them — every production incident is a case worth keeping.
Results are stored
Eval results land in the run store like any other run, so GET /api/evals and the dashboard show
history — which case regressed, when, and against which change. A single failing run tells you
something is wrong; the history tells you what you did.