Answer-only rubrics
Scoring the last message hides bad tool choices, skipped confirmations, and unsafe shortcuts.
AI agent evaluation
Test what your AI agent says and does.
A correct-sounding reply can hide a wrong tool call or an unfinished task. We check the whole conversation, including actions and handoffs, and compare your current agent with the proposed change.
The final reply doesn't tell the whole story.
Check which tools the agent used, how it handled errors, and whether it should have handed the task to a person.
Simulated-only suites
Scripted paths rarely match live user language, partial tool results, or messy state.
Prompt patches as evaluation
Shipping another instruction does not prove the agent recovers on the next similar request.
No approved replacement
A failed grade identifies the problem. Reviewers also need to record the correct reply or actions so the team can test a fix.
Review the mistake and test the correction
We review conversation and tool logs against agreed scoring rules. Your reviewers approve corrections, then we test whether retrieving those examples helps on similar tasks.
Check every step
Score whether the agent understood the request, used tools correctly, recovered from errors, and followed policy.
Reviewed corrections
Reviewers write the correct reply or actions and explain why they're right and where they apply.
Measured before/after
Compare task success, edits, escalations, latency, and cost on one focused agent workflow.
What agent evaluation must inspect
Review the tool calls and intermediate steps alongside the final reply. A good answer doesn't excuse an unsafe action earlier in the task.
Rules for passing a task
Define pass/fail for the full path: required checks, allowed tools, stop conditions, and when escalation is the correct outcome.
Tool and recovery review
Score whether the agent chose the right tools, handled empty or conflicting results, and recovered without inventing state.
Repeatable release tests
Version scenarios, prompts, tools, and expected trajectories so a release can be compared to a stable agent baseline.
Connect agent scoring to broader LLM evaluation servicesStore approved trajectories as golden datasets for AI agents
Tests for the agent's actions
Review each action leading to the reply, including tool calls and required checks.
Failure categories
Wrong tools or inputs, lost information, unrecovered errors, late handoffs, policy mistakes, and unsafe actions before a correct reply.
Agent-action scorecard
Use the CSV to score results, tool use, information handling, error recovery, handoffs, safety, cost, and response time.
Unsafe-action example
An agent that skips a required verification fails the test even if its final reply sounds right.
Evaluation modes
Offline replay of production traces, controlled simulation, and online human review of live runs.
Release checks
Record scenario and tool versions. Compare against the existing setup and approve corrections for recurring mistakes.
Free template
Download the agent-action scorecard
Score task outcome, tool selection and arguments, state, recovery, escalation, safety, cost, and latency.
This template uses made-up examples, not client data. Adapt it to your policies, privacy requirements, and approval process.
Who this is for
- Teams that need to test an agent's tool calls as well as its replies
- Support, ops, and voice agents with repeating multi-step failure modes
- Teams that need release checks based on real tasks
- Teams comparing prompt changes with retrieved examples
Custom Model Sprint for evaluation
Review
Sample logs, group recurring mistakes, and agree how to score actions, tool use, and handoffs.
Prepare examples
Review and approve the correct actions for the agreed prompt, retrieval, tool, or training test.
Evaluate
Compare the existing and changed agent and recommend whether to deploy.
Evidence before broader rollout
We compare results on one task before recommending a wider rollout. We can't promise improvement before reviewing the data.
Limitations
- Answer-only graders miss unsafe trajectories that end with a plausible reply.
- Simulation coverage is incomplete without production language and tool noise.
- Release gates are weak if scenarios are not versioned with prompts and tools.
- A failed grade without an approved replacement does not improve the next similar run.
How we test changes
We review real conversations with your team and agree what the agent should do differently. Approved corrections can become training data, test cases, or examples the agent looks up. We compare each change with your best existing setup on separate test cases, then check for new failures in the software you plan to deploy.
References:
Common questions
What does the work include?
We review conversation and tool logs against agreed scoring rules. Your reviewers approve corrections, then we test whether retrieving those examples helps on similar tasks.
Do you fine-tune our model?
If it's worth testing after we review the problem. We try your best prompt and consider better examples or tool fixes first. We test an adapter on cases it wasn't trained on and check for new failures. The contract sets delivery and usage rights; the base model's license still applies.
What do we need to start?
Logs or conversation exports for one task, examples of good results, and a reviewer who can approve corrections. We agree data handling before you send anything. Remove unnecessary personal details.
How is success measured?
We test the existing and changed agent on the same task. Where the data is available, we compare completed tasks, human edits, handoffs, response time, and cost.
Related
Discuss the tasks your agent needs to get right.
Bring examples of failed runs. We'll discuss what to test and what reviewers need to assess a fix.