Context
A customer asks whether a travel booking has completed after an agent tool call.
How-to
How to build a golden dataset from production traces in 5 steps.
Choose recurring mistakes, write the correct replies, get them approved, and test whether the examples help your agent. Keep a record of the source and rules for reuse.
Steps
- 01
Collect traces
Export OpenTelemetry spans or conversation logs that include requests, tools, intermediate steps, final responses, and outcome where available.
Choose one task and a specific date range. Remove or mask unnecessary personal data before review. Keep a source ID so reviewers can trace each example back to the original record.
- 02
Cluster repeated failures
Group recurring problems by request type, policy mistake, tool error, misunderstanding, or missed handoff.
Start with a recurring mistake that costs time or causes harm. Ask a reviewer to explain the correct result. Include enough examples to cover the different ways this mistake happens.
- 03
Write the approved replacement
Write the reply or actions that should have happened and explain the correction.
The correction should state what changed and why: missing fact, wrong tool choice, incomplete policy step, unsafe claim, or weak handoff. Use an escalation state rather than inventing a confident answer.
- 04
Store outside weights
Keep context, rewrite, scope, version, and approval status in a library you can filter and disable.
Record the original log, task, failed reply, approved correction, reviewer, approval time, policy version, and limits on reuse. These details let the team check, update, or remove the example later.
- 05
Retrieve and measure
Inject relevant examples at runtime and compare success, edits, escalations, latency, and cost.
Use separate test cases or a controlled test environment. Check whether the examples help on the target task and cause new failures on other tasks before expanding their use.
Common mistakes
- Treating raw logs as approved examples without checking the correct reply
- Building huge synthetic sets before sampling real failures
- Skipping approval gates and letting stale examples into production
- Optimizing a score that is not tied to a business outcome
Example experience record
Observed failure
The agent states that a ticket was purchased without confirming the result or providing a safe next step.
Approved correction
Confirm the completed state only when the trace supports it, state the relevant booking detail, and invite the customer to return if anything changes.
Controls
Tag as travel support, completion confirmation, approved; restrict retrieval to matching workflow and policy scope.
How we prepare training dataGet help preparing examples from your logs
Implementation checklist
Use this before expanding beyond a single workflow.
Copyable schema
Keep source trace, failed output, approved replacement, critique, scope, version, and approval status.
Separate test cases
Set aside cases for testing. Don't measure success only on the examples you already corrected.
Reviewer calibration
Score a shared sample first so acceptance criteria stay consistent.
Approval rules
Don't approve an answer without evidence. Set aside disagreements and exceptions that aren't safe to reuse.
Negative retrieval tests
Confirm that out-of-scope examples are not retrieved into unrelated workflows.
Privacy checklist
Mask identifiers, record how long data will be kept, and leave originals in the approved storage system.
Free template
Start from a filled golden-dataset record
Download an illustrative schema with production trace, failure, correction, critique, scope, version, and approval fields.
This template uses made-up examples, not client data. Adapt it to your policies, privacy requirements, and approval process.
How less than three runs this
The Custom Model Sprint applies this method to one workflow: review the data, establish the prompt baseline, prepare approved examples, and evaluate the agreed retrieval or training experiment.
Limitations
- Following the steps without a held-out comparison can create false confidence.
- Retrieval without scope filters can transfer one customer’s exception to another.
- Schedule reviews of approved examples and remove those that no longer apply.
- Synthetic examples can cover extra scenarios. Include real cases too when testing risks your users face.
How we test changes
We review real conversations with your team and agree what the agent should do differently. Approved corrections can become training data, test cases, or examples the agent looks up. We compare each change with your best existing setup on separate test cases, then check for new failures in the software you plan to deploy.
References:
Related
Need help building the dataset?
Book a call about one task your agent gets wrong. We'll discuss the data, review process, and tests it needs.