Context
A customer asks whether a travel booking has completed after an agent tool call.
How-to
How to build a golden dataset from production traces in 5 steps.
A practical, production-first method for selecting failures, writing approved replacements, adding governance, and testing whether the resulting examples help.
Steps
- 01
Collect traces
Export OpenTelemetry spans or conversation logs that include requests, tools, intermediate steps, final responses, and outcome where available.
Start with a bounded workflow and a repeatable time window. Remove or mask unnecessary personal data before review, then retain a trace identifier so every selected example can be audited.
- 02
Cluster repeated failures
Group by intent, policy break, tool error, misunderstanding, or escalation—not by one-off noise.
Prioritize a cluster only when the failure repeats, has a measurable cost, and a reviewer can explain what a good result looks like. A small, coherent sample beats a large undifferentiated export.
- 03
Write the approved replacement
Capture the response or trajectory that should have happened, plus a short critique.
The correction should state what changed and why: missing fact, wrong tool choice, incomplete policy step, unsafe claim, or weak handoff. Use an escalation state rather than inventing a confident answer.
- 04
Store outside weights
Keep context, rewrite, scope, version, and approval status in a library you can filter and disable.
A minimum record includes source trace, workflow, failed output, approved replacement, reviewer, approval timestamp, policy scope, and retrieval restrictions. That provenance lets teams revise or retire an experience safely.
- 05
Retrieve and measure
Inject relevant examples at runtime and compare success, edits, escalations, latency, and cost.
Test against a held-out slice or a controlled workflow. Measure whether retrieved examples help the target scenario without degrading unrelated intents, then expand only if the evidence supports it.
Common mistakes
- Treating raw logs as gold without an approved rewrite
- Building huge synthetic sets before sampling real failures
- Skipping approval gates and letting stale examples into production
- Optimizing a score that is not tied to a business outcome
Example experience record
Observed failure
The agent states that a ticket was purchased without confirming the result or providing a safe next step.
Approved correction
Confirm the completed state only when the trace supports it, state the relevant booking detail, and invite the customer to return if anything changes.
Controls
Tag as travel support, completion confirmation, approved; restrict retrieval to matching workflow and policy scope.
See how this governed record fits a golden-dataset serviceUse a production-trace sprint when you need help applying the method
Implementation checklist
Use this before expanding beyond a single workflow.
Copyable schema
Keep source trace, failed output, approved replacement, critique, scope, version, and approval status.
Coverage and held-out split
Reserve a held-out slice for measurement; do not evaluate only on the examples you already corrected.
Reviewer calibration
Score a shared sample first so acceptance criteria stay consistent.
Acceptance and rejection
Reject plausible-but-unverified answers, unresolved disagreements, and unsafe-to-retrieve exceptions.
Negative retrieval tests
Confirm that out-of-scope examples are not retrieved into unrelated workflows.
Privacy checklist
Mask identifiers, document retention, and keep originals in the approved system of record.
Reusable resource
Start from a filled golden-dataset record
Download an illustrative schema with production trace, failure, correction, critique, scope, version, and approval fields.
Illustrative template—not client data. Adapt it to your policies, privacy controls, and approval process.
How less than three runs this
The Agent Improvement Sprint compresses this loop for one workflow: audit, human-corrected dataset, retrieval prototype, and before/after evidence.
Limitations
- Following the steps without a held-out comparison can create false confidence.
- Retrieval without scope filters can transfer one customer’s exception to another.
- Gold sets age; schedule review and retirement.
- Synthetic examples can complement production gold but should not replace it for live risk.
Methodology by LTTAI LTD
This page describes less than three’s production-trace review method: reconstruct failures, capture human-approved corrections, store them outside model weights, and measure whether scoped retrieval improves a defined workflow.
Primary references used for terminology and risk framing:
Related
Want this done on your traces?
Book a call. Bring a sample of production traces and we will outline a focused golden-dataset sprint.