Unlabeled failures
Errors without an ideal response cannot train or retrieve well.
Production traces → golden data
Turn production AI traces into a governed golden dataset.
Transform real failures into normalized, reviewable records with an approved replacement, provenance, policy scope, and controls for evaluation or retrieval.
Raw traces are not a golden dataset.
Production history shows outcomes. Without the chosen rewrite, critique, and approval status, traces cannot teach the next request.
Synthetic-only sets
Made-up examples miss your tools, policies, and edge cases.
Exports that rot
CSV dumps lack versioning, scope, and runtime retrieval design.
Fine-tune pressure
Pushing every fix into weights slows iteration and hides provenance.
The production-to-library loop
Connect traces → human review → approved experiences → runtime retrieval → measured outcome.
Ingest what you already have
OpenTelemetry preferred; conversation or observability exports work for a first sprint.
Correct with experts
Internal reviewers, trained annotators, or both—your rubric, your approvals.
Ship a usable library
Each record keeps context, failed output, rewrite, critique, and status.
A record needs enough context to be reused safely
The artifact is a governed experience record, not a row of text. The fields below make it possible to audit retrieval and avoid applying one customer’s exception to another workflow.
Source context
Trace ID, user intent, relevant tools, policy state, and the failed output or trajectory.
Approved correction
The replacement response or action path, reviewer critique, approval status, and any escalation note.
Retrieval controls
Workflow, tenant, policy, confidence, and date metadata that limits where an experience can be used.
See the commercial golden-datasets service and its dataset design approachFollow the practical build process step by step
Trace intake and transformation
Raw traces become gold only after normalization, review, and approval.
Accepted formats
OpenTelemetry spans, conversation exports, ticket threads, and observability dumps with stable identifiers.
Field mapping
Map request, tools, intermediate state, final output, outcome, and version fields into a reviewable record.
Eligibility checklist
Use the trace-readiness checklist to reject incomplete, unsafe, or unresolvable candidates.
Transformation path
Raw trace → normalized candidate → human-approved gold → scoped retrieval or evaluation set.
Incomplete traces
Missing tool results, conflicting state, or retired policies are quarantined—not forced into gold.
Reusable resource
Check whether your production traces are ready
A print-ready checklist for required fields, privacy handling, eligibility, and rejection criteria.
Illustrative template—not client data. Adapt it to your policies, privacy controls, and approval process.
Who this is for
- Teams with production traffic and repeating request patterns
- Orgs that already review or edit agent output
- Builders who want golden data aligned to live failures
- Leaders who need measurable lift before architecture changes
Sprint deliverables
A — Audit
Coverage estimate and failure clustering from your trace sample.
B — Dataset
Human-corrected golden set for one workflow plus retrieval prototype.
C — Prove
Baseline vs corrected comparison and next-step recommendation.
Why this converts better than generic data
Every example points back to a real failure your users hit—so retrieval stays relevant and auditable.
Limitations
- Incomplete traces cannot become approved gold without inventing missing evidence.
- Exports without version metadata make later auditing unreliable.
- PII minimization can remove context that reviewers still need; design exports carefully.
- A large unlabeled history is not a golden dataset.
Methodology by LTTAI LTD
This page describes less than three’s production-trace review method: reconstruct failures, capture human-approved corrections, store them outside model weights, and measure whether scoped retrieval improves a defined workflow.
Primary references used for terminology and risk framing:
Common questions
What does production traces → golden data include?
Connect traces → human review → approved experiences → runtime retrieval → measured outcome.
Do you fine-tune our model?
No. Approved corrections stay outside model weights and can be retrieved at runtime. Fine-tuning can still complement the system for stable, common behavior.
What do we need to start?
A sample of production traces or conversation exports for one workflow, plus a working definition of success for that workflow.
How is success measured?
We compare a fixed baseline against a changed agent on the same workflow using task success, edits, escalations, latency, and cost where available.
Related
Send traces. Get a golden-dataset plan.
Book a call with a sample of production traces—we will map repeated failures and one sprint path.