Follow one low Evaluation score into the customer exchange and execution logs. The worked example traces an incorrect 60-day return promise against a 30-day policy and an ineligible order lookup.
Before you begin
- A scored Evaluation and access to its saved Transcript.
- The expected policy or behavior. The example uses the Return policy accuracy criterion.
Trace the result to its evidence
Find a low-scoring conversation
Open Return policy accuracy from Analytics. Confirm the Environment and dates in Results. Click the table’s Result heading to put low scores first, then choose a 1 result.
Read the Evaluation log
Click the result row to open its Evaluation log. Read the judgment and reasoning alongside the conversation. For this case, the agent promised a 60-day window although the policy permits 30 days.
Use the Prompt tab when you need to check the exact criterion. Return to Results and reopen the same row before continuing.
Inspect the same Transcript and lookup
Click View full transcript. Confirm that it contains the same customer request and response as the Evaluation log. Click Show logs (the bug icon), then expand Look up order to inspect its returned data.
In this example, order VF-4821 arrived 35 days earlier. The lookup returned a 30-day return window and return_eligible: false, but the response said the order was eligible within 60 days. The failure is in the agent’s response; the lookup supplied the correct restriction.
Record a reproducible finding
Record the conversation identifier, time, Environment, expected behavior and observed response. Include the relevant log result.
For the example: “The agent promised eligibility after 35 days using a 60-day window. The policy and order lookup specify 30 days. The expected response is to explain the restriction and offer a specialist review.”
Verify the investigation
- The finding cites the selected conversation and its execution evidence.
- The policy criterion and lookup support the finding; it is not inferred from the aggregate score.
- The evidence identifies which response or operation needs correction.
Act on the finding
- For response guidance, update the owning Instructions or Playbook.
- For incorrect tool data, investigate the tool’s input and returned result.
- For a correct response that was penalized, refine and retest the Evaluation criteria.
- Preserve a confirmed agent failure as a Test before making the correction.
If the conversation is missing
Recheck the Environment and dates, then review Transcript saving settings. Enabling saving now cannot recreate a conversation that was not saved.