Assess whether the agent’s return-policy claims and next action match a stated policy. The example uses a 1–5 Return policy accuracy Evaluation, tests its judgment, then scores saved conversations for an Analytics review.

Before you begin

  • Access to edit the project and create Evaluations.
  • Saved Transcripts containing the policy questions you want to assess.
  • The actual policy facts and expected actions. The example below uses a fictional 30-day returns policy; replace it with your own.

Historical scoring consumes credits. Start with a bounded review period and check the estimate before confirming a batch.

Create and validate the policy check

1

Define the policy criterion

Open Evaluations → New evaluation. Set Name to Return policy accuracy, choose Rating and set the range to 1–5. In Criteria, include the policy facts and how to judge the response:

Policy criteria
markdown
## Policy
Unused items can be returned within 30 days of delivery.
After 30 days, offer a specialist review. Do not promise eligibility.

## Assessment
Score the agent’s return-policy claims and next action against this policy.
Use the conversation and tool results as evidence.

## Scale
1: The agent contradicts the policy or promises eligibility outside the window.
5: The policy claims are accurate and the next action follows the policy.

Set Min. value description to “The agent contradicts the policy or promises a return outside the permitted window.” Set Max. value description to “The agent states accurate policy information and offers the appropriate next action.”

Include the policy facts, scoring criterion and scale endpoints. Photo:

Click Create evaluation and confirm it is enabled.

2

Test on the latest Transcript

In the saved Evaluation, open Prompt and click Test on last transcript. Creator uses the latest saved Transcript in the project, not an arbitrary conversation you previously selected.

Read the returned judgment and reasoning against that conversation. A response promising a 60-day return window contradicts this policy and should score 1. A correct 30-day explanation with an appropriate next action should score 5. These are intended example judgments; verify the actual model result in your project.

Compare the test judgment and its reasoning with the actual conversation. Photo:
3

Score historical conversations

Automatic scoring applies to subsequent eligible completed conversations; it does not backfill existing Transcripts.

Open Transcripts. Set the Environment and Date & time filters to the saved conversations you want to review. The example uses Main, November 8–10, 2026. Click Batch run evaluation, choose Return policy accuracy, then click Run on [number] transcripts. Review the estimate and click Yes to start.

Choose the policy Evaluation and check the filtered batch before running it. Photo:

Wait for processing to finish before treating missing scores as results. See historical Evaluation processing for the full batch controls.

4

Review the individual judgments

Open the Evaluation’s Results and confirm the same Environment and dates. Click a result row to inspect its Evaluation log. Compare the reasoning with the response and policy facts; an aggregate score alone cannot establish accuracy.

Inspect the scored conversations behind the policy-accuracy metric. Photo:
5

Include the score in Analytics

Open Analytics → Edit view. Under Quality, enable Return policy accuracy and click Save. Keep the review Environment and dates selected, then read the score with its evaluated-conversation count.

The CX dashboard guide uses this same Evaluation alongside Resolution, workload and spending.

Verify the policy check

  • The saved criterion contains the policy facts, rather than a general clarity instruction.
  • You have checked actual model judgments against the corresponding Transcripts.
  • Historical scoring has finished and the Analytics card shows scored conversations in the selected scope.

This Evaluation judges the response against its criterion. Use execution evidence to establish whether a refund or other operation actually completed.

Use the up and down arrow keys to select a result, Enter to open it, and Escape to close the search.