Overview

An Evaluation uses an AI model to assess a Transcript against criteria you define. It returns a result, such as a rating or category, with reasoning that explains the judgment. Open Evaluations in the project sidebar to configure criteria and review conversation quality.

Manage the criteria used to evaluate conversations. Photo:

Examples of when to use Evaluations

Use an Evaluation to apply a consistent review question across conversations. You can:

  • Measure resolution: assess whether the customer received a useful answer or completed action.
  • Check policy explanations: rate whether the agent clearly explained a return window and the available next step.
  • Categorize outcomes: distinguish resolved requests, successful escalations and conversations that ended too early.
  • Review an existing sample: apply new criteria to historical Transcripts before deciding how to use the results.

Enabled Evaluations can run automatically on new Transcripts. Existing conversations need a batch run; enabling an Evaluation does not automatically backfill them. Review individual judgments against their Transcripts before using aggregate scores to assess a release.

Each Evaluation has a name, metric type, criteria and AI settings. Its table shows the configured Evaluations, evaluated-log counts and average credit use. The enable toggle controls future automatic runs.

How to use Evaluations

1

Open a new Evaluation

Open Evaluations, then click New evaluation. Start with one clear review question.

Manage the criteria used to evaluate conversations. Photo:
2

Define the metric and criteria

Give the Evaluation a descriptive Name, choose its Metric, and write Criteria that explain how to judge the Transcript. For a Rating, define the range and describe what its endpoints mean.

Define the criterion and describe both ends of the Rating scale. Photo:
3

Review before enabling

Use Test on last transcript to check the criteria against an existing conversation. Compare the result and reasoning with the Transcript, refine the criteria if needed, then click Create evaluation. Check its enabled state in the list.

Define the criterion and describe both ends of the Rating scale. Photo:

The populated form is an illustrative Return policy clarity Rating. The screenshots show configuration, not evidence of a completed model run. Verify your own result before relying on the scores.

Defining criteria

Choose the question, output format and rules used to judge a Transcript.

Click New evaluation and give it a name that identifies the question it answers. Choose a Metric that fits the decision, then write Criteria specific enough that someone reviewing the same Transcript could understand the intended judgment.

Define the criterion and describe both ends of the Rating scale. Photo:

Use Test on last transcript before saving, and read the model’s reasoning alongside the actual conversation. After creation, review the Evaluation’s model and generation controls in its AI settings. Model choice affects judgments and cost; test a change against representative conversations rather than assuming two models will agree.

Return policy clarity criteria

Adapt an illustrative Rating with clear endpoints and review cases.

This example assumes the agent handles retail returns with a documented return window and an exception-review path. It assesses the explanation, not whether the customer is eligible for a refund.

Evaluation criteria
markdown
Score how clearly the agent explained the return window
and the next available action.
Use only evidence from the conversation.

Minimum (1): The agent gives no usable explanation of
 the return policy or next step.
Maximum (5): The agent clearly explains the return window
 and offers an appropriate next action.

Review a clear in-window explanation, an out-of-window request with a useful exception path, and a reply that states a rule without a next action. Compare the results with a human review. These are review cases for the sample criteria, not verified scores or a built-in Evaluation.

Criteria best practices

Make scores interpretable and test the boundaries between outcomes.
  • Ask one coherent question. Combining unrelated goals in one rating makes the score difficult to act on.
  • Define boundaries. Describe what distinguishes neighboring ratings or categories, including missing evidence.
  • Judge the Transcript’s evidence. Do not require information that was never available in the conversation.
  • Review contrasting cases. Compare clear successes, clear failures and ambiguous cases before relying on the aggregate.
  • Record criteria changes. When changing a metric or prompt, account for that change before comparing old and new scores.

Reviewing results

Inspect aggregate scores and the reasoning behind individual judgments.

Open an Evaluation to see its Results. The detail view combines summary information, a chart where applicable, and individual scored Transcripts. Use filters to choose the period, Environment and other available conversation attributes.

Open an Evaluation to investigate the conversations behind its aggregate score. Photo:

Open a result to read the model’s reasoning, then compare it with the conversation. The Prompt tab lets you inspect the criteria that explain the result. Read the evaluated-log count and average credit use alongside the score; a small or filtered sample can differ from overall traffic.

Results also appear in Transcripts, where available, and in Analytics. A score is a model judgment under the configured criteria, not independent confirmation that the underlying task succeeded.

Built-in Evaluations

Start with conversation-quality criteria and review their definitions.

Voiceflow includes Evaluations for Customer satisfaction, Deflection rate and Resolution rate. Review their prompts before treating the result as a business metric. Customer satisfaction uses a 1–5 assessment of the conversation; Deflection assesses self-service without human intervention; Resolution classifies the conversation’s outcome.

Manage the criteria used to evaluate conversations. Photo:

You can edit criteria or create a custom Evaluation when your review question differs. The capture shows one configured Resolution rate Evaluation in an illustrative project; the list depends on your project’s configuration.

Resolution categories

The Resolution criteria distinguish these five outcomes:

A customer leaving after a usable answer is Resolved, not Abandoned. A correctly declined out-of-scope request is Unrelated, not Abandoned. Buttons, Cards and Carousels count as agent responses: leaving after clarifying buttons without selecting one can be Abandoned rather than Unresolved.

Check the Evaluation’s Prompt for the exact criteria in your project. Customizing those criteria can change the interpretation of a result.

Historical Evaluations

Apply selected criteria to conversations that already exist.

Enabling an Evaluation applies it to new Transcripts. To score older conversations, open Transcripts and click Batch run evaluation. Choose the Evaluations to apply and set filters for the conversations to include, then review the scope before clicking Run.

Choose Evaluations and filters before running them on historical Transcripts. Photo:

Use a historical sample when validating a new criterion or reassessing conversations after an edit. Allow processing to finish before reading the aggregate, and inspect individual results to check how the updated criteria behaved. A batch run consumes credits.

Keep the criteria and sample consistent when comparing results. If you change both, the difference cannot be attributed to the criteria alone.

Evaluation credits

Understand the additional cost of scoring conversations.

Evaluations consume credits because an AI model analyzes the Transcript. Cost depends on the chosen model and the conversation’s length. Running several Evaluations or backfilling many historical conversations adds scoring work beyond the agent’s original conversation.

Manage the criteria used to evaluate conversations. Photo:

Review Avg. credits in the Evaluations table and the cost information in an Evaluation’s results. Use a representative sample to assess both usefulness and cost before extending a new criterion to a large historical set. An average is a review aid, not a fixed price for every Transcript.

Planning Evaluation coverage

Decide which questions justify automatic or historical scoring.

Keep automatically enabled Evaluations focused on decisions your team will use. Consider the length and variety of Transcripts when assessing a sample’s average cost, and check the batch filters before starting a backfill. Review model changes for both judgment quality and credit impact; a lower-cost model is useful only if its scores remain suitable for the review.

Use the up and down arrow keys to select a result, Enter to open it, and Escape to close the search.