> ## Documentation Index
> Fetch the complete documentation index at: https://www.voiceflow.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluate assistance quality

> Evaluate Voiceflow agent assist with case-grounded criteria, staff feedback, latency and access-boundary checks.

Evaluate the answer the human support agent actually receives, including case access and delivery. Begin with a configured staff deployment, an identified Voiceflow release and a reviewer who knows the policies. This guide defines a practical evaluation process; it does not promise a productivity gain.

## Run an evidence-based pilot

### 1. Create a representative case set

Select anonymized or fictional cases covering supported policies, exceptions, missing facts, source changes, different staff entitlements and unsupported requests. Include case text that tries to override instructions. For summaries, include unresolved promises and actions that were only discussed.

Record expected policy IDs, required conditions, permitted facts and the correct abstention/escalation. Keep authoring cases separate from a held-out review set. Sample by case type and staff experience when these affect the work; choose the sample plan with the policy owner. This prioritization is editorial judgment, not a claim about which CX job is most common.

### 2. Define the decision before testing

| Criterion | Evidence needed |
| --- | --- |
| Policy correctness | The applicable current source and all required conditions support the answer. |
| Attribution | Claimed source IDs/URLs exist, are accessible to the intended staff and support the specific claim. |
| Case faithfulness | Every customer fact appears in the authorized snapshot; no other case leaks in. |
| Useful next step | The guidance identifies the missing fact or permitted action without inventing completion. |
| Human control | Staff can edit, reject or ignore the draft; no send/write happens automatically. |
| Delivery and latency | A complete result reaches the taught staff interface within the team’s wait budget; failures clear partial output. |

Choose release thresholds and stop conditions with the support lead before looking at results. Treat access leaks and unauthorized writes as stop conditions. Do not use a fluency score to certify policy correctness. Measure p50/p95 time to completed draft separately from network time; the starter’s 20-second upstream deadline is not a response-time guarantee.

### 3. Exercise the deployed workflow

Run each case through the actual installed staff entry using permitted and denied staff accounts. Compare the visible answer with the expected evidence. Include switching cases mid-stream, logout/account change, expired session, stale source, case-service failure and interrupted delivery. Verify no runtime request occurs after an authorization denial.

Use [Transcripts](/docs/documentation/measure/transcripts) and [Evaluations](/docs/documentation/measure/evaluations) for supported Voiceflow inspection, while retaining separate host/platform delivery and access evidence. Each question creates a fresh runtime conversation, so do not assume a transcript is a complete provider interaction. The local fixture server returns fixed text and cannot establish answer quality.

### 4. Collect useful staff feedback

During a controlled pilot, record accepted unchanged, edited, rejected, no-answer and failed requests with the case type and agent release. Ask staff for a reason such as wrong policy, stale source, missing fact, unhelpful next step or excessive wait. Review a sample of accepted drafts too; acceptance alone is not accuracy.

The supplied panel has no feedback persistence endpoint. Collect pilot feedback in your approved QA tracker. If adding an in-product feedback control, authenticate it, bind it to the completed request/case/release, limit free-text retention and keep raw customer content out of analytics by default.

### 5. Make and record the release decision

Have the policy reviewer assess the held-out cases and unresolved failures. Record the staff-host build, platform configuration, source versions, Voiceflow Environment/release, sample composition and acceptance decision. Expand only after the promised route and permissions pass; retain a rollback owner and a way for staff to continue without assistance.

For any productivity comparison, use comparable cases and staff groups, record the observation window and account for learning and case-mix changes. Research on support assistants shows context-dependent effects; it does not supply a Voiceflow forecast. The original [Generative AI at Work study](https://www.nber.org/papers/w31161) is useful background, not a benchmark for this deployment.

### Averages look good but some cases fail

Inspect consequential slices and separate content failures from delivery failures.

Break results down by policy, missing-context cases, staff entitlement, experience and source version where the sample supports it. Investigate a leak or false action claim even if most drafts were accepted. Label small or selected samples and reviewer disagreement. Fix the owning policy, retrieval, prompt, host or platform path, then rerun the affected cases and the retained regression set.

## Repeat after meaningful changes

Re-evaluate after prompt, model, Knowledge, case-field, authorization, SDK or platform-configuration changes. Keep feedback and failure recovery understandable to staff. The [human–AI interaction research guidelines](https://www.microsoft.com/en-us/research/project/guidelines-for-human-ai-interaction/overview/) inform that editorial recommendation; they are not a certification of this application.
