Voiceflow vs Claude: where CX decisions go wrong

Voiceflow Flash 4.2 returned the correct support action and refund amount in 79 of 80 attempts, compared with 73 for Core 4.2, 51 for Claude Sonnet 4.6, and 32 for Claude Haiku 4.5. These results come from 40 synthetic customer experience (CX) cases, repeated twice per model inside Voiceflow.

Every model made at least one refund-amount error. For a CX team, that is the practical finding: choosing a model with a high score does not remove the need to validate the amount an agent proposes to refund.

What the comparison showed

A successful attempt required both the correct action and the exact amount in cents. Approving an eligible return with the wrong amount counted as unsuccessful. So did an output without a usable final action-and-amount line.

Each model saw the same 40 cases. The 80 attempts per model include repetitions; they are not 80 distinct scenarios. The last column counts cases answered correctly on both attempts.

Flash had the highest observed score under this shared configuration. That identifies a candidate for further evaluation on this task. The study does not establish which model is best for production CX overall.

Why this matters for CX agents

Policy decisions and refund arithmetic are different responsibilities. Core selected the correct action in 79/80 attempts, but returned the correct action and amount in 73/80. Sonnet selected the correct action in 68/80, while its fully correct decisions fell to 51/80.

Those gaps matter when a model's response drives an automated action. A plausible explanation for approving a return is insufficient if the proposed refund is wrong—even by one cent.

The evidence suggests three engineering responses to test:

• Calculate amounts in code. Let a verified calculation handle quantities, discounts, taxes, and rounding. Model selection should not be the only protection against arithmetic errors.
• Validate the proposed action against policy and account state. An amount check cannot catch an expired return being approved or an identity check being skipped.
• Reject unusable outputs before acting. Validate the action code and required fields, then retry or escalate when that validation fails.

These are recommendations from the observed failure types. This trial did not test the safeguards themselves or measure their effect on customer outcomes.

Harder cases reveal different weaknesses

The cases covered five challenges, with eight distinct cases in each. They included leap-year and holiday boundaries, partial returns, several item lines, defect exceptions, conflicting identity information, missing fields, pressure to override policy, and Spanish, French, and German inputs.

Every cell counts correct actions and exact amounts out of 16 attempts: eight cases repeated twice. A category can contain amount errors as well as action errors.

Refund calculations exposed weaknesses across all four models: Core returned 9/16 fully correct answers, Flash 15/16, Sonnet 5/16, and Haiku 4/16. Core and Flash answered all tested cases in the other four categories correctly on both repetitions; their remaining challenge in this suite was refund calculation and its required inputs.

Haiku's 0/16 result on restriction and defect cases requires care. It selected the correct action in 8/16 of those attempts. The fully correct score also requires the right amount; it does not mean every action decision was wrong.

The failures need different fixes

The breakdown separates an incorrect action from an incorrect amount after a correct action. Outputs without a recognized final action-and-amount line form a third class.

The counts are mutually exclusive. Output usability is checked first, then the action, then the amount. All failures remain in the aggregate scores.

Haiku had eight unusable final outputs. Some contained the correct decision earlier in the response but followed it with explanatory text. Under the prespecified final-line contract, that failed parsing; it is not evidence by itself that the model misunderstood the policy.

A one-cent error can change the result

The eight-item return case required a total of 67,885 cents ($678.85). Flash returned that amount on the first attempt and 67,886 cents on the second. Sonnet also returned 67,886 cents on its second attempt. The correct action with that one-cent difference was still an unsuccessful decision.

In the partial-return case, the policy required rounding the discounted item-line amount before adding tax. Two units at 3,799 cents, a 17.5% discount, and 220 cents of tax per unit give:

3,799 × 2 × 0.825 = 6,268.35 → 6,268 cents; plus 440 cents tax = 6,708 cents.

Core returned 6,696 cents on both attempts. Sonnet returned 6,709 and 6,934 cents; Flash and Haiku returned the expected 6,708 on both attempts. This is a concrete reason to test amount calculations separately from the action selected.

These illustrative cases were selected before the corrected comparison run. Their results do not replace the aggregate; all 40 cases are included in every overall score.

How the controlled trial worked

The agents applied a fixed policy for a fictional retailer, Meridian. The policy specified return windows, membership and holiday exceptions, defect precedence, gift credit, order lookup states, identity checks, and refund calculations in integer cents.

All four model variants ran in the dedicated Voiceflow Research workspace. Each trial started with fresh test state. The instruction text, cases, expected answers, configured temperature of 0, and 2,000-token output cap were shared. Variant order rotated between repetitions within each of four frozen batches. No knowledge base, retrieval, web search, or live tools were available.

The comparison used Voiceflow Core 4.2, Voiceflow Flash 4.2, Claude Sonnet 4.6, and Claude Haiku 4.5 as routed inside Voiceflow. Claude traces used the corresponding Bedrock model aliases. This was not a direct Anthropic API benchmark or a comparison of every Anthropic model. Provider reasoning defaults were not independently equalized, and model IDs do not pin exact weight snapshots.

Expected action-and-amount answers were mechanically derived from the policy. Approved refund amounts were crosschecked with independent integer arithmetic. Independent subject-matter-expert review of the labels is still pending.

Scoring and control checks

The last nonempty response line had to contain an allowed action code, a vertical bar, and an integer cent amount. Scoring ignored whitespace, surrounding bold or code-fence markup, and explanations before the final decision. It then compared both fields with the expected answer.

Native exact-string checks were retained as a separate metric. They require the entire response to match the expected string and therefore mix answer correctness with stricter formatting requirements; they are not a pure measure of formatting alone.

The final audit verified all 320 saved native run identities, complete case coverage, routed model identities, and the settings and stored instructions of all 16 comparison agents. No provider errors were recorded in the final comparison.

An earlier development run was excluded because Voiceflow's stored prompt changed multiplication symbols in the refund formula. The corrected run expressed the arithmetic in plain words and clarified that the output must use a concrete action code. That correction was applied uniformly and frozen before the rerun. All 16 stored prompts then matched each other exactly and matched the submitted instructions after whitespace normalization. Cases and expected answers were unchanged; earlier run history was retained.

What remains unproven

The 40 synthetic cases were used during instrument development. They are not unseen or held-out data, and the repetitions are not independent additions to the case set. Scores are descriptive observations, with no confidence intervals, significance tests, or population estimates.

The supplied order and account records were test inputs. There were no real tool calls, payments, refunds, retrieved documents, or customers. The study does not measure production resolution, conversational quality, dollar cost, or customer response latency.

A separate 96-trial multi-turn diagnostic was excluded from model ranking because its native judge was uncalibrated and a transcript appeared to disagree with its grade. That disagreement has not been independently adjudicated. Those results were not pooled into this comparison.

Before applying these findings to a live CX agent, the next test should use new cases, an independently reviewed policy oracle, and verifiable tool outcomes. Anthropic's evaluation guidance similarly distinguishes task grading from actual environmental outcomes and explains why model-based graders need human calibration. That guidance informs the next study; it does not validate these results.

The primary evidence is the frozen Voiceflow Research study, including native trial records and batch hashes. The final dataset manifest is eb83bb801f02005150ee7c92a760de699ff129e3360e7771fc5878ed0e6ae92f; development, smoke, and uncalibrated conversational runs are excluded from the reported 320 trials.