> ## Documentation Index
> Fetch the complete documentation index at: https://www.voiceflow.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Playback and responses

> Play agent audio, acknowledge finished playback and handle interrupted speech.

Enable `config.audioEvents` in [`client.start`](/docs/api-reference/conversations-api/interact-socket#client-start) and select an output `audioEncoding` that your playback pipeline supports. Input audio is configured separately in [Audio input and transcription](/docs/api-reference/agent-runtime/voice/audio-input).

## Receive agent output

Register listeners before sending an action. The outer `messageID` on `action.trace` identifies the action; an audio trace’s `payload.messageID` identifies its audio message.

```js title="Receive traces and action status"
socket.on('action.trace', ({ trace, messageID }) => {
  // Route audio traces to your playback queue, retaining the action messageID.
  // Dispatch other trace types to their corresponding interface handlers.
  console.log(messageID, trace);
});
socket.on('action.status', ({ status, messageID, reason }) => {
  // completed means processing finished, not that audio finished playing.
  console.log(messageID, status, reason);
});
```

Use [audio traces](/docs/api-reference/agent-runtime/voice/playback#audio-trace) for the exact start, content and end payloads. Decode the negotiated output format and play chunks in order. An action can finish generating before its queued audio finishes playing.

### Decode audio content

For an `audio` trace whose `payload.state` is `content`, `payload.content` contains base64-encoded audio. Decode it into bytes, then feed those bytes to a player for `payload.encoding` (`audio/mp3`, `audio/x-mulaw` or `audio/pcm`). It is not a URL or the raw binary input frame sent through `asr.audio.send`.

```js title="Decode one audio content trace in a browser"
function decodeAudioContent(trace) {
  if (trace.type !== 'audio' || trace.payload.state !== 'content') return;
  const bytes = Uint8Array.from(atob(trace.payload.content), char => char.charCodeAt(0));
  return { bytes, encoding: trace.payload.encoding };
}
```

This excerpt decodes a chunk; your playback queue still owns format-specific decoding, chunk order and the [playback acknowledgement](#acknowledge-playback-and-handle-interruptions).

## Acknowledge playback and handle interruptions

- **Playback finished:** after all audio for the action has played, emit `playback.idle` with the outer action `messageID`.
- **Speech detected:** `asr.interrupt` reports speech detection; it does not establish that a turn was cancelled.
- **Turn interrupted:** on `asr.barge-in`, stop playback and clear queued agent audio. The runtime emits this only when interruption is allowed in server-managed turns.

```js title="Clear interrupted output"
socket.on('asr.barge-in', () => {
  // Stop the current output and clear your audio queue.
});
// Call from your playback queue only when that action's audio is fully played:
function acknowledgePlayback(actionMessageID) {
  socket.emit('playback.idle', { messageID: actionMessageID });
}
```

For network interruptions, use the [connection recovery sequence](/docs/api-reference/conversations-api/interact-socket#reconnect-after-an-interruption). A reconnect does not confirm the previous action completed.

## Playback events

### `asr.interrupt`

**Server → client.** The recognizer detected speech. This event alone does not confirm that the agent’s running turn was interrupted.

Payload: `{}`.

### `asr.barge-in`

**Server → client.** With server-managed turns, the runtime interrupted the agent turn. Stop playback and clear queued agent audio. This occurs only when the runtime allows interruption.

Payload: `{}`.

### `playback.idle`

**Client → server.** Acknowledge that all audio for the named action messageID has finished playing. Use the outer action messageID, not an individual audio trace’s payload.messageID.

| Field | Type | Required | Default |
| --- | --- | --- | --- |
| `messageID` | string | yes | — |

## Trace payloads

Dispatch on `trace.type`. The definitions below cover spoken output, its text and conversation controls. Phone keypad collection requires your telephony client to implement the `dtmf` instruction; the socket does not collect phone digits itself. Call transfer is documented under [Human agent hand-off](/docs/api-reference/agent-runtime/voice/handoff).

<a id="messages-ui" />

## Response text

### `text` trace

Returned by the [Message step](/docs/documentation/build/steps/message), [Playbooks](/docs/documentation/build/steps/playbook), and no-match and no-reply reprompts.

```json
{
  "type": "text",
  "time": 1720552033,
  "payload": {
    "slate": {
      "id": "unique-id",
      "content": [
        {
          "children": [{ "text": "Hello there!" }]
        },
        {
          "children": [{ "text": "Select an option or ask me a question" }]
        }
      ],
      "messageDelayMilliseconds": 1000
    },
    "message": "Hello there!\n\nSelect an option or ask me a question",
    "messageID": "unique-id",
    "delay": 1000
  }
}
```

The `message` field contains the plain text content. The `slate` field contains the same content in a structured rich-text format. When present, `delay` gives the pause in milliseconds. Use the supplied value rather than assuming a fixed delay; Voice output does not inherit the project’s chat message delay.

#### text payload schema

Look up nested fields, conditional requirements and constraints.

This is the OpenAPI payload schema, not an example response. Referenced definitions are included under `components.schemas`.

```json title="OpenAPI payload schema"
{
  "type": "object",
  "properties": {
    "ai": {
      "type": "boolean"
    },
    "message": {
      "type": "string"
    },
    "ref": {
      "allOf": [
        {
          "$ref": "#/components/schemas/DebugTraceRef"
        }
      ]
    },
    "slate": {
      "type": "object",
      "properties": {
        "id": {
          "type": "string"
        },
        "content": {
          "type": "array",
          "items": {}
        }
      },
      "required": [
        "id",
        "content"
      ]
    },
    "delay": {
      "type": "number"
    },
    "voice": {
      "type": "string"
    },
    "audio": {
      "type": "object",
      "properties": {
        "src": {
          "type": "string"
        }
      },
      "required": [
        "src"
      ]
    },
    "messageID": {
      "type": "string"
    },
    "sourceUrls": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "url": {
            "type": "string"
          },
          "name": {
            "type": "string"
          }
        },
        "required": [
          "url"
        ]
      }
    }
  },
  "required": [
    "message",
    "slate",
    "messageID"
  ],
  "components": {
    "schemas": {
      "DebugTraceRef": {
        "oneOf": [
          {
            "$ref": "#/components/schemas/DebugTraceNodeRef"
          },
          {
            "$ref": "#/components/schemas/DebugTraceAgentRef"
          },
          {
            "$ref": "#/components/schemas/DebugTracePromptRef"
          },
          {
            "$ref": "#/components/schemas/DebugTraceAPIToolRef"
          },
          {
            "$ref": "#/components/schemas/DebugTraceFunctionRef"
          },
          {
            "$ref": "#/components/schemas/DebugTraceIntegrationToolRef"
          },
          {
            "$ref": "#/components/schemas/DebugTraceMcpIntegrationToolRef"
          }
        ]
      },
      "DebugTraceNodeRef": {
        "type": "object",
        "properties": {
          "type": {
            "type": "string",
            "enum": [
              "node"
            ],
            "x-enumNames": [
              "NODE"
            ]
          },
          "nodeID": {
            "type": "string"
          },
          "nodeType": {
            "type": "string"
          },
          "diagramID": {
            "type": "string"
          },
          "diagramName": {
            "type": "string"
          }
        },
        "required": [
          "type",
          "nodeID",
          "nodeType",
          "diagramID"
        ]
      },
      "DebugTraceAgentRef": {
        "type": "object",
        "properties": {
          "type": {
            "type": "string",
            "enum": [
              "agent"
            ],
            "x-enumNames": [
              "AGENT"
            ]
          },
          "nodeID": {
            "type": "string"
          },
          "agentID": {
            "type": "string"
          },
          "diagramID": {
            "type": "string"
          },
          "agentName": {
            "type": "string"
          },
          "diagramName": {
            "type": "string"
          }
        },
        "required": [
          "type",
          "nodeID",
          "agentID",
          "diagramID"
        ]
      },
      "DebugTracePromptRef": {
        "type": "object",
        "properties": {
          "type": {
            "type": "string",
            "enum": [
              "prompt"
            ],
            "x-enumNames": [
              "PROMPT"
            ]
          },
          "nodeID": {
            "type": "string"
          },
          "promptID": {
            "type": "string"
          },
          "diagramID": {
            "type": "string"
          },
          "diagramName": {
            "type": "string"
          }
        },
        "required": [
          "type",
          "nodeID",
          "promptID",
          "diagramID"
        ]
      },
      "DebugTraceAPIToolRef": {
        "type": "object",
        "properties": {
          "type": {
            "type": "string",
            "enum": [
              "api-tool"
            ],
            "x-enumNames": [
              "API_TOOL"
            ]
          },
          "nodeID": {
            "type": "string"
          },
          "diagramID": {
            "type": "string"
          },
          "apiToolID": {
            "type": "string"
          },
          "apiToolName": {
            "type": "string"
          },
          "diagramName": {
            "type": "string"
          }
        },
        "required": [
          "type",
          "nodeID",
          "diagramID",
          "apiToolID"
        ]
      },
      "DebugTraceFunctionRef": {
        "type": "object",
        "properties": {
          "type": {
            "type": "string",
            "enum": [
              "function"
            ],
            "x-enumNames": [
              "FUNCTION"
            ]
          },
          "nodeID": {
            "type": "string"
          },
          "diagramID": {
            "type": "string"
          },
          "functionID": {
            "type": "string"
          },
          "functionName": {
            "type": "string"
          },
          "diagramName": {
            "type": "string"
          }
        },
        "required": [
          "type",
          "nodeID",
          "diagramID",
          "functionID"
        ]
      },
      "DebugTraceIntegrationToolRef": {
        "type": "object",
        "properties": {
          "type": {
            "type": "string",
            "enum": [
              "integration-tool"
            ],
            "x-enumNames": [
              "INTEGRATION_TOOL"
            ]
          },
          "nodeID": {
            "type": "string"
          },
          "diagramID": {
            "type": "string"
          },
          "diagramName": {
            "type": "string"
          },
          "integrationToolName": {
            "type": "string"
          }
        },
        "required": [
          "type",
          "nodeID",
          "diagramID",
          "integrationToolName"
        ]
      },
      "DebugTraceMcpIntegrationToolRef": {
        "type": "object",
        "properties": {
          "type": {
            "type": "string",
            "enum": [
              "mcp-integration-tool"
            ],
            "x-enumNames": [
              "MCP_INTEGRATION_TOOL"
            ]
          },
          "nodeID": {
            "type": "string"
          },
          "diagramID": {
            "type": "string"
          },
          "mcpToolName": {
            "type": "string"
          },
          "diagramName": {
            "type": "string"
          }
        },
        "required": [
          "type",
          "nodeID",
          "diagramID",
          "mcpToolName"
        ]
      }
    }
  }
}
```

<a id="voice-telephony" />

## Audio and phone input

### `audio` trace

An audio response event. The payload's `state` field selects the shape: `start` announces a `messageID`, `content` carries base64-encoded audio in `content` with its `encoding`, and `end` carries the state alone.

| Field | Type | Required | Present when |
| --- | --- | --- | --- |
| `content` | string | no | `state` is `content` |
| `delay` | number | no | `state` is `start` |
| `duration` | number | no | `state` is `content` |
| `encoding` | `"audio/mp3"` or `"audio/x-mulaw"` or `"audio/pcm"` | no | `state` is `content` |
| `messageID` | string | no | `state` is `start` |
| `state` | `"start"` or `"content"` or `"end"` | yes | always |

“Required” above means required in every payload shape. A field can be required for a particular state or event even when it is not required in every shape; the schema below defines those conditions.

#### audio payload schema

Look up nested fields, conditional requirements and constraints.

This is the OpenAPI payload schema, not an example response. Referenced definitions are included under `components.schemas`.

```json title="OpenAPI payload schema"
{
  "anyOf": [
    {
      "type": "object",
      "properties": {
        "state": {
          "type": "string",
          "enum": [
            "start"
          ],
          "x-enumNames": [
            "START"
          ]
        },
        "messageID": {
          "type": "string"
        },
        "delay": {
          "type": "number"
        }
      },
      "required": [
        "state",
        "messageID"
      ]
    },
    {
      "type": "object",
      "properties": {
        "state": {
          "type": "string",
          "enum": [
            "content"
          ],
          "x-enumNames": [
            "CONTENT"
          ]
        },
        "content": {
          "type": "string"
        },
        "encoding": {
          "$ref": "#/components/schemas/AudioEncoding"
        },
        "duration": {
          "type": "number"
        }
      },
      "required": [
        "state",
        "content",
        "encoding"
      ]
    },
    {
      "type": "object",
      "properties": {
        "state": {
          "type": "string",
          "enum": [
            "end"
          ],
          "x-enumNames": [
            "END"
          ]
        }
      },
      "required": [
        "state"
      ]
    }
  ],
  "components": {
    "schemas": {
      "AudioEncoding": {
        "type": "string",
        "enum": [
          "audio/mp3",
          "audio/x-mulaw",
          "audio/pcm"
        ],
        "x-enumNames": [
          "MP_3",
          "MULAW",
          "PCM"
        ]
      }
    }
  }
}
```

### `speak` trace

A spoken response for voice channels, carrying the message text and optionally a synthesized audio source and voice name.

| Field | Type | Required |
| --- | --- | --- |
| `ai` | boolean | no |
| `isPrompt` | boolean | no |
| `message` | string | yes |
| `src` | string | no |
| `type` | `"audio"` or `"message"` | yes |
| `voice` | string | no |

#### speak payload schema

Look up nested fields, conditional requirements and constraints.

This is the OpenAPI payload schema, not an example response. Referenced definitions are included under `components.schemas`.

```json title="OpenAPI payload schema"
{
  "type": "object",
  "properties": {
    "ai": {
      "type": "boolean"
    },
    "message": {
      "type": "string"
    },
    "type": {
      "$ref": "#/components/schemas/TraceSpeakType"
    },
    "src": {
      "nullable": true,
      "type": "string"
    },
    "voice": {
      "type": "string"
    },
    "isPrompt": {
      "type": "boolean"
    }
  },
  "required": [
    "message",
    "type"
  ],
  "components": {
    "schemas": {
      "TraceSpeakType": {
        "type": "string",
        "enum": [
          "audio",
          "message"
        ],
        "x-enumNames": [
          "AUDIO",
          "MESSAGE"
        ]
      }
    }
  }
}
```

### `stream` trace

An audio stream event. The payload carries the source in `src`, a `token`, and an `action` of `LOOP`, `PLAY`, `PAUSE`, or `END`; `title`, `description`, `iconImage`, `backgroundImage`, and `loop` are optional.

| Field | Type | Required |
| --- | --- | --- |
| `action` | `"LOOP"` or `"PLAY"` or `"PAUSE"` or `"END"` | yes |
| `backgroundImage` | string | no |
| `description` | string | no |
| `iconImage` | string | no |
| `loop` | boolean | no |
| `src` | string | yes |
| `title` | string | no |
| `token` | string | yes |

#### stream payload schema

Look up nested fields, conditional requirements and constraints.

This is the OpenAPI payload schema, not an example response. Referenced definitions are included under `components.schemas`.

```json title="OpenAPI payload schema"
{
  "type": "object",
  "properties": {
    "src": {
      "type": "string"
    },
    "token": {
      "type": "string"
    },
    "action": {
      "$ref": "#/components/schemas/TraceStreamAction"
    },
    "loop": {
      "type": "boolean"
    },
    "title": {
      "type": "string"
    },
    "iconImage": {
      "type": "string"
    },
    "description": {
      "type": "string"
    },
    "backgroundImage": {
      "type": "string"
    }
  },
  "required": [
    "src",
    "token",
    "action"
  ],
  "components": {
    "schemas": {
      "TraceStreamAction": {
        "type": "string",
        "enum": [
          "LOOP",
          "PLAY",
          "PAUSE",
          "END"
        ],
        "x-enumNames": [
          "LOOP",
          "PLAY",
          "PAUSE",
          "END"
        ]
      }
    }
  }
}
```

### `dtmf` trace

Configures keypad (DTMF) input collection on a call: whether it is enabled, the timeout, the ending delimiter, and the digit cap.

| Field | Type | Required |
| --- | --- | --- |
| `delimiter` | (`"#"` or `"*"`)[] | yes |
| `enabled` | boolean | yes |
| `maxDigits` | number | yes |
| `timeoutInSeconds` | number | yes |

#### dtmf payload schema

Look up nested fields, conditional requirements and constraints.

This is the OpenAPI payload schema, not an example response. Referenced definitions are included under `components.schemas`.

```json title="OpenAPI payload schema"
{
  "$ref": "#/components/schemas/DTMFTracePayload",
  "components": {
    "schemas": {
      "DTMFTracePayload": {
        "type": "object",
        "properties": {
          "enabled": {
            "type": "boolean",
            "description": "Enables the new DTMF collection and configuration"
          },
          "timeoutInSeconds": {
            "type": "number",
            "minimum": 0,
            "maximum": 10,
            "description": "How long to wait before processing the DTMF input. Set to 0 to process only after a delimiter is provided."
          },
          "delimiter": {
            "type": "array",
            "items": {
              "$ref": "#/components/schemas/DTMFDelimiter"
            },
            "description": "Which key to use to end DTMF input. Defaults to '#'"
          },
          "maxDigits": {
            "type": "number",
            "minimum": 1,
            "maximum": 20,
            "description": "The maximum number of digits before we finish their input"
          }
        },
        "required": [
          "enabled",
          "timeoutInSeconds",
          "delimiter",
          "maxDigits"
        ]
      },
      "DTMFDelimiter": {
        "type": "string",
        "enum": [
          "#",
          "*"
        ],
        "x-enumNames": [
          "#",
          "*"
        ]
      }
    }
  }
}
```

<a id="conversation-control" />

## Conversation control

### `no-reply` trace

Returned when a No Reply timeout is active. The `timeout` value is in seconds.

```json
{
  "type": "no-reply",
  "time": 1720552033,
  "payload": {
    "timeout": 10
  }
}
```

If the user doesn't respond within the timeout window, send a `no-reply` action to retrieve the configured reprompt:

```json
{
  "action": {
    "type": "no-reply"
  }
}
```

#### no-reply payload schema

Look up nested fields, conditional requirements and constraints.

This is the OpenAPI payload schema, not an example response. Referenced definitions are included under `components.schemas`.

```json title="OpenAPI payload schema"
{
  "type": "object",
  "properties": {
    "timeout": {
      "type": "number"
    }
  },
  "required": [
    "timeout"
  ]
}
```

### `end` trace

Returned when the conversation reaches an End step. On receiving this trace, your application should treat the session as closed.

```json
{
  "type": "end",
  "time": 1720552033,
  "payload": null
}
```

#### end payload schema

Look up nested fields, conditional requirements and constraints.

This is the OpenAPI payload schema, not an example response. Referenced definitions are included under `components.schemas`.

```json title="OpenAPI payload schema"
{}
```

### `goto` trace

Instructs the client to re-send the wrapped request on the next turn.

| Field | Type | Required |
| --- | --- | --- |
| `request` | BaseRequest | yes |

#### goto payload schema

Look up nested fields, conditional requirements and constraints.

This is the OpenAPI payload schema, not an example response. Referenced definitions are included under `components.schemas`.

```json title="OpenAPI payload schema"
{
  "type": "object",
  "properties": {
    "request": {
      "$ref": "#/components/schemas/BaseRequest"
    }
  },
  "required": [
    "request"
  ],
  "components": {
    "schemas": {
      "BaseRequest": {
        "type": "object",
        "properties": {
          "type": {
            "type": "string"
          },
          "payload": {},
          "diagramID": {
            "type": "string"
          },
          "time": {
            "type": "number"
          },
          "metadata": {
            "type": "object",
            "additionalProperties": {}
          }
        },
        "required": [
          "type"
        ]
      }
    }
  }
}
```

### `entity-filling` trace

Requests a value for a named entity, wrapping the intent request being filled.

| Field | Type | Required |
| --- | --- | --- |
| `entityToFill` | string | yes |
| `intent` | IntentRequest | yes |

#### entity-filling payload schema

Look up nested fields, conditional requirements and constraints.

This is the OpenAPI payload schema, not an example response. Referenced definitions are included under `components.schemas`.

```json title="OpenAPI payload schema"
{
  "type": "object",
  "properties": {
    "entityToFill": {
      "type": "string"
    },
    "intent": {
      "$ref": "#/components/schemas/IntentRequest"
    }
  },
  "required": [
    "entityToFill",
    "intent"
  ],
  "components": {
    "schemas": {
      "IntentRequest": {
        "type": "object",
        "properties": {
          "type": {
            "type": "string",
            "enum": [
              "intent"
            ],
            "x-enumNames": [
              "INTENT"
            ]
          },
          "payload": {
            "$ref": "#/components/schemas/IntentRequestPayload"
          },
          "diagramID": {
            "type": "string"
          },
          "time": {
            "type": "number"
          },
          "metadata": {
            "type": "object",
            "additionalProperties": {}
          }
        },
        "required": [
          "type",
          "payload"
        ],
        "description": "Trigger a specific intent from your conversation model."
      },
      "IntentRequestPayload": {
        "type": "object",
        "properties": {
          "actions": {
            "type": "array",
            "items": {
              "$ref": "#/components/schemas/BaseAction"
            }
          },
          "label": {
            "type": "string"
          },
          "data": {
            "type": "object",
            "additionalProperties": {}
          },
          "query": {
            "default": "",
            "type": "string"
          },
          "intent": {
            "type": "object",
            "properties": {
              "name": {
                "type": "string"
              }
            },
            "required": [
              "name"
            ],
            "additionalProperties": {}
          },
          "entities": {
            "default": [],
            "type": "array",
            "items": {
              "$ref": "#/components/schemas/IntentRequestEntity"
            }
          },
          "confidence": {
            "type": "number"
          }
        },
        "required": [
          "intent"
        ]
      },
      "BaseAction": {
        "type": "object",
        "properties": {
          "type": {
            "type": "string"
          },
          "payload": {}
        },
        "required": [
          "type",
          "payload"
        ]
      },
      "IntentRequestEntity": {
        "type": "object",
        "properties": {
          "name": {
            "type": "string"
          },
          "value": {
            "type": "string"
          },
          "query": {
            "type": "string"
          },
          "verboseValue": {
            "type": "array",
            "items": {
              "$ref": "#/components/schemas/VerboseValue"
            }
          }
        },
        "required": [
          "name",
          "value"
        ]
      },
      "VerboseValue": {
        "type": "object",
        "properties": {
          "rawText": {
            "type": "string"
          },
          "canonicalText": {
            "type": "string"
          },
          "startIndex": {
            "type": "number"
          }
        },
        "required": [
          "rawText",
          "canonicalText",
          "startIndex"
        ]
      }
    }
  }
}
```

### `channel-action` trace

A channel-specific action, named by the channel integration, with an action-defined payload.

| Field | Type | Required |
| --- | --- | --- |
| `name` | string | yes |
| `payload` | object | yes |

#### channel-action payload schema

Look up nested fields, conditional requirements and constraints.

This is the OpenAPI payload schema, not an example response. Referenced definitions are included under `components.schemas`.

```json title="OpenAPI payload schema"
{
  "type": "object",
  "properties": {
    "name": {
      "type": "string"
    },
    "payload": {
      "type": "object",
      "additionalProperties": {}
    }
  },
  "required": [
    "name",
    "payload"
  ]
}
```
