Enable config.audioEvents in client.start and select an output audioEncoding that your playback pipeline supports. Input audio is configured separately in Audio input and transcription.
Receive agent output
Register listeners before sending an action. The outer messageID on action.trace identifies the action; an audio trace’s payload.messageID identifies its audio message.
Use audio traces for the exact start, content and end payloads. Decode the negotiated output format and play chunks in order. An action can finish generating before its queued audio finishes playing.
Decode audio content
For an audio trace whose payload.state is content, payload.content contains base64-encoded audio. Decode it into bytes, then feed those bytes to a player for payload.encoding (audio/mp3, audio/x-mulaw or audio/pcm). It is not a URL or the raw binary input frame sent through asr.audio.send.
This excerpt decodes a chunk; your playback queue still owns format-specific decoding, chunk order and the playback acknowledgement.
Acknowledge playback and handle interruptions
- Playback finished: after all audio for the action has played, emit
playback.idlewith the outer actionmessageID. - Speech detected:
asr.interruptreports speech detection; it does not establish that a turn was cancelled. - Turn interrupted: on
asr.barge-in, stop playback and clear queued agent audio. The runtime emits this only when interruption is allowed in server-managed turns.
For network interruptions, use the connection recovery sequence. A reconnect does not confirm the previous action completed.
Playback events
asr.interrupt
Server → client. The recognizer detected speech. This event alone does not confirm that the agent’s running turn was interrupted.
Payload: {}.
asr.barge-in
Server → client. With server-managed turns, the runtime interrupted the agent turn. Stop playback and clear queued agent audio. This occurs only when the runtime allows interruption.
Payload: {}.
playback.idle
Client → server. Acknowledge that all audio for the named action messageID has finished playing. Use the outer action messageID, not an individual audio trace’s payload.messageID.
Trace payloads
Dispatch on trace.type. The definitions below cover spoken output, its text and conversation controls. Phone keypad collection requires your telephony client to implement the dtmf instruction; the socket does not collect phone digits itself. Call transfer is documented under Human agent hand-off.
Response text
text trace
Returned by the Message step, Playbooks, and no-match and no-reply reprompts.
The message field contains the plain text content. The slate field contains the same content in a structured rich-text format. When present, delay gives the pause in milliseconds. Use the supplied value rather than assuming a fixed delay; Voice output does not inherit the project’s chat message delay.
text payload schema
Look up nested fields, conditional requirements and constraints.
This is the OpenAPI payload schema, not an example response. Referenced definitions are included under components.schemas.
Audio and phone input
audio trace
An audio response event. The payload’s state field selects the shape: start announces a messageID, content carries base64-encoded audio in content with its encoding, and end carries the state alone.
“Required” above means required in every payload shape. A field can be required for a particular state or event even when it is not required in every shape; the schema below defines those conditions.
audio payload schema
Look up nested fields, conditional requirements and constraints.
This is the OpenAPI payload schema, not an example response. Referenced definitions are included under components.schemas.
speak trace
A spoken response for voice channels, carrying the message text and optionally a synthesized audio source and voice name.
speak payload schema
Look up nested fields, conditional requirements and constraints.
This is the OpenAPI payload schema, not an example response. Referenced definitions are included under components.schemas.
stream trace
An audio stream event. The payload carries the source in src, a token, and an action of LOOP, PLAY, PAUSE, or END; title, description, iconImage, backgroundImage, and loop are optional.
stream payload schema
Look up nested fields, conditional requirements and constraints.
This is the OpenAPI payload schema, not an example response. Referenced definitions are included under components.schemas.
dtmf trace
Configures keypad (DTMF) input collection on a call: whether it is enabled, the timeout, the ending delimiter, and the digit cap.
dtmf payload schema
Look up nested fields, conditional requirements and constraints.
This is the OpenAPI payload schema, not an example response. Referenced definitions are included under components.schemas.
Conversation control
no-reply trace
Returned when a No Reply timeout is active. The timeout value is in seconds.
If the user doesn’t respond within the timeout window, send a no-reply action to retrieve the configured reprompt:
no-reply payload schema
Look up nested fields, conditional requirements and constraints.
This is the OpenAPI payload schema, not an example response. Referenced definitions are included under components.schemas.
end trace
Returned when the conversation reaches an End step. On receiving this trace, your application should treat the session as closed.
end payload schema
Look up nested fields, conditional requirements and constraints.
This is the OpenAPI payload schema, not an example response. Referenced definitions are included under components.schemas.
goto trace
Instructs the client to re-send the wrapped request on the next turn.
goto payload schema
Look up nested fields, conditional requirements and constraints.
This is the OpenAPI payload schema, not an example response. Referenced definitions are included under components.schemas.
entity-filling trace
Requests a value for a named entity, wrapping the intent request being filled.
entity-filling payload schema
Look up nested fields, conditional requirements and constraints.
This is the OpenAPI payload schema, not an example response. Referenced definitions are included under components.schemas.
channel-action trace
A channel-specific action, named by the channel integration, with an action-defined payload.
channel-action payload schema
Look up nested fields, conditional requirements and constraints.
This is the OpenAPI payload schema, not an example response. Referenced definitions are included under components.schemas.