HeyGen Interactive Avatar Realtime API

Summary

HeyGen’s Interactive Avatar Realtime API enables you to build a low latency, visual conversation experience for your customers. You bring your speech to speech stack (we’re starting with support for pipecat), we’ll provide a human like visual agent and end user video delivery.

Architecture Diagram

Sequence Diagram.

Getting Started

• NextJS Demo
  • https://github.com/HeyGen-Official/InteractiveAvatarNextJSDemo/tree/realtime-alpha-demo
• Pipecat Demo
  • https://github.com/HeyGen-Official/pipecat-realtime-demo

Endpoint

Interactive Avatar Realtime API is a stateful, server-side, event-based API that communicates over a WebSocket. The websocket address and port will be provided when The WebSocket connection requires the following parameters:

• URL: wss://{heygen-server}:{port}/v1-alpha/realtime/{session_id}

The server ip and port will be assigned at the start of each new session and returned as part of the /v1/streaming.new api call in a field called: realtime_endoint

Event Base Body:

All events will be json and have these base fields:

{
	"type": "<event_type>",
	"event_id": "<event_id>"
}

Client Events (events you send):

• agent.audio_buffer_append

{
	"type": "agent.audio_buffer_append",
	"event_id": "<event_id>",
	"audio": "{Send base64 encoded PCM 16bit 24khz audio segments}"
}
# to optimize inferencing speeds, we suggest limiting segments to 1-4 seconds

• agent.audio_buffer_commit

{
	"type": "agent.audio_buffer_commit",
	"event_id": "<event_id>",
	"audio": "{Send base64 encoded PCM 16bit 24khz audio segments}"
}

• agent.audio_buffer_clear

{
	"type": "agent.audio_buffer_clear",
	"event_id": "<event_id>",
}

• agent.interrupt - stops agent’s current task to return back to silent animation

{
	"type": "agent.interrupt",
	"event_id": "<event_id>",
}

• agent.start_listening - starts listening animation, only possible if agent is already silent

{
	"type": "agent.start_listening",
	"event_id": "<event_id>",
}

• agent.stop_listening - ends listening animation, only possible if agent is in listening animation

{
	"type": "agent.end_listening",
	"event_id": "<event_id>",
}

This API is under continuous development for easier integration and better performance.
If any feedback please share with us!