Streaming Avatar SDK API Reference

The @heygen/streaming-avatar package provides a TypeScript SDK for interacting with HeyGen's streaming avatar service. For detailed information about the available methods, interfaces, enums, and event handling, refer to the API reference provided below.

Classes

StreamingAvatar

This class is the core of the SDK, responsible for managing avatar streaming sessions, including session creation, controlling avatars, and handling real-time communication.

const avatar = new StreamingAvatar({ token: "access-token" });
constructor({ token, basePath }: StreamingAvatarApiConfig)

Interfaces

Configuration object for initializing StreamingAvatar.

PropertyTypeDescription
tokenstringAuthentication token.
basePathstringBase API URL (optional, defaults to https://api.heygen.com).
const config: StreamingAvatarApiConfig = {
  token: "access-token",
};

Request payload to initiate a new avatar streaming session.

PropertyTypeDescription
avatarNamestringInteractive Avatar ID for the session. (default, default)
qualityAvatarQuality(Optional) The desired quality level of the avatar stream.
voiceVoiceSetting(Optional) Voice settings for the avatar.
knowledgeIdstring(Optional) Knowledge base ID for the avatar's knowledge.
const startRequest: StartAvatarRequest = {
  avatarName: "MyAvatar",
  quality: AvatarQuality.High,
};

The response received when an avatar session is successfully started.

PropertyTypeDescription
session_idstringThe unique ID of the streaming session.
access_tokenstringToken to authenticate further interactions.
urlstringWebSocket URL for establishing the streaming session.
is_paidbooleanIndicates whether the session is under a paid plan.
session_duration_limitnumberMaximum allowed duration for the session in seconds.
{
  "session_id": "eba59f0d-71f5-11ef-b8af-d2e5560124bc",
  "sdp": null,
  "access_token": "eyJhbGc...",
  "url": "wss://heygen-feapbkvq.livekit.cloud",
  "ice_servers": null,
  "ice_servers2": null,
  "is_paid": true,
  "session_duration_limit": 600
}

Request payload for sending a speaking command to the avatar.

PropertyTypeDescription
textstringThe text the avatar should speak.
sessionIdstringThe active session's identifier.
const speakRequest: SpeakRequest = {
  text: "Hello, there!",
  sessionId: "session-id",
};

Request structure to interrupt the avatar's current speaking session.

PropertyTypeDescription
sessionIdstringThe active session's identifier.
const interruptRequest: InterruptRequest = { sessionId: sessionId };

Request structure to stop the avatar session.

PropertyTypeDescription
sessionIdstringThe active session's identifier.
const stopRequest: StopAvatarRequest = { sessionId: sessionId };

Methods

createStartAvatar

Starts a new avatar session using the provided configuration and returns session information.

createStartAvatar(requestData: StartAvatarRequest): Promise<StartAvatarResponse>
const response = await createStartAvatar(startRequest);
console.log(response.session_id); // Store session ID

startSession

Starts an existing avatar session.

startSession(requestData: StartSessionRequest): Promise<any>
const response = await startSession(sessionRequest);
console.log("Session started:", response);

speak

Sends a text command to the avatar to speak.

speak(requestData: SpeakRequest): Promise<any>
await speak({ text: "Hello, world!", sessionId });

startListening

Starts listening to the avatar for incoming messages.

startListening(requestData: {sessionId: string}): Promise<any>
await startListening({ sessionId: "session-id" });

stopListening

Stops listening to the avatar.

stopListening(requestData: {sessionId: string}): Promise<any>
await stopListening({ sessionId: "session-id" });

interrupt

Interrupts the current speaking task.

interrupt(requestData: InterruptRequest): Promise<any>
await interrupt({ sessionId: "session-id" });

stopAvatar

Stops the avatar session.

stopAvatar(requestData: StopAvatarRequest): Promise<any>
await stopAvatar({ sessionId: "session-id" });

on

Registers an event listener for specific streaming events.

on(eventType: string, listener: EventHandler): this
on('messageReceived', (message) => {
  console.log("Received message:", message);
});

off

Unregisters an event listener.

off(eventType: string, listener: EventHandler): this
off('messageReceived', listenerFunction);

Types and Enums

Defines the quality settings for the avatar.

  • Low: 'low' - 2000kbps and 720p.
  • Medium: 'medium' - 1000kbps and 480p.
  • High: 'high' - 500kbps and 360p.

Enumerates the event types for streaming. See Event Handling for details.

  • AVATAR_START_TALKING: Emitted when the avatar starts speaking.
  • AVATAR_STOP_TALKING: Emitted when the avatar stops speaking.
  • AVATAR_TALKING_MESSAGE: Triggered when the avatar sends a speaking message.
  • AVATAR_END_MESSAGE: Triggered when the avatar finishes sending messages.
  • USER_TALKING_MESSAGE: Emitted when the user sends a speaking message.
  • USER_END_MESSAGE: Triggered when the user finishes sending messages.
  • USER_START: Indicates when the user starts interacting.
  • USER_STOP: Indicates when the user stops interacting.
  • STREAM_READY: Indicates that the stream is ready for display.
  • STREAM_DISCONNECTED: Triggered when the stream disconnects.

Event Handling

The SDK emits a variety of events during a streaming session, which can be captured to update the UI or trigger additional logic. Use the on and off methods to manage event listeners.

AVATAR_START_TALKING

This event is emitted when the avatar begins speaking.

avatar.on(StreamingEvents.AVATAR_START_TALKING, (event) => {
  console.log('Avatar has started talking:', event);
  // You can update the UI to reflect that the avatar is talking
});

AVATAR_STOP_TALKING

avatar.on(StreamingEvents.AVATAR_STOP_TALKING, (event) => {
  console.log('Avatar has stopped talking:', event);
  // You can reset the UI to indicate the avatar has stopped speaking
});

AVATAR_TALKING_MESSAGE

Fired when the avatar sends a message while talking. This event can be useful for real-time updates on what the avatar is currently saying.

avatar.on(StreamingEvents.AVATAR_TALKING_MESSAGE, (message) => {
  console.log('Avatar talking message:', message);
  // You can display the message in the UI
});

AVATAR_END_MESSAGE

Fired when the avatar sends the final message before ending its speech.

avatar.on(StreamingEvents.AVATAR_END_MESSAGE, (message) => {
  console.log('Avatar end message:', message);
  // Handle the end of the avatar's message, e.g., indicate the end of the conversation
});

USER_TALKING_MESSAGE

Fired when the user sends a message to the avatar.

avatar.on(StreamingEvents.USER_TALKING_MESSAGE, (message) => {
  console.log('User talking message:', message);
  // Handle the user's message input to the avatar
});

USER_END_MESSAGE

Fired when the user has finished sending their message to the avatar.

avatar.on(StreamingEvents.USER_END_MESSAGE, (message) => {
  console.log('User end message:', message);
  // Handle the end of the user's message, e.g., process the user's response
});

USER_START

Fired when the user has finished sending their message to the avatar.

avatar.on(StreamingEvents.USER_START, (event) => {
  console.log('User has started interaction:', event);
  // Handle the start of the user's interaction, such as activating a listening indicator
});

USER_STOP

Triggered when the user stops interacting or speaking with the avatar.

avatar.on(StreamingEvents.USER_STOP, (event) => {
  console.log('User has stopped interaction:', event);
  // Handle the end of the user's interaction, such as deactivating a listening indicator
});

STREAM_READY

Fired when the avatar's streaming session is ready.

avatar.on(StreamingEvents.STREAM_READY, (event) => {
  console.log('Stream is ready:', event.detail);
  // Use event.detail to attach the media stream to a video element, for example
});

STREAM_DISCONNECTED

Triggered when the streaming connection is lost or intentionally disconnected.

avatar.on(StreamingEvents.STREAM_DISCONNECTED, () => {
  console.log('Stream has been disconnected');
  // Handle the disconnection, e.g., clean up the UI or try to reconnect the session
});

Error Handling

Always handle errors gracefully when dealing with asynchronous requests to avoid disruptions in the user experience.

try {
  await avatar.speak({ text: 'Hello!', sessionId: 'session-id' });
} catch (error) {
  console.error('Error sending speak command:', error);
}

Conclusion

This reference offers a comprehensive overview of HeyGen's streaming avatar SDK, complete with examples and descriptions of methods, events, and configuration options.