The @heygen/streaming-avatar package provides a TypeScript SDK for interacting with HeyGen's streaming avatar service. For detailed information about the available methods, interfaces, enums, and event handling, refer to the API reference provided below.

Classes

StreamingAvatar

This class is the core of the SDK, responsible for managing avatar streaming sessions, including session creation, controlling avatars, and handling real-time communication.

const avatar = new StreamingAvatar({ token: "access-token" });

constructor({ token, basePath }: StreamingAvatarApiConfig)

Interfaces

StreamingAvatarApiConfig

Configuration object for initializing StreamingAvatar.

const config: StreamingAvatarApiConfig = {
  token: "access-token",
};

StartAvatarRequest↗

Request payload to initiate a new avatar streaming session.

const startRequest: StartAvatarRequest = {
  quality: AvatarQuality.High,
  avatarName: avatarId,
  knowledgeId: knowledgeId,
  // knowledgeBase: knowledgeBase,
  voice: {
    voiceId: voiceId,
    rate: 1.5, // 0.5 ~ 1.5
    emotion: VoiceEmotion.EXCITED,
  },
  language: language,
  disableIdleTimeout: true
};

StartAvatarResponse

The response received when an avatar session is successfully started.

{
  "session_id": "eba59f0d-71f5-11ef-b8af-d2e5560124bc",
  "sdp": null,
  "access_token": "eyJhbGc...",
  "url": "wss://heygen-feapbkvq.livekit.cloud",
  "ice_servers": null,
  "ice_servers2": null,
  "is_paid": true,
  "session_duration_limit": 600
}

SpeakRequest↗

Request payload for sending a speaking command to the avatar.

const speakRequest: SpeakRequest = {
  text: "Hello, there!",
  task_type: TaskType.REPEAT
};

Methods

createStartAvatar

Starts a new avatar session using the provided configuration and returns session information.

createStartAvatar(requestData: StartAvatarRequest): Promise<any>

const startRequest = { /* configuration settings */ };
const response = await createStartAvatar(startRequest);
console.log(response.session_id); // Store session ID for future operations

startVoiceChat

Starts a voice chat within the active avatar session. You can optionally enable or disable silence prompts during the chat by setting the useSilencePrompt flag

startVoiceChat(requestData: { useSilencePrompt?: boolean } = {}): Promise<any>

await startVoiceChat({ useSilencePrompt: false });

closeVoiceChat

Ends the active voice chat session within the avatar interaction.

closeVoiceChat(): Promise<any>

await closeVoiceChat();

newSession

Creates and starts a new session using the provided StartAvatarRequest data, returning detailed session information such as the session ID and other metadata.

newSession(requestData: StartAvatarRequest): Promise<StartAvatarResponse>

const sessionData = await newSession(startRequest);
console.log("New session created:", sessionData);

startSession

Starts an existing avatar session by using the previously stored session ID or configuration from a StartAvatarRequest.

startSession(): Promise<any>

const response = await startSession(sessionRequest);
console.log("Session started:", response);

speak

Sends a command to the avatar to speak the provided text. Additional parameters like task_type allow for more advanced control, like repeating or talking.

speak(requestData: SpeakRequest): Promise<any>

await speak({ text: "Hello, world!", task_type: TaskType.REPEAT });

startListening

Activates the avatar’s listening mode, allowing it to process incoming audio or messages from the user.

startListening(): Promise<any>

await startListening({});

stopListening

Stops the avatar from listening to incoming audio or messages.

stopListening(): Promise<any>

await stopListening({});

interrupt

Interrupts the current speaking task.

interrupt(): Promise<any>

await interrupt({});

stopAvatar

Stops the avatar session.

stopAvatar(): Promise<any>

await stopAvatar({});

on

Registers an event listener for specific streaming events.

on(eventType: string, listener: EventHandler): this

on('messageReceived', (message) => {
  console.log("Received message:", message);
});

off

Unregisters an event listener.

off(eventType: string, listener: EventHandler): this

off('messageReceived', listenerFunction);

Types and Enums

AvatarQuality

Defines the quality settings for the avatar.

• High: 'high' - 2000kbps and 720p.
• Medium: 'medium' - 1000kbps and 480p.
• Low: 'low' - 500kbps and 360p.

VoiceEmotion

• EXCITED: Excited voice emotion.
• SERIOUS: Serious voice emotion.
• FRIENDLY: Friendly voice emotion.
• SOOTHING: Soothing voice emotion.
• BROADCASTER: Broadcaster voice emotion.

TaskType

• TALK: Avatar will talk in response to the Text sent in tasks of this type; the response will be provided by HeyGen's connection to GPT-4o mini, and influenced by the KnowledgeID or KnowledgeBasethat were provided when calling the StartAvatarRequest method.
• REPEAT: Avatar will simply repeat the Text sent in tasks of this type; this task type is commonly used by developers who process a user's input independently, via an LLM of their choosing, and send the LLM's response as a Repeat task for the Avatar to say.

StreamingEvents

Enumerates the event types for streaming. See Event Handling for details.

• AVATAR_START_TALKING: Emitted when the avatar starts speaking.
• AVATAR_STOP_TALKING: Emitted when the avatar stops speaking.
• AVATAR_TALKING_MESSAGE: Triggered when the avatar sends a speaking message.
• AVATAR_END_MESSAGE: Triggered when the avatar finishes sending messages.
• USER_TALKING_MESSAGE: Emitted when the user sends a speaking message.
• USER_END_MESSAGE: Triggered when the user finishes sending messages.
• USER_START: Indicates when the user starts interacting.
• USER_STOP: Indicates when the user stops interacting.
• USER_SILENCE: Indicates when the user is silent.
• STREAM_READY: Indicates that the stream is ready for display.
• STREAM_DISCONNECTED: Triggered when the stream disconnects.

Event Handling

The SDK emits a variety of events during a streaming session, which can be captured to update the UI or trigger additional logic. Use the on and off methods to manage event listeners.

AVATAR_START_TALKING

This event is emitted when the avatar begins speaking.

avatar.on(StreamingEvents.AVATAR_START_TALKING, (event) => {
  console.log('Avatar has started talking:', event);
  // You can update the UI to reflect that the avatar is talking
});

AVATAR_STOP_TALKING

avatar.on(StreamingEvents.AVATAR_STOP_TALKING, (event) => {
  console.log('Avatar has stopped talking:', event);
  // You can reset the UI to indicate the avatar has stopped speaking
});

AVATAR_TALKING_MESSAGE

Fired when the avatar sends a message while talking. This event can be useful for real-time updates on what the avatar is currently saying.

avatar.on(StreamingEvents.AVATAR_TALKING_MESSAGE, (message) => {
  console.log('Avatar talking message:', message);
  // You can display the message in the UI
});

AVATAR_END_MESSAGE

Fired when the avatar sends the final message before ending its speech.

avatar.on(StreamingEvents.AVATAR_END_MESSAGE, (message) => {
  console.log('Avatar end message:', message);
  // Handle the end of the avatar's message, e.g., indicate the end of the conversation
});

USER_TALKING_MESSAGE

Fired when the user sends a message to the avatar.

avatar.on(StreamingEvents.USER_TALKING_MESSAGE, (message) => {
  console.log('User talking message:', message);
  // Handle the user's message input to the avatar
});

USER_END_MESSAGE

Fired when the user has finished sending their message to the avatar.

avatar.on(StreamingEvents.USER_END_MESSAGE, (message) => {
  console.log('User end message:', message);
  // Handle the end of the user's message, e.g., process the user's response
});

USER_START

Fired when the user has finished sending their message to the avatar.

avatar.on(StreamingEvents.USER_START, (event) => {
  console.log('User has started interaction:', event);
  // Handle the start of the user's interaction, such as activating a listening indicator
});

USER_STOP

Triggered when the user stops interacting or speaking with the avatar.

avatar.on(StreamingEvents.USER_STOP, (event) => {
  console.log('User has stopped interaction:', event);
  // Handle the end of the user's interaction, such as deactivating a listening indicator
});

USER_SILENCE

Triggered when the user is silent for a certain period.

avatar.on(StreamingEvents.USER_SILENCE, () => {
  console.log('User is silent');
});

STREAM_READY

Fired when the avatar's streaming session is ready.

avatar.on(StreamingEvents.STREAM_READY, (event) => {
  console.log('Stream is ready:', event.detail);
  // Use event.detail to attach the media stream to a video element, for example
});

STREAM_DISCONNECTED

Triggered when the streaming connection is lost or intentionally disconnected.

avatar.on(StreamingEvents.STREAM_DISCONNECTED, () => {
  console.log('Stream has been disconnected');
  // Handle the disconnection, e.g., clean up the UI or try to reconnect the session
});

Error Handling

Always handle errors gracefully when dealing with asynchronous requests to avoid disruptions in the user experience.

try {
  await avatar.speak({ text: 'Hello!' });
} catch (error) {
  console.error('Error sending speak command:', error);
}

Conclusion

This reference offers a comprehensive overview of HeyGen's streaming avatar SDK, complete with examples and descriptions of methods, events, and configuration options.