The Streaming Avatars API Suite is a powerful tool designed to bring digital avatars to life in real time. These APIs open up a realm of possibilities for dynamic and interactive experiences by streaming avatar videos directly from the server to your application. Whether you're building a virtual assistant, an interactive game, a real-time training module, or any application that requires a responsive, visually engaging interface, these APIs provide you the capability to make it more immersive.

Streaming Avatar APIs are still in beta.

Video Tutorials

Streaming Protocol

HeyGen's Live Streaming protocol leverages WebRTC (Web Real-Time Communication), a powerful technology enabling direct, low-latency communication between browsers or compatible applications. This technology is fundamental to our APIs, which use WebRTC to stream avatar data in real time, providing developers the tools needed to integrate interactive, virtual avatars into their applications. For an in-depth understanding of WebRTC, visit the WebRTC website.

Quick Guide: Use Streaming Avatar in 5 Steps

In this section, we will guide you through the five steps to establish a WebRTC connection and drive real-time speech using an API. Here's a concise summary, You can also directly access this repository to view the demo code.

step 1: new session

To establish a Real-time WebRTC connection, start by calling the newSession API endpoint to retrieve the server's offer SDP and ICE information. You can use the following code to make the API call:

const response = await fetch(`${https://api.heygen.com}/v1/streaming.new`, {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "X-Api-Key": apiKey,
  },
  body: JSON.stringify({ quality: "high" }),
});

you can get the response like these:

{
    "code": "<CODE>",
    "data": {
        "ice_servers": ["<ICE_SERVERS>"],
        "sdp": {
            "sdp": "<SDP_DATA>",
            "type": "<SDP_TYPE>"
        },
        "session_id": "<SESSION_ID>"
    },
    "message": "<MESSAGE>"
}

In your main control logic, you can create a new WebRTC connection using the information.

step 2: start session

In the previous section, we obtained the necessary connection information from the server. Now, we can proceed to send the answer SDP to the server in order to establish the connection.

To submit the answer SDP and establish the connection, use the startSession API endpoint. You can use the following code to make the API call:

const response = await fetch(`${https://api.heygen.com}/v1/streaming.start`, {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "X-Api-Key": apiKey,
  },
  body: JSON.stringify({
    session_id: "<SESSION_ID>",
    sdp: {
      type: "<SDP_TYPE>",
      sdp: "<SDP_DATA>"
    }
  }),
});

In general, you can do the following steps to complete the exchanging SDP process:

Create a new WebRTC peer connection object and set the necessary callback functions, such as ontrack, to handle incoming audio and video streams.
Retrieve the SDP offer from the newSession API response and set it as the remote description using the setRemoteDescription() method.
Generate the SDP answer by calling the createAnswer() method on the peer connection.
Set the generated SDP answer as the local description using the setLocalDescription() method.
Send the answer SDP to the server using the startSession API to establish the connection.

you can get more detail from Streaming Avatar demo

step 3: submit network information

After exchanging SDP, WebRTC requires the exchange of network information to establish a connection.

Within the peerConnection.onicecandidate() callback function, you can obtain the network information. Then, you can use this information to call the submit-ice API endpoint for submitting the network information like the following code:

peerConnection.onicecandidate = ({ candidate }) => {
  if (candidate) {
    const { candidate: iceCandidate, sdpMid, sdpMLineIndex } = candidate.toJSON();

    fetch(`${https://api.heygen.com}/v1/streaming.ice`, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'X-Api-Key': apiKey,
      },
      body: JSON.stringify({
        candidate: iceCandidate,
        sdpMid,
        sdpMLineIndex,
        session_id: sessionInfo.session_id,
      }),
    });
  }
};

step 4: driver avatar to speak

Once the connection is established, you can utilize the task API to drive the real-time speech of the avatar.The avatar will echo and repeat your text content. You can use the following code to make the API call:

const response = await fetch(`${https://api.heygen.com}/v1/streaming.task`, {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "X-Api-Key": apiKey,
  },
  body: JSON.stringify({
    session_id: "<SESSION_ID>",
    text: "<TEXT_PROMPT>"
  }),
});

step 5: close session

To close session and close the connection, you can call Close session API. This will terminate the session. If no messages are sent within the session for 3 minutes, it will be automatically terminated.

You can use the following code to make the API call:

const response = await fetch(`${https://api.heygen.com}/v1/streaming.stop`, {  
  method: "POST",  
  headers: {  
    "Content-Type": "application/json",  
    "X-Api-Key": apiKey,  
  },  
  body: JSON.stringify({ session_id }),  
});

Start with Demo Code

Pre-Requisites

Grab Your API Token (Trial Token works too)
- please follow this page to get your API token
Node.js and npm installed on your system.

Run the Demo

Clone the repository.

 git clone https://github.com/HeyGen-Official/StreamingAvatar

Open the api.json file and replace 'YourApiKey' with your API key:
```
"apiKey": "YourApiKey";
```
open a terminal in the folder and then install the express and run the server.js:
```
npm install express
node server.js
```
you will see App is listening on port 3000!.

Use the Demo

Open the web browser and enter the http://localhost:3000to start the demo.
Click the "New" button to create a new session. The status updates will be displayed on the screen.
After the session is created successfully, click the "Start" button to start streaming the avatar.
To send a task to the avatar, type the text in the provided input field and click the "Repeat Text" button.
Once done, click the "Close Connection" button to close the session.

get more details on here

Use Cases

With the API, you can unlock a multitude of possibilities, including:

Real-time Voice Conversations: Build applications that facilitate seamless voice conversations in real-time between users and digital avatars, providing immersive and interactive experiences.
Interactive GPT Conversations: Integrate the GPT model to enable dynamic and interactive conversations with AI-powered chatbots or virtual assistants, enhancing user engagement and satisfaction.
Live Stream Shopping: Develop captivating live stream shopping experiences where customers can interact with avatars or chatbots, obtain product recommendations, and make purchases instantly.
Language Learning: Create interactive language learning platforms that enable users to practice speaking with virtual language tutors or engage in role-playing conversations, fostering effective language acquisition.

If you have innovative ideas and require technical support, feel free to reach out to us. We are here to assist you. Additionally, the possibilities for exploration are limitless!