In this section, we will guide you through the five steps to establish a WebRTC connection and drive real-time speech using an API. Here's a concise summary, You can also directly access this repository to view the demo code.

Step 1: Create a New Session

To establish a real-time WebRTC connection, call the newSession API endpoint to retrieve the server's offer SDP and ICE information. Use the following code to make the API call:

const response = await fetch(`${https://api.heygen.com}/v1/streaming.new`, {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "X-Api-Key": apiKey,
  },
  body: JSON.stringify({ quality: "high" }),
});

{
    "code": "<CODE>",
    "data": {
        "ice_servers": ["<ICE_SERVERS>"],
        "sdp": {
            "sdp": "<SDP_DATA>",
            "type": "<SDP_TYPE>"
        },
        "session_id": "<SESSION_ID>"
    },
    "message": "<MESSAGE>"
}

You will receive a response containing the necessary information to create a new WebRTC connection.

Step 2: Start the Session

After obtaining the connection information, submit the answer SDP to the server to establish the connection. Utilize the startSession API endpoint for this purpose. Here's a sample code snippet:

const response = await fetch(`${https://api.heygen.com}/v1/streaming.start`, {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "X-Api-Key": apiKey,
  },
  body: JSON.stringify({
    session_id: "<SESSION_ID>",
    sdp: {
      type: "<SDP_TYPE>",
      sdp: "<SDP_DATA>"
    }
  }),
});

In general, you can do the following steps to complete the exchanging SDP process:

1. Create a new WebRTC peer connection object and set the necessary callback functions, such as ontrack, to handle incoming audio and video streams.
2. Retrieve the SDP offer from the newSession API response and set it as the remote description using the setRemoteDescription() method.
3. Generate the SDP answer by calling the createAnswer() method on the peer connection.
4. Set the generated SDP answer as the local description using the setLocalDescription() method.
5. Send the answer SDP to the server using the startSession API to establish the connection.

You can get more details at Streaming Avatar demo.

Step 3: Submit Network Information

After exchanging SDP, it's essential to exchange network information to establish a connection fully. Within the peerConnection.onicecandidate() callback function, obtain the network information and submit it to the server using the submit-ice API endpoint. Here's how you can do it:

peerConnection.onicecandidate = ({ candidate }) => {
  if (candidate) {
    const { candidate: iceCandidate, sdpMid, sdpMLineIndex } = candidate.toJSON();

    fetch(`${https://api.heygen.com}/v1/streaming.ice`, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'X-Api-Key': apiKey,
      },
      body: JSON.stringify({
        candidate: iceCandidate,
        sdpMid,
        sdpMLineIndex,
        session_id: sessionInfo.session_id,
      }),
    });
  }
};

Step 4: Drive Avatar to Speak

Once the connection is established, utilize the task API to drive real-time speech of the avatar. The avatar will echo and repeat your text content. Here's a sample code snippet:

const response = await fetch(`${https://api.heygen.com}/v1/streaming.task`, {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "X-Api-Key": apiKey,
  },
  body: JSON.stringify({
    session_id: "<SESSION_ID>",
    text: "<TEXT_PROMPT>"
  }),
});

Step 5: Close Session

To terminate the session and close the connection, call the Close session API. This will gracefully end the session. If no messages are sent within the session for 3 minutes, it will be automatically terminated. Here's a code snippet for closing the session:

const response = await fetch(`${https://api.heygen.com}/v1/streaming.stop`, {  
  method: "POST",  
  headers: {  
    "Content-Type": "application/json",  
    "X-Api-Key": apiKey,  
  },  
  body: JSON.stringify({ session_id }),  
});

Follow these steps to effectively utilize the Streaming Avatar API and drive real-time interactions with avatars in your applications. For further details and implementation, refer to the Streaming Avatar demo.

Conclusion

Utilizing the Streaming Avatar API opens up a world of possibilities for developers to create immersive and interactive experiences in their applications. By following the five steps outlined in this guide, you can establish a WebRTC connection, drive real-time speech, and seamlessly integrate dynamic avatars into your projects. Whether you're building virtual assistants, interactive e-learning platforms, or immersive entertainment experiences, the Streaming Avatar API provides the tools needed to enhance user engagement and interactivity.