Discussions

so below i will be attaching the process of how i am starting the things and let me know if there is any problem in the approach because my session starts correctly and working completely fine but then disconnects suddently after 10 or so seconds and i see in the disconnect room section console log a 5 text only.

HeyGen Session & Room Connection Process

Step 1: Authentication & Session Token
First, we authenticate with HeyGen by creating a session token. This token is obtained by making a POST request to the /v1/streaming.create_token endpoint with our API key. This token serves as the foundation for all subsequent API calls and has a limited lifespan, so we implement an automatic refresh mechanism that renews it every 25 seconds.

// Get initial session token
async function getSessionToken() {
const response = await fetch(${API_CONFIG.heyGen.serverUrl}/v1/streaming.create_token, {
method: "POST",
headers: {
"Content-Type": "application/json",
"X-Api-Key": API_CONFIG.heyGen.apiKey,
},
});
const data = await response.json();
state.sessionToken = data.data.token;
}

Step 2: Session Creation
Once we have a valid session token, we create a new streaming session by calling the /v1/streaming.new endpoint. This request includes configuration parameters such as video quality (set to "high"), the avatar ID for the virtual character, voice settings (voice ID and speech rate), and video encoding format (H264). The response provides us with session information including a unique session ID and access token.

// Create new streaming session
async function createNewSession() {
const response = await fetch(${API_CONFIG.heyGen.serverUrl}/v1/streaming.new, {
method: "POST",
headers: {
"Content-Type": "application/json",
Authorization: Bearer ${state.sessionToken},
},
body: JSON.stringify({
quality: "high",
avatar_id: userData.avatarID,
voice: { voice_id: elements.voiceIDInput.value, rate: 1.0 },
version: "v2",
video_encoding: "H264",
}),
});
state.sessionInfo = data.data;
}

Step 3: LiveKit Room Setup
We initialize a LiveKit room instance with specific configurations for adaptive streaming and dynamic casting. The room is configured with video capture defaults using H720 resolution presets. We then set up event handlers to manage media stream subscriptions, track events, and connection state changes. The room is prepared for connection using the URL and access token from the session information.

// Initialize LiveKit room
state.room = new LivekitClient.Room({
adaptiveStream: true,
dynacast: true,
videoCaptureDefaults: {
resolution: LivekitClient.VideoPresets.h720.resolution
},
});

// Set up room event handlers
setupRoomEventHandlers();

// Prepare connection
await state.room.prepareConnection(state.sessionInfo.url, state.sessionInfo.access_token);

Step 4: WebSocket Connection for Chat
A WebSocket connection is established to enable real-time chat functionality. The WebSocket URL is constructed using the HeyGen server hostname and includes query parameters for session ID, session token, language settings, and other chat-specific configurations. This connection handles bidirectional communication for text-based interactions.

// Connect WebSocket for chat
async function connectWebSocket(sessionId) {
const params = new URLSearchParams({
session_id: sessionId,
session_token: state.sessionToken,
silence_response: false,
opening_text: "",
stt_language: "en",
});

const wsUrl = `wss://${new URL(API_CONFIG.heyGen.serverUrl).hostname}/v1/ws/streaming.chat?${params}`;
state.webSocket = new WebSocket(wsUrl);

}

Step 5: Streaming Session Activation
The streaming session is activated by calling the /v1/streaming.start endpoint with the session ID. This initiates the actual video streaming. We then connect to the LiveKit room using the provided URL and access token, which establishes the media streaming connection for the avatar video.

// Start the streaming session
async function startStreamingSession() {
await fetch(${API_CONFIG.heyGen.serverUrl}/v1/streaming.start, {
method: "POST",
headers: {
"Content-Type": "application/json",
Authorization: Bearer ${state.sessionInfo.access_token},
},
body: JSON.stringify({ session_id: state.sessionInfo.session_id }),
});

// Connect to room
await state.room.connect(state.sessionInfo.url, state.sessionInfo.access_token);

}