Discussions

Hi there,

We’re currently working on integrating a fully interactive video avatar using HeyGen, powered by an external LLM-based assistant (currently OpenAI via API). The goal is to have a dynamic, real-time conversation with the avatar that is driven entirely by responses generated by our own assistant logic.

Here’s what we’re aiming for (see attached visual for clarity):
• We use a custom assistant built with OpenAI’s API.
• We stream user input to our LLM backend (via a streaming API layer).
• The LLM generates a response.
• That response is converted to speech using ElevenLabs.
• Finally, the HeyGen avatar plays both the audio and syncs the lips accordingly in real-time (or near real-time), creating the illusion of a live assistant.

The challenge: How can we achieve this connection between the HeyGen avatar and our external LLM/assistant in real-time or near real-time? Does HeyGen currently support this level of integration, and if so, how should we technically approach this?

We’d love to know:
1. Whether this workflow is possible with HeyGen right now (and if not, if it’s on the roadmap).
2. What kind of API/webhook/event-based control (if any) we can use to stream inputs/outputs in and out of HeyGen avatars.
3. Any best practices or examples for this kind of integration?

This is a key use case for us, and we’d love to collaborate to make this work.

Thanks in advance!

Best regards,
Marcel