This guide walks you through adding a chroma key effect to your streaming avatar demo. This feature will allow you to make the green background of your avatar transparent, making it appear as if the avatar is floating on the webpage.

In our implementation, we'll:

Detect green pixels in the avatar video
Replace those pixels with transparency
Allow real-time toggling of this effect

Step 1: Update the HTML

First, we need to add a canvas element that will display the processed video and a checkbox to toggle the chroma key effect.


<!-- Video Section - Add a canvas element to the video container -->
<article style="width: fit-content">
  <video id="avatarVideo" autoplay playsinline></video>
  <canvas id="avatarCanvas"></canvas> <!-- This is new -->
</article>

<!-- Add this after your other controls -->
<!-- Chroma Key Toggle -->
<div class="chromakey-toggle">
  <input type="checkbox" id="chromaKeyToggle" />
  <label for="chromaKeyToggle">Enable Chroma Keying</label>
</div>

Step 2: Create the Chroma Key Module

You can use Chroma Keying Preview tool to generate this code snippet. Create a new file called src/chromaKey.ts to contain the chroma keying functionality:

/**
 * Apply chroma key effect to a video frame on canvas
 * @param sourceVideo - Source video element
 * @param targetCanvas - Target canvas element
 * @param options - Chroma key options
 */
export function applyChromaKey(
  sourceVideo: HTMLVideoElement,
  targetCanvas: HTMLCanvasElement,
  options: {
    minHue: number; // 60 - minimum hue value (0-360)
    maxHue: number; // 180 - maximum hue value (0-360)
    minSaturation: number; // 0.10 - minimum saturation (0-1)
    threshold: number; // 1.00 - threshold for green detection
  }
): void {
  // Get canvas context
  const ctx = targetCanvas.getContext("2d", {
    willReadFrequently: true,
    alpha: true,
  });

  if (!ctx || sourceVideo.readyState < 2) return;

  // Set canvas dimensions to match video
  targetCanvas.width = sourceVideo.videoWidth;
  targetCanvas.height = sourceVideo.videoHeight;

  // Clear canvas
  ctx.clearRect(0, 0, targetCanvas.width, targetCanvas.height);

  // Draw video frame to canvas
  ctx.drawImage(sourceVideo, 0, 0, targetCanvas.width, targetCanvas.height);

  // Get image data for processing
  const imageData = ctx.getImageData(
    0,
    0,
    targetCanvas.width,
    targetCanvas.height
  );
  const data = imageData.data;

  // Process each pixel
  for (let i = 0; i < data.length; i += 4) {
    const r = data[i];
    const g = data[i + 1];
    const b = data[i + 2];

    // Convert RGB to HSV
    const max = Math.max(r, g, b);
    const min = Math.min(r, g, b);
    const delta = max - min;

    // Calculate hue
    let h = 0;
    if (delta === 0) {
      h = 0;
    } else if (max === r) {
      h = ((g - b) / delta) % 6;
    } else if (max === g) {
      h = (b - r) / delta + 2;
    } else {
      h = (r - g) / delta + 4;
    }

    h = Math.round(h * 60);
    if (h < 0) h += 360;

    // Calculate saturation and value
    const s = max === 0 ? 0 : delta / max;
    const v = max / 255;

    // Check if pixel is in the green screen range
    const isGreen =
      h >= options.minHue &&
      h <= options.maxHue &&
      s > options.minSaturation &&
      v > 0.15 &&
      g > r * options.threshold &&
      g > b * options.threshold;

    // Apply transparency for green pixels
    if (isGreen) {
      const greenness = (g - Math.max(r, b)) / (g || 1);
      const alphaValue = Math.max(0, 1 - greenness * 4);
      data[i + 3] = alphaValue < 0.2 ? 0 : Math.round(alphaValue * 255);
    }
  }

  // Put processed image data back to canvas
  ctx.putImageData(imageData, 0, 0);
}

/**
 * Setup continuous chroma keying
 * @param sourceVideo - Source video element
 * @param targetCanvas - Target canvas element
 * @param options - Chroma key options
 * @returns - Function to stop the processing
 */
export function setupChromaKey(
  sourceVideo: HTMLVideoElement,
  targetCanvas: HTMLCanvasElement,
  options: {
    minHue: number; // Minimum hue value (0-360)
    maxHue: number; // Maximum hue value (0-360)
    minSaturation: number; // Minimum saturation (0-1)
    threshold: number; // Threshold for green detection
  }
): () => void {
  let animationFrameId: number | null = null;

  // Processing function
  const render = () => {
    applyChromaKey(sourceVideo, targetCanvas, options);
    animationFrameId = requestAnimationFrame(render);
  };

  // Start rendering
  render();

  // Return cleanup function
  return () => {
    if (animationFrameId !== null) {
      cancelAnimationFrame(animationFrameId);
    }
  };
}

Step 3: Update Main Application Logic

Now we need to update src/main.ts to integrate the chroma keying functionality:

1. Import the chroma key module

Add the following import at the top of your file:

import { setupChromaKey } from "./chromaKey";

2. Add new DOM element references

Add these DOM element references at the beginning of your file:

const canvasElement = document.getElementById(
  "avatarCanvas"
) as HTMLCanvasElement;
const chromaKeyToggle = document.getElementById(
  "chromaKeyToggle"
) as HTMLInputElement;

3. Add a state variable for tracking the chroma key process

Add this with your other state variables:

let stopChromaKeyProcessing: (() => void) | null = null;

4. Update the stream ready handler

Modify your handleStreamReady function to initialize chroma key when the stream is ready:

function handleStreamReady(event: any) {
  if (event.detail && videoElement) {
    videoElement.srcObject = event.detail;
    videoElement.onloadedmetadata = () => {
      videoElement.play().catch(console.error);

      // Setup chroma key if enabled
      updateChromaKeyState();
    };
    voiceModeBtn.disabled = false;
  } else {
    console.error("Stream is not available");
  }

  // Rest of your existing function...
}

5. Update stream disconnection handling

Update your disconnection handlers to clean up chroma key processing:

function handleStreamDisconnected() {
  console.log("Stream disconnected");
  if (videoElement) {
    videoElement.srcObject = null;
  }

  // Stop chroma key processing if active
  if (stopChromaKeyProcessing) {
    stopChromaKeyProcessing();
    stopChromaKeyProcessing = null;
  }

  // Enable start button and disable end button
  startButton.disabled = false;
  endButton.disabled = true;
}

async function terminateAvatarSession() {
  if (!avatar || !sessionData) return;

  await avatar.stopAvatar();
  videoElement.srcObject = null;

  // Stop chroma key processing if active
  if (stopChromaKeyProcessing) {
    stopChromaKeyProcessing();
    stopChromaKeyProcessing = null;
  }

  avatar = null;
}

6. Add the chroma key toggle function

Add this new function to handle toggling chroma key on/off:

function updateChromaKeyState() {
  if (!videoElement.srcObject) return;

  // Stop any existing chroma key processing
  if (stopChromaKeyProcessing) {
    stopChromaKeyProcessing();
    stopChromaKeyProcessing = null;
  }

  if (chromaKeyToggle.checked) {
    // Show canvas, hide video
    canvasElement.style.display = "block";
    videoElement.style.display = "none";

    // Start chroma key processing
    stopChromaKeyProcessing = setupChromaKey(videoElement, canvasElement, {
      minHue: 60,
      maxHue: 180,
      minSaturation: 0.1,
      threshold: 1.0,
    });
  } else {
    // Show video, hide canvas
    videoElement.style.display = "block";
    canvasElement.style.display = "none";
  }
}

7. Add the event listener

Add this with your other event listeners:

chromaKeyToggle.addEventListener("click", updateChromaKeyState);

Advanced Customization

The chroma key parameters can be adjusted to fine-tune the effect:

minHue and maxHue: Control the range of green hues that will be detected (60-180 covers most greens)
minSaturation: Sets a minimum saturation for detection (helps avoid detecting unsaturated grays/whites)
threshold: Controls how much "greener" a pixel must be compared to its red/blue components

You can adjust these values live using the Chroma Keying Preview tool.

Conclusion

This implementation provides a functional chroma keying solution for streaming avatars using standard web technologies. The client-side processing is compatible with most modern browsers and requires no additional libraries.