H264BSD decoder

Decode and render H.264 streams in Web browsers using H264BSD, a WebAssembly build of the Android H264BSD software decoder.

It's slow, and only supports H.264 Baseline profile at level 4, but works on most browsers.

npm

npm install @yume-chan/scrcpy-decoder-h264bsd

Yarn

yarn add @yume-chan/scrcpy-decoder-h264bsd

pnpm

pnpm add @yume-chan/scrcpy-decoder-h264bsd

Bun

bun add @yume-chan/scrcpy-decoder-h264bsd

warning

Vite's dependency "optimizer" will break this package: https://github.com/vitejs/vite/issues/8427

Add this to your vite.config.js/vite.config.ts:

import { defineConfig } from "vite";

export default defineConfig({
  optimizeDeps: {
    exclude: [
      "@yume-chan/scrcpy-decoder-h264bsd",
    ],
    include: [
      "@yume-chan/scrcpy-decoder-h264bsd > yuv-canvas",
    ],
  },
});

Performance

There are two aspects of performance:

Decoding

@yume-chan/h264bsd package is used for decoding, which compiles the C code into WebAssembly, and runs it in a Web Worker. This way, the main thread is not blocked by the decoding process.

H.264 is an old codec and the decoding algorithm is relatively simple, so the performance is acceptable on most devices. But on low-end devices, it may still be too slow to decode high-resolution video at high frame rates. You can reduce the resolution and/or frame rate to improve performance.

Rendering

H264BSD decoder outputs buffers containing pixel colors in YUV color space, which needs to be converted to RGB for rendering.

yuv-canvas package is used to do the conversion and rendering. When supported, it uses a WebGL shader to accelerate the conversion, so it's very fast. But on unsupported devices, it falls back to a software implementation, which is super slow.

Limit profile/level

Because it only supports H.264 Baseline profile at level 4, but many newer devices default to higher profiles/levels, you must limit it using codecOptions option(until v2.0)videoCodecOptions option(since v2.0):

import { ScrcpyOptions1_24, ScrcpyCodecOptions } from "@yume-chan/scrcpy";
import { H264BsdDecoder } from "@yume-chan/scrcpy-decoder-h264bsd";

const H264Capabilities = H264BsdDecoder.capabilities.h264;

const options = new ScrcpyOptions1_24({
  // other options...
  codecOptions: new ScrcpyCodecOptions()
    .setProfile(H264Capabilities.maxProfile)
    .setLevel(H264Capabilities.maxLevel),
});

However, it will fail on some very old devices that doesn't even support Baseline level 4 codec. If that happens, you can retry starting the server without the codecOptions option(until v2.0)videoCodecOptions option(since v2.0).

import { ScrcpyOptions1_24, ScrcpyCodecOptions } from "@yume-chan/scrcpy";
import { H264BsdDecoder } from "@yume-chan/scrcpy-decoder-h264bsd";

const H264Capabilities = H264BsdDecoder.capabilities.h264;

try {
  await startServer(
    new ScrcpyOptions1_24({
      // other options...
      codecOptions: new ScrcpyCodecOptions()
        .setProfile(H264Capabilities.maxProfile)
        .setLevel(H264Capabilities.maxLevel),
    })
  );
} catch (e) {
  await startServer(
    new ScrcpyOptions1_24({
      // other options...
    })
  );
}

Create a decoder

export declare class H264BsdDecoder implements ScrcpyVideoDecoder {
  static readonly capabilities: Record<string, ScrcpyVideoDecoderCapability>;

  constructor(options?: H264BsdDecoder.Options);

  get type(): "software";
  get canvas(): HTMLCanvasElement | OffscreenCanvas;
  get rendererType(): "software" | "hardware";
  get paused(): boolean;
  get writable(): WritableStream<ScrcpyMediaStreamPacket>;
  get width(): number;
  get height(): number;
  get sizeChanged(): Event<{ width: number; height: number }>;

  get decoderResetCount(): number;
  get framesDecoded(): number;
  get framesSkippedDecoding(): number;
  get framesRendered(): number;
  get framesDisplayed(): number;
  get framesSkippedRendering(): number;

  pause(): void;
  resume(): void;
  trackDocumentVisibility(document: Document): () => void;
  dispose(): Promise<void>;
}

export declare namespace H264BsdDecoder {
  interface Options {
    /**
     * Optional render target canvas element or offscreen canvas.
     * If not provided, a new `<canvas>` (when DOM is available)
     * or a `OffscreenCanvas` will be created.
     */
    canvas?: HTMLCanvasElement | OffscreenCanvas | undefined;

    /**
     * Whether to create a Web Worker to run the decoder.
     *
     * - `"auto"`: Create a Web Worker if currently running in the main thread.
     * - `true`: Always create a Web Worker. If already running in a Web Worker,
     *   `canvas` must be an `OffscreenCanvas`, which will be transferred to the new Worker.
     * - `false`: Never create a Web Worker.
     *
     * Defaults to `"auto"`.
     */
    worker?: "auto" | boolean;
  }
}

H264BSD decoder can render to an HTML <canvas> element, or an OffscreenCanvas object. The render target can be accessed via the canvas property.

Create a new canvas

If the canvas option is not provided, it automatically creates a <canvas> element if there is DOM API, or an OffscreenCanvas otherwise. The created render target can be retrieved from the canvas property.

JavaScript

import { H264BsdDecoder } from "@yume-chan/scrcpy-decoder-h264bsd";

const decoder = new H264BsdDecoder();
document.body.appendChild(decoder.canvas);

TypeScript

import { H264BsdDecoder } from "@yume-chan/scrcpy-decoder-h264bsd";

const decoder = new H264BsdDecoder();
document.body.appendChild(decoder.canvas as HTMLCanvasElement);

The newly created canvas needs to be inserted into the page to display the video.

Using an existing canvas

When using MVVM frameworks like React.js, it might be simpler to attach the decoder to an existing element, for example:

JavaScript

import React from "react";
import { H264BsdDecoder } from "@yume-chan/scrcpy-decoder-h264bsd";

export function Renderer(props) {
  const canvasRef = React.useRef(null);

  React.useEffect(() => {
    const decoder = new H264BsdDecoder({ canvas: canvasRef.current });
    void props.videoStream.pipeTo(decoder.writable).catch(() => {});

    return () => {
      void decoder.dispose();
    };
  }, [props.videoStream]);

  return (
    <div className="container">
      <canvas ref={canvasRef} />
    </div>
  );
}

TypeScript

import React from "react";
import { H264BsdDecoder } from "@yume-chan/scrcpy-decoder-h264bsd";
import type { ScrcpyMediaStreamPacket } from "@yume-chan/scrcpy";

export function Renderer(props: { videoStream: ReadableStream<ScrcpyMediaStreamPacket> }) {
  const canvasRef = React.useRef<HTMLCanvasElement | null>(null);

  React.useEffect(() => {
    const decoder = new H264BsdDecoder({ canvas: canvasRef.current });
    void props.videoStream.pipeTo(decoder.writable).catch(() => {});

    return () => {
      void decoder.dispose();
    };
  }, [props.videoStream]);

  return (
    <div className="container">
      <canvas ref={canvasRef} />
    </div>
  );
}

Web Worker

The worker option controls whether decoding runs in a Web Worker. It accepts "auto" (default), true, or false. The actual behavior depends on both the option value and the thread the decoder is constructed on:

worker	Current thread	Behavior
"auto"	Main thread	Spawns a Web Worker; decoding and rendering run off the main thread. The canvas is transferred to the worker (an HTMLCanvasElement becomes an OffscreenCanvas via transferControlToOffscreen) and can no longer be drawn to from the main thread.
"auto"	Web Worker	Runs on the current thread. No worker is created.
true	Main thread	Same as "auto" on the main thread: spawns a Web Worker and transfers the canvas.
true	Web Worker	Spawns a nested Web Worker. canvas must be an OffscreenCanvas; it is transferred to the new worker (throws otherwise).
false	Main thread	Runs on the current main thread; decoding blocks the main thread. No worker is created.
false	Web Worker	Runs on the current thread. No worker is created.

The default "auto" is recommended for most cases, as it keeps the main thread responsive while working out of the box on both the main thread and inside Web Workers.

import { H264BsdDecoder } from "@yume-chan/scrcpy-decoder-h264bsd";

const decoder = new H264BsdDecoder({
  worker: false,
});

Common decoder operations

The H264BSD decoder implements the ScrcpyVideoDecoder interface. See the following pages for common decoder functionality:

• Pipe video stream
• Handle size changes
• Pause decoding
• Rendering metrics