Skip to main content
Version: next

Handle video stream

The video socket from Scrcpy server contains the encoded video frames. Tango provides two levels of APIs:

  • High-level (via @yume-chan/adb-scrcpy): Parses the stream automatically and integrates with the client lifecycle. You get the parsed metadata, a stream of typed packets, and automatic video size tracking. Start here if you're unsure which to use.
  • Low-level (via @yume-chan/scrcpy): Parse the raw byte stream directly with full control over the packet format. Useful when you already have a ReadableStream<Uint8Array> and want to parse it yourself.

High-level usage​

With @yume-chan/adb-scrcpy​

When the video option is not false, (since v2.1)AdbScrcpyClient.videoStream is a Promise that resolves to an AdbScrcpyVideoStream:

import type { Event } from "@yume-chan/event";

// import type { ScrcpyVideoSizeChangedEvent } from "@yume-chan/scrcpy";
interface ScrcpyVideoSizeChangedEvent {
  width: number;
  height: number;
  isClientResize?: boolean | undefined;
}

interface AdbScrcpyVideoStream {
  readonly metadata: ScrcpyVideoStreamMetadata;
  readonly stream: ReadableStream<ScrcpyVideoStreamPacket>;
  readonly width: number;
  readonly height: number;
  readonly sizeChanged: Event<ScrcpyVideoSizeChangedEvent>;
}

Metadata​

// import type { ScrcpyVideoStreamMetadata } from "@yume-chan/scrcpy";
interface ScrcpyVideoStreamMetadata {
  deviceName?: string | undefined;
  width?: number | undefined;
  height?: number | undefined;
  codec: ScrcpyVideoCodecId;
}

When is each field available?

  • deviceName: Always present (no option to disable)(until v1.22), unless sendDeviceMeta is false(since v1.22).
  • width and height: The initial video resolution. Always present (no option to disable)(until v1.22)Present unless sendDeviceMeta is false(between v1.22 and v1.25)Present unless sendCodecMeta is false(between v2.0 and v3.3.4)Not included in metadata — sent as a session packet instead(since v4.0).
  • codec: Always available — Tango determines it from the stream or falls back to the default ScrcpyVideoCodecId.H264(until v1.25)videoCodec option value(since v2.0).

See the Low-level metadata format table for the exact byte layout of each element.

Why is there both a static metadata and a live stream?

The metadata is only sent once at connection start. AdbScrcpyVideoStream stores it as a static snapshot — the width and height in metadata reflect only the initial video size. When the screen rotates or the encoder restarts, the new size arrives as stream packets, not as updated metadata. That's why AdbScrcpyVideoStream has separate metadata (static, initial values) and stream (continuous, dynamic) properties. See Video size for how the live size is tracked.

import type { AdbScrcpyClient } from "@yume-chan/adb-scrcpy";

declare const client: AdbScrcpyClient;

if (client.videoStream) {
  const videoStream = await client.videoStream;
  const { deviceName, width, height, codec } = videoStream.metadata;
  console.log(deviceName, width, height, codec);
}

Stream​

AdbScrcpyVideoStream.stream is a ReadableStream<ScrcpyVideoStreamPacket> produced by parsing the video socket with createMediaStreamTransformer. See Low-level stream packets for details on packet types and format.

import type { ScrcpyVideoStreamPacket } from "@yume-chan/scrcpy";
import type { AdbScrcpyClient } from "@yume-chan/adb-scrcpy";

declare const client: AdbScrcpyClient;

if (client.videoStream) {
  const videoStream = await client.videoStream;

  videoStream.stream
    .pipeTo(
      new WritableStream({
        write(packet: ScrcpyVideoStreamPacket) {
          switch (packet.type) {
            case "session":
              console.log(packet.width, packet.height);
              break;
            case "configuration":
              console.log(packet.data);
              break;
            case "data":
              console.log(packet.keyframe, packet.pts, packet.data);
              break;
          }
        },
      }),
    )
    .catch((e) => {
      console.error(e);
    });
}
How to handle each packet type
  • Session (type: "session"): Informs about video size changes. If you're using AdbScrcpyVideoStream's width and height properties (see Video size), you can ignore these packets — the class tracks them internally.
  • Configuration (type: "configuration"): Codec initialization data (e.g., SPS/PPS for H.264). A video decoder needs this before it can decode data packets. The built-in decoders handle this automatically — just pipe the whole stream into them.
  • Data (type: "data"): One encoded video frame. Send these to a video decoder for rendering, or write them to a file for recording.
danger

Don't await the Promise returned by pipeTo. The pipeTo promise only resolves when the source stream ends (the scrcpy server disconnects). Awaiting it before consuming other required streams (e.g., audio or clipboard) blocks those streams from being consumed, causing a deadlock — the server waits for all streams to be read, but your code is stuck waiting for one to end.

Similar to options.clipboard, you should store the returned promise in a variable and handle it asynchronously (e.g., using .catch() for errors) rather than awaiting it.

Video size​

When session packets are enabled, AdbScrcpyVideoStream will track the up-to-date video size by inspecting the session packets. Otherwise,(since v4.0) it parses the video stream to extract the video size.

The width and height properties are the current video size, and the sizeChanged event is fired when the video size changes. The isClientResize property indicates whether the resize was initiated by the client using resizeDisplay control message.(since v4.0)

import type { AdbScrcpyClient } from "@yume-chan/adb-scrcpy";

declare const client: AdbScrcpyClient;

if (client.videoStream) {
  const videoStream = await client.videoStream;
  console.log(videoStream.width, videoStream.height);

  const dispose = videoStream.sizeChanged(({ width, height, isClientResize }) => {
    console.log(width, height, isClientResize);
  });

  setTimeout(dispose, 1000);
}

Decode and render in browsers​

We provide two packages to decode and render the video stream in Web browsers:

Both decoders implement the common ScrcpyVideoDecoder interface for piping video streams, handling size changes, pausing, and monitoring rendering metrics.

To use them, the sendFrameMeta option must be true (the default value) to enable configuration packets.

Decode and render outside browsers​

Decoding and playing video outside the browser is out of this library's scope. It will depend on the runtime environment, UI framework, and media library you use.

However, we have an example of using ffplay (from ffmpeg) to play the video stream (without audio) at https://github.com/tango-adb/scrcpy-ffplay.

Low-level usage​

If you already have a ReadableStream<Uint8Array> that reads from the video socket, @yume-chan/scrcpy provides methods to parse it.

Metadata​

Metadata is sent at the start of the video stream. The availability of each field depends on the Scrcpy server version and the specified option values.

Format​

ElementSizeTypeDescriptionRelated Option
Device name64 bytesNull-terminated stringThe device's model namesendDeviceMeta(since v1.22)
Codec ID4 bytesEnumThe codec IDsendCodecMeta(between v2.0 and v3.3.4)
sendStreamMeta(since v4.0)
Initial video size4 bytes2 bytes width
2 bytes height		Width and height of the videosendDeviceMeta(between v1.22 and v1.25)
sendCodecMeta(since v2.0)

Size changes​

The metadata is only sent once.

When device screen size changes (for example, when device orientation changes, or a foldable device unfolds), the server restarts the video encoder, but does NOT send another metadata with the new size. To track video resolution, parsing the video stream is required. Built-in video decoders have a sizeChanged event, and AdbScrcpyClient also contains size information.(until v3.3.4)

When encoder restarts on screen size changes, the server sends a session packet with the new size. Consumers can inspect the packet to get the new video size. The sizeChanged event on video decoders and AdbScrcpyClient.videoStream will also be updated accordingly.(since v4.0)

Parsing metadata​

ScrcpyOptionsX_YY.prototype.parseVideoStreamMetadata parses the metadata from a raw video stream:

import { ScrcpyOptions2_1 } from "@yume-chan/scrcpy";

const options = new ScrcpyOptions2_1({
  // use the same version and options when starting the server
});

const videoSocket: ReadableStream<Uint8Array>; // get the stream yourself

const { metadata: videoMetadata, stream: videoStream } =
  await options.parseVideoStreamMetadata(videoSocket);

The method returns the parsed metadata, and the remaining stream after the metadata, so you can continue reading the video stream.

Stream packets​

Various packets are sent after the metadata. The packet format depends on the Scrcpy server version and the specified option values.

Format​

ElementSizeDescriptionRelated Option
Session packet12 bytesResize reason and new video sizesendStreamMeta(since v4.0)
Configuration packet8 bytes header
Variable data		Codec-specific configuration datasendFrameMeta
Data packet8 bytes header
Variable data		Encoded video frame, with keyframe flag and presentation timestampsendFrameMeta

Packet types​

Tango parses them into different packet types:

interface ScrcpyMediaStreamConfigurationPacket {
  type: "configuration";
  data: Uint8Array;
}

interface ScrcpyMediaStreamDataPacket {
  type: "data";
  keyframe?: boolean;
  pts?: bigint;
  data: Uint8Array;
}

interface ScrcpyVideoStreamSessionPacket {
  type: "session";
  isClientResize: boolean;
  width: number;
  height: number;
}

type ScrcpyAudioStreamPacket =
  ScrcpyMediaStreamConfigurationPacket | ScrcpyMediaStreamDataPacket;

type ScrcpyVideoStreamPacket =
  ScrcpyAudioStreamPacket | ScrcpyVideoStreamSessionPacket;

Session packet​

A session packet is sent after the metadata, and when the video encoder restarts (due to screen rotation, resetVideo control message, or resizeDisplay control message with flexDisplay option).

interface ScrcpyVideoStreamSessionPacket {
  type: "session";
  isClientResize: boolean;
  width: number;
  height: number;
}
  • isClientResize: Whether the resize was initiated by the client using resizeDisplay control message.
  • width/height: The new video size.

Configuration packet​

If the chosen video codec has configuration data, a configuration packet will be sent before the first encoded video frame, and each time the encoder restarts.

The format of configuration packets depends on the codec. More information can be found at https://developer.android.com/reference/android/media/MediaCodec#CSD

H.264 and H.265: For H.26x codecs, the configuration packets include:

  • H.264: Sequence Parameter Set (SPS) and Picture Parameter Set (PPS), in Annex B format.
  • H.265: Video Parameter Set (VPS), Sequence Parameter Set (SPS), and Picture Parameter Set (PPS), in Annex B format.

These are essential for proper decoding, so they will still be sent through the video stream even when configuration packets are disabled. Enabling configuration packets only adds a mark to them, so the client can more easily find and handle them.

When a client receives a configuration packet, it should create a new video decoder with the data.

info

The two built-in decoders handle configuration packets and re-create video decoders for you.

You only need to pipe the whole video stream into them.

AV1: The configuration packet contains the first 3 bytes of AV1CodecConfigurationRecord (https://aomediacodec.github.io/av1-isobmff/#av1codecconfigurationbox). The remaining configuration OBUs are in the next data packet.

These 3 bytes are required when saving the stream to video files, although they can also be inferred from the configuration OBUs.

VP8: VP8 does not have configuration data, so no configuration packet will be sent.

VP9: A VP9 CodecPrivate Data.

Data packet​

Each data packet represents exactly one encoded frame, with extra information when the related options are enabled:

  • keyframe: true if the current packet is a keyframe. Many decoders can decode the video stream without knowing if each frame is a keyframe, but some decoders require this information.
  • pts: Presentation timestamp in nanoseconds. When rendering the video in real-time, generally you want to present the decoded frames as they arrive to minimize latency, but this information can be used to remove processing time deviations when recording.
  • data: Contains exactly one encoded frame. For H.264 and H.265, in Annex B format.

Parsing packets​

ScrcpyOptionsX_YY.prototype.createMediaStreamTransformer creates a TransformStream that parses the video stream into packets.

parseVideoStreamMetadata and createMediaStreamTransformer are separate methods, because createMediaStreamTransformer is also used to parse the audio stream.

import type { ScrcpyVideoStreamPacket } from "@yume-chan/scrcpy";

declare const options: ScrcpyOptions2_1;
declare const videoStream: ReadableStream<Uint8Array>;

const videoPacketStream: ReadableStream<ScrcpyVideoStreamPacket> = videoStream.pipeThrough(
  options.createMediaStreamTransformer(),
);

videoPacketStream
  .pipeTo(
    new WritableStream({
      write(packet: ScrcpyVideoStreamPacket) {
        switch (packet.type) {
          case "session":
            console.log(packet.width, packet.height);
            break;
          case "configuration":
            console.log(packet.data);
            break;
          case "data":
            console.log(packet.keyframe, packet.pts, packet.data);
            break;
        }
      },
    }),
  )
  .catch((e) => {
    console.error(e);
  });

Raw mode​

As the metadata format table shows, since v1.22, if sendDeviceMeta and sendFrameMeta options are false, and sendCodecMeta option is also false(between v2.0 and v3.3.4), and sendStreamMeta option is also false(since v4.0), all elements will be disabled.

This is also called raw mode. In this mode, the video socket only contains codec-specific, encoded video data.

Raw mode is more difficult to process because it requires the client to parse the codec-specific format directly.

Tango has partial support for raw mode: all omitted fields will be undefined, and built-in video decoders don't work with raw video streams.

Advanced decoders, like ffmpeg, can decode raw video streams. If you only need to pipe the video stream to ffmpeg, raw mode can save some bandwidth and processing time.

When data packet headers are not present, the keyframe and pts fields will be undefined. The data field contains the bytes from one read call; since there are no packet boundaries, it might contain partial or multiple frames.