moq-encoder-player

MOQT version: draft-18 (negotiated via ALPN token moqt-18). Media packaging can be CMSF (CMAF packaging, draft-ietf-moq-cmsf), which is what the encoder and the player default to, or LOC (Low Overhead Media Container) draft-04 + Codecstring; both ends have to be set to the same one, see Packager

This project provides a minimal implementation (inside the browser) of a live video and audio encoder and video / audio player based on MOQT draft, media packaging is based on draft-ietf-moq-cmsf or draft-ietf-moq-loc, the exact versions of the drafts implemented are shown in the UI of the endoder and the player.

The goal if ths code is to provide a minimal live platform implementation that helps learning on low latency trade offs and facilitates experimentation.

It is NOT optimized for performance / production at all since the 1st goal is experimenting / learning.

Fig1: Main block diagram

For the server/relay side we have used moxygen.

Note: You need to be careful and check that protocol versions implemented by this code and moxygen matches

The source code is written in TypeScript and lives under src/. It is compiled with tsc into native ES modules under dist/ (mirroring the src/ tree). The browser demos under demo/ load the compiled output from dist/ directly as ES module Web Workers / AudioWorklets, so you must build the project before running the demos (see Development).

moq-encoder-player/
├── demo/                   # Browser demos (HTML). They load the compiled code from dist/
│   ├── encoder/            #   index.html (full encoder), simple.html
│   ├── player/             #   index.html (full player), simple.html
│   └── shared/             #   demo.css (styles shared by the demos)
├── src/                    # TypeScript source code
│   ├── index.ts            #   Library entry point (re-exports the reusable modules)
│   ├── capture/            #   a_capture.ts, v_capture.ts        (Web Workers)
│   ├── encode/             #   a_encoder.ts, v_encoder.ts        (Web Workers)
│   ├── decode/             #   audio_decoder.ts, video_decoder.ts (Web Workers)
│   ├── moq/                #   moq.ts (high-level Moq/Track/Subscription client),
│   │                       #   moqt.ts (wire protocol), varint.ts, byte_utils.ts, buffer_utils.ts,
│   │                       #   network_simulator.ts (send-path drop/hold impairments), README.md
│   ├── sender/             #   moq_sender.ts (worker shell) + moq/moq_sender_internals.ts   (MOQT publisher)
│   ├── receiver/           #   moq_demuxer_downloader.ts (worker shell) + moq/moq_receiver_internals.ts (MOQT subscriber)
│   ├── packager/           #   media_packager.ts (common interface + factory),
│   │   │                   #   loc_packager.ts                   (LOC media packager)
│   │   └── cmaf/           #   cmaf_packager.ts / cmaf_depackager.ts, cmaf_init_segment.ts,
│   │                       #   box_writer.ts / box_reader.ts        (CMSF media packager)
│   ├── overlay_processor/  #   overlay_encoder.ts / overlay_decoder.ts (pixel latency overlay)
│   ├── render/             #   audio_player.ts (Web Audio renderer), playback_rate_controller.ts,
│   │                       #   video_render_buffer.ts
│   ├── utils/              #   jitter_buffer.ts, ts_queue.ts, avg_last_n_items.ts, utils.ts,
│   │                       #   media_dumper.ts (save packaged objects to a local file),
│   │   └── media/          #   avcc_parser.ts, avc_decoder_configuration_record_parser.ts
│   └── types/              #   globals.d.ts (ambient types for WebTransport / WebCodecs)
├── tests/                  # Jest unit tests for the pure utilities
├── dist/                   # Compiled JavaScript + type declarations (generated, git-ignored)
├── .github/workflows/      # CI: lint + build + test
├── tsconfig.json           # TypeScript compiler options
├── jest.config.js          # Test runner configuration
├── eslint.config.js        # ESLint configuration (flat config)
├── .prettierrc             # Prettier configuration
└── package.json            # NPM dependencies, scripts and metadata

Requirements: Node.js 18+ (for the toolchain) and Python 3 (for the local dev web server). The included dev server also sets cross-origin-isolation headers, but the player no longer requires them (audio playback dropped SharedArrayBuffer); any static server over HTTPS works.

Install dependencies once:

npm install

# 1. Compile TypeScript -> dist/ and start the cross-origin-isolated web server on :8080
npm run dev

npm run dev runs npm run build followed by npm run serve. While iterating on the TypeScript you can keep the compiler running in watch mode in one terminal and the server in another:

npm run build:watch   # terminal 1: re-compile on every change
npm run serve         # terminal 2: serve the repo on http://localhost:8080

Then open the demos (see Testing below for the full flow):

• Encoder: http://localhost:8080/demo/encoder/?local
• Player: http://localhost:8080/demo/player/?local

npm run build     # type-checks and emits dist/*.js + dist/*.d.ts (declarations)

The contents of dist/ are everything needed at runtime (the demos and any external consumer import from there). npm run clean removes the dist/ folder.

npm test          # run the Jest unit test suite once
npm run test:watch

npm run lint          # ESLint
npm run lint:fix
npm run format        # Prettier (write)
npm run format:check

CI (GitHub Actions, see .github/workflows/main.yml) runs lint, build and test on every push / pull request.

Two media packagers are implemented behind a common pair of interfaces in src/packager/media_packager.ts: MediaPackager on the publisher and MediaDepackager on the subscriber. The encoder can publish with either one and the player can receive either one, chosen from the "Media packager" dropdown in the encoder demo and the "Media packager expected" selector in the player. Both demos default to CMSF (the library falls back to LOC when packagerFormat is not set), and there is no catalog to negotiate the choice, so the two ends have to agree.

It uses draft-ietf-moq-loc draft-04 plus the Codecstring property (ID 0x11), which draft-04 does not register.

That addition is required, not optional: LOC puts no media type on the wire (a catalog is meant to supply it) and this project implements no catalog, so Codecstring is the only thing that tells the player which codec to configure its decoders with. A plain draft-04 peer will not interoperate with this implementation. See src/packager/loc_packager.ts for the full property set.

It follows draft-ietf-moq-cmsf (CMAF packaging for MoQ, written against the individual draft the working group adopted, draft-wilaw-moq-cmafpackaging-01) with the box syntax of CMAF (ISO/IEC 23000-19) and ISOBMFF: each MoQ object payload is a self-describing ISOBMFF media fragment and no MoQ Object Properties are sent at all. See src/packager/cmaf/ for the mapping and the two documented deviations from the draft.

The encoder implements MOQT publisher role. It is based on Webcodecs, and AudioContext, see the block diagram in fig3

Fig3: Encoder block diagram

Note: We have used WebTransport, so the underlying transport is QUIC (QUIC streams to be more accurate)

Video encoding config:

// Video encoder config
const videoEncoderConfig = {
        encoderConfig: {
            codec: 'avc1.42001e', // Baseline = 66, level 30 (see: https://en.wikipedia.org/wiki/Advanced_Video_Coding)
            width: 320,
            height: 180,
            bitrate: 1_000_000, // 1 Mbps
            framerate: 30,
            latencyMode: 'realtime', // Sends 1 chunk per frame
        },
        encoderMaxQueueSize: 2,
        keyframeEvery: 60,
    };

Audio encoder config:

// Audio encoder config
const audioEncoderConfig = {
        encoderConfig: {
            codec: 'opus', // AAC NOT implemented YET (it is in their roadmap)
            sampleRate: 48000, // To fill later
            numberOfChannels: 1, // To fill later
            bitrate: 32000,
            opus: { // See https://www.w3.org/TR/webcodecs-opus-codec-registration/
                frameDuration: 10000 // In us. Lower latency than default = 20000
            }
        },
        encoderMaxQueueSize: 10,
    };

Muxer config:

const muxerSenderConfig = {
        urlHostPort: '',
        urlPath: '',

        keepAlivesEveryMs: 5000,

        certificateHash: null,

        // Announce each namespace once with PUBLISH_NAMESPACE and serve tracks
        // lazily on subscribe, instead of one PUBLISH per track.
        usePublishNamespace: true,

        // Media packaging format for every track: "cmaf" (what the demo's
        // "Media packager" dropdown selects by default) or "loc", which is also
        // the fallback when this field is missing (see Packager)
        packagerFormat: 'cmaf',

        moqTracks: {
            "audio": {
                namespace: ["vc"],               // namespace tuple (array of segments)
                name: "audio0",
                maxInFlightRequests: 20,          // caps the per-track send queue
                maxOpenStreams: 60,               // caps concurrent open subgroup streams
                isHipri: true,
                authInfo: "secret",
                moqMapping: MOQ_MAPPING_SUBGROUP_PER_GROUP, // or MOQ_MAPPING_OBJECT_PER_DATAGRAM
            },
            "video": {
                namespace: ["vc"],
                name: "video0",
                maxInFlightRequests: 10,
                maxOpenStreams: 39,
                isHipri: false,
                authInfo: "secret",
                moqMapping: MOQ_MAPPING_SUBGROUP_PER_GROUP,
            }
        },
    }

moqMapping selects how objects hit the QUIC wire (see src/moq/README.md): MOQ_MAPPING_SUBGROUP_PER_GROUP (one unidirectional stream per group) or MOQ_MAPPING_OBJECT_PER_DATAGRAM (one datagram per object).

Video is always one subgroup per GOP (an IFrame does not fit in a datagram, so offering datagrams there would mean dropping almost every IFrame). Audio frames are all independent, so how many of them share a group is a free transport choice: newSubgroupEvery (the "MOQ audio packager" dropdown: 1, 5 or 10 frames per subgroup) trades fewer streams and less per-object overhead against losing a whole group at once. With CMSF the dropdown drops the datagram option and defaults to 10 frames — its objects carry ~140 bytes of boxes each, so a stream per 20ms frame is wasteful — and the sender rejects a CMSF track configured for datagrams.

Main encoder webpage and also glues all encoder pieces together

• Before capture starts:

  • Creates the audio and video capture workers and waits for a ready message from both
  • Only then creates and transfers the MediaStreamTrackProcessor streams. This keeps remote worker-module download time out of the A/V timeline
  • Records a shared wall-clock origin immediately before starting both capture readers

• When it receives an audio OR video raw frame from a_capture or v_capture:

  • Records one capture anchor per stream: that stream's first WebCodecs timestamp paired with the wall clock sampled when its capture worker read the frame
  • Maps each stream onto the common presentation timeline as (timestamp - firstTimestamp) + (firstCaptureClock - sharedOrigin). Audio and video may start at different times and may use unrelated raw timestamp origins
  • Sends it to the encoder (every video frame also carries its frame-read wall clock so the latency overlay can stamp it — see overlay_processor)

• When it receives an audio OR video encoded chunk from a_encoder or v_encoder:

  • Computes its A/V-aligned presentation timestamp from that stream's anchor
  • Sends the chunk (augmented with seqId and metadata) to the muxer

It also owns the "Media packager" dropdown (CMSF, the UI name of the CMAF packaging, by default, or LOC — see Packager), which is where the published format is selected; the player has its own "Media packager expected" selector that has to match. Selecting CMSF also locks the video QUIC mapping to subgroup per GOP and rebuilds the audio mapping options (no datagrams, 10 frames per subgroup by default), which is the grouping the CMAF mapping assumes.

To analyse what the selected packager actually puts on the wire, the encoder can save the packaged objects to a file. Set "Save the first N seconds" in Advanced > Save media to a local file (dumper) (0, the default, disables it) and pick which media types to capture.

A second limit applies at the same time: the capture also stops at DUMP_MAX_OBJECTS objects per media type (one object = one frame), so a long / high-frame-rate session cannot exhaust memory. The box shows that cap and, once a file is written, how many objects and how many seconds it actually holds (flagged when the object cap truncated it). The N seconds are media seconds, measured from the chunk timestamps of the stream being captured.

Each media type is downloaded when its N seconds are captured or when you press Stop (capturing both means two downloads, so the browser may ask to allow multiple files). With CMSF the file is cmaf-<mediaType>.mp4, with LOC it is loc-<mediaType>.bin. The capture is independent of the transport: chunks are packaged and written to the dump even when there is no MoQ session or no subscriber, so a file can be produced with no relay running at all. The same capture can be driven by hand mid-session from the console:

armMediaDump('video'); // start capturing at the next group boundary (also 'audio')
dumpMedia('video'); // downloads everything captured so far

Either way the capture starts on a group boundary, so for CMSF it carries the CMAF Header and the downloaded concatenation of object payloads is a playable fragmented MP4: ffprobe -count_frames cmaf-video.mp4, ffplay cmaf-video.mp4, or any MP4 box analyzer. The capture logic itself lives in src/utils/media_dumper.ts (MediaDumper) and works with any packager.

Stamps an integer value (the wall-clock epoch sampled when the capture worker read the frame, in ms) into the top rows of a raw video frame by writing one bright/dark pixel run per bit, prefixed with a marker sequence. The value survives H.264 encode/decode as image content, so the player can recover it and estimate capture-read-to-render latency without any side-channel metadata (see OverlayDecoder on the player side). It requires an NV12 raw frame; the encoder toggles it live from the "Add latency information in video" checkbox (on by default) and falls back to the un-overlaid frame if the source format differs.

WebWorker that announces when its module is ready, then waits for RGB or YUV frames from the capture device. Each frame is sent to the main encoder page with the wall clock sampled when the processor read completed; the page forwards the frame and clock to the video encoder.

WebWorker that announces when its module is ready, then reads PCM audio frames (typically 10–25 ms of samples) from the capture device. Each frame is structured-cloned to the main encoder page with the wall clock sampled when the processor read completed; the page forwards it to the audio encoder.

WebWorker Encodes RGB or YUV video frames into encoded video chunks

• Receives the video RGB or YUV frame from v_capture.ts
• Adds the video frame to a queue. And it keeps the queue smaller than encodeQueueSize (that helps when encoder is overwhelmed)
• Specifies I frames based on config var keyframeEvery
• It delivers the encoded chunks to the next stage (muxer)

Note: We configure VideoEncoder in realtime latency mode, so it delivers a chunk per video frame

WebWorker Encodes PCM audio frames (samples) into encoded audio chunks

• Receives the audio PCM frame from a_capture.ts
• Adds the audio frame to a queue. And it keeps the queue smaller than encodeQueueSize (that helps when encoder is overwhelmed)
• It delivers the encoded chunks to the next stage (muxer)

Note: opus.frameDuration and opus.application: 'lowdelay' setting helps keeping encoding latency low

• Implements draft-ietf-moq-loc draft-04 + Codecstring (the Codecstring property is a required addition here, not an optional extra — see Packager)

The MoQ Object Payload is the LOC Payload: the "internal data" of an EncodedVideoChunk / EncodedAudioChunk, with no extra framing. The metadata describing it travels as MoQ Object Properties:

CODECSTRING is the non-draft-04 addition; every other property is registered in the draft's IANA table.

• VIDEO_FRAME_MARKING is the 1-byte RFC 9626 short form; its Independent bit is what tells the player a key frame from a delta frame
• VIDEO_CONFIG / AUDIO_CONFIG carry the WebCodecs decoder description (an AVCDecoderConfigurationRecord, an OpusHead, or an AAC AudioSpecificConfig). The player recovers the audio sample rate and channel count from it — see src/utils/media/audio_decoder_config_parser.ts
• For video the codec is described twice: by CODECSTRING and, implicitly, by the profile / constraint flags / level inside VIDEO_CONFIG. The player configures from CODECSTRING and logs a warning if the two disagree
• LOC has no media type on the wire (that is a catalog's job), so the publisher and the player both take it from their own per-track config
• LOC covers audio and video only. The data track used by the simple.html demos is an opaque payload with no properties

• Implements draft-ietf-moq-cmsf (written against draft-wilaw-moq-cmafpackaging-01), boxes per CMAF (ISO/IEC 23000-19) and ISOBMFF (ISO/IEC 14496-12), on both the publisher and the subscriber side

Mapping (draft §4.2, "CMAF Chunk to MOQT Object"):

1 encoded frame = 1 CMAF chunk (moof + mdat, one sample) = 1 MOQT Object
1 CMAF fragment (GOP)                                    = 1 MOQT Group

That is the same grouping the LOC path already uses (a new group starts on every key frame), so the MoQ / QUIC layer, the priorities and the moqMapping options are unchanged.

Each object payload is [ftyp moov] styp moof mdat, and there are two deliberate deviations from the draft:

• Initialization header (§6). The draft delivers the CMAF Header out of band (§6.1) or as a dedicated init MOQT track (§6.2). This project has no catalog and no out-of-band channel, so the header is instead prepended to the first object of a group, which makes that group self-initializing. It is repeated at most every initRepeatEveryMs (1s by default) — without that limit, per-frame audio groups would carry ~600 bytes of moov for every 20ms of Opus
• Object payload (§3). A styp opens every object, as the draft requires, but the self-initializing prefix above sits in front of it on the objects that carry a header

Other details:

• CMAF is self-describing, so no MoQ Object Properties are sent: timing lives in tfdt / trun, the codec in the sample entry, and key frames in the trun sample flags (sample_depends_on / sample_is_non_sync_sample)
• Media timescale: video keeps the source (WebCodecs, microsecond) timebase, which the caller must provide; audio uses its sample rate, as CMAF §7.5.13 recommends. Timestamps are converted per chunk from the absolute source timestamp, so rounding cannot accumulate
• Sample duration comes from the WebCodecs chunk when the encoder reports one, otherwise from the interval since the previous chunk (a live stream has no lookahead). tfdt is exact in both cases
• The samples go into mdat untouched, which requires the WebCodecs default AVC format (avc: length-prefixed AVCC, not Annex-B)
• CMAF covers audio and video only; asking for any other media type (an opaque data track) is an error rather than a silent fallback to another format
• If the object carrying a header is dropped by the send queue / stream caps, the subscriber simply waits for the next repeat

WebWorker that implements the MOQT publisher role and sends video and audio packets (see loc_packager.ts / cmaf/cmaf_packager.ts) to the server / relay following MOQT and the selected packaging format.

moq_sender.ts is a thin worker shell; the publisher logic lives in MoqSender (src/sender/moq/moq_sender_internals.ts), which drives the shared, media-free Moq client in src/moq/moq.ts (fully documented in src/moq/README.md).

• Opens a WebTransport session against the relay (MOQT version negotiated via ALPN)
• Announces its track(s): either one PUBLISH per track, or a single PUBLISH_NAMESPACE per namespace serving tracks lazily on subscribe (usePublishNamespace)
• Receives audio and video chunks from a_encoder.ts and v_encoder.ts and publishes each as a MoQ object via track.sendObject(...)
• Packages them with LOC or CMAF depending on packagerFormat (the demo sends "cmaf"; an unset field falls back to "loc"). One packager instance is kept per media type for the whole session, because the CMAF one is stateful (moof sequence numbers, initialization header)
• Object → QUIC wire mapping is configurable per track (moqMapping): SubgroupPerGroup opens one unidirectional QUIC stream per group (a video keyframe starts a new group/stream), while ObjectPerDatagram sends one datagram per object
• Send priority uses the MoQ publisher priority carried on each group; audio is published at a higher priority than video (lower numeric value = higher priority)
• It keeps the per-track send queue below maxInFlightRequests and the concurrent open subgroup streams below maxOpenStreams (objects / whole groups are dropped once the respective cap is reached). Two stats are reported per track: numQueued (objects waiting in the send queue) and numOpenStreams (open QUIC subgroup streams)
• Optional send-path impairments (src/moq/network_simulator.ts) can drop or hold bursts of wire units to test A/V sync and loss recovery; both are exposed from the encoder UI

The encoder implements MOQT subscriber role. It uses Webcodecs and AudioContext (audio is scheduled with AudioBufferSourceNode on the AudioContext clock — no SharedArrayBuffer or AudioWorklet)

Fig5: Player block diagram

The packaging it expects is selected with the "Media packager expected" dropdown under the track name (CMSF by default, LOC being the other option) and has to match what the encoder publishes: nothing on the wire announces the format and there is no catalog to negotiate it.

To keep the audio and video in-sync the following strategy is applied:

• Audio renderer (audio_player.ts, GapTolerantPlayer) schedules each decoded AudioData frame on the AudioContext clock and exposes the media timestamp currently sounding at the speakers (already latency-adjusted) via its playingTimestamp stat. The player page mirrors it into timingInfo.renderer.currentAudioTS.
• Every time the stats callback fires (and in the render loop) the video renderer video_render_buffer (who contains YUV/RGB frames + timestamps) gets called and:
  • Returns / paints the oldest closest (or equal) frame to current audio ts (timingInfo.renderer.currentAudioTS)
  • Discards (frees) all frames older current ts (except the returned one)
• AudioDecoder does NOT track timestamps, it just uses the 1st one sent and at every decoded audio sample adds 1/fs (so sample time). Rather than compute an explicit gap offset, audio_decoder.ts mirrors the decoder's input queue and reconciles it on the dequeue event so each output frame carries the true source timestamp of the chunk that produced it; GapTolerantPlayer re-anchors media time to that timestamp whenever a new contiguous segment starts (after an underrun/gap).

WebWorker entry point. It is a thin shell that forwards worker messages to the MoqReceiver class in src/receiver/moq/moq_receiver_internals.ts, mirroring the publisher layout (src/sender/).

The MOQT subscriber logic is split in two layers:

• src/moq/moq.ts — the high-level, media-free Moq client (shared with the publisher). It owns the WebTransport session, the control loop, the SUBSCRIBE handshake (Moq.subscribe → Subscription), and the incoming stream / datagram receive loops. Received object payloads are routed to the matching Subscription by track alias.
• src/receiver/moq/moq_receiver_internals.ts — MoqReceiver translates worker messages into Moq calls and parses the received payloads with the depackager the packagerFormat config asks for (loc_packager.ts or cmaf/cmaf_depackager.ts, see Packager) into EncodedVideoChunk / EncodedAudioChunk for the rest of the player pipeline. One depackager instance is kept per track for the whole subscription, because the CMSF one is stateful (it remembers the CMAF Header).

It implements MOQT and extracts video and audio packets from the server / relay following MOQT and the selected packaging format:

• Opens WebTransport session
• Implements MOQT subscriber handshake for 2 tracks (video and audio)
• Waits for incoming unidirectional (Server -> Player) QUIC streams (and datagrams)
• For every received chunk (QUIC stream) we:
  • Parse it with the track's depackager (loc_packager.ts or cmaf/cmaf_depackager.ts)
  • Video: Create EncodedVideoChunk
  • Audio: Create EncodedAudioChunk
  • The timestamp is converted from the timescale the publisher stated (LOC Timescale property, or the CMSF mdhd) into the per-track timebase the player pipeline runs at

With CMSF the track description (timescale, codec, decoder configuration) travels inside the media, in the CMAF Header that rides the objects starting a group. A player that joins mid-group therefore has nothing to configure its decoders with: those objects are dropped, with one log line per media type, until the first header arrives (at most ~1s later, see initRepeatEveryMs).

The encoder measures what it costs to put each subgroup on the wire and reports it per track: the instant figures (payload bytes, overhead bytes and overhead %) live in the "Data overhead" tab, and a "<track name> overhead (last 60s)" chart per track plots the trend (overhead % on the left axis, overhead bytes on the right). The percentage is relative to the payload (overhead / payload), so it goes over 100% when the overhead is bigger than the media itself — which is the normal case for small audio frames.

• Payload = the encoded media bytes the encoder produced
• Overhead = the packager (CMSF boxes and the periodic CMAF Header; LOC adds nothing to the payload) + the MoQ signaling the track counted (subgroup header, per-object headers, object properties, end-of-group marker)

Both come from the bytes actually written — Track accumulates them per wire unit from the byte counts the moqSend* helpers return (see SubgroupBytes in src/moq/moq.ts) — so objects the send queue, the open-stream cap or a drop simulator skipped are not counted.

For 40 byte Opus frames, one frame per subgroup, that works out at roughly 47 bytes of MoQ signaling per object with LOC (its properties carry the timestamp, timescale, codec string and the audio config), and ~140 bytes of boxes per object with CMSF. Grouping 10 frames per subgroup amortizes the subgroup header but not the per-object cost.

Since we do not have any guarantee that QUIC streams are delivered in order we need to order them before sending them to the decoder. This is the function of the deJitter. We create one instance per track, in this case one for Audio, one for video

• Receives the chunks from moq_demuxer_downloader.ts
• Adds them into a sorted list ordered by the MoQ transport-native key (groupId, objId) (lexicographic), which reproduces the publisher's send order
• When list length (in ms is > bufferSizeMs) we deliver (remove) the 1st element in the list
• It also keeps track of the last delivered (groupId, objId) detecting:
  • Gaps / discontinuities
  • Total QUIC Stream lost (not arrived in time)

WebWorker that decodes each audio chunk and posts the decoded AudioData frames (with a timestamp) to the main-thread renderer. AudioDecoder does NOT track timestamps on decoded data, it just uses the 1st one sent and at every decoded audio sample adds 1/fs (so sample time). That means a dropped audio packet would collapse the timeline and desync A/V.

To recover the true position it mirrors the decoder's input queue (pendingTs) and reconciles it on the dequeue event:

• Receives audio chunk → push chunk.timestamp to pendingTs, then decode().
• On dequeue: consumed = pendingTs.length - decodeQueueSize; the most recent consumed chunk's timestamp becomes the timestamp for the frames about to be output.
• Posts { type: 'aframe', frame, ts } — ts is the true source timestamp of the chunk that produced the frame. The renderer uses it to anchor media time on a resume, superseding the old explicit gap-offset compensation.

GapTolerantPlayer — the Web Audio renderer. It keeps a nextPlayTime cursor on the AudioContext clock and schedules each decoded frame with an AudioBufferSourceNode:

• addFrame(audioData, ts): converts the AudioData to an AudioBuffer and starts it at nextPlayTime, then advances the cursor by the frame's duration.
• Gap tolerance: if the network stalls, nextPlayTime falls into the past; the player resumes at currentTime (clamped) so late audio plays immediately instead of piling up. A real gap re-pads the jitterDelay cushion and re-anchors media time to the incoming frame's ts.
• Exposes playingTimestamp (media time currently at the speakers, already latency-adjusted) via its onStats callback — the A/V master clock.

No SharedArrayBuffer, Atomics, or AudioWorklet are used, so the player no longer needs cross-origin isolation.

• Reports last PTS rendered (this is used to sync video to the audio track, so to keep A/V in sync)

PlaybackRateController keeps the audio render buffer (a proxy for latency) near a configurable target by nudging the playback speed. It is a hysteresis controller (decision-only): when the buffer leaves an on-target band it commands GapTolerantPlayer.setPlaybackSpeed to speed up (drain an over-full buffer) or slow down (refill an under-full one), holding the correction until the buffer crosses back to the target. The player UI exposes the target latency, on-target band, and speed-up / slow-down rates, and lets you toggle speed compensation on/off. (Note: changing playbackRate also shifts pitch.)

WebWorker, Decodes video chunks and sends the decoded data (YUV or RGB) to the next stage (video_render_buffer.ts)

• Initializes video decoder with init segment
• Sends video chunks to video decoder
  • If it detects a discontinuity drops all video frames until next IDR frame
• Sends the decoded frame to video_render_buffer.ts

Buffer that stores video decoded frames

• Received video decoded frames
• Allows the retrieval of video decoded frames via timestamps
  • Automatically drops all video frames that older than the currently requested

Recovers the integer value (frame-read wall-clock epoch in ms) that OverlayEncoder wrote into the top rows of a frame, reading one bright/dark pixel run per bit. It only trusts the value when the marker sequence is present (so ordinary, non-overlaid frames are ignored) and returns a confidence flag alongside the value. It requires a decoded I420 frame and does not close it (the caller still owns it).

Video capture-read-to-render latency is estimated with the pixel overlay, not a side-channel:

• The encoder stamps the wall-clock epoch sampled when the capture worker read the frame (ms) into each frame's top rows (OverlayEncoder), enabled from the encoder's "Add latency information in video" checkbox.
• The player recovers it from the displayed frame (OverlayDecoder) and computes videoLatencyMs = Date.now() - recoveredEpoch.
• Because the overlay carries a marker sequence, the player only trusts a value when the marker is present; it shows a rolling recovery-confidence % and stops the extractor on error.

This excludes capture-device and browser pipeline time before the processor yields the frame, so it is not a full physical glass-to-glass measurement. Encoder and player clocks must also be synchronized; using the same computer avoids inter-machine clock skew.

• Clone this repo

git clone [email protected]:facebookexperimental/moq-encoder-player.git

• Install Node.js 18+ and Python

• Install dependencies and build the TypeScript into dist/:

npm install
npm run build

• Run local webserver by calling:

./start-http-server-cross-origin-isolated.py

Note: It is better to run webserver using this script (or npm run serve) but you can use any webserver you like to publish the . directory (repo directory). The demos load the compiled code from dist/, so remember to (re)run npm run build after changing any TypeScript.

• Load encoder webpage, url: http://localhost:8080/demo/encoder/?local
  • Click "Start"
• Load player webpage, url: http://localhost:8080/demo/player/?local
  • Copy Track Name from encoder webpage and paste it into Receiver demuxer Track Name
  • Check that "Media packager expected" matches the encoder's "Media packager" (both default to CMSF, nothing on the wire negotiates it)
  • Click "Start"

ENJOY YOUR POCing!!! :-)

Fig6: Encoder UI

Fig7: Player UI

Note: This is an experimentation code, we plan the evolve it quick, so those screenshots could be a bit outdated

• Create key, certificate, and certificate fingerprint by running following script

./create_self_signed_certs.sh

Note: The trick here is that this script will create a self signed certificate for localhost with EDCSA and validity of 10 days (<15), this is the type Chrome will accept.

• Follow the installation instructions of moxygen.

  • Remember to use key and certificate created on the previous step to run moxygen

• Clone this repo

git clone [email protected]:facebookexperimental/moq-encoder-player.git

• Install Node.js 18+ and Python

• Install dependencies and build the TypeScript into dist/:

npm install
npm run build

• Run local webserver by calling:

./start-http-server-cross-origin-isolated.py

Note: this script adds cross-origin-isolation headers. The player no longer requires them (audio playback dropped SharedArrayBuffer), so any static HTTPS server works — but this script remains a convenient default.

• Load encoder webpage, url: http://localhost:8080/demo/encoder/?local
  • Click "Start"
• Load player webpage, url: http://localhost:8080/demo/player/?local
  • Copy Track Name from encoder webpage and paste it into Receiver demuxer Track Name
  • Click "Start"

ENJOY YOUR POCing!!! :-)

You should see same UI that is shown in testing section above

• Check token in all messages, not just when encoder receives SUBSCRIBE

• Encoder: Cancel QUIC stream after some reasonable time (?) in mode live

• Player: Do not use main thread for anything except reporting

• Player/server: Cancel QUIC stream if arrives after jitter buffer

• Accept B frames (DTS)

• DONE When it drops 100+ audio streams it breaks not recovering (I’m guessing it is trying to send 100 streams at same time hitting browser limit):

  • Moxygen issue: Moxygen relay doesn't queue when it's out of stream credit, instead it fails

moq-encoder-player is released under the MIT License.