Webspace Lip Syncer

A standalone browser lip-sync engine that converts streaming mono PCM audio into timestamped viseme frames. It has no dependency on Webspace Engine, WebRTC, insertable streams, or networked-aframe.

The package is Apache-2.0. Bundled third-party components are permissively licensed and listed in THIRD_PARTY_NOTICES.txt. See PROVENANCE.md for the authorship and relicensing review.

Try the record/playback example

npm install
npm run dev

The dev command builds the package and serves /example/ at the URL printed in the terminal. Grant microphone access, record a short sample, then play it back. The example records PCM from the same AudioWorklet sample clock used for inference, so playback audio and viseme timestamps share an exact origin rather than relying on MediaRecorder container timing. It animates the included mouth SVGs during both live recording and playback. The SVGs use a vertical correction for their original sphere-texture presentation; adjust scaleY(0.72) in example/style.css if a different avatar surface needs another correction.

Microphone access requires localhost or HTTPS in current browsers.

Web Audio API

import { WebAudioLipSyncer } from "webspace-lip-syncer";

const context = new AudioContext();
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
const source = context.createMediaStreamSource(stream);
const lipSyncer = await WebAudioLipSyncer.create(context, { sourceNode: source });

lipSyncer.addEventListener("viseme", event => {
  const { viseme, timestamp, effectiveTimestamp, confidence, logit, speaking, levelDb, changed } = event.detail;
  // Apply `viseme` to an avatar or serialize the frame for transport.
});

// Later:
lipSyncer.destroy();
stream.getTracks().forEach(track => track.stop());

WebAudioLipSyncer captures audio through an AudioWorkletNode, transfers chunks directly to an inference worker over a MessagePort, and keeps the capture branch silent. It does not alter or own the source node's other audio connections. By default it also reproduces the legacy Webspace analysis conditioning: 3x gain followed by a Web Audio compressor with a -12 dB threshold, 0 dB knee, 20:1 ratio, 5 ms attack, and 50 ms release. Pass conditionAudio: false if the source has already been conditioned or an integration needs unmodified PCM. This graph was part of the historical browser deployment, but it is not part of the recovered torchaudio training transform.

createPcmTap() returns a MessagePort carrying the exact conditioned PCM chunks sent to inference ({ type: "audio", samples, timestamp }). Call detachPcmTap() to flush the final partial chunk and receive a stopped message. The example uses this tap to make its synchronized WAV recording.

Raw PCM API

Use PcmLipSyncer when an application already has PCM samples:

import { PcmLipSyncer } from "webspace-lip-syncer";

const lipSyncer = await PcmLipSyncer.create({ inputSampleRate: 48000 });
lipSyncer.addEventListener("viseme", event => console.log(event.detail));

// Push continuous mono samples. `timestamp` is the time of the first sample.
lipSyncer.push(float32Samples, timestampSeconds);

The frontend accepts any positive integer input sample rate and performs streaming sinc resampling to the model's native 16 kHz rate. It emits causal 40-bin log-mel frames every 10 ms. The model runs at 50 Hz and its prediction is held for the intervening 10 ms frame, so applications still receive events at 100 Hz. Successive pushes form one continuous stream; call reset() after a seek or discontinuity. setSpeaking(true | false) overrides energy-based silence detection, and setSpeaking(null) restores the internal detector. Raw-PCM integrations should make an explicit conditioning choice and keep it consistent between evaluation and production.

If a bundler does not preserve package-relative assets, pass explicit modelUrl, workerUrl, workletUrl, and wasmBaseUrl options. Keeping those URLs configurable is what lets the inference core work with ordinary static hosting as well as application-specific asset pipelines.

Viseme event

Each viseme event contains:

interface VisemeFrame {
  viseme: number;       // 0-11; 0 is neutral during silence
  timestamp: number;    // seconds on the input/audio-context timeline
  effectiveTimestamp: number; // onset estimate, corrected for target delay and smoothing
  confidence: number;   // winning softmax probability
  logit: number | null; // raw winning score; null for silence-gate output
  speaking: boolean;
  levelDb: number;
  changed: boolean;     // true when the smoothed output changes
}

Architecture

The reusable seam is deliberately below application transport:

AudioNode or Float32 PCM
        |
AudioWorklet capture (Web Audio adapter only)
        |
PCM worker: sinc resample -> causal log-mel -> stride-2 GRU -> smoothing
        |
timestamped VisemeFrame events
        |
application-owned avatar, WebRTC metadata, recording, etc.

The package owns only PCM-to-viseme inference. Third parties can choose how to render or transport the resulting events without importing any Webspace Engine audio graph or networking code. RNNoise VAD, acoustic echo cancellation, WebRTC metadata packing, and insertable-stream handling remain outside this package boundary.

Build and verification

npm test
npm run build
npm run check

The build bundles the worker and ONNX Runtime, copies the single-threaded WASM asset used by inference, and generates notices from the dependency inputs actually included in the bundles. Model and frontend regression tests compare against PyTorch and torchaudio reference outputs.

To reproduce model/model.onnx from a compatible training checkpoint, use a Python environment with PyTorch and ONNX installed:

npm run export-model -- path/to/best.pt