Repository navigation
perf(web-renderer): overlap HTML-in-canvas capture with worker frame construction - #11861
alec-watts wants to merge 7 commits into
Conversation
![pullfrog[bot]](https://avatars.githubusercontent.com/in/1768019?s=60&v=4){kind=small}
There was a problem hiding this comment.
ℹ️ No critical issues found — a couple of rough edges worth a look, nothing blocking.
Reviewed changes
- New worker capture path (
html-in-canvas-worker.ts) — transfers the layout canvas offscreen, drawsElementImagesnapshots and constructsVideoFrames in a worker; owns startup handshake, bounded pending requests, and cancellation/error cleanup. - Deferred context acquisition (
html-in-canvas.ts) —setupHtmlInCanvasno longer obtains a 2D context up front;waitForPaintis now signal-aware and shared between the main-thread and worker paths. - Overlapped render loop (
render-media-on-web.tsx) — singleunsubmittedFrameslot,pendingEncodingbarrier before current-frame audio mixing/submission, and a drain-on-exitfinallythat outlives encoder sources. - Shared fallback decision (
take-screenshot.ts) —getHtmlInCanvasFallbackReasonextracted and reused by both capture paths. - Docs (
html-in-canvas.mdx) — eligibility, callback limits, and worker-error behavior.
The browser-API assumptions check out: OffscreenCanvasRenderingContext2D includes the drawElementImage/reset mixins, VideoFrame.close() on a transferred (detached) frame is a spec-defined no-op, and requestPaint()/paint remain usable on the main-thread canvas after transferControlToOffscreen(). Frame ownership on the worker path is also clean — the worker closes the transferred frame, the !request branch closes orphan responses, the unsubmittedFrame slot and addVideoSampleAndCloseFrame are mutually exclusive closers, and Promise.allSettled(activeEncodingPromises) keeps sources alive on early exit.
ℹ️ Worker startup failure is now fatal where the prior path recovered
The worker is created eagerly whenever captureElementImage exists (new lines 404–415), and this runs before the main try block. If new Worker(url) or transferControlToOffscreen() throws — for example a strict CSP that blocks blob: workers — the export rejects before the first frame, even though the DOM composer is fully available and nothing has been encoded yet. Before this change, native capture failures were per-frame and fell back to the DOM composer. This is documented as intentional, but the startup case is recoverable, unlike a mid-render worker failure.
Technical details
# Worker startup failure aborts the export instead of falling back
## Affected sites
- `packages/web-renderer/src/render-media-on-web.tsx:404-415` — worker created outside the `try` at line 505; a throw here rejects the whole render.
- `packages/web-renderer/src/html-in-canvas-worker.ts:47` — `new Worker(url)` (blob URL, subject to `worker-src` CSP).
- `packages/web-renderer/src/html-in-canvas-worker.ts:91` — `transferControlToOffscreen()` can throw.
## Required outcome
A failure to start the worker (before any frame is captured) should degrade to the DOM composer for the whole export, matching the previous best-effort behavior, rather than failing the render.
## Suggested approach (optional)
Wrap worker creation (or the `ready` handshake) and on startup rejection set `htmlInCanvasWorker = null` so the loop uses the existing fallback branch (which already handles `htmlInCanvasContext` correctly when the canvas has been transferred). Mid-render worker failures can continue to reject, since already-encoded frames cannot be replayed.
## Open questions for the human
Is the all-or-nothing behavior for startup intended for an opt-in, flag-gated feature, or is graceful fallback preferred?ℹ️ Nitpicks
- Worker creation could also check
getHtmlInCanvasFallbackReason(div) === null, so compositions that always fall back (nested<HtmlInCanvas>, URL masks) don't start a worker or permanently transfer the canvas for zero benefit. URL masks can appear mid-render, so this is only safe for the statically-detectable nested-canvas case. - In the
onFramepath,unsubmittedFramestill references the original frame whenvalidateVideoFramethrows, so the newfinallycallsclose()a second time.VideoFrame.close()is idempotent, so this is harmless, but nullingunsubmittedFramebefore validation would remove the doubt.
DeepSeek Flash ｜ 𝕏
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes since the prior pullfrog review
- Brand CI lint fixes —
packages/brand/src/AudioWaveformProgressComposition.tsxdrops a redundant single-child fragment, andpackages/brand/src/NewComposition.tsxswitches toimport type Reactsince the file only referencesReactas a type (it returnsnull, so there is no JSX/value use). Verified locally with the brand ESLint config: both files lint clean.
No changes to the web-renderer worker/loop code or docs since the previous review; the findings there stand as previously reported.
DeepSeek Flash ｜ 𝕏
Co-authored-by: Cursor <cursoragent@cursor.com>
|
We benchmarked this against a simpler main-thread variant to understand how much of the gain requires a worker. We are still a bit sceptical that introducing a capture worker is justified by the performance evidence so far. We compared three paths:
On an Apple M4 with 24 GiB RAM, headless Chrome 152.0.7977.42 / ANGLE Metal, Remotion 4.0.532 and Mediabunny 1.56.1:
The worker's mean advantage over the persistent main-thread canvas was only 31 ms, or about 0.25%, with much larger variation between runs. The worker beat the existing path in four of six rounds; the persistent canvas beat it in three of six. Individual rounds and methodology
Each export was 1920×1080, 30 fps, 1,800 frames, H.264 MP4, very-high bitrate, muted audio, allowHtmlInCanvas=true, no callbacks, default hardware acceleration preference, and medium page responsiveness. Two generated H.264 videos at distinct URLs filled the frame and a 960×540 region at (960,0), respectively; GOP 60 and two B-frames. One full-length warmup per variant was discarded. Six measured rounds used all permutations of the three variants, with fresh browser contexts/pages and serial execution. Timers included source loading, render setup, worker startup, capture, encoding and finalization, and excluded bundling, page creation, verification and disk saving. Frame-rate limiting, GPU vsync and background throttling were disabled. Detailed profiling was excluded from these export timings. All 21 full-length outputs, including warmups, used native capture for every frame and passed decoded frame-count, timestamp, composition-marker and both source-video-marker checks. Separate exports also produced matching full-frame RGBA hashes at twelve positions across all three paths, for both two-video content and moving semitransparent graphics. Worker requests never exceeded two pending captures. Separate instrumented 300-frame profiles of the two-video workload measured average synchronous VideoFrame construction at roughly 0.13 ms for the existing path, 0.52 ms for the persistent main-thread canvas, and 1.79 ms in the worker. These are call durations, potentially including deferred drawing/synchronization, rather than GPU traces. The canvas backings differ, and these profiling exports are excluded from the throughput comparison. The existing full-length path also spent only about 0.14 s awaiting video/audio submission, averaged across the six runs. This does not disprove the reported 31.1% reduction on your M2 Max / hidden Electron setup: our hardware, host, launch settings and generated assets differ. We checked the baseline revisions, and the relevant renderer/media implementation is unchanged between your benchmark baseline and ours. Your reported saving corresponds to about 4.36 ms per frame, versus about 0.28 ms here, so identifying which work or waiting is being hidden would help explain the difference. Our concern is whether the incremental benefit warrants the additional worker lifecycle, transfers, frame ownership, queue and failure/cancellation handling. Removing the extra canvas copy looks like a simpler candidate to evaluate first, although our results do not establish a consistent speedup for that either. The three-path comparison also does not isolate thread placement from the PR's changes to capture and submission scheduling. Could you share the exact benchmark harness, generated assets and Electron launch flags, and compare against a persistent main-thread canvas? Ideally, stage measurements or a trace would also separate the benefit of submission overlap from moving capture into a worker. That would give us a stronger basis for deciding whether the worker itself is worth introducing. |
# Conflicts: # packages/docs/docs/client-side-rendering/html-in-canvas.mdx # packages/web-renderer/src/take-screenshot.ts
This branch was successfully deployed
3 participants
Native HTML-in-canvas capture can keep the main thread occupied with snapshot drawing and VideoFrame construction while the next React frame is ready to be prepared. This PR moves that capture work to a worker and keeps bounded submission overlap through the existing encoder, muxer, and audio mixer.
Maintainer-requested comparison
The revision includes a standalone replay harness, generated-asset recipe, source hashes, original Electron launch flags, and every individual result in
packages/web-renderer/benchmarks/html-in-canvas-capture. Current main is merged, preserving its Chrome 157 nested-capture capability check.Fresh controlled measurements use the original generated two-video assets and source head
f096ab1826cff02082e39c6f207964d0c84da419, with serial baselinec320056a980972de109ef27a40bede9660a46931. Host: Apple M2 Max, Electron 44.3.0 / Chromium 152.0.7977.78, ANGLE Metal, React 19.2.3, public Mediabunny 1.56.1. Every export is 1920×1080, 30 fps, 1,800 frames, H.264 MP4, very-high bitrate, muted audio, default hardware acceleration preference, and medium page responsiveness.One discarded full-length warmup per variant and four balanced measured rounds, fresh hidden windows:
The worker is 36.4% faster than persistent main-thread capture with submission overlap in this cohort. Copy removal and submission overlap alone do not explain the measured benefit on this host.
A second cohort isolates placement more closely. Both variants use the same transferred canvas, immutable ElementImage snapshots, reset/draw/VideoFrame sequence, conditional resizing, and submission overlap. One warmup per variant and four alternating measured pairs: 22.988 s main thread versus 14.895 s worker. An initial setup that reset main-thread canvas dimensions on every frame was stopped and excluded; the retained comparison matches the worker's resize behavior.
All 30 full-length outputs, including warmups, pass frame count, timestamps, composition markers, and both source-video markers: 54,000 verified output frames. Worker runs capture all 1,800 frames in the worker, keep at most two requests pending, and terminate the worker. Results include phase totals, which overlap and must not be interpreted as additive CPU time. No GPU trace or process-memory measurement is claimed.
The preserved Electron flags enable ANGLE Metal and CanvasDrawElement, and each window disables background throttling. They do not disable frame-rate limiting or GPU vsync. This differs from the maintainer's headless Chrome/M4 run and may help explain the different benefit. These measurements do not establish a universal speedup or contradict that host's roughly 0.25% incremental gain.
Behavior and validation
allowHtmlInCanvasopt-in. Requires transferable captureElementImage snapshots and workers; no new public option.git diff --checkpass. The dry run verifies the replay tools do not ship in the npm package. Full upstream CI remains a separate check.Why each file changes
packages/web-renderer/.npmignorepackages/web-renderer/src/html-in-canvas-worker.tspackages/web-renderer/src/html-in-canvas.tspackages/web-renderer/src/render-media-on-web.tsxpackages/web-renderer/src/take-screenshot.tspackages/docs/docs/client-side-rendering/html-in-canvas.mdxpackages/web-renderer/benchmarks/html-in-canvas-capture/browser.tsxpackages/web-renderer/benchmarks/html-in-canvas-capture/build.tspackages/web-renderer/benchmarks/html-in-canvas-capture/run.cjspackages/web-renderer/benchmarks/html-in-canvas-capture/generate-fixtures.pypackages/web-renderer/benchmarks/html-in-canvas-capture/results.jsonpackages/web-renderer/benchmarks/html-in-canvas-capture/README.mdDiff cleanup retains the render loop's try/finally indentation because it ensures outstanding work drains before encoder-source disposal. Benchmark variants are isolated evidence tools requested by the maintainer; they do not add a second production encoder, muxer, queue, or renderer.
Shared CI test repair
Includes the one-line caption-inspector track selector fix from #12179. Current main fails this test before it can exercise caption persistence. Both caption-inspector E2E cases pass in the actual Studio with this repair; restoring the old selector reproduces the failure.
packages/example/e2e/captions-inspector.test.mts