feat(upload): reject video outside the board's hardware decode envelope - #3309
Open
mickzijdel wants to merge 6 commits into
Open
feat(upload): reject video outside the board's hardware decode envelope#3309mickzijdel wants to merge 6 commits into
mickzijdel wants to merge 6 commits into
Conversation
The codec gate answers "can this board hardware-decode this codec",
and stops there. It has no opinion on the stream itself, so a
3840x2160 H.264 clip uploaded to a Pi 4 passes cleanly: H.264 is in
pi4-64's supported set, the row lands in rotation, and the screen
plays it at roughly 4 fps. Nothing in the pipeline says a word, so the
first anyone hears about it is the screen.
Measured on a Pi 4B, a 4K High L5.1 clip at 116 Mbps against a 1080p
re-encode of the same content:
4K 1080p
hardware decode REFUSED 75 fps
software, 4 cores 10 fps 37 fps
software, 1 thread 3 fps 12 fps
The 4K clip does not decode slowly in hardware, it never gets in.
h264_v4l2m2m fails outright because 3840 is past the device's frame
bound, so libavcodec drops to software, where 10 fps on an idle
four-core box is already under the 25 fps the clip needs — before the
viewer spends anything on presentation.
This adds the missing check as a standalone module, with no callers
yet. Two tiers, because the evidence differs in kind. BLOCKING covers
driver-enforced facts: bcm2835-v4l2-codec.c pins MAX_W_CODEC and
MAX_H_CODEC at 1920 and restricts the capture queue to 8-bit 4:2:0, so
an oversized or High 10 stream is refused by VIDIOC_S_FMT rather than
merely being slow. Confirmed by asking the hardware — v4l2-ctl reports
"Stepwise 32x32 - 1920x1920 with step 2/2" and YU12/YV12/NV12/NV21/
NC12. Every board decoding H.264 through that device inherits it: pi2
and pi3 via GStreamer, pi3-64 via the kmssink overlay, pi4-64 via
QtMultimedia. ADVISORY covers judgement calls — currently only
software H.264 above 1080p on a Pi 5, which has no H.264 block at all.
Deliberately NOT gating on the declared H.264 level, which is the
obvious-looking rule and the wrong one. The driver exposes
V4L2_CID_MPEG_VIDEO_H264_LEVEL read-only and never validates the
bitstream against it. Plenty of 1080p files carry an inflated level
tag and play perfectly, so the level is a symptom of an oversized
stream, never the cause. Recorded for diagnostics; not branched on.
Deliberately no bitrate rule either. One was drafted at the Level 4.2
ceiling and measurement killed it: 1080p25 through h264_v4l2m2m runs
93 fps at ~9 Mbps and still 62 fps at ~137 Mbps, so it would only have
flagged files that play fine.
Every predicate fails open. An unmeasured dimension, an unparseable
pixel format or an uncharacterised board yields no warning, and the
pixel-format check is a denylist of formats known to be unsupported
rather than an allowlist of the ones we thought of. A false positive
costs an operator a working asset and teaches them to ignore the
badge; a false negative leaves them where they are today.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HB1hhpAJcgtFueWqnC1z2K
The decode envelope needs a field the probe summary was throwing away. video_pix_fmt drives the blocking 8-bit-4:2:0 check, since the VideoCore capture queue has no 10-bit or 4:2:2 fourcc and a High 10 source therefore falls to software decode. video_bit_rate, video_level and video_profile are recorded too, and nothing branches on any of them. They are here so an operator looking at a misbehaving asset can see what they actually uploaded, and so the next person who wonders whether the level is the problem can find the answer in the metadata instead of guessing. Bitrate reads the video stream's own figure and falls back to the container's, because Matroska and some MP4 muxers omit the per-stream value entirely. ffprobe writes -99 for a container carrying no level, so non-positive values normalise to None rather than surviving as a real reading. Every new field collapses to None when ffprobe cannot supply it, including on the probe-failure path, which the envelope reads as "not measured" and stays quiet about. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HB1hhpAJcgtFueWqnC1z2K
A 3840x2160 H.264 upload to a Pi 4 passed every gate we had. H.264 is in pi4-64's supported codec set, and the 1080p resolution cap is guarded by is_low_ram_device(), which is false on any Pi 4 with 2 GB or more — measured at 3766308 kB on the device that hit this. So the row went is_processing=False, joined the rotation, and played at a few frames per second. The first report of the problem came from someone looking at the screen. The envelope check now runs after the codec gate on the accepted path, and only its BLOCKING tier rejects: frames over 1920 on either axis and pixel formats outside 8-bit 4:2:0, both refusals by the driver rather than predictions about speed. The advisory tier is deliberately not consulted here; it annotates the asset list without ever stopping an upload. Existing assets are untouched. This runs during normalisation, so it only ever sees a new upload; rows already on disk keep playing exactly as they do today. The rejection reuses the codec gate's UnsupportedVideoCodecError, so the operator gets the UI they already know: the reason inline, a copy-pasteable ffmpeg recipe, and HandBrake steps for anyone who would rather not open a terminal. Two details in the recipe matter. The downscale clause is emitted only when the frame is what failed, so a 10-bit 1080p file is not told to resize for no reason. And that case gets -pix_fmt yuv420p, because libx264 preserves the source bit depth by default — without it the operator would follow the recipe exactly and produce a second file that fails the same gate. Verified end to end against real encodes rather than fixtures: a 3840x2160 High L5.1 file is rejected, running the emitted recipe verbatim produces a 1920x1080 L4.0 file, and that output passes the gate. A 1920x1080 file carrying an inflated level=51 tag passes untouched, which is the false positive this must never produce. The module was also run on the production Pi itself, where resolve_device_key() returns 'pi4-64' — the key the rules are written against, and a mismatch there would have made every rule silently match nothing while the tests still passed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HB1hhpAJcgtFueWqnC1z2K
…code The upload gate only ever sees new uploads, so every asset that landed before it existed keeps playing exactly as it did — including the 4K clip that started this. Deleting or disabling those rows behind the operator's back would be worse than the problem; what they need is to be told which asset is making the screen look wrong. The asset list now carries a chip next to the name for any video outside this board's decode envelope, and the edit modal spells out every finding with its fix. Blocking findings read "Will not play well" and advisory ones "May not play well", which is the honest distinction: one is a format the decoder refuses, the other is a judgement about speed. Warnings are computed server-side and travel in the asset payload _to_dict already builds for the modal, so the per-board rules stay in one module instead of being reimplemented in Alpine. Two things the browser caught that the markup did not. The chip sits in a flex column, which blockifies inline-flex and then stretches it to the full column width — the pill ran the whole width of the name cell, fixed with align-self: flex-start. And the advisory variant was drawn as an outline with no fill, which made its label unreadable: --color-warning-on-wash is contrast-matched to the wash, not to the page, and the asset list resolves the light amber tokens over a dark surface. The softer tier is now signalled with a dashed edge and lighter weight, and both variants keep the wash the text needs to be legible against. Measured after that fix at 6.37 contrast in light and 12.84 in dark. Verified in a browser against seeded rows covering four cases: the 4K clip shows a blocking chip, a 115 Mbps 1080p clip an advisory one, and both a 1080p file tagged level 5.1 and a portrait 1080x1920 file show nothing at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HB1hhpAJcgtFueWqnC1z2K
board-enablement.md gains the second half of what enabling a board means. Codec support alone never settled playback, and the doc only described the codec set, so anyone adding a board had no reason to think about frame size or pixel format. It now carries the two tiers, which boards are in each and why x86 and rockpi4 are deliberately in neither, and the v4l2-ctl probe to characterise a new one — ask the decoder for its bounds rather than timing a clip, because an out-of-range frame cannot be set at all, so there is nothing to time. Corrects an actively misleading line in the anthias-hardware skill. It listed "level>4.x" alongside 4K30 / High10 / 4:2:2 as things that stop a Pi 4 opening /dev/video10. The driver never looks at the level; the real bound is 1920 per axis plus an 8-bit-4:2:0 capture queue, and the level control is read-only. That distinction is the whole reason the gate keys on frame size, so it is corrected in place with the mechanism and flagged as a previous error rather than quietly edited. Records what the hardware actually said, so the next person does not have to rediscover it: the enumerated frame bounds and capture formats, the 4K-vs-1080p decode comparison, the bitrate sweep that retired a drafted rule, and the fact that /dev/video19 (rpi-hevc-dec) is bound by default with no dtoverlay=rpivid-v4l2 anywhere in config.txt — so the gate's HEVC entry for pi4-64 is correct and the older docs demanding that overlay are out of date. The Pi 4 HEVC node is also stateless and advertises 10-bit capture formats, which is why the 8-bit restriction is scoped to the H.264 path and not applied board-wide. Two traps worth not repeating are written down. Synthetic benchmark clips (testsrc2 + noise, ultrafast, no B-frames) software-decode roughly 5x faster than real High-profile content, so extrapolating 4K software decode from a synthetic 1080p figure is wrong by about 2x. And fuser /dev/video10 run from the host reports no holders while the viewer is actively decoding, because it cannot see across the container's namespace — scan /proc/[0-9]*/fd instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HB1hhpAJcgtFueWqnC1z2K
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3309 +/- ##
=========================================
Coverage ? 90.48%
=========================================
Files ? 86
Lines ? 10064
Branches ? 1119
=========================================
Hits ? 9106
Misses ? 706
Partials ? 252 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Codecov reported 89.34% patch coverage, 13 lines short. Every one of them was a real branch rather than a coverage-metric artefact, so they are now tested rather than waived. The two that matter most are ffprobe parse failures. ffprobe writes the literal string "N/A" instead of omitting the key on plenty of containers, and _ffprobe_summary runs for every video upload — so an unhandled ValueError there would fail the whole normalisation task over a diagnostic field that nothing branches on. Both bit_rate and level now have tests proving they collapse to None, that a stream-level "N/A" still falls back to the container figure, and that one bad field does not poison the rest of the summary. The rest are helper guards in playback_envelope, which is now at 100%: _as_positive_int against unparseable and bool inputs (bools are ints in Python, and treating True as 1 would invent a dimension out of a flag), _dimensions_label's unmeasured-dimension fallback, and PlaybackWarning's __eq__/__repr__ — the first is what lets tests compare findings by value and defer sanely on foreign types, the second is what pytest prints when one fails. Also covers the HEVC branch of _ffmpeg_reencode_recipe. My pix_fmt change landed inside a branch that no test reached, because no board ships an HEVC-only codec set any more (Pi 5 gained an H.264 software fallback). It stays reachable-in-principle because those sets are per-board data that can change, and a recipe silently emitting libx264 for an HEVC-only board would hand the operator a file that fails the same gate again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HB1hhpAJcgtFueWqnC1z2K
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issues Fixed
A 4K H.264 clip uploaded to a Pi 4 was accepted with no warning and then played
at very low fps on the screen. H.264 is in
pi4-64's supported codec set, andthe existing 1080p resolution cap only applies to boards under 1.5 GiB of RAM,
so nothing in the pipeline had an opinion about it. The first sign of trouble
was the screen itself.
Description
The codec gate answers "can this board decode this codec". It says nothing
about the stream, so this adds a second check for whether the frame is inside
what the decoder can take.
Measured on a Pi 4B, a 4K High L5.1 clip at 116 Mbps against a 1080p re-encode
of the same content:
4K does not decode slowly in hardware, it never gets in:
h264_v4l2m2mfailsoutright because 3840 is past the device's frame bound, so libavcodec drops to
software, where 10 fps is already under the 25 fps the clip needs before the
viewer spends anything on presentation.
Two tiers, split by how good the evidence is:
the existing ffmpeg recipe and HandBrake steps. Covers frames over 1920 on
either axis and pixel formats outside 8-bit 4:2:0, both confirmed by asking
the hardware (
v4l2-ctlreportsStepwise 32x32 - 1920x1920andYU12/YV12/NV12/NV21/NC12).software H.264 above 1080p on a Pi 5.
Existing assets are never touched. The gate only runs during normalisation, so
anything already on disk keeps playing and gets a chip in the asset list
instead, with the full explanation in the edit modal.
Not gating on the declared H.264 level, which is the obvious-looking rule
and the wrong one. The driver exposes
V4L2_CID_MPEG_VIDEO_H264_LEVELread-only and never validates the bitstream against it, and plenty of ordinary
1080p files carry an inflated level tag and play fine. A 1080p file tagged
level 5.1 is a test case here precisely because it must not be flagged.
A bitrate rule was drafted and then removed after measuring it: 1080p through
h264_v4l2m2mstill runs at 62 fps at ~137 Mbps, so it would only ever haveflagged files that play fine.
Known gaps
pi4-64.pi2,pi3andpi3-64areextrapolated from the same driver constant. The error direction is safe (a
lower real bound means we under-block, never falsely reject), but a
maintainer with testbed access could confirm with
v4l2-ctl -d /dev/video10 --list-framesizes=H264on each, plusresolve_device_key()inside the server container — a mismatch there wouldmake every rule silently match nothing.
x86androckpi4are deliberately in neither tier; their decode paths arenot characterised, and a guess would produce false rejections.
Checklist
On the Pi testing: run on a live Pi 4B — the module against real asset
metadata,
resolve_device_key()returningpi4-64, thev4l2-ctlcapabilityprobes, and the decode measurements above. Not tested was a full upload through
a patched build on the device.
On x86: no rules fire there by design, verified in a dev container where
DEVICE_TYPE=x86produces no warnings, so behaviour is unchanged.🤖 Generated with Claude Code