Skip to content

Pin HEAD-versioned packages to commit SHAs for reproducible builds #2270

Description

@widgetii

Split out from #2036, which is now closed — the 404 flakiness that motivated it is fixed, but this follow-up was never done.

Problem

29 packages under general/package/ are pinned to a moving ref:

_VERSION = HEAD

including the three called out in #2036motors, ipctool, ingenic-opensdk — plus divinus, msposd, go2rtc, mavfwd, yaml-cli, venc-openipc, and others.

A moving ref keeps a constant download filename whose contents change underneath it. Two consequences:

  1. Not content-addressable. The output/dl cache is keyed by month, and actions/cache only saves on a key miss, so a HEAD tarball fetched on the 1st stays frozen for the rest of the month. Nightly then silently ships stale code.
  2. Unstable URLs across runs, which is what made these packages disproportionately represented in the original 404 failures.

What already mitigates this

build.yml has a Refresh moving-ref package downloads step that deletes *[-.](HEAD|master|main|dist).tar.(gz|bz2|xz) from output/dl before each build, forcing a refetch. That fixes staleness — it does not give us reproducibility. Two builds of the same firmware SHA can still produce different images, which makes contrib/openipc-bisect unreliable across any window where an unpinned upstream moved.

There is also a sharper failure mode, demonstrated on 2026-08-12 (run 31649720825): upstream majestic-webui deleted two files, the package's fixup did a bare rm on them, and all 99 boards failed at once. A moving ref means an unrelated repo can break every platform in this one, with no commit here. Pinning converts that from "nightly is red tomorrow" into "the bump PR is red".

Proposed work

  • Pin motors, ipctool, ingenic-opensdk to explicit commit SHAs (the three named in Nightly CI is flaky on master — package download 404s break ~30% of scheduled runs #2036).
  • Triage the remaining 26 — some are ours and bump often, some are effectively dormant. Pin at least the dormant ones; they cost nothing to pin and are pure downside unpinned.
  • Decide the bump mechanism. Manual bump PRs are the honest option; a scheduled auto-bump PR (Renovate-style) keeps freshness without sacrificing reproducibility. Either beats a moving ref.
  • Once a package is pinned, it no longer needs the moving-ref refresh regex — narrow the pattern as packages graduate.

Why it is worth doing

Reproducible builds are the actual prize. Right now, a nightly image cannot be rebuilt byte-for-byte from its SHA, which undercuts bisect and makes "works on the nightly, not on my build" hard to investigate.

Metadata

Metadata

Assignees

No one assigned

    Labels

    ciContinuous Integration: workflows, build flakiness, infra

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions