Skip to content

feat(gateware): pick this build's image by summary schema; keep its name in custom gateware debs - #203

Merged
vishnutskumar merged 5 commits into
mainfrom
vts/gateware-versioned-bit-name
Nov 11, 2026
Merged

vishnutskumar merged 5 commits into
mainfrom
vts/gateware-versioned-bit-name

Conversation

@vishnutskumar

@vishnutskumar vishnutskumar commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

This PR also carries the two commits from #202, which is closed in favour of this one.

Why

synapsectl peripherals build has two defects that stop custom gateware from reaching a probe.

1. It can't find the image current SDKs build

After axon-peripheral-sdk build, run_gateware_build looks only for sdk_*_extracted.bit. Current SDKs write a native sdk_*.bit instead:

  • the released SDK 1.0.8 does so for nerv512u-devkit;
  • newer SDK builds do so for every profile.

So on main, such a build succeeds in Radiant and then fails in synapse:

axon-peripheral-sdk build completed but no sdk_*_extracted.bit was emitted under ...

Looking for the native image instead isn't enough on its own:

  • Some SDK builds still write only sdk_*_extracted.bit. The released 1.0.8 SDK does this for via-devkit. Users who haven't upgraded the SDK must keep working.
  • build/bitstreams/ persists between builds. A name rule alone can pick a leftover image. For example, a native .bit from an earlier build beats the extracted image an older SDK just wrote, and the stale image gets packaged and flashed. Copilot reported this on feat(gateware): prefer the native SDK .bit, fall back to the extracted image #202.
  • synapse and the SDK release independently. synapse can't assume which SDK a user has, so it needs a signal from the build rather than a guess.

2. The image it packages can't be flashed on wired probes

build_gateware_deb installs the image as /opt/scifi/bitstreams/custom/<identifier>.bit. That discards the build's file name, for example sdk_nerv512u-devkit_gateware_v0.3.7_<hash>_<date>_<HHMM>.bit.

On wired probes such as the NeRV devkit, the device's probe updater (scifi-probe-updater) takes the DFU version from the file name and refuses an image without a v<M.m.p> token. The probe enforces anti-rollback against that version, so an unversioned image would be rejected anyway. So NeRV custom gateware packaged by synapse fails at Probe Updates.

3. Smaller additions

  • Deploy timeout. deploy_package(..., timeout=...) lets an automated caller fail a hung install instead of blocking on it.
  • Error reporting. The new build errors print cleanly instead of as a traceback.
  • Exit codes. synapsectl peripherals build and peripherals deploy now exit non-zero when they fail. See Exit codes below.

What

Image selection (gateware.py, run_gateware_build)

  1. Only this build's output counts. build/bitstreams/ is snapshotted before the build (name → mtime_ns, size). Afterwards, only files that are new or changed are considered, so a leftover image is never chosen whatever its name.

  2. The build summary's schema_version decides:

    Summary Selection
    1, or no schema_version (every SDK released so far) the newest new native sdk_*.bit, else the newest new sdk_*_extracted.bit
    2 (the next axon-peripheral-sdk release) <stem>.summary.json names its sibling <stem>.bit in bitstream. synapse uses that image, which must be from this build; any other value is a ValueError
    newer than 2 a ValueError asking to upgrade science-synapse; never guessed
  3. Nothing emitted raises FileNotFoundError, and the message says older files were ignored.

peripherals._run_gateware_half also catches ValueError, so these errors print as Error: Gateware build failed: ....

Gateware .deb layout (peripherals.py, build_gateware_deb)

  • The image keeps its build file name: /opt/scifi/bitstreams/custom/artifacts/<identifier>/<basename>.bit.
  • The manifest fragment's artifact points there: custom/artifacts/<identifier>/<basename>.bit.
  • Unchanged:
    • the package name axon-gateware-<identifier>;
    • the fragment file custom/<identifier>.manifest.json and its name, usb_pid and git_hash;
    • scifi-probe-updater --name and the systemd unit.
  • The per-identifier directory keeps images of different identifiers from sharing a path.

Deploy (deploy.py)

deploy_package(ip_address, deb_package_path, timeout=None) passes timeout, in seconds, to the gRPC DeployApp call. None keeps the current no-deadline behaviour.

Exit codes (peripherals.py, build_cmd and deploy_cmd)

synapsectl peripherals build and peripherals deploy used to exit 0 on every failure. Their handlers returned None, and run_action exits 1 only for False.

Every failure path now returns False:

  • Docker missing;
  • a missing or invalid manifest;
  • a failed driver or gateware build or packaging step;
  • a missing --package;
  • a failed deploy.

A deploy without -u still exits 0, because its packages are built and nothing was asked to be deployed.

Version

This PR bumps the version to 2.10.0 (setup.py). Publishing runs on a v* tag; merging alone does not publish.

Backward compatibility

  • Old and new SDKs. Schema 1, a missing schema and schema 2 all work. A future schema fails with an upgrade message.
  • Release order. Ship this synapse before the schema-2 SDK. SDKs released so far are covered by the schema-1 rule.
  • Devices. Probe updaters with custom gateware support flash the fragment's artifact path as given, so the nested path needs no SciFi update. This was confirmed on hardware; see below.
  • Installed packages. The package name is unchanged, so a dpkg upgrade removes the old custom/<identifier>.bit.
  • Callers. Existing deploy_package(ip, deb) calls are unchanged. Scripts that relied on peripherals build or deploy exiting 0 after a failure now see exit 1.

Tests

The tests were written from the contract by a separate tester who didn't read the implementation.

  • test_gateware_runner.py (19 of these fail against the previous gateware.py):
    • a stale native .bit when only an extracted image was built;
    • sequential builds in one directory;
    • a rewritten pre-existing file counts;
    • an unchanged file with a future mtime is ignored;
    • schema 1 and a missing schema behave the same;
    • schema 2 picks its sibling over a newer native image;
    • schema 2 rejects a different image, a path, a non-.bit name, a missing or non-string value, a missing file and a pre-existing file;
    • a stale schema-2 or schema-3 summary is ignored;
    • schema 3 and above ask for an upgrade;
    • nothing new raises.
  • test_gateware_profiles.py: the per-profile build asserts --project is the selected profile's project (Copilot review on feat(gateware): prefer the native SDK .bit, fall back to the extracted image #202).
  • test_custom_gateware_packaging.py, test_half_selectors.py:
    • versioned and _extracted file names are kept;
    • the fragment's artifact resolves to exactly the staged image;
    • that image is the only .bit under custom/;
    • two identifiers never share a path.
  • test_deploy_timeout.py: the timeout reaches DeployApp, and two-argument and None calls set no deadline.
  • test_peripherals_exit_codes.py (80 cases):
    • It drives the real synapsectl main() over every build and deploy failure for each half, with Docker, the builds, fpm and the device faked.
    • 69 of them fail against the previous code. The 11 success cases pass both before and after.
  • Result: pytest, 367 passed; the ruff and ruff-format pre-commit hooks pass.

Some hunks in peripherals.py and gateware.py (line rewraps, quote style) are formatting from the ruff-format pre-commit hook, not behaviour changes.

Real builds and hardware (2026-11-10)

Real gateware builds

Each build ran this branch's run_gateware_build and build_gateware_deb against a real axon-peripheral-sdk build.

The checks were:

  • the deb holds exactly one .bit, at custom/artifacts/<id>/<basename>.bit;
  • that image is byte-identical to the image this build wrote, and is the newest one;
  • the basename carries v<M.m.p>;
  • the fragment's artifact, usb_pid and git_hash match the summary.
SDK Profile Image the SDK writes Summary schema Result
released 1.0.8, installed by the example's gateware.Dockerfile nerv512u-devkit native 1 pass
released 1.0.8 via-devkit _extracted 1 pass (extracted fallback)
released 1.0.8, rebuilt in the same directory with the previous image left behind nerv512u-devkit native 1 pass (the new image is packaged)
development build nerv512u-devkit native 1 pass
development build via-devkit native 1 pass
schema-2 pre-release nerv512u-devkit native 2 pass (the image the summary names)
schema-2 pre-release via-devkit native 2 pass (the image the summary names)
released 1.0.8 + released synapse 2.9.0 (control) nerv512u-devkit native 1 fails as expected: no sdk_*_extracted.bit was emitted

The control shows that released synapse 2.9.0 cannot build NeRV gateware with the released SDK today.

The stale-image build covers only a stale native image next to a newer native one. Two cases rest on the unit tests above:

  • a stale native image beside a fresh extracted one;
  • a stale schema-2 summary.

Hardware

A NeRV devkit on a SciFi installed this branch's packages with deploy_package(..., timeout=300). The probe updater then flashed custom/artifacts/<id>/<basename>.bit through Probe Updates, reading version 1.0.14 from the file name, and the unit ended with result success.

On a Via devkit the same install succeeded. The flash failed for a reason outside this PR: the probe updater requires a scIR image whose first word is 0x32cac2c2, and that SDK build's Via image has an LSCC header.

Notes

  • Old-SDK NeRV images cannot reach production probes, even though this PR packages them correctly.
    • Released-SDK NeRV images are named …_gateware_v0.1.1_<hash>_<date>_<time>.bit.
    • The probe updater reads the DFU version from that name, so it gets 0.1.1, which is below a production probe's rollback floor.
    • Later SDK builds name images …_sdk0.1.1_v1.0.14_….
    • This was found from the file-name format; no hardware flash of an old-SDK deb was attempted.

…d image

Newer axon-peripheral-sdk builds emit a native sdk_*.bit and no longer
write sdk_*_extracted.bit, so the gateware runner's extracted-only glob
failed with "no sdk_*_extracted.bit was emitted" after a successful build.

The runner now returns the newest native sdk_*.bit. When a build emitted
no native image, as older SDK releases do, it returns the newest
sdk_*_extracted.bit instead, so synapsectl works with both SDK versions
whichever is released first. Packaging stages the selected image and its
same-stem summary unchanged.
…a-2 summary

Build directories persist, so the name rule could return a native .bit
left by an earlier build when the current SDK emits only the extracted
image. run_gateware_build now snapshots build/bitstreams before the build
and considers only files the build created or changed (mtime_ns, size).

The SDK and synapse release independently, so selection follows the
summary's schema_version:
- 2: <stem>.summary.json names its sibling <stem>.bit in "bitstream";
  that image is returned, and any other value is rejected.
- 1 or absent: the native-first name rule, with the extracted fallback.
- newer than 2: ValueError asking to upgrade science-synapse.
The per-profile build test again asserts the SDK runs with --project set
to the selected profile's project.
…deploy timeout

build_gateware_deb staged the image as custom/<identifier>.bit. For wired
probes, scifi-probe-updater's Lattice DFU reads the image version from the
file name and refuses an image without a v<M.m.p> token (anti-rollback), so
renamed NeRV custom gateware could not be flashed. The image now keeps its
build basename under custom/artifacts/<identifier>/ and the manifest
fragment's artifact points there. Every updater with custom-gateware
support flashes the fragment's artifact path, so devices need no update;
the package, fragment name and updater unit are unchanged, and a dpkg
upgrade replaces the old custom/<identifier>.bit.

deploy_package gains an optional timeout (seconds) for the whole DeployApp
call; None keeps the current no-deadline behaviour.
Copilot AI balanced review requested due to automatic review settings November 5, 2026 23:40

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

A maintainer must confirm that deployed headstage updaters accept nested artifact paths and parse DFU versions from basenames, since updater sources were inaccessible and local tests mock deployment rather than verify on-device DFU flashing.

No open issues 🎉

What changed in this PR

This PR preserves custom gateware build filenames so wired-probe updaters can read the version required for DFU flashing. It also adds an optional deployment timeout while retaining the existing no-deadline default.

Changes:

  • Stage images under custom/artifacts/<identifier>/<basename> and point manifest fragments to that path.
  • Forward deploy_package(..., timeout=...) to DeployApp.
  • Expand tests for filename preservation, manifest paths, identifier isolation, and timeout compatibility.
File Description
synapse/​tests/​cli/​test_half_selectors.py Updates build-flow assertions for the new image layout.
synapse/​tests/​cli/​test_deploy_timeout.py Tests timeout forwarding and no-deadline defaults.
synapse/​tests/​cli/​test_custom_gateware_packaging.py Tests preserved filenames, manifest resolution, and isolated artifact paths.
synapse/​cli/​peripherals.py Preserves image basenames in identifier-specific directories and updates manifest paths.
synapse/​cli/​gateware.py Clarifies the identifier docstring to refer to the manifest name.
synapse/​cli/​deploy.py Adds and documents the optional DeployApp timeout.

Review effort: Balanced

@vishnutskumar
vishnutskumar changed the base branch from vts/sdk-native-bit-lookup to main November 5, 2026 23:57
@vishnutskumar vishnutskumar changed the title feat(gateware): keep the image's build name in custom gateware debs; deploy timeout feat(gateware): pick this build's image by summary schema; keep its name in custom gateware debs Nov 5, 2026
`synapsectl peripherals build` and `peripherals deploy` returned None on every
failure (Docker missing, invalid manifest, a failed driver or gateware build,
a missing --package, a failed deploy), so run_action exited 0 and scripts
could not tell a failed build from a good one. Each failure now returns False,
which run_action maps to exit 1. A deploy with no URI still exits 0: the
packages are built and nothing was asked to be deployed.

test_peripherals_exit_codes.py drives the real synapsectl main() over every
failure for each half, with Docker, the builds, fpm and the device faked.
…re debs, deploy timeout, peripherals exit codes
Copilot AI balanced review requested due to automatic review settings November 10, 2026 23:07

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Both timeout handlers access the undefined device.verbose attribute, causing an AttributeError that masks the timeout instead of returning False, while unrestricted mocks leave this path untested.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Timeout handlers access undefined device.verbose and mask the error

synapse/​cli/​deploy.py:126

When the new deadline expires, both exception handlers access device.verbose, but Device never defines that attribute, so deployment raises AttributeError instead of returning False and masks the timeout. Use error_message(e) in both handlers (lines 186 and 207), matching the non-verbose device configuration. Add an expired-deadline test with a Device-shaped stub; the unrestricted MagicMock hides this mismatch.

Review effort: Balanced

@vishnutskumar
vishnutskumar merged commit f9f80bb into main Nov 11, 2026
3 checks passed
@vishnutskumar
vishnutskumar deleted the vts/gateware-versioned-bit-name branch November 11, 2026 18:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants