Tufaceous v2: project rewrite / RFD 621 implementation by iliana · Pull Request #45 · oxidecomputer/tufaceous

iliana · 2026-04-21T18:16:42Z

I don't think this is 100% to where I want it to be yet but I have to cut a PR at some point.

This is essentially a complete redo of Tufaceous. I have been developing it in parallel with an Omicron changeset that integrates it (which you can see the WIP tree here: https://github.com/oxidecomputer/omicron/tree/iliana/rfd621).

Here are the main ideas:

All-in-one API

Everything a consumer needs for creating and reading repositories is in Tufaceous; there's no need to directly use APIs from tough. (The one exception to this currently is using non-ed25519 signing keys; when I get more of the signing tooling done I will probably add this.)

Artifact tags

RFD 621 goes into detail here, but the name-version-kind triple really doesn't do a good job of differentiating artifacts in practice. This system was developed in order to future-proof Nexus against future kinds of artifacts, but when we started this system we didn't fully understand what types of artifacts we need in the repo.

All artifacts that can go in the repo are described with KnownArtifactTags, an enum type that serializes to a mapping of string keys to string values. It is not possible to use the library to create a repository with "unknown" tags. When an older Nexus, using an older Tufaceous v2 library, interacts with a repo from the future with unknown tags, it will still store them in the database but ignore them for planning purposes.

`ArtifactSet`

This is basically a BTreeSet<Artifact> which has an index based on an artifact's known tags. This structure replaces most of the code in update-common currently in Omicron (apart from reading ZIP archives, below).

Builder API for creating / editing archives

In v1, repos were generated from manifests, but you couldn't automatically generate a manifest from a repo. This meant that patching a repo to add/change artifacts was extremely difficult, and it was usually easier to just use our releng tooling.

Instead we use a builder API to create and edit archives using the RepositoryEditor. You can start with an empty repository, or a fake repository with one of every artifact in it, or an already-loaded repository in order to modify it.

A significant amount of the code (and related abstractions) in the edit module is used for guessing artifact tags. This was done to make the CLI tooling bearable. I've added a note to the new README that indicates that future artifacts should be "guessable" by this tooling.

ZIP archive implementation

Tufaceous now has a ZIP transport implementation for tough, so Omicron no longer needs to unpack ZIP archives to disk in order to load the repository. There is a lengthy comment at the start of lib/src/zip_transport.rs on some of the pitfalls of using ZIP archives and the mitigations we've taken.

v1 compatibility

The v2 repository format is not compatible with the v1 format.

RepositoryLoader can read v1 repos, unpacking composite artifacts along the way. This library is not intended for creating v1 repos, and we expect the releng tool will carry a dependency on both Tufaceous v1 and v2 until we stop producing v1 repos. After that happens, we can start to remove v1 compatibility code from v2.

The tufaceous verify subcommand has v1 compatibility enabled and can be used to check for problems in the conversion (in fact, writing the problem checker and using it on our v1 repos made me find problems in the compatibility code!).

Currently this code cannot import a v1 repository into a [RepositoryEditor], and thus cannot directly convert a v1 archive into a v2 archive. I would like to add this functionality at some point (there is some additional complexity around handling the composite artifacts).

the primary purpose of this change is `ArtifactsExt::fake`, which is guaranteed to return the same set of `Artifacts` that you would get from generating a repository, but without requiring async. this also replaced the struct variants of `KnownArtifactTags` with named structs (RFD 643).

iliana · 2026-06-16T05:36:24Z

I think I've now covered all review feedback.

sunshowers

other than the spawn_blocking panic safety things we discussed...

sunshowers · 2026-06-30T22:32:34Z

+        // Collect all the sha256 hashes and lengths for each source. For file
+        // and fake sources, we want to calculate the hashes in parallel, so
+        // we spawn their calculation tasks on a JoinSet. Sources from borrowed
+        // repositories can't be moved into a task, but we already know their
+        // hash.
+        let mut all_targets = Vec::new();
+        let mut tasks = JoinSet::new();
+        for (target_name, sources) in self.targets {


do we want to bound the number of tasks running in parallel here? probably doesn't matter much.

I pushed an implementation which adds a semaphore -- it doesn't bound the number of tasks but it does limit the number of actively-running tasks. Seems like a reasonable thing to do.

sunshowers · 2026-07-01T22:03:51Z


    fn from_str(s: &str) -> std::result::Result<Self, Self::Err> {
-        let mut out = [0u8; 32];
-        hex::decode_to_slice(s, &mut out)?;


This looks like infinite recursion?

sunshowers · 2026-07-01T22:05:00Z

+                let mut stream = transport
+                    .fetch(root_url.clone())
+                    .await
+                    .map_err(ErrorKind::Fetch)?;


does this enforce a size limit? should it? probably not a huge deal.

No but it absolutely should, good catch.

sunshowers · 2026-07-01T22:05:40Z

+        let mut file = File::open(&phase_2_path).await.ok()?;
+        // Read the header block from the image and guess whether it's a
+        // recovery image based on the image name.
+        let mut buf = [0; 4096];
+        file.read_exact(&mut buf).await.ok()?;


is the ok()? on both of these calls required?

Yes but it could probably stand to be more specific.

sunshowers · 2026-07-01T22:06:30Z

+        let Ok(archive) = input.source.read_hubris_archive().await else {
+            return Ok(ControlFlow::Continue(input));
+        };
+        let Ok(caboose) = archive.read_caboose() else {
+            return Ok(ControlFlow::Continue(input));
+        };


should these handle errors more specifically?

In the first one, sure, we could return errors related to reading the file, but we deliberately want to not error on if it happens to be a ZIP file that isn't a valid Hubris archive.

sunshowers · 2026-07-01T22:07:26Z

+    pub fn remove_target(mut self, target_name: impl AsRef<str>) -> Self {
+        let target_name = target_name.as_ref();
+        self.targets.remove(target_name);
+        self.artifacts.remove(target_name);
+        self
+    }


should this error out if the target wasn't found?

If this one does, should remove_artifacts error if the target isn't found either?

I kind of would like to punt on this question because I'm genuinely not sure.

Yeah, I'm worried about a typo or something leading to a confusing experience.

sunshowers · 2026-07-02T00:13:55Z

+            return Some(Err(ErrorKind::ReadFile {
+                source,
+                path: Some(path.to_owned()),
+            }


do you mean to use phase_2_path here?

sunshowers · 2026-07-02T01:19:01Z

+        let phase_2_path = path.join(PHASE_2_PATH);
+        let mut file = File::open(&phase_2_path).await.ok()?;


Sorry, I apologize for not fully understanding this, but why is ok()? fine here? Shouldn't we handle not-found vs other errors differently?

I apologize for not fully understanding this

At this point I think you understand it better than I do! :)

sunshowers

Thank you for all your hard work on this!

iliana · 2026-07-02T17:55:37Z

Thank you all for the extremely helpful reviews! This is going to be great to integrate.

iliana added 30 commits December 17, 2025 17:35

wip

ea2dc59

define and wire up metadata

ece6da9

editor refactor; allow the CLI to guess kinds

bc122f2

todo -> todone

09fd212

add edit/ls/cat commands

2e4f484

match tokio-util version with omicron

e5f270e

impl Eq for Artifacts

b6a79ba

wip changes while working on omicron integration

2d998ad

load repo from opened zip; flatten io error variants

243782f

fix incorrect conversion into TransportErrorKind::FileNotFound

10f3578

ability to retrieve archive path/sha256 from loaded repo

fdd859e

impl From<T> for Sign

cae592a

refactor RepositoryEditor::fake

734c376

add Repository::fake shorthand method

9153796

add Artifacts::filter_tags

a05e82a

clean up unwraps from fake artifact generation

49fd3ff

clippy pedantic cleanup and explicit unwraps

2089329

small nit

1f329c4

ugh clippy

9a48652

add a Repository::verify_targets method

fb729bc

ensure target streams impl Sync

846b468

add ErrorKind::is_repository_error

8afcc64

add ArtifactId and ArtifactHandle

b487607

i fought the borrow checker and won

e69319a

create Root struct

3d48760

various tweaks

6225be4

add v1_compatibility tests

1aeda4a

a few tweaks and tests

24868ed

metadata will always be string:string map

8d88948

iliana added 10 commits June 15, 2026 21:45

reject repos with bad target metadata

9f301a0

verify subcommand should print repo hash

cff1fc4

fix added/changed flags in readme

a0f34b8

make Repository::check_problems fallible

3f740da

list problem if metadata cannot convert to mapping

509557c

make unix.z/cpio.z optional as intended

1dedde5

comments for v1

89c4f39

document GuessInput/GuessResult

e81ad2a

consistently use tuf_repo: tough::Repository

0975ac6

avoid overrelying on RotSlot::fmt

f503e6a

iliana added 2 commits June 25, 2026 06:29

ensure TargetSource::stream is Send

5bd5bd0

correctly use the friendly CA name in the target name

52d37f2

iliana mentioned this pull request Jun 25, 2026

[RFD 621 1/n] releng: generate Tufaceous v2 repos oxidecomputer/omicron#10676

Open

sunshowers reviewed Jun 30, 2026

View reviewed changes

iliana added 2 commits June 30, 2026 19:19

properly handle JoinError in the blocking stream pattern

53b20ab

max parallelism in RepositoryEditor::finish

e37c399

sunshowers reviewed Jul 1, 2026

View reviewed changes

iliana added 3 commits July 1, 2026 15:35

wow

67e9ea2

return io errors while guessing artifacts

470478c

check max_root_size when blindly trusting 1.root.json

85ba134

sunshowers reviewed Jul 2, 2026

View reviewed changes

fixup! return io errors while guessing artifacts

50e55db

sunshowers reviewed Jul 2, 2026

View reviewed changes

iliana added 2 commits July 2, 2026 09:37

further guessing about guess errors

d692cc7

fix os image guess error

2c4c642

sunshowers approved these changes Jul 2, 2026

View reviewed changes

iliana merged commit 72705fd into main Jul 2, 2026
5 checks passed

iliana deleted the iliana/rfd621 branch July 2, 2026 17:55

		let phase_2_path = path.join(PHASE_2_PATH);
		let mut file = File::open(&phase_2_path).await.ok()?;

Uh oh!

Conversation

iliana commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

All-in-one API

Artifact tags

ArtifactSet

Builder API for creating / editing archives

ZIP archive implementation

v1 compatibility

Uh oh!

iliana commented Jun 16, 2026

Uh oh!

sunshowers left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sunshowers left a comment

Choose a reason for hiding this comment

Uh oh!

iliana commented Jul 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

iliana commented Apr 21, 2026 •

edited

Loading

`ArtifactSet`