Repository navigation

Benchmark #46

Workflow file for this run

.github/workflows/benchmark.yml at 1a1d197

	name: "Benchmark"

	# First milestone of the OpenMS benchmarking effort (Issue #8788).
	#
	# Flow:
	# fixed OpenMS SHA -> Linux build in GitHub Actions -> generated binaries
	# -> smoke benchmark (Comet -> Percolator) -> results + resource info
	# -> OpenSwath DIA benchmark -> results + resource info
	#
	# The MS2Rescore choice is plumbed through as a workflow input and recorded in
	# the results, but the actual MS2Rescore execution is part of the extended
	# benchmark (a later milestone) and is NOT run here. See README.md.

	on:
	workflow_dispatch:
	inputs:
	runtime_source:
	description: "OpenMS runtime for the benchmarks: 'source' builds openms_sha from source (exact-SHA semantics, ONNX/ProSE/PeptDeep included); 'package' benchmarks a published Debian nightly .deb (Smoke + OpenSwath only; package provenance recorded; ONNX unavailable today)"
	required: false
	default: "source"
	type: choice
	options:
	- source
	- package
	openms_sha:
	description: "OpenMS commit SHA to build and benchmark (source mode). ONNX/ProSE jobs additionally require a SHA carrying PR #9975 ONNX support (cmake/Modules/FindONNXRuntime.cmake, added after f1768367); the default satisfies both. In package mode this input is optional: if set, the package's embedded OpenMS SHA must match it or the run fails loudly."
	required: false
	default: "e4b9609c95cac0c64b7d991d0c047d5fac2c7a27"
	package_date:
	description: "Package mode only: archive date (YYYY.MM.DD, the upload-day directory; the .deb inside is the prior night's build). Empty = newest available at dispatch."
	required: false
	default: ""
	type: string
	use_ms2rescore:
	description: "Request the extended (MS2Rescore) benchmark mode"
	required: false
	default: false
	type: boolean
	nightly_deb_probe:
	description: "Opt-in nightly Debian .deb probe (develop-snapshot runtime-health check via the Smoke benchmark; NOT PR-SHA validation, no ONNX)"
	required: false
	default: false
	type: boolean
	nightly_deb_date:
	description: "Optional archive date (YYYY.MM.DD, the upload-day directory; the .deb inside is the prior night's build). Empty = newest available at dispatch."
	required: false
	default: ""
	type: string

	jobs:
	build-openms:
	name: "Build OpenMS (Linux, fixed SHA)"
	runs-on: ubuntu-24.04
	# Package mode consumes the published .deb instead of a source build; the
	# build job is skipped entirely. Guarding here keeps every dependent job's
	# `needs:` intact (skipped, not unfulfilled) so the workflow graph stays valid.
	if: ${{ inputs.runtime_source != 'package' }}
	outputs:
	openms_sha: ${{ steps.sha.outputs.sha }}
	build_minutes: ${{ steps.build.outputs.minutes }}
	vcpkg_cache_restored: ${{ steps.vcpkg-cache.outputs.cache-hit != '' && steps.vcpkg-cache.outputs.cache-hit == 'true' }}
	steps:
	- name: Checkout OpenMS at pinned SHA
	uses: actions/checkout@v6
	with:
	repository: OpenMS/OpenMS
	ref: ${{ inputs.openms_sha }}

	# Resolve the full SHA so the results record exactly what was built,
	# even when the input is a short SHA or a branch name.
	- name: Resolve pinned SHA
	id: sha
	shell: bash
	run: echo "sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"

	# OpenMS changed its CI dependency model ("Only use dependencies from
	# vcpkg", upstream e7e2297241): newer SHAs no longer install libraries
	# (xerces, boost, ...) via tools/ci/deps-ubuntu.sh — they come from the
	# vcpkg manifest during CMake configure instead. Detect which model
	# applies from the checked-out SHA itself; never hardcode a date or SHA
	# boundary here.
	- name: Detect dependency model
	id: depmodel
	shell: bash
	run: \|
	set -euo pipefail

	# Capability: does this SHA ship the complete vcpkg mechanism
	# (manifest + CMake opt-in + vcpkg submodule declaration)?
	capable=false
	if [[ -f vcpkg.json ]] \
	&& grep -q "OPENMS_USE_VCPKG" CMakeLists.txt \
	&& grep -Eq '^[[:space:]]path[[:space:]]=[[:space:]]vcpkg[[:space:]]$' .gitmodules; then
	capable=true
	fi

	# Does the SHA's own dependency script still install libraries via
	# apt? While that holds, the legacy path is exactly the recipe that
	# produced our stored baselines, so keep it untouched.
	legacy_libs=false
	if grep -Eqi 'libxerces\|boost\|libsvm\|libzip\|hdf5\|eigen\|seqan\|coinor\|glpk\|wildmagic' tools/ci/deps-ubuntu.sh; then
	legacy_libs=true
	fi

	if [[ "$legacy_libs" == true ]]; then
	echo "model=legacy" >> "$GITHUB_OUTPUT"
	echo "OPENMS_CMAKE_VCPKG_ARGS=" >> "$GITHUB_ENV"
	echo "Dependency model: legacy (deps script installs libraries via apt)"
	elif [[ "$capable" == true ]]; then
	echo "model=vcpkg" >> "$GITHUB_OUTPUT"
	# Flags mirror OpenMS' own linux presets: the vcpkg community
	# triplet is dynamic (the builtin x64-linux triplet is static,
	# which cannot link libOpenMS as a shared library), and upstream
	# sets BOOST_USE_STATIC=OFF with it.
	# wnetalign: WITH_WNETALIGN defaults ON upstream; its vcpkg branch find_path()s headers
	# provided only by the wnetalign manifest feature (same feature upstream CI presets set).
	echo "OPENMS_CMAKE_VCPKG_ARGS=-DOPENMS_USE_VCPKG=ON -DVCPKG_TARGET_TRIPLET=x64-linux-dynamic -DBOOST_USE_STATIC=OFF -DVCPKG_MANIFEST_FEATURES=wnetalign" >> "$GITHUB_ENV"
	echo "Dependency model: vcpkg manifest (deps script installs no libraries)"
	else
	echo "::error::OpenMS at ${{ steps.sha.outputs.sha }} neither installs libraries via tools/ci/deps-ubuntu.sh nor ships the vcpkg mechanism (vcpkg.json / OPENMS_USE_VCPKG in CMakeLists.txt / vcpkg submodule in .gitmodules). No workable dependency model for this SHA." >&2
	exit 1
	fi

	# Reuse OpenMS' own dependency script (the same one its CI uses) instead
	# of duplicating the package list here. --skip-gui-deps keeps the build
	# lean: TOPP tools do not need Qt, so WITH_GUI is disabled below.
	- name: Install dependencies
	shell: bash
	run: bash tools/ci/deps-ubuntu.sh --skip-gui-deps

	# ccache: the same compiler-cache strategy OpenMS CI uses (see
	# .github/actions/build/action.yml in OpenMS), transplanted here. The
	# cache is keyed by the pinned SHA, so repeated benchmarks of the same
	# SHA skip the ~70 min cold build entirely, while benchmarks of nearby
	# SHAs still hit most entries because ccache keys on file content, not
	# on the commit. This is what maintainer @jpfeuffer suggested.
	- name: Restore compiler cache
	uses: actions/cache@v5
	with:
	path: .ccache
	key: openms-ccache-${{ runner.os }}-${{ runner.arch }}-${{ steps.sha.outputs.sha }}
	restore-keys: \|
	openms-ccache-${{ runner.os }}-${{ runner.arch }}-${{ steps.sha.outputs.sha }}
	openms-ccache-${{ runner.os }}-${{ runner.arch }}-

	- name: Configure ccache
	shell: bash
	run: \|
	mkdir -p "$GITHUB_WORKSPACE/.ccache"
	{
	echo "CCACHE_COMPRESS=true"
	echo "CCACHE_COMPRESSLEVEL=6"
	echo "CCACHE_DIR=$GITHUB_WORKSPACE/.ccache"
	echo "CCACHE_MAXSIZE=1G"
	} >> "$GITHUB_ENV"

	# ---- vcpkg dependency model (newer OpenMS SHAs) ----------------------
	# Mirrors OpenMS' own dependency setup (.github/actions/dependencies):
	# vcpkg is a submodule of OpenMS and is bootstrapped with the same
	# upstream-supported action; the manifest install itself runs during
	# CMake configure through the toolchain file that OPENMS_USE_VCPKG=ON
	# selects. Port/triplet overlays come from vcpkg-configuration.json.
	# Binary artifacts are cached locally, keyed by the manifest content so
	# a dependency change in any benchmarked SHA gets its own entry; the
	# restore-key fallback is safe because vcpkg binary caches are
	# content-addressed per port (stale entries are simply unused).
	- name: Restore vcpkg binary cache
	id: vcpkg-cache
	if: steps.depmodel.outputs.model == 'vcpkg'
	uses: actions/cache@v5
	with:
	path: .vcpkg-cache
	key: openms-vcpkg-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('vcpkg.json', 'vcpkg-configuration.json') }}
	restore-keys: \|
	openms-vcpkg-${{ runner.os }}-${{ runner.arch }}-

	# Cold-cache fallback: OpenMS publishes its own vcpkg binary cache at
	# archive.openms.de under the SAME manifest hash our cache key computes
	# (hashFiles over the OpenMS checkout = sha256 of the concatenated
	# per-file sha256 digests). Seeding the files-cache dir from it turns a
	# ~90 min from-source provisioning into minutes. Sharded zips are
	# content-addressed, so seeding is safe; misses simply build from source.
	- name: Seed vcpkg cache from upstream archive (on miss)
	if: steps.depmodel.outputs.model == 'vcpkg' && steps.vcpkg-cache.outputs.cache-hit != 'true'
	shell: bash
	run: \|
	set -euo pipefail
	manifest_hash="${{ hashFiles('vcpkg.json', 'vcpkg-configuration.json') }}"
	base="https://archive.openms.de/openms/vcpkg-cache/Linux-X64-g++-vcpkg-${manifest_hash}"
	if ! curl -fsSL --max-time 30 "${base}/" -o /tmp/upstream-index.html 2>/dev/null; then
	echo "upstream vcpkg cache not reachable (continuing without seed)"
	exit 0
	fi
	shards=$(grep -oE 'href="[0-9a-f]{2}/"' /tmp/upstream-index.html \| sed 's/href="$.*$"/\1/' \| tr -d '/' \| sort -u)
	if [ -z "$shards" ]; then
	echo "upstream cache layout unexpected (no shards found); continuing without seed"
	exit 0
	fi
	for shard in $shards; do
	curl -fsSL --max-time 30 "${base}/${shard}/" -o "/tmp/shard-${shard}.html" \|\| continue
	zips=$(grep -oE 'href="[0-9a-f]{64}\.zip"' "/tmp/shard-${shard}.html" \| sed 's/href="$.*$"/\1/')
	for zip_name in $zips; do
	dest="$GITHUB_WORKSPACE/.vcpkg-cache/${shard}"
	mkdir -p "$dest"
	curl -fsSL --retry 2 -o "${dest}/${zip_name}" "${base}/${shard}/${zip_name}" \|\| rm -f "${dest}/${zip_name}"
	done
	rm -f "/tmp/shard-${shard}.html"
	done
	seeded=$(find "$GITHUB_WORKSPACE/.vcpkg-cache" -name '*.zip' 2>/dev/null \| wc -l)
	echo "seeded ${seeded} vcpkg binary-cache zips from upstream archive"

	- name: Set up vcpkg
	if: steps.depmodel.outputs.model == 'vcpkg'
	shell: bash
	run: \|
	git submodule update --init vcpkg
	mkdir -p "$GITHUB_WORKSPACE/.vcpkg-cache"
	{
	echo "VCPKG_BINARY_SOURCES=clear;files,$GITHUB_WORKSPACE/.vcpkg-cache,readwrite"
	} >> "$GITHUB_ENV"

	- name: Bootstrap vcpkg
	if: steps.depmodel.outputs.model == 'vcpkg'
	uses: lukka/run-vcpkg@v11
	with:
	doNotCache: true

	# Search engines (Comet, Percolator, ...) ship as the THIRDPARTY
	# submodule of OpenMS; stage them exactly like OpenMS' CI does so the
	# adapters can find them at runtime.
	- name: Stage search engines
	shell: bash
	run: \|
	git submodule update --init THIRDPARTY
	mkdir -p _thirdparty
	cp -R THIRDPARTY/Linux/$(uname -m)/* _thirdparty/
	cp -R THIRDPARTY/All/* _thirdparty/

	# Headless build: no Qt, no docs. ENABLE_DOCS=OFF is required because the
	# doc toolchain (TOPPDocumenter et al.) includes QtCore/QString even
	# though we never build the docs themselves. The compiler launchers make
	# every compile go through ccache.
	- name: Configure (Release, Ninja)
	shell: bash
	env:
	SEARCH_ENGINES_DIRECTORY: ${{ github.workspace }}/_thirdparty
	run: \|
	cmake -S . -B build \
	-G Ninja \
	-DCMAKE_BUILD_TYPE=Release \
	-DCMAKE_C_COMPILER_LAUNCHER=ccache \
	-DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
	-DWITH_GUI=OFF \
	-DENABLE_DOCS=OFF \
	-DSEARCH_ENGINES_DIRECTORY="$SEARCH_ENGINES_DIRECTORY" \
	$OPENMS_CMAKE_VCPKG_ARGS

	- name: Build
	id: build
	shell: bash
	run: \|
	start=$(date +%s)
	cmake --build build
	end=$(date +%s)
	echo "minutes=$(( (end - start) / 60 ))" >> "$GITHUB_OUTPUT"

	# Ship the shared library and shared data too: TOPP binaries link
	# against libOpenMS.so (build/lib) and need OpenMS' share/OpenMS data
	# at runtime. Neither the build-tree RPATH nor the compiled-in data
	# paths exist on the runner, so without them the smoke job gets exit
	# 127 / "Cannot find shared data" on every tool.
	- name: Upload OpenMS binaries and search engines
	uses: actions/upload-artifact@v7
	with:
	name: openms-bin-${{ steps.sha.outputs.sha }}
	path: \|
	build/bin
	build/lib
	build/vcpkg_installed/x64-linux-dynamic/lib
	share/OpenMS
	_thirdparty
	if-no-files-found: error

	# ---------------------------------------------------------------------------
	# Package runtime provider (runtime_source=package).
	#
	# Generalizes the nightly probe's download/extract/dependency steps into a
	# runtime for the benchmark jobs: resolves the dated nightly archive dir,
	# downloads the .deb (SHA256 recorded), satisfies the declared Depends,
	# extracts into $RUNNER_TEMP (never installed system-wide), and resolves
	# exact provenance from the package's own openms_package_version.h
	# (OPENMS_GIT_SHA1 / OPENMS_GIT_BRANCH / OPENMS_PACKAGE_VERSION). The
	# benchmark runners consume the extracted prefix via the same env contract
	# the source-build path uses (OPENMS_BIN/ENGINES_DIR/OPENMS_SHA/RESULTS_DIR).
	#
	# Strict SHA guard: if the dispatch requested a specific openms_sha, the
	# package's embedded SHA must match it (full or unambiguous prefix) or the
	# job fails — a package may only satisfy an SHA request by actually being
	# that revision. A nightly package can never silently replace a requested
	# custom SHA.
	#
	# ONNX capability: published nightlies are built WITHOUT ONNX Runtime
	# (payload-verified), so ProSE/PeptDeep jobs do not run in package mode.
	# ---------------------------------------------------------------------------
	package-runtime:
	name: "Resolve package runtime (nightly .deb)"
	runs-on: ubuntu-24.04
	timeout-minutes: 30
	if: ${{ inputs.runtime_source == 'package' }}
	outputs:
	package_filename: ${{ steps.package.outputs.filename }}
	package_sha256: ${{ steps.package.outputs.sha256 }}
	package_date: ${{ steps.package.outputs.pkg_date }}
	archive_dir: ${{ steps.archive.outputs.dir }}
	package_bytes: ${{ steps.package.outputs.bytes }}
	openms_sha: ${{ steps.provenance.outputs.pkg_sha }}
	openms_branch: ${{ steps.provenance.outputs.pkg_branch }}
	openms_version: ${{ steps.provenance.outputs.pkg_version }}
	steps:
	- name: Checkout benchmarking repo
	uses: actions/checkout@v6

	# Same archive-dir resolution as the nightly probe: explicit date or
	# newest available; the resolved dir is recorded so every run is
	# reproducible from its provenance.
	- name: Resolve nightly archive directory (explicit date or newest)
	id: archive
	env:
	REQUESTED_DATE: ${{ inputs.package_date }}
	shell: bash
	run: \|
	set -euo pipefail
	index=$(curl -fsSL --max-time 30 https://archive.openms.de/openms/OpenMSInstaller/nightly/)
	requested="${REQUESTED_DATE:-}"
	if [[ -n "$requested" ]]; then
	if ! printf '%s' "$requested" \| grep -qE '^[0-9]{4}\.[0-9]{2}\.[0-9]{2}$'; then
	echo "ERROR: package_date must be YYYY.MM.DD (archive upload-day directory), got: '$requested'" >&2
	exit 1
	fi
	if ! printf '%s' "$index" \| grep -qF "href=\"$requested/\""; then
	echo "ERROR: requested package archive dir '$requested' not found in archive index" >&2
	exit 1
	fi
	dir="$requested"
	echo "Using requested archive dir: $dir"
	else
	dir=$(printf '%s' "$index" \| grep -oE 'href="[0-9]{4}\.[0-9]{2}\.[0-9]{2}/"' \
	\| grep -oE '[0-9]{4}\.[0-9]{2}\.[0-9]{2}' \| sort \| tail -1)
	if [[ -z "$dir" ]]; then
	echo "ERROR: no dated nightly directory found in archive index" >&2
	exit 1
	fi
	echo "Resolved newest archive dir: $dir"
	fi
	echo "dir=$dir" >> "$GITHUB_OUTPUT"

	- name: Download .deb and record filename, size and SHA256
	id: package
	shell: bash
	run: \|
	set -euo pipefail
	base="https://archive.openms.de/openms/OpenMSInstaller/nightly/${{ steps.archive.outputs.dir }}"
	listing=$(curl -fsSL --max-time 30 "$base/")
	# Security: the filename comes from remote HTML; restrict to a
	# shell-safe charset at capture time and anchor-validate (same
	# policy as the nightly probe).
	filename=$(printf '%s' "$listing" \| grep -oE 'OpenMS-[-A-Za-z0-9._+~]+-Debian-Linux-x86_64\.deb' \| head -1)
	if [[ -z "$filename" ]] \|\| ! [[ "$filename" =~ ^OpenMS-[-A-Za-z0-9._+~]+-Debian-Linux-x86_64\.deb$ ]]; then
	echo "ERROR: no x86_64 Debian .deb with the expected shell-safe filename in $base/" >&2
	exit 1
	fi
	curl -fsSL --max-time 900 -o "$RUNNER_TEMP/openms-nightly.deb" "$base/$filename"
	bytes=$(stat -c %s "$RUNNER_TEMP/openms-nightly.deb")
	sha=$(sha256sum "$RUNNER_TEMP/openms-nightly.deb" \| cut -d' ' -f1)
	pkg_date=$(printf '%s' "$filename" \| grep -oE '[0-9]{4}-[0-9]{2}-[0-9]{2}')
	echo "Downloaded $filename ($bytes bytes)"
	echo "SHA256: $sha"
	{
	echo "filename=$filename"
	echo "pkg_date=$pkg_date"
	echo "bytes=$bytes"
	echo "sha256=$sha"
	} >> "$GITHUB_OUTPUT"

	- name: Install the package's declared dependencies (as dpkg would)
	shell: bash
	run: \|
	set -euo pipefail
	sudo apt-get update -qq
	depends=$(dpkg-deb -f "$RUNNER_TEMP/openms-nightly.deb" Depends)
	echo "Depends: $depends"
	sudo apt-get -y -qq satisfy --no-install-recommends "$depends"

	- name: Extract package into $RUNNER_TEMP (no system install)
	id: extract
	shell: bash
	run: \|
	set -euo pipefail
	prefix="$RUNNER_TEMP/deb-prefix"
	dpkg-deb -x "$RUNNER_TEMP/openms-nightly.deb" "$prefix"
	for d in usr/bin usr/lib usr/share/OpenMS; do
	[[ -d "$prefix/$d" ]] \|\| { echo "ERROR: $d missing after extraction" >&2; exit 1; }
	done
	test -x "$prefix/usr/share/OpenMS/THIRDPARTY/Comet/comet.exe"
	test -x "$prefix/usr/share/OpenMS/THIRDPARTY/Percolator/percolator"
	echo "prefix=$prefix" >> "$GITHUB_OUTPUT"

	# Exact provenance from the package itself: OpenMS embeds its git
	# revision in the installed openms_package_version.h
	# (OPENMS_GIT_SHA1, OPENMS_GIT_BRANCH, OPENMS_PACKAGE_VERSION). The
	# short SHA is resolved to the full SHA via the GitHub API; a package
	# with an unresolvable SHA is recorded with the short form only and
	# fails any strict SHA request (fail-closed, never invented).
	- name: Resolve package provenance (embedded OpenMS git SHA)
	id: provenance
	env:
	REQUESTED_SHA: ${{ inputs.openms_sha }}
	shell: bash
	run: \|
	set -euo pipefail
	# The input's non-empty default is treated as unset in package
	# mode: workflow_dispatch cannot distinguish "user set it" from
	# "default left in place", and a default package dispatch must
	# never be required to match an arbitrary source-mode SHA.
	PINNED_DEFAULT_SHA="e4b9609c95cac0c64b7d991d0c047d5fac2c7a27"
	hdr="$RUNNER_TEMP/deb-prefix/usr/include/OpenMS/openms_package_version.h"
	if [[ ! -f "$hdr" ]]; then
	echo "ERROR: $hdr not found in package - cannot establish provenance" >&2
	exit 1
	fi
	pkg_short=$(sed -n 's/^#define OPENMS_GIT_SHA1 "$[0-9a-fA-F]$"./\1/p' "$hdr" \| head -1)
	pkg_branch=$(sed -n 's/^#define OPENMS_GIT_BRANCH "$[^"]$"./\1/p' "$hdr" \| head -1)
	pkg_version=$(sed -n 's/^#define OPENMS_PACKAGE_VERSION "$[^"]$"./\1/p' "$hdr" \| head -1)
	if [[ -z "$pkg_short" ]] \|\| ! printf '%s' "$pkg_short" \| grep -qE '^[0-9a-fA-F]{7,40}$'; then
	echo "ERROR: package header carries no usable OPENMS_GIT_SHA1 (got: '$pkg_short')" >&2
	exit 1
	fi
	full=$(curl -fsSL --max-time 30 \
	-H "Accept: application/vnd.github+json" \
	"https://api.github.com/repos/OpenMS/OpenMS/commits/$pkg_short" \
	\| python3 -c 'import json,sys; print(json.load(sys.stdin).get("sha", ""))' \|\| true)
	printf '%s' "$full" \| grep -qE '^[0-9a-f]{40}$' \|\| full=""
	if [[ -z "$full" ]]; then
	# Fail closed on a strict SHA request; otherwise record the short
	# form with an explicit caveat. The pinned default counts as no
	# request (see top of step).
	if [[ -n "${REQUESTED_SHA:-}" && "$REQUESTED_SHA" != "$PINNED_DEFAULT_SHA" ]]; then
	echo "ERROR: could not resolve package SHA '$pkg_short' to a full SHA; cannot verify it against the requested SHA. Aborting (strict SHA guard)." >&2
	exit 1
	fi
	pkg_sha="$pkg_short"
	sha_note="short SHA from package header; full-SHA resolution unavailable (GitHub API)"
	else
	pkg_sha="$full"
	sha_note="resolved to full SHA via GitHub API"
	fi
	echo "Package provenance: sha=$pkg_sha branch=$pkg_branch version=$pkg_version ($sha_note)"
	# Strict SHA guard: a requested openms_sha must match the package.
	# The pinned default counts as no request (see top of step).
	# Charset-validate the caller-controlled input before it reaches the
	# commit-resolution API (defense in depth; a malformed SHA would
	# otherwise be interpolated into the curl URL verbatim).
	if [[ -n "${REQUESTED_SHA:-}" && "$REQUESTED_SHA" != "$PINNED_DEFAULT_SHA" ]]; then
	printf '%s' "$REQUESTED_SHA" \| grep -qE '^[0-9a-fA-F]{7,40}$' \|\| {
	echo "ERROR: requested OpenMS SHA '$REQUESTED_SHA' is not a plausible git SHA (7-40 hex chars)" >&2
	exit 1
	}
	req_full=$(curl -fsSL --max-time 30 \
	-H "Accept: application/vnd.github+json" \
	"https://api.github.com/repos/OpenMS/OpenMS/commits/$REQUESTED_SHA" \
	\| python3 -c 'import json,sys; print(json.load(sys.stdin).get("sha", ""))' \|\| true)
	printf '%s' "$req_full" \| grep -qE '^[0-9a-f]{40}$' \|\| req_full="$REQUESTED_SHA"
	if [[ "$pkg_sha" != "$req_full" ]]; then
	echo "ERROR: strict SHA guard failed - requested OpenMS SHA '$REQUESTED_SHA' (resolves to ${req_full:-unresolvable}) does not match the package's embedded SHA '$pkg_sha'. A package can only satisfy an SHA request by actually being that revision. Use runtime_source=source to benchmark a specific SHA, or clear openms_sha / set it to the package's SHA." >&2
	exit 1
	fi
	echo "Strict SHA guard: requested SHA matches the package ($pkg_sha)"
	fi
	{
	echo "pkg_sha=$pkg_sha"
	echo "pkg_branch=$pkg_branch"
	echo "pkg_version=$pkg_version"
	} >> "$GITHUB_OUTPUT"

	# Capability policy: benchmark support is NEVER inferred from binary
	# availability. A component counts as supported in package mode only
	# once its actual benchmark runner has executed successfully against a
	# package runtime. Today the package-mode job graph runs Smoke and
	# OpenSwath; the ProSE/PeptDeep jobs are structurally skipped (their
	# ONNX source build does not run in package mode). No capability claim
	# is recorded beyond what the executed runners prove.


	run-smoke-benchmark:
	name: "Smoke benchmark (Comet -> Percolator)"
	runs-on: ubuntu-24.04
	# One of build-openms (source mode) or package-runtime (package mode)
	# provides the runtime; the other is skipped by its mode guard, which is
	# not an error. Real failures still block via the needs[...] result check.
	needs: [build-openms, package-runtime]
	if: ${{ !cancelled() && needs.build-openms.result != 'failure' && needs.package-runtime.result != 'failure' && (needs.build-openms.result == 'success' \|\| needs.package-runtime.result == 'success') }}
	env:
	USE_MS2RESCORE: ${{ inputs.use_ms2rescore && 'true' \|\| 'false' }}
	steps:
	- name: Checkout benchmarking repo
	uses: actions/checkout@v6

	# Package mode only: fetch, verify, and extract the .deb on THIS runner
	# (the provider job's prefix lives on a different runner; only metadata
	# crosses jobs). Source mode skips this step entirely — its runtime
	# paths are the pre-Stage-1 literals in the env blocks below.
	- name: Resolve package runtime (package mode)
	id: runtime
	if: ${{ inputs.runtime_source == 'package' }}
	shell: bash
	run: \|
	set -euo pipefail
	curl -fsSL --max-time 900 -o /tmp/openms-nightly.deb \
	"https://archive.openms.de/openms/OpenMSInstaller/nightly/${{ needs.package-runtime.outputs.archive_dir }}/${{ needs.package-runtime.outputs.package_filename }}"
	echo "Expected package SHA256: ${{ needs.package-runtime.outputs.package_sha256 }}"
	sha256sum /tmp/openms-nightly.deb
	echo "$(sha256sum /tmp/openms-nightly.deb)" \| grep -q "^${{ needs.package-runtime.outputs.package_sha256 }} " \
	\|\| { echo "ERROR: downloaded package does not match the SHA256 resolved by package-runtime" >&2; exit 1; }
	sudo apt-get update -qq
	depends=$(dpkg-deb -f /tmp/openms-nightly.deb Depends)
	echo "Depends: $depends"
	sudo apt-get -y -qq satisfy --no-install-recommends "$depends"
	prefix="$RUNNER_TEMP/deb-prefix"
	dpkg-deb -x /tmp/openms-nightly.deb "$prefix"
	rm -f /tmp/openms-nightly.deb
	echo "artifact_bytes=$(du -sb "$prefix" \| awk '{print $1}')" >> "$GITHUB_OUTPUT"

	# Source mode: the downloaded binaries need OpenMS' runtime libraries
	# (xerces, hdf5, libzip, boost, ...) from OpenMS' own deps script at the
	# exact pinned SHA.
	- name: Install runtime dependencies
	if: ${{ inputs.runtime_source != 'package' }}
	shell: bash
	run: \|
	curl -fsSL "https://raw.githubusercontent.com/OpenMS/OpenMS/${{ needs.build-openms.outputs.openms_sha }}/tools/ci/deps-ubuntu.sh" -o deps-ubuntu.sh
	bash deps-ubuntu.sh --skip-gui-deps

	- name: Download OpenMS binaries
	if: ${{ inputs.runtime_source != 'package' }}
	uses: actions/download-artifact@v8
	with:
	name: openms-bin-${{ needs.build-openms.outputs.openms_sha }}
	path: openms

	# GitHub's artifact pipeline does not reliably restore the Unix
	# executable bit, so the downloaded binaries need chmod before use.
	- name: Make downloaded binaries executable
	if: ${{ inputs.runtime_source != 'package' }}
	shell: bash
	run: chmod -R +x openms/build/bin openms/_thirdparty

	- name: Run smoke benchmark
	shell: bash
	env:
	# Both modes feed the same runner contract. Source mode resolves to
	# the pre-Stage-1 literal paths; package mode to the extracted prefix
	# (which resolves its own libs via RUNPATH and its own shared data,
	# proven by the nightly probe — hence empty LD_LIBRARY_PATH/
	# OPENMS_DATA_PATH there).
	OPENMS_BIN: ${{ inputs.runtime_source == 'package' && format('{0}/deb-prefix/usr/bin', runner.temp) \|\| format('{0}/openms/build/bin', github.workspace) }}
	ENGINES_DIR: ${{ inputs.runtime_source == 'package' && format('{0}/deb-prefix/usr/share/OpenMS/THIRDPARTY', runner.temp) \|\| format('{0}/openms/_thirdparty', github.workspace) }}
	# Note the inverted condition: `cond && '' \|\| fallback` never yields
	# an empty value (GHA treats '' as falsy), so package mode would leak
	# the source-mode paths. Inverted form genuinely emits '' in package
	# mode; the source paths are always truthy, so source mode is unchanged.
	LD_LIBRARY_PATH: ${{ inputs.runtime_source != 'package' && format('{0}/openms/build/lib:{0}/openms/build/vcpkg_installed/x64-linux-dynamic/lib', github.workspace) \|\| '' }}
	OPENMS_DATA_PATH: ${{ inputs.runtime_source != 'package' && format('{0}/openms/share/OpenMS', github.workspace) \|\| '' }}
	OPENMS_SHA: ${{ inputs.runtime_source == 'package' && needs.package-runtime.outputs.openms_sha \|\| needs.build-openms.outputs.openms_sha }}
	RESULTS_DIR: ${{ github.workspace }}/results
	run: bash benchmark/run_smoke_benchmark.sh

	# Compute artifact size for the normalization step. Source mode: the
	# binaries layout as before (vcpkg runtime lib dir contributes nothing
	# on legacy-model builds; du skips missing dirs).
	- name: Compute artifact size
	id: artifact-size
	if: ${{ inputs.runtime_source != 'package' }}
	shell: bash
	run: \|
	bytes=$(du -sb openms/build/bin openms/build/lib $( [ -d openms/build/vcpkg_installed/x64-linux-dynamic/lib ] && printf '%s ' openms/build/vcpkg_installed/x64-linux-dynamic/lib ) openms/share/OpenMS openms/_thirdparty 2>/dev/null \| awk '{s+=$1} END{print s}')
	echo "bytes=${bytes:-0}" >> "$GITHUB_OUTPUT"

	# Package mode: the extracted prefix size (recorded during resolution).
	- name: Compute package artifact size
	id: package-artifact-size
	if: ${{ inputs.runtime_source == 'package' }}
	shell: bash
	run: echo "bytes=${{ steps.runtime.outputs.artifact_bytes }}" >> "$GITHUB_OUTPUT"

	# Real cache state: report "warm" only when the vcpkg binary cache was
	# actually restored in the build job (legacy builds have no vcpkg cache
	# and are recorded as "none", matching the previous hardcoding).
	- name: Derive cache state
	id: cache-state
	if: ${{ inputs.runtime_source != 'package' }}
	shell: bash
	run: \|
	cache="none"
	if [ "${{ needs.build-openms.outputs.vcpkg_cache_restored }}" = "true" ]; then
	cache="warm"
	fi
	echo "cache=$cache" >> "$GITHUB_OUTPUT"

	- name: Normalize smoke results to v2 (package mode)
	if: ${{ inputs.runtime_source == 'package' }}
	shell: bash
	run: \|
	python benchmark/report/report_generate.py normalize smoke \
	results/smoke.json \
	--run-id "${{ github.run_id }}" \
	--run-at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
	--cache none \
	--build-time 0 \
	--artifact-bytes "${{ steps.package-artifact-size.outputs.bytes }}" \
	--runtime-source package \
	--package-filename "${{ needs.package-runtime.outputs.package_filename }}" \
	--package-sha256 "${{ needs.package-runtime.outputs.package_sha256 }}" \
	--package-archive-dir "${{ needs.package-runtime.outputs.archive_dir }}" \
	--package-date "${{ needs.package-runtime.outputs.package_date }}" \
	--results-dir results

	- name: Normalize smoke results to v2
	if: ${{ inputs.runtime_source != 'package' }}
	shell: bash
	run: \|
	python benchmark/report/report_generate.py normalize smoke \
	results/smoke.json \
	--run-id "${{ github.run_id }}" \
	--run-at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
	--cache "${{ steps.cache-state.outputs.cache }}" \
	--build-time "$(echo '${{ needs.build-openms.outputs.build_minutes }} * 60' \| bc)" \
	--artifact-bytes "${{ steps.artifact-size.outputs.bytes }}" \
	--results-dir results

	- name: Upload benchmark results
	uses: actions/upload-artifact@v7
	with:
	name: benchmark-results-${{ inputs.runtime_source == 'package' && needs.package-runtime.outputs.openms_sha \|\| needs.build-openms.outputs.openms_sha }}
	path: results
	if-no-files-found: error

	run-openswath-benchmark:
	name: "OpenSwath DIA benchmark"
	runs-on: ubuntu-24.04
	needs: [build-openms, package-runtime]
	if: ${{ !cancelled() && needs.build-openms.result != 'failure' && needs.package-runtime.result != 'failure' && (needs.build-openms.result == 'success' \|\| needs.package-runtime.result == 'success') }}
	steps:
	- name: Checkout benchmarking repo
	uses: actions/checkout@v6

	# Package mode only: fetch, verify, and extract the .deb on THIS runner
	# (same procedure as the smoke job's package step). Source mode skips
	# this step entirely.
	- name: Resolve package runtime (package mode)
	id: runtime
	if: ${{ inputs.runtime_source == 'package' }}
	shell: bash
	run: \|
	set -euo pipefail
	curl -fsSL --max-time 900 -o /tmp/openms-nightly.deb \
	"https://archive.openms.de/openms/OpenMSInstaller/nightly/${{ needs.package-runtime.outputs.archive_dir }}/${{ needs.package-runtime.outputs.package_filename }}"
	echo "$(sha256sum /tmp/openms-nightly.deb)" \| grep -q "^${{ needs.package-runtime.outputs.package_sha256 }} " \
	\|\| { echo "ERROR: downloaded package does not match the SHA256 resolved by package-runtime" >&2; exit 1; }
	sudo apt-get update -qq
	depends=$(dpkg-deb -f /tmp/openms-nightly.deb Depends)
	echo "Depends: $depends"
	sudo apt-get -y -qq satisfy --no-install-recommends "$depends"
	prefix="$RUNNER_TEMP/deb-prefix"
	dpkg-deb -x /tmp/openms-nightly.deb "$prefix"
	rm -f /tmp/openms-nightly.deb
	echo "artifact_bytes=$(du -sb "$prefix" \| awk '{print $1}')" >> "$GITHUB_OUTPUT"

	- name: Install runtime dependencies
	if: ${{ inputs.runtime_source != 'package' }}
	shell: bash
	run: \|
	curl -fsSL "https://raw.githubusercontent.com/OpenMS/OpenMS/${{ needs.build-openms.outputs.openms_sha }}/tools/ci/deps-ubuntu.sh" -o deps-ubuntu.sh
	bash deps-ubuntu.sh --skip-gui-deps

	- name: Download OpenMS binaries
	if: ${{ inputs.runtime_source != 'package' }}
	uses: actions/download-artifact@v8
	with:
	name: openms-bin-${{ needs.build-openms.outputs.openms_sha }}
	path: openms

	- name: Make downloaded binaries executable
	if: ${{ inputs.runtime_source != 'package' }}
	shell: bash
	run: chmod -R +x openms/build/bin openms/_thirdparty

	- name: Run OpenSwath benchmark
	shell: bash
	env:
	OPENMS_BIN: ${{ inputs.runtime_source == 'package' && format('{0}/deb-prefix/usr/bin', runner.temp) \|\| format('{0}/openms/build/bin', github.workspace) }}
	ENGINES_DIR: ${{ inputs.runtime_source == 'package' && format('{0}/deb-prefix/usr/share/OpenMS/THIRDPARTY', runner.temp) \|\| format('{0}/openms/_thirdparty', github.workspace) }}
	# Note the inverted condition: `cond && '' \|\| fallback` never yields
	# an empty value (GHA treats '' as falsy), so package mode would leak
	# the source-mode paths. Inverted form genuinely emits '' in package
	# mode; the source paths are always truthy, so source mode is unchanged.
	LD_LIBRARY_PATH: ${{ inputs.runtime_source != 'package' && format('{0}/openms/build/lib:{0}/openms/build/vcpkg_installed/x64-linux-dynamic/lib', github.workspace) \|\| '' }}
	OPENMS_DATA_PATH: ${{ inputs.runtime_source != 'package' && format('{0}/openms/share/OpenMS', github.workspace) \|\| '' }}
	OPENMS_SHA: ${{ inputs.runtime_source == 'package' && needs.package-runtime.outputs.openms_sha \|\| needs.build-openms.outputs.openms_sha }}
	RESULTS_DIR: ${{ github.workspace }}/results-openswath
	run: bash benchmark/run_openswath_benchmark.sh

	- name: Compute artifact size
	id: artifact-size
	if: ${{ inputs.runtime_source != 'package' }}
	shell: bash
	run: \|
	bytes=$(du -sb openms/build/bin openms/build/lib $( [ -d openms/build/vcpkg_installed/x64-linux-dynamic/lib ] && printf '%s ' openms/build/vcpkg_installed/x64-linux-dynamic/lib ) openms/share/OpenMS openms/_thirdparty 2>/dev/null \| awk '{s+=$1} END{print s}')
	echo "bytes=${bytes:-0}" >> "$GITHUB_OUTPUT"

	- name: Compute package artifact size
	id: package-artifact-size
	if: ${{ inputs.runtime_source == 'package' }}
	shell: bash
	run: echo "bytes=${{ steps.runtime.outputs.artifact_bytes }}" >> "$GITHUB_OUTPUT"

	# Main's normalize contract: raw CI openswath.json (v1) + required
	# build-time/artifact-bytes provenance, normalized output written into
	# the results dir and shipped in the artifact.
	- name: Derive cache state
	id: cache-state
	if: ${{ inputs.runtime_source != 'package' }}
	shell: bash
	run: \|
	cache="none"
	if [ "${{ needs.build-openms.outputs.vcpkg_cache_restored }}" = "true" ]; then
	cache="warm"
	fi
	echo "cache=$cache" >> "$GITHUB_OUTPUT"

	- name: Normalize OpenSwath results to v2 (package mode)
	if: ${{ inputs.runtime_source == 'package' }}
	shell: bash
	run: \|
	python benchmark/report/report_generate.py normalize openswath \
	results-openswath/openswath.json \
	--run-id "${{ github.run_id }}" \
	--run-at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
	--cache none \
	--build-time 0 \
	--artifact-bytes "${{ steps.package-artifact-size.outputs.bytes }}" \
	--runtime-source package \
	--package-filename "${{ needs.package-runtime.outputs.package_filename }}" \
	--package-sha256 "${{ needs.package-runtime.outputs.package_sha256 }}" \
	--package-archive-dir "${{ needs.package-runtime.outputs.archive_dir }}" \
	--package-date "${{ needs.package-runtime.outputs.package_date }}" \
	--results-dir results-openswath

	- name: Normalize OpenSwath results to v2
	if: ${{ inputs.runtime_source != 'package' }}
	shell: bash
	run: \|
	python benchmark/report/report_generate.py normalize openswath \
	results-openswath/openswath.json \
	--run-id "${{ github.run_id }}" \
	--run-at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
	--cache "${{ steps.cache-state.outputs.cache }}" \
	--build-time "$(echo '${{ needs.build-openms.outputs.build_minutes }} * 60' \| bc)" \
	--artifact-bytes "${{ steps.artifact-size.outputs.bytes }}" \
	--results-dir results-openswath

	- name: Upload OpenSwath benchmark results
	uses: actions/upload-artifact@v7
	with:
	name: benchmark-openswath-${{ inputs.runtime_source == 'package' && needs.package-runtime.outputs.openms_sha \|\| needs.build-openms.outputs.openms_sha }}
	path: results-openswath
	if-no-files-found: error
	# ---------------------------------------------------------------------------
	# ProSE + PeptDeep ONNX pipeline validation.
	#
	# Separate ONNX-enabled build job so the existing build-openms artifact
	# (consumed by smoke + OpenSwath) stays byte-identical: different configure
	# flags, two extra downloads, separate ccache key prefix.
	#
	# The fixture is pipeline-validation-only: the downstream job asserts that
	# ProSE executes, Percolator really runs (q-values), the PeptDeep arm really
	# runs ONNX inference (five features), entrapment accessions survive, and
	# prose.json + normalization are produced. It does NOT require q <= 0.01
	# survivors (0 on this CID fixture; documented).
	#
	# Dependency model: identical detection to build-openms. Post-vcpkg SHAs
	# receive the libraries (including Arrow) from the vcpkg manifest; legacy
	# SHAs receive the apt libraries plus prebuilt conda-forge Arrow (>= 23 is a
	# hard requirement of the ONNX support and Ubuntu 24.04 apt ships Arrow 5).
	# ---------------------------------------------------------------------------
	build-openms-onnx:
	name: "Build OpenMS with ONNX (ProSE + PeptDeep, fixed SHA)"
	runs-on: ubuntu-24.04
	timeout-minutes: 90
	# ONNX benchmarking requires a source build with WITH_ONNX=ON; published
	# Debian packages do not contain ONNX Runtime (verified: no libonnxruntime
	# and no share/OpenMS/models/*.onnx in the nightly payload).
	if: ${{ inputs.runtime_source != 'package' }}
	outputs:
	openms_sha: ${{ steps.sha.outputs.sha }}
	steps:
	- name: Checkout OpenMS at pinned SHA
	uses: actions/checkout@v6
	with:
	repository: OpenMS/OpenMS
	ref: ${{ inputs.openms_sha }}

	- name: Resolve pinned SHA
	id: sha
	shell: bash
	run: echo "sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"

	- name: Detect dependency model
	id: depmodel
	shell: bash
	run: \|
	set -euo pipefail

	capable=false
	if [[ -f vcpkg.json ]] \
	&& grep -q "OPENMS_USE_VCPKG" CMakeLists.txt \
	&& grep -Eq '^[[:space:]]path[[:space:]]=[[:space:]]vcpkg[[:space:]]$' .gitmodules; then
	capable=true
	fi

	legacy_libs=false
	if grep -Eqi 'libxerces\|boost\|libsvm\|libzip\|hdf5\|eigen\|seqan\|coinor\|glpk\|wildmagic' tools/ci/deps-ubuntu.sh; then
	legacy_libs=true
	fi

	if [[ "$legacy_libs" == true ]]; then
	echo "model=legacy" >> "$GITHUB_OUTPUT"
	echo "OPENMS_CMAKE_VCPKG_ARGS=" >> "$GITHUB_ENV"
	echo "Dependency model: legacy (deps script installs libraries via apt)"
	elif [[ "$capable" == true ]]; then
	echo "model=vcpkg" >> "$GITHUB_OUTPUT"
	# onnxruntime: deliberately NOT a vcpkg feature - the overlay port builds ORT
	# from source (dbg+rel), which exhausted the runner disk and the 90-minute
	# budget (runs 35700644257, 35713031426, 35736568407). find_package(onnxruntime
	# CONFIG REQUIRED) instead resolves the pinned prebuilt release archive via
	# -Donnxruntime_DIR (staged in the download step for the config package's
	# lib64 / include-onnxruntime layout). wnetalign stays: WITH_WNETALIGN defaults ON.
	echo "OPENMS_CMAKE_VCPKG_ARGS=-DOPENMS_USE_VCPKG=ON -DVCPKG_TARGET_TRIPLET=x64-linux-dynamic -DBOOST_USE_STATIC=OFF -DVCPKG_MANIFEST_FEATURES=wnetalign" >> "$GITHUB_ENV"
	echo "Dependency model: vcpkg manifest (deps script installs no libraries)"
	else
	echo "::error::OpenMS at ${{ steps.sha.outputs.sha }} neither installs libraries via tools/ci/deps-ubuntu.sh nor ships the vcpkg mechanism (vcpkg.json / OPENMS_USE_VCPKG in CMakeLists.txt / vcpkg submodule in .gitmodules). No workable dependency model for this SHA." >&2
	exit 1
	fi

	- name: Install dependencies
	shell: bash
	run: bash tools/ci/deps-ubuntu.sh --skip-gui-deps

	# Arrow >= 23 is a hard requirement of the ONNX support and Ubuntu 24.04
	# apt ships Arrow 5. Legacy-model builds get prebuilt conda-forge
	# libarrow via micromamba; post-vcpkg SHAs receive Arrow from the vcpkg
	# manifest instead. Versions pinned to the exact set that produced the
	# locally validated binaries.
	- name: Install prebuilt Arrow 25 (micromamba, legacy model only)
	if: steps.depmodel.outputs.model == 'legacy'
	shell: bash
	run: \|
	curl -fsSL -o "$RUNNER_TEMP/mm.tar.bz2" \
	https://anaconda.org/conda-forge/micromamba/2.9.0/download/linux-64/micromamba-2.9.0-0.tar.bz2
	mkdir -p "$RUNNER_TEMP/mm"
	tar -xjf "$RUNNER_TEMP/mm.tar.bz2" -C "$RUNNER_TEMP/mm" bin/micromamba
	MM="$RUNNER_TEMP/mm/bin/micromamba"
	"$MM" create -y -p "$RUNNER_TEMP/arrow-pref" -c conda-forge \
	libarrow=25.0.0 libparquet=25.0.0 xz zstd
	# conda-forge ships the runtime liblzma; Arrow's static link wants
	# the dev-style -llzma symlink (seen in the local build).
	if [ ! -e "$RUNNER_TEMP/arrow-pref/lib/liblzma.so" ] \
	&& [ -e "$RUNNER_TEMP/arrow-pref/lib/liblzma.so.5" ]; then
	ln -s liblzma.so.5 "$RUNNER_TEMP/arrow-pref/lib/liblzma.so"
	fi

	- name: Set legacy-model CMake extras
	if: steps.depmodel.outputs.model == 'legacy'
	shell: bash
	run: echo "OPENMS_ONNX_MODEL_FLAGS=-DOPENMS_USE_VCPKG=OFF -DCMAKE_PREFIX_PATH=$RUNNER_TEMP/arrow-pref" >> "$GITHUB_ENV"

	- name: Set vcpkg-model CMake extras
	if: steps.depmodel.outputs.model == 'vcpkg'
	shell: bash
	run: echo "OPENMS_ONNX_MODEL_FLAGS=" >> "$GITHUB_ENV"

	# Same ccache strategy as build-openms but a SEPARATE key prefix: the
	# ONNX configure adds include paths, so object files differ from the
	# non-ONNX build flavor. CCACHE_BASEDIR keeps paths comparable across
	# checkouts.
	- name: Restore compiler cache (ONNX flavor)
	uses: actions/cache@v5
	with:
	path: .ccache
	key: openms-onnx-ccache-${{ runner.os }}-${{ runner.arch }}-${{ steps.sha.outputs.sha }}
	restore-keys: \|
	openms-onnx-ccache-${{ runner.os }}-${{ runner.arch }}-${{ steps.sha.outputs.sha }}
	openms-onnx-ccache-${{ runner.os }}-${{ runner.arch }}-

	- name: Configure ccache
	shell: bash
	run: \|
	mkdir -p "$GITHUB_WORKSPACE/.ccache"
	{
	echo "CCACHE_COMPRESS=true"
	echo "CCACHE_COMPRESSLEVEL=6"
	echo "CCACHE_DIR=$GITHUB_WORKSPACE/.ccache"
	echo "CCACHE_MAXSIZE=1G"
	echo "CCACHE_BASEDIR=$GITHUB_WORKSPACE"
	} >> "$GITHUB_ENV"

	# ---- vcpkg dependency model (same mechanism as build-openms) ----------
	- name: Restore vcpkg binary cache (ONNX flavor)
	id: vcpkg-cache
	if: steps.depmodel.outputs.model == 'vcpkg'
	uses: actions/cache@v5
	with:
	path: .vcpkg-cache
	key: openms-vcpkg-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('vcpkg.json', 'vcpkg-configuration.json') }}
	restore-keys: \|
	openms-vcpkg-${{ runner.os }}-${{ runner.arch }}-

	# Cold-cache fallback, identical to build-openms (see comment there).
	- name: Seed vcpkg cache from upstream archive (on miss)
	if: steps.depmodel.outputs.model == 'vcpkg' && steps.vcpkg-cache.outputs.cache-hit != 'true'
	shell: bash
	run: \|
	set -euo pipefail
	manifest_hash="${{ hashFiles('vcpkg.json', 'vcpkg-configuration.json') }}"
	base="https://archive.openms.de/openms/vcpkg-cache/Linux-X64-g++-vcpkg-${manifest_hash}"
	if ! curl -fsSL --max-time 30 "${base}/" -o /tmp/upstream-index.html 2>/dev/null; then
	echo "upstream vcpkg cache not reachable (continuing without seed)"
	exit 0
	fi
	shards=$(grep -oE 'href="[0-9a-f]{2}/"' /tmp/upstream-index.html \| sed 's/href="$.*$"/\1/' \| tr -d '/' \| sort -u)
	if [ -z "$shards" ]; then
	echo "upstream cache layout unexpected (no shards found); continuing without seed"
	exit 0
	fi
	for shard in $shards; do
	curl -fsSL --max-time 30 "${base}/${shard}/" -o "/tmp/shard-${shard}.html" \|\| continue
	zips=$(grep -oE 'href="[0-9a-f]{64}\.zip"' "/tmp/shard-${shard}.html" \| sed 's/href="$.*$"/\1/')
	for zip_name in $zips; do
	dest="$GITHUB_WORKSPACE/.vcpkg-cache/${shard}"
	mkdir -p "$dest"
	curl -fsSL --retry 2 -o "${dest}/${zip_name}" "${base}/${shard}/${zip_name}" \|\| rm -f "${dest}/${zip_name}"
	done
	rm -f "/tmp/shard-${shard}.html"
	done
	seeded=$(find "$GITHUB_WORKSPACE/.vcpkg-cache" -name '*.zip' 2>/dev/null \| wc -l)
	echo "seeded ${seeded} vcpkg binary-cache zips from upstream archive"

	- name: Set up vcpkg
	if: steps.depmodel.outputs.model == 'vcpkg'
	shell: bash
	run: \|
	git submodule update --init vcpkg
	mkdir -p "$GITHUB_WORKSPACE/.vcpkg-cache"
	{
	echo "VCPKG_BINARY_SOURCES=clear;files,$GITHUB_WORKSPACE/.vcpkg-cache,readwrite"
	} >> "$GITHUB_ENV"

	- name: Bootstrap vcpkg
	if: steps.depmodel.outputs.model == 'vcpkg'
	uses: lukka/run-vcpkg@v11
	with:
	doNotCache: true

	- name: Stage search engines
	shell: bash
	run: \|
	git submodule update --init THIRDPARTY
	mkdir -p _thirdparty
	cp -R THIRDPARTY/Linux/$(uname -m)/* _thirdparty/
	cp -R THIRDPARTY/All/* _thirdparty/

	# Prebuilt onnxruntime 1.23.2 from the official GitHub release; checksum
	# pinned to the byte-identical tarball used for the locally validated
	# build.
	- name: Download onnxruntime 1.23.2 (pinned)
	shell: bash
	run: \|
	mkdir -p "$RUNNER_TEMP/ort"
	curl -fsSL -o "$RUNNER_TEMP/ort.tgz" \
	https://github.com/microsoft/onnxruntime/releases/download/v1.23.2/onnxruntime-linux-x64-1.23.2.tgz
	echo "1fa4dcaef22f6f7d5cd81b28c2800414350c10116f5fdd46a2160082551c5f9b $RUNNER_TEMP/ort.tgz" \| sha256sum -c -
	tar -xzf "$RUNNER_TEMP/ort.tgz" -C "$RUNNER_TEMP/ort" --strip-components=1
	# Stage a config-package-compatible prefix: the release archive ships lib/ and
	# include/, but its generated onnxruntimeConfig.cmake references lib64/ and
	# include/onnxruntime/ (vcpkg-style layout) - re-layout so the CONFIG search
	# resolves real paths. The raw extraction stays in ort/ for the legacy-model
	# flags and the runtime-lib bundling step.
	O="$RUNNER_TEMP/ort-prefix"
	mkdir -p "$O/lib64" "$O/lib/cmake/onnxruntime" "$O/include/onnxruntime"
	cp -a "$RUNNER_TEMP/ort/lib/libonnxruntime"* "$O/lib64/"
	cp -a "$RUNNER_TEMP/ort/lib/cmake/onnxruntime/." "$O/lib/cmake/onnxruntime/"
	cp -a "$RUNNER_TEMP/ort/include/." "$O/include/onnxruntime/"

	# WITH_THERMO_RAW=OFF: vendor .raw reading needs the .NET SDK and is
	# irrelevant for mzML inputs. WITH_ONNX=ON makes configure download the
	# three PeptDeep models from archive.openms.de and verify their pinned
	# SHA256s (configure fails hard on mismatch). Legacy model links Arrow
	# from the micromamba prefix via OPENMS_ONNX_MODEL_FLAGS; vcpkg model
	# links it from the manifest via OPENMS_CMAKE_VCPKG_ARGS, and ORT from the
	# staged prebuilt release archive via -Donnxruntime_DIR (CONFIG mode).
	- name: Configure (Release, Ninja, ONNX)
	shell: bash
	env:
	SEARCH_ENGINES_DIRECTORY: ${{ github.workspace }}/_thirdparty
	run: \|
	cmake -S . -B build \
	-G Ninja \
	-DCMAKE_BUILD_TYPE=Release \
	-DCMAKE_C_COMPILER_LAUNCHER=ccache \
	-DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
	-DWITH_GUI=OFF \
	-DENABLE_DOCS=OFF \
	-DWITH_THERMO_RAW=OFF \
	-DWITH_ONNX=ON \
	-DONNXRuntime_INCLUDE_DIR="$RUNNER_TEMP/ort/include" \
	-DONNXRuntime_LIBRARY="$RUNNER_TEMP/ort/lib/libonnxruntime.so" \
	-Donnxruntime_DIR="$RUNNER_TEMP/ort-prefix/lib/cmake/onnxruntime" \
	-DSEARCH_ENGINES_DIRECTORY="$SEARCH_ENGINES_DIRECTORY" \
	$OPENMS_CMAKE_VCPKG_ARGS \
	$OPENMS_ONNX_MODEL_FLAGS

	# Only what the benchmark needs (the full TOPP set is built by the
	# non-ONNX job; this flavor exists for ProSE).
	- name: Build ProSE + PercolatorAdapter + FileMerger
	shell: bash
	run: cmake --build build --target ProSE PercolatorAdapter FileMerger

	# Lay out the build-tree share the way ProSE resolves relative model
	# paths (<exe>/../share/OpenMS): CMake downloads the models into the
	# src/openms build subtree, so copy them next to the binaries.
	- name: Place PeptDeep models in build-tree share
	shell: bash
	run: \|
	MODEL_SRC=$(find build -path "*/share/OpenMS/models/peptdeep_ms2_dynamic.onnx" -printf "%h\n" \| head -n1)
	if [ -z "$MODEL_SRC" ]; then
	echo "ERROR: PeptDeep models not found in build tree" >&2
	exit 1
	fi
	mkdir -p build/share/OpenMS/models
	cp "$MODEL_SRC"/*.onnx build/share/OpenMS/models/
	sha256sum build/share/OpenMS/models/*.onnx

	# Ship the onnxruntime shared object(s) with build/lib so the run jobs'
	# single LD_LIBRARY_PATH covers them (cp -a keeps the .so.1 symlinks).
	- name: Bundle onnxruntime runtime libs
	shell: bash
	run: cp -a "$RUNNER_TEMP/ort/lib/libonnxruntime.so"* build/lib/

	- name: Upload OpenMS ONNX binaries, models, engines
	uses: actions/upload-artifact@v7
	with:
	name: openms-onnx-bin-${{ steps.sha.outputs.sha }}
	path: \|
	build/bin
	build/lib
	build/vcpkg_installed/x64-linux-dynamic/lib
	build/share/OpenMS/models
	share/OpenMS
	_thirdparty
	if-no-files-found: error

	run-prose-peptdeep-benchmark:
	name: "ProSE + PeptDeep ONNX benchmark (pipeline validation)"
	runs-on: ubuntu-24.04
	timeout-minutes: 30
	needs: build-openms-onnx
	# Capability gate: ONNX benchmarks require the source ONNX build. Package
	# mode provides no ONNX Runtime today (verified from the package payload),
	# so these jobs are skipped there rather than failing or pretending.
	if: ${{ !cancelled() && needs.build-openms-onnx.result == 'success' }}
	steps:
	- name: Checkout benchmarking repo
	uses: actions/checkout@v6

	# The downloaded binaries need OpenMS' runtime libraries; install the
	# same non-GUI deps the build job uses. The deps script lives in the
	# OpenMS repo (not this one), so fetch it at the exact built SHA — the
	# same adaptive pattern the smoke and OpenSwath jobs use.
	- name: Install runtime dependencies
	shell: bash
	run: \|
	curl -fsSL "https://raw.githubusercontent.com/OpenMS/OpenMS/${{ needs.build-openms-onnx.outputs.openms_sha }}/tools/ci/deps-ubuntu.sh" -o deps-ubuntu.sh
	bash deps-ubuntu.sh --skip-gui-deps

	- name: Download OpenMS ONNX binaries
	uses: actions/download-artifact@v8
	with:
	name: openms-onnx-bin-${{ needs.build-openms-onnx.outputs.openms_sha }}
	path: openms

	- name: Make downloaded binaries executable
	shell: bash
	run: chmod -R +x openms/build/bin openms/_thirdparty

	# Source tree (pinned SHA, sparse: only the files the preparation script
	# reads) provides P02769.fasta, the E. coli target_decoy FASTA and the
	# three BSA mzML acquisitions. Generated inputs land in the git-ignored
	# benchmark/generated/prose/ and are never committed.
	- name: Checkout OpenMS sources (pinned, sparse)
	uses: actions/checkout@v6
	with:
	repository: OpenMS/OpenMS
	ref: ${{ needs.build-openms-onnx.outputs.openms_sha }}
	path: openms-src
	fetch-depth: 1
	sparse-checkout: \|
	doc/pyopenms/src/data/P02769.fasta
	share/OpenMS/examples/BSA
	share/OpenMS/examples/TOPPAS/data/Identification/target_decoy_Ecoli_K12_TaxID_83333.proteomes.fasta

	- name: Generate entrapment FASTA + merged mzML
	shell: bash
	env:
	# The artifact ships libOpenMS.so in build/lib (plus vcpkg runtime
	# libs on post-vcpkg builds), but the binaries' baked RUNPATH points
	# at the build job's layout (<ws>/build/lib), not this job's
	# (<ws>/openms/build/lib) - so point the dynamic linker there for
	# FileMerger, exactly like the run step below.
	LD_LIBRARY_PATH: ${{ github.workspace }}/openms/build/lib:${{ github.workspace }}/openms/build/vcpkg_installed/x64-linux-dynamic/lib
	# OpenMS tools abort without the shared data (schemas, CHEMISTRY);
	# the artifact's share/OpenMS mirrors the source tree the smoke and
	# OpenSwath jobs already point OPENMS_DATA_PATH at.
	OPENMS_DATA_PATH: ${{ github.workspace }}/openms/share/OpenMS
	run: \|
	bash benchmark/prepare_prose_inputs.sh \
	"$GITHUB_WORKSPACE/openms-src" \
	"$GITHUB_WORKSPACE/openms/build/bin" \
	"$GITHUB_WORKSPACE/benchmark/generated/prose"
	cat "$GITHUB_WORKSPACE/benchmark/generated/prose/PROVENANCE.txt"

	# PROSE_EXTRA_SEARCH_ARGS: the fixture is LTQ-Orbitrap ion-trap CID, so
	# the 20 ppm fragment default yields zero IDs (verified control);
	# 0.5 Da is the instrument-matched tolerance validated locally. The
	# runner applies this env var IDENTICALLY to both arms, so arms differ
	# only in PeptDeep enablement (benchmark semantics unchanged).
	- name: Run ProSE + PeptDeep benchmark (both arms)
	shell: bash
	env:
	OPENMS_BIN: ${{ github.workspace }}/openms/build/bin
	PROSE_INPUT: ${{ github.workspace }}/benchmark/generated/prose/bsa_merged.mzML
	PROSE_FASTA: ${{ github.workspace }}/benchmark/generated/prose/entrapment_target.fasta
	ENGINES_DIR: ${{ github.workspace }}/openms/_thirdparty
	PROSE_EXTRA_SEARCH_ARGS: "-Search:fragment:mass_tolerance 0.5 -Search:fragment:mass_tolerance_unit Da"
	LD_LIBRARY_PATH: ${{ github.workspace }}/openms/build/lib:${{ github.workspace }}/openms/build/vcpkg_installed/x64-linux-dynamic/lib
	OPENMS_DATA_PATH: ${{ github.workspace }}/openms/share/OpenMS
	OPENMS_SHA: ${{ needs.build-openms-onnx.outputs.openms_sha }}
	RESULTS_DIR: ${{ github.workspace }}/results-prose
	run: bash benchmark/run_prose_benchmark.sh

	# Guard step (unchanged runner semantics, new consumer): fails the job
	# unless ProSE executed, both arms carry q-values (Percolator really
	# ran), the peptdeep arm carries all five ONNX features, entrapment
	# accessions survive into the idXML, and the verdict is pass.
	- name: Validate pipeline guards
	shell: bash
	run: \|
	python3 - <<'PY'
	import json, sys, xml.etree.ElementTree as ET
	sys.path.insert(0, "benchmark")
	import prose_metrics as pm

	prefix = ""
	with open("results-prose/meta.txt") as fh:
	for line in fh:
	if line.startswith("entrapment_prefix="):
	prefix = line.split("=", 1)[1].strip()

	raw = json.load(open("results-prose/prose.json"))
	assert raw.get("verdict") == "pass", "verdict: %r" % raw.get("verdict")
	stages = {s["name"]: s for s in raw["stages"]}
	for name in ("prose_baseline", "prose_peptdeep"):
	assert stages[name]["status"] == "pass", stages[name]

	for arm in ("baseline", "peptdeep"):
	a = raw["arms"][arm]
	assert "error" not in a, "%s arm error: %s" % (arm, a.get("error"))
	assert a["psms_total"] > 0, "%s arm: no PSMs" % arm
	parsed = pm.parse_idxml("results-prose/%s.idXML" % arm, prefix)
	assert parsed["score_type"] == "q-value", \
	"%s arm score_type %r (Percolator did not run?)" % (arm, parsed["score_type"])
	assert parsed["psms"], "%s arm: no parseable PSMs" % arm
	root = ET.parse("results-prose/%s.idXML" % arm).getroot()
	accs = {ph.get("accession") for ph in root.iter("ProteinHit")}
	assert any(x and x.startswith(prefix) for x in accs), \
	"%s arm: no %s ProteinHits" % (arm, prefix)

	missing = raw["arms"]["peptdeep"].get("peptdeep_features_missing")
	assert missing == [], "PeptDeep features missing: %s" % missing
	print("prose-peptdeep CI validation: all pipeline guards passed")
	PY

	- name: Normalize ProSE+PeptDeep results to v2
	shell: bash
	run: \|
	python benchmark/report/report_generate.py normalize prose-peptdeep \
	--run-id "${{ github.run_id }}" \
	--run-at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
	--cache none \
	--results-dir results-prose

	- name: Upload ProSE+PeptDeep benchmark results
	uses: actions/upload-artifact@v7
	with:
	name: benchmark-prose-${{ needs.build-openms-onnx.outputs.openms_sha }}
	path: results-prose
	if-no-files-found: error

	# ---------------------------------------------------------------------------
	# PXD028735 scientific-dataset benchmark (Phase-2c). Same pipeline-validation
	# scope as run-prose-peptdeep-benchmark above: the guards assert execution
	# integrity (verdict, stages, q-values, entrapment survival, five PeptDeep
	# features), NOT a q <= 0.01 result. Dataset: ProteoBench QE HF-X HYE (CC0,
	# doi:10.1038/s41597-022-01216-6); inputs are downloaded byte-pinned by
	# benchmark/prepare_pxd028735_inputs.sh (nothing is vendored or committed).
	#
	# The runner is single-input and wipes its RESULTS_DIR at startup, so the
	# two validated Alpha_01 runs execute as fully isolated invocations: each
	# gets its own mzML, RESULTS_DIR, guard execution and normalization.
	# ---------------------------------------------------------------------------
	run-prose-pxd028735-benchmark:
	name: "ProSE + PeptDeep ONNX benchmark (PXD028735, pipeline validation)"
	runs-on: ubuntu-24.04
	timeout-minutes: 60
	needs: build-openms-onnx
	# Capability gate: see run-prose-peptdeep-benchmark.
	if: ${{ !cancelled() && needs.build-openms-onnx.result == 'success' }}
	steps:
	- name: Checkout benchmarking repo
	uses: actions/checkout@v6

	- name: Install runtime dependencies
	shell: bash
	run: \|
	curl -fsSL "https://raw.githubusercontent.com/OpenMS/OpenMS/${{ needs.build-openms-onnx.outputs.openms_sha }}/tools/ci/deps-ubuntu.sh" -o deps-ubuntu.sh
	bash deps-ubuntu.sh --skip-gui-deps

	- name: Download OpenMS ONNX binaries
	uses: actions/download-artifact@v8
	with:
	name: openms-onnx-bin-${{ needs.build-openms-onnx.outputs.openms_sha }}
	path: openms

	- name: Make downloaded binaries executable
	shell: bash
	run: chmod -R +x openms/build/bin openms/_thirdparty

	# Source tree (pinned SHA, sparse): only the targets-only S. cellulosum
	# FASTA the PXD preparation script reads. The HYE target FASTA and the
	# two mzML runs are downloaded from archive.openms.de by the script
	# itself (pinned byte size + sha256, fail-closed on mismatch).
	- name: Checkout OpenMS sources (pinned, sparse)
	uses: actions/checkout@v6
	with:
	repository: OpenMS/OpenMS
	ref: ${{ needs.build-openms-onnx.outputs.openms_sha }}
	path: openms-src
	fetch-depth: 1
	sparse-checkout: \|
	doc/pyopenms/src/data/18Protein_SoCe_Tr_detergents_trace.fasta

	# No OPENMS_DATA_PATH / LD_LIBRARY_PATH needed here: unlike the BSA
	# preparation this script invokes no OpenMS binary (pinned downloads +
	# deterministic FASTA assembly only). Generated inputs land in the
	# git-ignored benchmark/generated/prose-pxd028735/ and are never
	# committed.
	- name: Prepare PXD028735 inputs (runs A and B)
	shell: bash
	run: \|
	bash benchmark/prepare_pxd028735_inputs.sh \
	"$GITHUB_WORKSPACE/openms-src" \
	"$GITHUB_WORKSPACE/benchmark/generated/prose-pxd028735" \
	"A B"
	cat "$GITHUB_WORKSPACE/benchmark/generated/prose-pxd028735/PROVENANCE.txt"

	# Run A. The 0.02 Da fragment tolerance is the published PXD028735/
	# ProteoBench parameter (validated through this runner locally);
	# precursor 10 ppm is the ProSE default. Search configuration is
	# identical for A and B and both arms (the runner applies
	# PROSE_EXTRA_SEARCH_ARGS to baseline and peptdeep alike).
	- name: Run ProSE + PeptDeep benchmark (PXD028735 run A)
	shell: bash
	env:
	OPENMS_BIN: ${{ github.workspace }}/openms/build/bin
	PROSE_INPUT: ${{ github.workspace }}/benchmark/generated/prose-pxd028735/LFQ_Orbitrap_DDA_Condition_A_Sample_Alpha_01.mzML
	PROSE_FASTA: ${{ github.workspace }}/benchmark/generated/prose-pxd028735/entrapment_hye.fasta
	ENGINES_DIR: ${{ github.workspace }}/openms/_thirdparty
	PROSE_EXTRA_SEARCH_ARGS: "-Search:fragment:mass_tolerance 0.02 -Search:fragment:mass_tolerance_unit Da"
	LD_LIBRARY_PATH: ${{ github.workspace }}/openms/build/lib:${{ github.workspace }}/openms/build/vcpkg_installed/x64-linux-dynamic/lib
	OPENMS_DATA_PATH: ${{ github.workspace }}/openms/share/OpenMS
	OPENMS_SHA: ${{ needs.build-openms-onnx.outputs.openms_sha }}
	RESULTS_DIR: ${{ github.workspace }}/results-prose-pxd028735-A
	run: bash benchmark/run_prose_benchmark.sh

	# Same pipeline guards as the BSA job, parameterized by RESULTS_DIR so
	# the run A and run B guard steps are identical apart from the value.
	- name: Validate pipeline guards (run A)
	shell: bash
	env:
	RESULTS_DIR: results-prose-pxd028735-A
	run: \|
	python3 - <<'PY'
	import json, os, sys, xml.etree.ElementTree as ET
	sys.path.insert(0, "benchmark")
	import prose_metrics as pm

	r = os.environ["RESULTS_DIR"]
	prefix = ""
	with open(os.path.join(r, "meta.txt")) as fh:
	for line in fh:
	if line.startswith("entrapment_prefix="):
	prefix = line.split("=", 1)[1].strip()

	raw = json.load(open(os.path.join(r, "prose.json")))
	assert raw.get("verdict") == "pass", "verdict: %r" % raw.get("verdict")
	stages = {s["name"]: s for s in raw["stages"]}
	for name in ("prose_baseline", "prose_peptdeep"):
	assert stages[name]["status"] == "pass", stages[name]

	for arm in ("baseline", "peptdeep"):
	a = raw["arms"][arm]
	assert "error" not in a, "%s arm error: %s" % (arm, a.get("error"))
	assert a["psms_total"] > 0, "%s arm: no PSMs" % arm
	parsed = pm.parse_idxml(os.path.join(r, "%s.idXML" % arm), prefix)
	assert parsed["score_type"] == "q-value", \
	"%s arm score_type %r (Percolator did not run?)" % (arm, parsed["score_type"])
	assert parsed["psms"], "%s arm: no parseable PSMs" % arm
	root = ET.parse(os.path.join(r, "%s.idXML" % arm)).getroot()
	accs = {ph.get("accession") for ph in root.iter("ProteinHit")}
	assert any(x and x.startswith(prefix) for x in accs), \
	"%s arm: no %s ProteinHits" % (arm, prefix)

	missing = raw["arms"]["peptdeep"].get("peptdeep_features_missing")
	assert missing == [], "PeptDeep features missing: %s" % missing
	print("%s: all pipeline guards passed" % r)
	PY

	- name: Normalize PXD028735 run A results to v2
	shell: bash
	run: \|
	python benchmark/report/report_generate.py normalize prose-peptdeep \
	--run-id "${{ github.run_id }}-A" \
	--run-at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
	--cache none \
	--results-dir results-prose-pxd028735-A

	# Run B: identical configuration; only the input mzML and RESULTS_DIR
	# differ from run A (the runner wipes its RESULTS_DIR at startup, so
	# A and B must not share one).
	- name: Run ProSE + PeptDeep benchmark (PXD028735 run B)
	shell: bash
	env:
	OPENMS_BIN: ${{ github.workspace }}/openms/build/bin
	PROSE_INPUT: ${{ github.workspace }}/benchmark/generated/prose-pxd028735/LFQ_Orbitrap_DDA_Condition_B_Sample_Alpha_01.mzML
	PROSE_FASTA: ${{ github.workspace }}/benchmark/generated/prose-pxd028735/entrapment_hye.fasta
	ENGINES_DIR: ${{ github.workspace }}/openms/_thirdparty
	PROSE_EXTRA_SEARCH_ARGS: "-Search:fragment:mass_tolerance 0.02 -Search:fragment:mass_tolerance_unit Da"
	LD_LIBRARY_PATH: ${{ github.workspace }}/openms/build/lib:${{ github.workspace }}/openms/build/vcpkg_installed/x64-linux-dynamic/lib
	OPENMS_DATA_PATH: ${{ github.workspace }}/openms/share/OpenMS
	OPENMS_SHA: ${{ needs.build-openms-onnx.outputs.openms_sha }}
	RESULTS_DIR: ${{ github.workspace }}/results-prose-pxd028735-B
	run: bash benchmark/run_prose_benchmark.sh

	- name: Validate pipeline guards (run B)
	shell: bash
	env:
	RESULTS_DIR: results-prose-pxd028735-B
	run: \|
	python3 - <<'PY'
	import json, os, sys, xml.etree.ElementTree as ET
	sys.path.insert(0, "benchmark")
	import prose_metrics as pm

	r = os.environ["RESULTS_DIR"]
	prefix = ""
	with open(os.path.join(r, "meta.txt")) as fh:
	for line in fh:
	if line.startswith("entrapment_prefix="):
	prefix = line.split("=", 1)[1].strip()

	raw = json.load(open(os.path.join(r, "prose.json")))
	assert raw.get("verdict") == "pass", "verdict: %r" % raw.get("verdict")
	stages = {s["name"]: s for s in raw["stages"]}
	for name in ("prose_baseline", "prose_peptdeep"):
	assert stages[name]["status"] == "pass", stages[name]

	for arm in ("baseline", "peptdeep"):
	a = raw["arms"][arm]
	assert "error" not in a, "%s arm error: %s" % (arm, a.get("error"))
	assert a["psms_total"] > 0, "%s arm: no PSMs" % arm
	parsed = pm.parse_idxml(os.path.join(r, "%s.idXML" % arm), prefix)
	assert parsed["score_type"] == "q-value", \
	"%s arm score_type %r (Percolator did not run?)" % (arm, parsed["score_type"])
	assert parsed["psms"], "%s arm: no parseable PSMs" % arm
	root = ET.parse(os.path.join(r, "%s.idXML" % arm)).getroot()
	accs = {ph.get("accession") for ph in root.iter("ProteinHit")}
	assert any(x and x.startswith(prefix) for x in accs), \
	"%s arm: no %s ProteinHits" % (arm, prefix)

	missing = raw["arms"]["peptdeep"].get("peptdeep_features_missing")
	assert missing == [], "PeptDeep features missing: %s" % missing
	print("%s: all pipeline guards passed" % r)
	PY

	- name: Normalize PXD028735 run B results to v2
	shell: bash
	run: \|
	python benchmark/report/report_generate.py normalize prose-peptdeep \
	--run-id "${{ github.run_id }}-B" \
	--run-at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
	--cache none \
	--results-dir results-prose-pxd028735-B

	- name: Upload PXD028735 benchmark results (runs A and B)
	uses: actions/upload-artifact@v7
	with:
	name: benchmark-prose-pxd028735-${{ needs.build-openms-onnx.outputs.openms_sha }}
	path: \|
	results-prose-pxd028735-A
	results-prose-pxd028735-B
	if-no-files-found: error

	# ---------------------------------------------------------------------------
	# Nightly Debian .deb probe (opt-in, via the nightly_deb_probe input).
	#
	# Runtime-health check of OpenMS' published nightly Debian package:
	# downloads a nightly archive .deb from an explicit date directory
	# (nightly_deb_date input) or, by default, the newest one available at
	# dispatch, extracts it into $RUNNER_TEMP (never installed system-wide),
	# and runs the existing Smoke benchmark (benchmark/run_smoke_benchmark.sh,
	# reused unmodified) against the packaged binaries.
	#
	# Scope, stated explicitly:
	# - validates the nightly develop snapshot packaging, NOT the PR SHA:
	# nightlies track OpenMS develop and are built WITHOUT ONNX, so they
	# can never replace the SHA-pinned source build above.
	# - Smoke-only evidence: says nothing about OpenSwath or ProSE/PeptDeep.
	# - fully isolated: no needs:, excluded from render-report, cannot gate
	# or block any other job in this workflow.
	#
	# Library/data resolution: the packaged binaries carry
	# RUNPATH=$ORIGIN/../lib/ and OpenMS resolves share/OpenMS relative to the
	# install prefix, so the extraction layout usr/bin + usr/lib +
	# usr/share/OpenMS is expected to run WITHOUT LD_LIBRARY_PATH or
	# OPENMS_DATA_PATH. The package's declared Debian Depends are apt-installed
	# first (what dpkg would do on a real system) so the probe tests the
	# package's own runtime resolution, not the runner's stock library set.
	# Both resolution assumptions are TESTED AT RUNTIME below (loader probe
	# under env -i, then a CHEMISTRY-data probe); OPENMS_DATA_PATH is exported
	# only if the probe proves it necessary, and that fact is recorded in
	# PROVENANCE.txt. The package itself is never installed system-wide.
	# ---------------------------------------------------------------------------
	run-nightly-deb-probe:
	name: "Nightly Debian .deb probe (develop snapshot, Smoke-only)"
	runs-on: ubuntu-24.04
	timeout-minutes: 30
	if: ${{ inputs.nightly_deb_probe }}
	steps:
	- name: Checkout benchmarking repo (for the Smoke runner)
	uses: actions/checkout@v6

	# Created before anything can fail, so the if: always() artifact upload
	# always has a place to put diagnostics (probe logs, archive index,
	# PROVENANCE.txt) even when an early step fails.
	- name: Initialize probe diagnostics directory
	shell: bash
	run: \|
	set -euo pipefail
	mkdir -p "$GITHUB_WORKSPACE/nightly-deb-probe-results"
	{
	echo "probe started: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
	echo "workflow_run_id = ${{ github.run_id }}"
	echo "workflow_sha = ${{ github.sha }}"
	} > "$GITHUB_WORKSPACE/nightly-deb-probe-results/probe_run.log"

	# The archive keeps upload-day directories nightly/YYYY.MM.DD/ (the
	# .deb filename carries the build night, one day earlier). An explicit
	# nightly_deb_date input pins the exact directory; empty selects the
	# newest one at dispatch. Either way the resolved date is recorded, so
	# every run is reproducible from its provenance.
	- name: Resolve nightly archive directory (explicit date or newest)
	id: archive
	env:
	REQUESTED_DATE: ${{ inputs.nightly_deb_date }}
	shell: bash
	run: \|
	set -euo pipefail
	results="$GITHUB_WORKSPACE/nightly-deb-probe-results"
	index=$(curl -fsSL --max-time 30 https://archive.openms.de/openms/OpenMSInstaller/nightly/)
	printf '%s' "$index" > "$results/archive_index.html"
	requested="${REQUESTED_DATE:-}"
	if [[ -n "$requested" ]]; then
	if ! printf '%s' "$requested" \| grep -qE '^[0-9]{4}\.[0-9]{2}\.[0-9]{2}$'; then
	echo "ERROR: nightly_deb_date must be YYYY.MM.DD (archive upload-day directory), got: '$requested'" >&2
	exit 1
	fi
	if ! printf '%s' "$index" \| grep -qF "href=\"$requested/\""; then
	echo "ERROR: requested nightly archive dir '$requested' not found in archive index (see archive_index.html in the probe artifact)" >&2
	exit 1
	fi
	dir="$requested"
	echo "Using requested archive dir: $dir"
	else
	dir=$(printf '%s' "$index" \| grep -oE 'href="[0-9]{4}\.[0-9]{2}\.[0-9]{2}/"' \
	\| grep -oE '[0-9]{4}\.[0-9]{2}\.[0-9]{2}' \| sort \| tail -1)
	if [[ -z "$dir" ]]; then
	echo "ERROR: no dated nightly directory found in archive index (see archive_index.html in the probe artifact)" >&2
	exit 1
	fi
	echo "Resolved newest archive dir: $dir"
	fi
	echo "dir=$dir" >> "$GITHUB_OUTPUT"

	- name: Download .deb and record filename, size and SHA256
	id: package
	shell: bash
	run: \|
	set -euo pipefail
	base="https://archive.openms.de/openms/OpenMSInstaller/nightly/${{ steps.archive.outputs.dir }}"
	listing=$(curl -fsSL --max-time 30 "$base/")
	printf '%s' "$listing" > "$GITHUB_WORKSPACE/nightly-deb-probe-results/archive_listing.html"
	# Security: this filename comes from remote HTML and is stored in a
	# step output. Restrict it to a shell-safe charset at capture time so
	# no shell metacharacter can ever ride along, and anchor-validate
	# before use. Consumers receive it via env:, never via textual
	# GitHub expression expansion into a generated script.
	filename=$(printf '%s' "$listing" \| grep -oE 'OpenMS-[-A-Za-z0-9._+~]+-Debian-Linux-x86_64\.deb' \| head -1)
	if [[ -z "$filename" ]] \|\| ! [[ "$filename" =~ ^OpenMS-[-A-Za-z0-9._+~]+-Debian-Linux-x86_64\.deb$ ]]; then
	echo "ERROR: no x86_64 Debian .deb with the expected shell-safe filename in $base/ (see archive_listing.html in the probe artifact)" >&2
	exit 1
	fi
	curl -fsSL --max-time 900 -o "$RUNNER_TEMP/openms-nightly.deb" "$base/$filename"
	bytes=$(stat -c %s "$RUNNER_TEMP/openms-nightly.deb")
	sha=$(sha256sum "$RUNNER_TEMP/openms-nightly.deb" \| cut -d' ' -f1)
	pkg_date=$(printf '%s' "$filename" \| grep -oE '[0-9]{4}-[0-9]{2}-[0-9]{2}')
	# Partial provenance now so it survives any later failure; the
	# provenance step after the runtime probes rewrites the complete
	# file (adding develop-SHA and the data-path finding).
	{
	echo "package = $filename"
	echo "package_sha256 = $sha"
	echo "package_date = $pkg_date (build night)"
	echo "archive_dir = ${{ steps.archive.outputs.dir }} (upload day)"
	echo "package_bytes = $bytes"
	echo "source = OpenMS nightly/develop (https://archive.openms.de/openms/OpenMSInstaller/nightly/)"
	echo "PR_SHA = ${{ github.sha }} (what this workflow checked out; the deb does NOT contain it)"
	echo "ONNX = unavailable (nightly packages are built without ONNX; PeptDeep arm impossible)"
	echo "validation = Smoke (DecoyDatabase -> CometAdapter -> PercolatorAdapter) only; no OpenSwath/ProSE evidence"
	} > "$GITHUB_WORKSPACE/nightly-deb-probe-results/PROVENANCE.txt"
	{
	echo "filename=$filename"
	echo "pkg_date=$pkg_date"
	echo "bytes=$bytes"
	echo "sha256=$sha"
	} >> "$GITHUB_OUTPUT"
	echo "Downloaded $filename ($bytes bytes)"
	echo "SHA256: $sha"

	- name: Resolve nightly provenance (best effort, never invented)
	id: provenance
	shell: bash
	run: \|
	set -euo pipefail
	dev_sha=$(curl -fsSL --max-time 30 \
	-H "Accept: application/vnd.github+json" \
	https://api.github.com/repos/OpenMS/OpenMS/commits/develop \
	\| python3 -c 'import json,sys; print(json.load(sys.stdin).get("sha", ""))' \
	\|\| true)
	# Only accept a 40-hex commit SHA from the API; anything unexpected
	# is treated as unresolved rather than recorded verbatim.
	printf '%s' "$dev_sha" \| grep -qE '^[0-9a-f]{40}$' \|\| dev_sha=""
	if [[ -z "$dev_sha" ]]; then
	dev_note="unresolved (GitHub API unavailable or rate-limited); provenance limited to the package build date"
	else
	dev_note="resolved from OpenMS/OpenMS@develop at probe time (at/near the build date, not verified per-package)"
	fi
	{
	echo "dev_sha=$dev_sha"
	echo "dev_note=$dev_note"
	} >> "$GITHUB_OUTPUT"

	# The deb declares its runtime library Depends (xerces, libzip, arrow
	# system pieces, Qt6 for GUI bits, ...). Satisfy them exactly as dpkg
	# would, so the loader probe below measures the package's own
	# resolution, not a stock-runner library gap. Only declared deps are
	# installed; the package itself is never installed system-wide.
	- name: Install the package's declared dependencies (as dpkg would)
	shell: bash
	run: \|
	set -euo pipefail
	sudo apt-get update -qq
	# Feed the verbatim Depends field to apt-get satisfy, which parses
	# Debian dependency syntax natively (alternatives "a \| b", version
	# constraints, commas) - no hand-rolled splitting that could mangle
	# alternative groups.
	depends=$(dpkg-deb -f "$RUNNER_TEMP/openms-nightly.deb" Depends)
	echo "Depends: $depends"
	sudo apt-get -y -qq satisfy --no-install-recommends "$depends"

	- name: Extract package into $RUNNER_TEMP (no system install)
	id: extract
	shell: bash
	run: \|
	set -euo pipefail
	prefix="$RUNNER_TEMP/deb-prefix"
	dpkg-deb -x "$RUNNER_TEMP/openms-nightly.deb" "$prefix"
	for d in usr/bin usr/lib usr/share/OpenMS; do
	[[ -d "$prefix/$d" ]] \|\| { echo "ERROR: $d missing after extraction" >&2; exit 1; }
	done
	# Engines: the deb ships Linux engines under share/OpenMS/THIRDPARTY
	# (Comet is comet.exe but is a statically linked Linux ELF).
	test -x "$prefix/usr/share/OpenMS/THIRDPARTY/Comet/comet.exe"
	test -x "$prefix/usr/share/OpenMS/THIRDPARTY/Percolator/percolator"
	echo "prefix=$prefix" >> "$GITHUB_OUTPUT"

	# Probe 1 (loader): run a TOPP tool under env -i to prove the bundled
	# libs load via RUNPATH=$ORIGIN/../lib/ with no environment help.
	# Probe 2 (shared data): Digestor reads CHEMISTRY data from
	# share/OpenMS, proving data-path resolution without OPENMS_DATA_PATH.
	- name: Runtime probes (native RPATH loader, then shared-data path)
	id: runtime
	shell: bash
	run: \|
	set -euo pipefail
	bin="${{ steps.extract.outputs.prefix }}/usr/bin"
	# --help, NOT --version: TOPP tools reject --version (exit 6,
	# "Unknown option") even on a healthy package - verified directly
	# against the 2026-09-24 nightly on Ubuntu 24.04. --help exercises
	# the same loader/RUNPATH path and exits 0.
	if ! env -i "$bin/FileMerger" --help >"$RUNNER_TEMP/loader_probe.log" 2>&1; then
	cp "$RUNNER_TEMP/loader_probe.log" "$GITHUB_WORKSPACE/nightly-deb-probe-results/"
	echo "FileMerger failed without env vars; loader probe log:" >&2
	cat "$RUNNER_TEMP/loader_probe.log" >&2
	exit 1
	fi
	cp "$RUNNER_TEMP/loader_probe.log" "$GITHUB_WORKSPACE/nightly-deb-probe-results/"
	echo "Loader probe: FileMerger runs with NO LD_LIBRARY_PATH (native RUNPATH OK)"
	printf '>probe_protein\nPEPTIDEKPROBESEQUENCEAAAAK\n' > "$RUNNER_TEMP/probe.fasta"
	data_env=not_required
	if ! env -i "$bin/Digestor" -in "$RUNNER_TEMP/probe.fasta" \
	-out "$RUNNER_TEMP/probe.idXML" >"$RUNNER_TEMP/data_probe.log" 2>&1; then
	cp "$RUNNER_TEMP/data_probe.log" "$GITHUB_WORKSPACE/nightly-deb-probe-results/"
	# Classify the failure: a genuine missing-shared-data abort mentions
	# the data root, its subdirectories, schema files, or the env var
	# itself. Anything else is a non-data-path failure and must fail
	# the probe loudly rather than be papered over with env vars.
	if grep -qiE "shared data\|OPENMS_DATA_PATH\|data file\|CHEMISTRY\|SCHEMAS\|\.xsd\|share/OpenMS" "$RUNNER_TEMP/data_probe.log"; then
	data_env=required
	echo "Data probe: share/OpenMS not auto-resolved; OPENMS_DATA_PATH required (recorded in provenance)"
	else
	echo "Digestor probe failed for a non-data-path reason; log:" >&2
	cat "$RUNNER_TEMP/data_probe.log" >&2
	exit 1
	fi
	else
	cp "$RUNNER_TEMP/data_probe.log" "$GITHUB_WORKSPACE/nightly-deb-probe-results/" 2>/dev/null \|\| true
	echo "Data probe: share/OpenMS auto-resolved (no OPENMS_DATA_PATH needed)"
	fi
	echo "data_env=$data_env" >> "$GITHUB_OUTPUT"

	- name: Write provenance metadata
	shell: bash
	env:
	PACKAGE_FILENAME: ${{ steps.package.outputs.filename }}
	run: \|
	set -euo pipefail
	results="$GITHUB_WORKSPACE/nightly-deb-probe-results"
	mkdir -p "$results"
	{
	echo "package = $PACKAGE_FILENAME"
	echo "package_sha256 = ${{ steps.package.outputs.sha256 }}"
	echo "package_date = ${{ steps.package.outputs.pkg_date }} (build night)"
	echo "archive_dir = ${{ steps.archive.outputs.dir }} (upload day)"
	echo "package_bytes = ${{ steps.package.outputs.bytes }}"
	echo "source = OpenMS nightly/develop (https://archive.openms.de/openms/OpenMSInstaller/nightly/)"
	echo "develop_head_sha = ${{ steps.provenance.outputs.dev_sha }}"
	echo "develop_note = ${{ steps.provenance.outputs.dev_note }}"
	echo "PR_SHA = ${{ github.sha }} (what this workflow checked out; the deb does NOT contain it)"
	echo "ONNX = unavailable (nightly packages are built without ONNX; PeptDeep arm impossible)"
	echo "validation = Smoke (DecoyDatabase -> CometAdapter -> PercolatorAdapter) only; no OpenSwath/ProSE evidence"
	echo "data_path_env = ${{ steps.runtime.outputs.data_env }}"
	} > "$results/PROVENANCE.txt"
	cp "$RUNNER_TEMP/loader_probe.log" "$RUNNER_TEMP/data_probe.log" "$results/" 2>/dev/null \|\| true

	- name: Run existing Smoke benchmark against the packaged binaries
	env:
	OPENMS_BIN: ${{ steps.extract.outputs.prefix }}/usr/bin
	ENGINES_DIR: ${{ steps.extract.outputs.prefix }}/usr/share/OpenMS/THIRDPARTY
	OPENMS_SHA: nightly-deb-${{ steps.package.outputs.pkg_date }}
	RESULTS_DIR: ${{ github.workspace }}/nightly-deb-probe-results/smoke
	run: \|
	# OPENMS_DATA_PATH is exported ONLY if the runtime probe proved it
	# necessary; otherwise the package must resolve its data on its own.
	if [[ "${{ steps.runtime.outputs.data_env }}" == "required" ]]; then
	export OPENMS_DATA_PATH="${{ steps.extract.outputs.prefix }}/usr/share/OpenMS"
	echo "OPENMS_DATA_PATH exported (required per runtime probe)"
	fi
	bash benchmark/run_smoke_benchmark.sh

	# Defense-in-depth: the Smoke runner already exits 1 when a required
	# stage fails; this step re-checks the recorded verdict from smoke.json
	# (guards against runner-side logic changes) and writes the summary.
	- name: Enforce probe verdict (explicit pass/fail)
	shell: bash
	env:
	PACKAGE_FILENAME: ${{ steps.package.outputs.filename }}
	run: \|
	set -euo pipefail
	v=$(python3 -c 'import json; print(json.load(open("nightly-deb-probe-results/smoke/smoke.json"))["verdict"])')
	echo "Nightly .deb probe verdict: $v"
	{
	echo "## Nightly Debian .deb probe"
	echo ""
	echo "Package: $PACKAGE_FILENAME"
	echo ""
	echo "SHA256: ${{ steps.package.outputs.sha256 }}"
	echo ""
	echo "Smoke verdict: $v - develop-snapshot packaging health only, not PR-SHA validation. ONNX unavailable in nightlies."
	} >> "$GITHUB_STEP_SUMMARY"
	[[ "$v" == "pass" ]] \|\| { echo "Probe FAILED" >&2; exit 1; }

	- name: Upload probe artifact
	if: always()
	uses: actions/upload-artifact@v7
	with:
	name: nightly-deb-probe-${{ steps.package.outputs.pkg_date }}
	path: nightly-deb-probe-results
	if-no-files-found: error

	# ---------------------------------------------------------------------------
	# Render the combined report from every benchmark result of this run.
	# Downloads the per-benchmark result artifacts, merges them into one results
	# tree, renders report.html, uploads it as an artifact and writes the
	# current-run verdict into the job summary.
	# ---------------------------------------------------------------------------
	render-report:
	name: "Render combined report"
	runs-on: ubuntu-24.04
	needs:
	- build-openms
	- package-runtime
	- run-smoke-benchmark
	- run-openswath-benchmark
	- run-prose-peptdeep-benchmark
	- run-prose-pxd028735-benchmark
	if: ${{ !cancelled() }}
	steps:
	# One label for the whole job: source mode keys everything by the built
	# SHA (pre-Stage-1 behavior, preserved exactly); package mode keys by
	# the package's embedded SHA (build-openms is skipped there, so its
	# output would be empty).
	- name: Resolve report label
	id: label
	shell: bash
	run: \|
	set -euo pipefail
	if [[ "${{ inputs.runtime_source }}" == "package" ]]; then
	echo "sha=${{ needs.package-runtime.outputs.openms_sha }}" >> "$GITHUB_OUTPUT"
	else
	echo "sha=${{ needs.build-openms.outputs.openms_sha }}" >> "$GITHUB_OUTPUT"
	fi

	- name: Checkout benchmarking repo
	uses: actions/checkout@v6

	# Artifact pattern is runtime-dependent: source mode keys artifacts by
	# the built SHA (pre-Stage-1 behavior, preserved exactly); package mode
	# keys by the package's embedded SHA, unknown at plan time.
	- name: Download benchmark result artifacts
	uses: actions/download-artifact@v8
	with:
	pattern: ${{ inputs.runtime_source == 'package' && 'benchmark-' \|\| format('benchmark--{0}', needs.build-openms.outputs.openms_sha) }}
	merge-multiple: true
	path: results

	- name: Render combined report
	id: render
	shell: bash
	run: \|
	set -euo pipefail
	python benchmark/report/report_generate.py render \
	--results-dir results \
	--out benchmark/reports/report.html
	# Current-run verdict: the newest run in the merged results tree.
	verdict=$(python - <<'PY'
	import sys
	sys.path.insert(0, "benchmark/report")
	from report_generate import load_results, _verdict
	runs, _ = load_results("results")
	if not runs:
	sys.exit("no runs found after artifact merge")
	print(_verdict(runs[-1]))
	PY
	)
	echo "verdict=$verdict" >> "$GITHUB_OUTPUT"

	- name: Upload report
	uses: actions/upload-artifact@v7
	with:
	name: benchmark-report-${{ steps.label.outputs.sha }}
	path: benchmark/reports/report.html
	if-no-files-found: error

	- name: Write job summary
	if: always()
	shell: bash
	run: \|
	{
	echo "## Benchmark report (${{ steps.label.outputs.sha }})"
	echo ""
	echo "Current-run verdict: ${{ steps.render.outputs.verdict }}"
	echo ""
	echo "report.html is attached to this run as artifact"
	echo "benchmark-report-${{ steps.label.outputs.sha }}."
	} >> "$GITHUB_STEP_SUMMARY"

Navigation Menu

Repository navigation

Benchmark #46

Workflow file

Benchmark #46

Uh oh!

Workflow file for this run