HAMi-core Initialization Topology Study

An independent, process-level study of concurrent cuInit() behavior when HAMi-core workers share one cache file or use multiple isolated cache files. The study is an evidence artifact, not a replacement design or an implementation PR.

Key result

Across three runs per condition, every HAMi worker returned a successful cuInit(). PID discovery behaved very differently:

Topology	Workers	Correct self PID	Peer-worker PID	Explicit discovery failure
1×250 — one shared cache	750	750	0	0
50×5 — 50 isolated caches	750	40	709	1

Every one of the 709 incorrect assigned PIDs matched another worker launched in the same run. The result is consistent with overlapping node-global NVML snapshot-difference operations that are serialized within one cache but not across independent caches.

This repository does not claim that downstream SM limiting or memory accounting was directly exercised. The workload performed initialization only; it did not allocate test memory or launch test kernels.

Performance context

External wall time, mean ± sample SD over three runs:

Mode	1×250	50×5
Native	16.48 ± 1.11 s	17.91 ± 0.13 s
HAMi	98.80 ± 1.24 s	89.91 ± 2.36 s

The topology change reduced HAMi completion time by about 9.0%, while the 50×5 HAMi condition remained about 5.02× slower than Native. Because launch orchestration, scheduling overlap, and the number of lock domains also changed, the experiment does not independently attribute the remaining delay to driver-global serialization.

Repository contents

• REPORT.md — methodology, interpretation, and boundaries.
• repro/run_topology_experiment.sh — the experiment driver used for the reported runs, with its known harness limitations documented in comments.
• analysis/summarize_results.py — standard library-only parser that rebuilds the compact CSVs from raw logs.
• results/summary.csv — one row per run and condition.
• results/pid-mapping-summary.csv — one row per HAMi worker, classified as self, peer, discovery_failed, external, or unresolved.
• results/external-wall-times.csv — sanitized transcription of shell-level timings used by the report.
• SHA256SUMS — checksums for the release archive and compact evidence files.

The raw 625-file log set is published separately as the evidence-v1 release to keep the Git history small.

Rebuild the summaries

Python 3.9 or later is sufficient; no third-party packages are required.

python3 analysis/summarize_results.py \
  --input-dir /path/to/17_experiment_results \
  --external-wall-times results/external-wall-times.csv \
  --output-dir /tmp/hami-topology-summary

diff -u results/summary.csv /tmp/hami-topology-summary/summary.csv
diff -u results/pid-mapping-summary.csv \
  /tmp/hami-topology-summary/pid-mapping-summary.csv

Collect a new run

The benchmark binaries come from HAMi-core PR #239. The reported run used HAMi-core 5496322f and the PR #239 benchmark tree at 9c233954.

One way to reconstruct that combination is:

git clone https://github.com/Project-HAMi/HAMi-core.git
cd HAMi-core
git fetch origin pull/239/head:bench-init-pr239
git checkout 5496322f2fb3e71bf1eca014fba3c9bc59ab8ffd
git restore --source=bench-init-pr239 -- bench

./build.sh
make -C bench
cp /path/to/run_topology_experiment.sh bench/
cd bench
bash run_topology_experiment.sh

Review the script before running it: it clears ../experiment_results/*.

Scope and related work

This is a process-level reproduction of a production-relevant cache topology, not an end-to-end Kubernetes or container-runtime validation. It complements the ongoing tiered host-PID discovery RFC by identifying a topology-aware correctness condition for both broker and fallback paths.

If later work demonstrates access to another tenant's data, device, or namespace, use HAMi's private vulnerability reporting process before public disclosure.