5.0 native puffer train hangs at init for CPU-host envs (containerized/virtualized hosts)

[IlyaasK]

Environment

• Branch: 5.0 @ 23c84bdb (HEAD as of 2026-08-27)
• Host: Thunder Compute A6000 (48 GB), CUDA 13.0, driver 580, Ubuntu 22.04
• Reproduced with stock envs: bash build.sh breakout && puffer train --train.total-timesteps=50000 (also cartpole, and a custom pinball env)

Symptom

puffer train never starts: after init it spins ~107% of one core, GPU at 0%, produces no output (buffered or line-buffered), workers futex-wait. CPU/GPU platform pieces verified fine (nvcc 13 build, cudart/cublas via cublas_init_handle, NVML init+handle, 3 GB cudaMalloc+memset, pinned-mem async copies, standalone OpenMP probe). Config knobs ruled out: --base.cudagraphs=-1 --base.async=0, --vec.num-threads/num-buffers matches. Marker traces placed in src/pufferl.cu localize the hang to env_start() — the one-time env reset runs under #pragma omp parallel for and never completes; worker OMP region (vec_thread_main) exhibits the same class of deadlock.

Blame (git blame, region ~1955-2010 + worker/init sites)

• 107717e7 "compact everything" — introduced #pragma omp parallel into this path (git log -S)
• d7cf8325 "Fix cudagraphs + async" — BUF_WAITING machinery
• 20d3b21a (2026-08-14), 95268f2c (2026-07-30), 8d1e0809, 88673c93 — region churn, all Joseph Suarez's WIP

Fix PR

• Fix native CPU-env training hang (OMP regions + missing log keys) #668: serialize the two OMP regions (one-time init + per-buffer stepping), new dict_get_default in ini.h, guard dashboard/log-history reads (util/*, uptime, agent_steps). Verified end-to-end on the same host: full epochs, live dashboard, ~80–110K SPS, 200K-step sanity completes.

Notes

• Standalone OpenMP works on the same box — the deadlocks are process-context-specific (likely interaction with the sandbox/monitoring threads), which is why a runtime swap (clang/libomp) does not help; removing OMP from these two loops does.
• The dashboard also hard-exits on missing log keys (util/gpu_percent first — under virtualized NVML these are legitimately absent/zero), which blocks even the first epoch render; guarded by the same PR.