Fix Launcher fail-stop races and normal completion

README.md

@@ -316,7 +316,7 @@ asset definition.
   "version": "0.1",
   "runtime": {
     "health_check_interval_sec": 1.0,
-    "health_check_timeout_sec": 0.8
+    "health_check_timeout_sec": 3.0
   },
   "defaults": {
     "cwd": ".",
@@ -407,7 +407,10 @@ with a `checks[]` array; an unsupported or malformed report is `UNKNOWN`.
 and terminates every managed process. `UNKNOWN` is reported as a warning and
 does not trigger destructive action. Every check has an outer subprocess
 timeout, configured by `runtime.health_check_timeout_sec`; the periodic interval
-is `runtime.health_check_interval_sec`.
+is `runtime.health_check_interval_sec`. The default outer timeout is 3 seconds
+to accommodate Windows process creation and security-scanner latency. The
+native `check` itself still uses a nonblocking OS try-lock, so increasing this
+outer watchdog does not turn runtime lock inspection into an unbounded wait.
 
 The Launcher never repairs or deletes a failed runtime automatically. The old
 `runtime.cleanup_mmap_on_start` field is retained only for input compatibility;
@@ -416,14 +419,18 @@ setting it to `true` now refuses startup. Inspect the evidence with
 explicitly. Launcher session files and asset logs remain available after a
 fail-stop event.
 
-An unexpected asset exit is also fail-stop: the asset name, PID, exit code or
-signal, cleanup exception, and post-cleanup process diagnostics are stored in
-`failure_report`. Successful cleanup does not replace `FAILED` with
-`TERMINATED`. `failure_report.cleanup.recover_safe` is true only when every
-managed process and descendant scope was verified stopped. In particular,
-Windows uses `taskkill /T /F` for native process trees but cannot independently
-enumerate descendants after a parent has already disappeared; that limitation
-is recorded and recovery remains unsafe until the operator checks it.
+An asset that exits with code 0 and no terminating signal is treated as a
+completed run: the Launcher stops the remaining assets and reaches
+`TERMINATED`, preserving the pre-fail-stop Launcher behavior used by finite
+show/scenario runners. A nonzero, signaled, or unclassifiable exit is fail-stop:
+the asset name, PID, exit code or signal, cleanup exception, and post-cleanup
+process diagnostics are stored in `failure_report`. Successful cleanup does
+not replace `FAILED` with `TERMINATED`. `failure_report.cleanup.recover_safe`
+is true only when every managed process and descendant scope was verified
+stopped. In particular, Windows uses `taskkill /T /F` for native process trees
+but cannot independently enumerate descendants after a parent has already
+disappeared; that limitation is recorded and recovery remains unsafe until the
+operator checks it.
 
 #### Notify section
 

docs/launcher-background-lifecycle.md

@@ -54,6 +54,18 @@ include cleanup status, exceptions, survivor diagnostics, and a conservative
 `TERMINATED`; only an explicit normal termination of a healthy session produces
 `TERMINATED`.
 
+A managed asset that finishes with exit code 0 and no terminating signal is a
+normal finite-run completion, not fail-stop. The Launcher terminates the other
+managed assets and records `TERMINATED`. Nonzero, signaled, and unclassifiable
+asset exits remain `unexpected_asset_exit` failures. This distinction keeps
+finite show/scenario runners compatible while retaining fail-stop behavior for
+actual crashes.
+
+Lifecycle state changes are guarded so that once a runtime or asset failure is
+recorded, a concurrent activation, `hako-cmd start`, after-start spawn, stop,
+reset, or cleanup cannot overwrite `FAILED`. A concurrent abort also prevents
+the next asset in an activation sequence from being spawned.
+
 The session file is retained after termination with `state: "TERMINATED"` for post-run inspection.
 
 ## Session ownership and stale files

src/hakoniwa_pdu/apps/launcher/hako_cli.py

@@ -129,11 +129,22 @@ def check(self, *, timeout: float = 1.0) -> RuntimeHealth:
                 "hako-cmd check returned invalid JSON",
                 {"returncode": proc.returncode, "stderr": proc.stderr},
             )
+        if not isinstance(payload, dict):
+            return RuntimeHealth(
+                "UNKNOWN",
+                "hako-cmd check returned a non-object JSON payload",
+                {
+                    "returncode": proc.returncode,
+                    "payload": payload,
+                    "stderr": proc.stderr,
+                },
+            )
         if payload.get("schema_version") != 1:
             return RuntimeHealth(
                 "UNKNOWN", "hako-cmd check returned an unsupported schema", payload
             )
         checks = payload.get("checks")
+        reason = payload.get("reason")
         if not isinstance(checks, list) or any(
             not isinstance(check, dict)
             or not isinstance(check.get("name"), str)
@@ -145,11 +156,40 @@ def check(self, *, timeout: float = 1.0) -> RuntimeHealth:
                 "UNKNOWN", "hako-cmd check returned invalid checks", payload
             )
         status = payload.get("status")
-        if status not in ("OK", "ABORT", "UNKNOWN"):
+        if (
+            status not in ("OK", "ABORT", "UNKNOWN")
+            or not isinstance(reason, str)
+        ):
             return RuntimeHealth(
                 "UNKNOWN", "hako-cmd check returned an invalid status", payload
             )
-        return RuntimeHealth(status, str(payload.get("reason", "")), payload)
+        check_statuses = {check["status"] for check in checks}
+        aggregate_status = (
+            "ABORT"
+            if "ABORT" in check_statuses
+            else "UNKNOWN"
+            if "UNKNOWN" in check_statuses
+            else "OK"
+        )
+        if status != aggregate_status:
+            return RuntimeHealth(
+                "UNKNOWN",
+                "hako-cmd check summary is inconsistent with checks",
+                payload,
+            )
+        expected_returncode = {"OK": 0, "ABORT": 2, "UNKNOWN": 3}[status]
+        if proc.returncode != expected_returncode:
+            return RuntimeHealth(
+                "UNKNOWN",
+                "hako-cmd check status/returncode combination is invalid",
+                {
+                    **payload,
+                    "reported_status": status,
+                    "returncode": proc.returncode,
+                    "stderr": proc.stderr,
+                },
+            )
+        return RuntimeHealth(status, reason, payload)
 
     # ---- internals ----
     def _resolve_cmd(self, env: dict[str, str]) -> str: