Add Launcher fail-stop runtime supervision

.github/workflows/tests.yml

@@ -9,6 +9,7 @@ on:
 jobs:
   test:
     runs-on: ubuntu-latest
+    timeout-minutes: 15
     steps:
       - uses: actions/checkout@v3
       - uses: actions/setup-python@v4
@@ -26,6 +27,7 @@ jobs:
       matrix:
         os: [ubuntu-latest, macos-latest, windows-latest]
     runs-on: ${{ matrix.os }}
+    timeout-minutes: 15
     steps:
       - uses: actions/checkout@v3
       - uses: actions/setup-python@v4
@@ -35,5 +37,9 @@ jobs:
         run: |
           python -m pip install --upgrade pip
           pip install .[test]
-      - name: Run platform-neutral Launcher lifecycle tests
-        run: pytest tests/test_launcher_background.py
+      - name: Run platform-neutral Launcher lifecycle and fail-stop tests
+        run: >-
+          pytest
+          tests/test_launcher_background.py
+          tests/test_launcher_cleanup.py
+          tests/test_launcher_fail_stop.py

README.md

@@ -315,7 +315,8 @@ asset definition.
 {
   "version": "0.1",
   "runtime": {
-    "cleanup_mmap_on_start": false
+    "health_check_interval_sec": 1.0,
+    "health_check_timeout_sec": 0.8
   },
   "defaults": {
     "cwd": ".",
@@ -350,7 +351,7 @@ asset definition.
 | Key        | Description |
 | ---------- | ----------- |
 | `version`  | Free-form version string for the specification.【F:src/hakoniwa_pdu/apps/launcher/model.py†L82-L90】 |
-| `runtime`  | Optional Launcher-owned runtime preparation. `cleanup_mmap_on_start` is disabled by default and removes only Hakoniwa mmap/lock files before any asset starts. Enable it only when this Launcher exclusively owns the configured runtime directory. |
+| `runtime`  | Runtime integrity supervision intervals. `cleanup_mmap_on_start` is deprecated and refused; recovery is always an explicit operator action. |
 | `defaults` | Shared defaults applied to every asset when the field is omitted at the asset level. Paths are resolved relative to the launch file.【F:src/hakoniwa_pdu/apps/launcher/loader.py†L47-L91】 |
 | `assets`   | Array of process definitions. Launch order is automatically sorted by `depends_on` while preserving the original order when there is no dependency.【F:src/hakoniwa_pdu/apps/launcher/model.py†L112-L167】 |
 | `notify`   | Optional notification that fires when an asset exits or the launcher aborts. Supports `webhook` and `exec` variants.【F:src/hakoniwa_pdu/apps/launcher/model.py†L58-L80】【F:src/hakoniwa_pdu/apps/launcher/hako_monitor.py†L83-L121】 |
@@ -399,12 +400,30 @@ keeps its slightly longer Python subprocess timeout as an outer watchdog. Older
 or unrecognized hako-cmd versions remain on the legacy command line, so existing
 installations continue to work without the new option.
 
-When `runtime.cleanup_mmap_on_start` is `true`, the Launcher resolves
-`HAKO_CONFIG_PATH`, reads `core_mmap_path`, and removes only `mmap-0x*.bin`,
-`flock.bin`, `pdu_init.lock`, and `pro_init.lock` before starting any asset. It
-does not delete runtime files on shutdown. This option is an explicit ownership
-declaration: do not enable it when another Launcher or Hakoniwa process may use
-the same mmap directory.
+With hako-cmd 1.1.0 or newer, the Launcher runs `hako-cmd check` before startup
+and periodically while active. The consumer requires the versioned v1 report
+with a `checks[]` array; an unsupported or malformed report is `UNKNOWN`.
+`ABORT` changes the Launcher state to `FAILED`
+and terminates every managed process. `UNKNOWN` is reported as a warning and
+does not trigger destructive action. Every check has an outer subprocess
+timeout, configured by `runtime.health_check_timeout_sec`; the periodic interval
+is `runtime.health_check_interval_sec`.
+
+The Launcher never repairs or deletes a failed runtime automatically. The old
+`runtime.cleanup_mmap_on_start` field is retained only for input compatibility;
+setting it to `true` now refuses startup. Inspect the evidence with
+`hako-cmd check`, stop all related processes, and run `hako-cmd recover`
+explicitly. Launcher session files and asset logs remain available after a
+fail-stop event.
+
+An unexpected asset exit is also fail-stop: the asset name, PID, exit code or
+signal, cleanup exception, and post-cleanup process diagnostics are stored in
+`failure_report`. Successful cleanup does not replace `FAILED` with
+`TERMINATED`. `failure_report.cleanup.recover_safe` is true only when every
+managed process and descendant scope was verified stopped. In particular,
+Windows uses `taskkill /T /F` for native process trees but cannot independently
+enumerate descendants after a parent has already disappeared; that limitation
+is recorded and recovery remains unsafe until the operator checks it.
 
 #### Notify section
 

docs/launcher-background-lifecycle.md

@@ -40,6 +40,20 @@ python -m hakoniwa_pdu.apps.launcher.hako_launcher_ctl \
 
 `terminate` calls the running Launcher's `LauncherService.terminate()` through a localhost control endpoint. Child-process cleanup therefore remains owned by the Launcher, including the existing OS-specific process-group behavior.
 
+With hako-cmd 1.1.0 or newer, the background Launcher also supervises runtime
+integrity. An `ABORT` result moves the session to `FAILED` and stops all managed
+children. It does not remove the session file, logs, mmap files, or lock files;
+recovery requires a separate, explicit `hako-cmd recover` after every related
+process has stopped. An `UNKNOWN` result is diagnostic only and is logged as a
+warning.
+
+Startup-time ABORT details are retained in the background session even when no
+control server was created. Runtime ABORT and unexpected asset exit reports
+include cleanup status, exceptions, survivor diagnostics, and a conservative
+`recover_safe` flag. Cleanup never changes an existing `FAILED` state to
+`TERMINATED`; only an explicit normal termination of a healthy session produces
+`TERMINATED`.
+
 The session file is retained after termination with `state: "TERMINATED"` for post-run inspection.
 
 ## Session ownership and stale files

src/hakoniwa_pdu/apps/launcher/hako_asset_runner.py

@@ -95,6 +95,8 @@ def __init__(self, *, env: Optional[Mapping[str, str]] = None) -> None:
         self.handle: Optional[ProcHandle] = None
         self._win_pid: Optional[int] = None
         self._posix_pgid: Optional[int] = None
+        self._managed_pid: Optional[int] = None
+        self._windows_tree_cleanup = False
 
     # ---------- helpers ----------
     @staticmethod
@@ -180,6 +182,8 @@ def spawn(
                 timeout=15.0
             )
             self._win_pid = int(out.strip())
+            self._managed_pid = self._win_pid
+            self._windows_tree_cleanup = True
             dummy = subprocess.Popen(["/bin/true"]) if IS_POSIX else subprocess.Popen(["cmd.exe", "/c", "exit", "0"])
             self.handle = ProcHandle(
                 popen=dummy,
@@ -211,6 +215,8 @@ def spawn(
             # even after the direct child exits: launch scripts may leave a
             # long-running descendant in the same group.
             self._posix_pgid = popen.pid
+        self._managed_pid = popen.pid
+        self._windows_tree_cleanup = IS_WIN
 
         self.handle = ProcHandle(
             popen=popen,
@@ -327,6 +333,19 @@ def kill(self) -> None:
         try:
             if IS_POSIX:
                 os.killpg(self._posix_pgid or h.popen.pid, signal.SIGKILL)
+            elif IS_WIN:
+                completed = subprocess.run(
+                    ["taskkill", "/PID", str(h.popen.pid), "/T", "/F"],
+                    stdout=subprocess.DEVNULL,
+                    stderr=subprocess.DEVNULL,
+                    timeout=5.0,
+                    check=False,
+                )
+                if completed.returncode not in (0, 128):
+                    raise RuntimeError(
+                        f"taskkill /T failed for pid={h.popen.pid}: "
+                        f"rc={completed.returncode}"
+                    )
             else:
                 h.popen.kill()
         finally:
@@ -338,6 +357,33 @@ def kill(self) -> None:
                 f"pgid={self._posix_pgid}"
             )
 
+    def diagnostics(self) -> dict:
+        """Return non-mutating lifecycle evidence for failure reports."""
+        info = self.exit_info()
+        if self._windows_tree_cleanup:
+            cleanup_scope = "windows_taskkill_tree"
+            descendants_verified = False
+        elif IS_POSIX:
+            cleanup_scope = "posix_process_group"
+            descendants_verified = not self.is_alive()
+        elif IS_WIN:
+            cleanup_scope = "windows_taskkill_tree"
+            # Windows taskkill /T is requested while the parent exists, but
+            # this runner does not enumerate an already-orphaned descendant.
+            descendants_verified = False
+        else:
+            cleanup_scope = "direct_process"
+            descendants_verified = False
+        return {
+            "pid": self._managed_pid if self._managed_pid is not None else info.pid,
+            "pgid": self._posix_pgid,
+            "alive": self.is_alive(),
+            "exit_code": info.exit_code,
+            "signal": info.signal,
+            "cleanup_scope": cleanup_scope,
+            "descendants_verified_stopped": descendants_verified,
+        }
+
     def exit_info(self) -> ExitInfo:
         h = self.handle
         # Windows プロセス優先で情報を埋める

src/hakoniwa_pdu/apps/launcher/hako_cli.py

@@ -5,12 +5,21 @@
 import shutil
 import subprocess
 import re
+import json
+from dataclasses import dataclass
 from typing import Optional, Literal
 
 from .effective_model import EffectiveSpec
 from .envmerge import merge_env
 
 
+@dataclass(frozen=True)
+class RuntimeHealth:
+    status: Literal["OK", "ABORT", "UNKNOWN"]
+    reason: str
+    details: dict
+
+
 class HakoCli:
     """
     Thin wrapper to run hako-cmd in the foreground.
@@ -81,6 +90,67 @@ def list_assets(self, *, timeout: Optional[float] = None) -> tuple[int, set[str]
         }
         return int(proc.returncode), names
 
+    def check(self, *, timeout: float = 1.0) -> RuntimeHealth:
+        """Run the non-mutating native integrity check with an outer timeout."""
+        env = merge_env(
+            defaults_env=self.defaults_env_ops,
+            asset_env=None,
+            asset_name="hako_cli",
+        )
+        try:
+            resolved = self._resolve_cmd(env)
+        except OSError as exc:
+            return RuntimeHealth("UNKNOWN", str(exc), {})
+        if not self._supports_health_check(resolved, env):
+            return RuntimeHealth(
+                "UNKNOWN",
+                "hako-cmd does not support runtime integrity checks",
+                {"version": self._detected_version},
+            )
+        try:
+            proc = subprocess.run(
+                [resolved, "check"],
+                cwd=str(self.spec.base_dir),
+                env=env,
+                check=False,
+                timeout=timeout,
+                capture_output=True,
+                text=True,
+            )
+        except subprocess.TimeoutExpired:
+            return RuntimeHealth("UNKNOWN", "hako-cmd check timed out", {})
+        except OSError as exc:
+            return RuntimeHealth("UNKNOWN", str(exc), {})
+        try:
+            payload = json.loads(proc.stdout)
+        except (json.JSONDecodeError, TypeError):
+            return RuntimeHealth(
+                "UNKNOWN",
+                "hako-cmd check returned invalid JSON",
+                {"returncode": proc.returncode, "stderr": proc.stderr},
+            )
+        if payload.get("schema_version") != 1:
+            return RuntimeHealth(
+                "UNKNOWN", "hako-cmd check returned an unsupported schema", payload
+            )
+        checks = payload.get("checks")
+        if not isinstance(checks, list) or any(
+            not isinstance(check, dict)
+            or not isinstance(check.get("name"), str)
+            or check.get("status") not in ("OK", "ABORT", "UNKNOWN")
+            or not isinstance(check.get("reason"), str)
+            for check in checks
+        ):
+            return RuntimeHealth(
+                "UNKNOWN", "hako-cmd check returned invalid checks", payload
+            )
+        status = payload.get("status")
+        if status not in ("OK", "ABORT", "UNKNOWN"):
+            return RuntimeHealth(
+                "UNKNOWN", "hako-cmd check returned an invalid status", payload
+            )
+        return RuntimeHealth(status, str(payload.get("reason", "")), payload)
+
     # ---- internals ----
     def _resolve_cmd(self, env: dict[str, str]) -> str:
         path = env.get("PATH")
@@ -125,6 +195,13 @@ def _supports_bounded_lock_wait(self, resolved: str, env: dict[str, str]) -> boo
         )
         return self._bounded_lock_wait_supported
 
+    def _supports_health_check(self, resolved: str, env: dict[str, str]) -> bool:
+        self._supports_bounded_lock_wait(resolved, env)
+        return bool(
+            self._detected_version is not None
+            and self._detected_version >= (1, 1, 0)
+        )
+
     def _command(
         self,
         resolved: str,