waybeam: do not SIGKILL the encoder when a graceful stop times out - #2337
waybeam: do not SIGKILL the encoder when a graceful stop times out#2337snokvist wants to merge 1 commit into
Conversation
The stop-timeout escalation added in OpenIPC#2332 fixes a real bug -- a stop that did not happen must not report OK, or restart runs start() against a survivor and silently does nothing -- but it escalates with killall -9, which is the one thing that must not be done to this daemon on SigmaStar. waybeam releases kernel-side state on its way out (MI_SYS_Exit / MI_VENC_DestroyChn). SIGKILL skips that, so the MI channel and binding slots stay occupied and nothing in userspace can reclaim them. The next start binds a half-open channel whose symptom is "no encoder data received" until the board is power-cycled. That is a bad state for an init script to be able to create, because restart is what people run when they cannot reach the hardware. Keep the part that mattered -- the honest failure report and "restart) stop && start", which together prevent the silent no-op -- and drop the kill. A stop that times out now reports the failure and returns 1, leaving the running instance streaming rather than starting a second one on top of wedged state. If the daemon is genuinely hung the SDK is already stuck, and a reboot is the recovery that returns the board to a known state; waybeam's own teardown watchdog already does exactly that when its MI flush wedges. Also raise the graceful window from 15s to 30s. waybeam's teardown can legitimately take ~12s when the MI flush is slow, so the old window could fire on a shutdown that was going to succeed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PR Summary by Qodowaybeam: avoid SIGKILL during timed-out graceful stops
AI Description
Diagram
High-Level Assessment
Files changed (1)
|
Code Review by Qodo🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)
Great, no issues found!Qodo reviewed your code and found no material issues that require reviewTip of the day💡 Did you know, you can enable the Remediation agent and Qodo fixes findings in a dedicated fix PR |
|
Withdrawing this patch — we'll carry it in our own fork rather than ask upstream to take it. To be clear about what is and isn't being withdrawn: the concern itself stands, and I'd leave the note in #2332 (#issuecomment-5466202529) for your consideration. No action needed from anyone here — closing to keep your queue clean. |
Follow-up to #2332, on the
stop()escalation added in2f85ed2d.What and why
That commit fixes a real bug and I am not proposing to undo it. A stop that did not happen must not report
OK, becauserestartthen runsstart()against a survivor, trips the duplicate-instance check, returns 0, and the requested restart silently never happens. Keeping the honest failure report andrestart) stop && startis right.What this PR changes is the escalation it reaches for.
killall -9is the one thing that must not be done to this daemon on SigmaStar.waybeam releases kernel-side state on its way out, through
MI_SYS_Exit/MI_VENC_DestroyChn. SIGKILL skips that path, so the MI channel and binding slots stay occupied and nothing in userspace can reclaim them. The nextstartbinds a half-open channel: snapshots time out, and the log fills withwaiting for encoder data.... In our experience only a power cycle clears it. That is a bad state for an init script to be able to create, becauserestartis precisely what someone runs remotely when they cannot get to the hardware — the command most likely to be used is the one that can require physical access to undo.This is not theoretical. We lost a Star6E bench to exactly this on 2026-05-14: a
killall -9on the encoder left the board so that a fresh start logged its init as OK while every snapshot timed out and the main stream sat atwaiting for encoder data..., and power-cycling was the only way back. The encoder's own source carries the same warning from another angle, instar6e_audio.c, describing a teardown hang "with the watchdog SIGKILL'ing us before MI_SYS_Exit could run" as the thing to avoid.There is a second-order cost too, which is what prompted me to send a patch rather than just a comment. The
OK (forced)path reports success and letsrestartproceed intostart()against poisoned kernel state, so the resulting symptom isno encoder data received. That is the same signature a board with a mismatched device tree produces. I spent a long session this week separating those two causes on Infinity6C hardware, and an init script that can manufacture the symptom makes that diagnosis harder for whoever hits it next.So: keep the fix that mattered, drop the kill. A stop that times out reports the failure and returns 1, which leaves the still-running instance streaming instead of starting a second one on top of wedged state. If the daemon is genuinely hung then the SDK is already stuck, and a reboot is the recovery that returns the board to a known state rather than an unknown one — waybeam's own teardown watchdog already does exactly that when its MI flush wedges, logging
teardown wedged for 12s (MI flush D-state) — forcing reboot for deterministic recovery. The daemon has a deterministic escape; the init script's job is to not pre-empt it.The window also goes from 15s to 30s. waybeam's teardown can legitimately take ~12s when the MI flush is slow, so the old window could fire on a shutdown that was going to succeed.
STOP_TIMEOUT_Sdrives both the loop and the message so the two cannot drift apart.Evidence
SSC338Q + IMX415, running an image built from
masterwith this package selected (Majestic off,BR2_PACKAGE_WAYBEAM=y). The modified script was run against the live daemon.Graceful stop — completes in 2s, well inside even the old window, and releases the kernel-side state that SIGKILL would have skipped:
Those
client [810] disconnectedlines are the point of the change: that is the MI teardown a SIGKILL never gets to run.Start after a graceful stop — binds cleanly, no half-open channel:
Restart — the path the original bug made silently no-op — and idempotent start:
Stream verified after the restart, at the mode the config selects, via
/api/v1/snapshot.jpg:which matches the pre-test reading of 87.84 on the same scene, so the board came back exactly as it was.
sh -npasses on the script.What I did not test
I have not exercised the timeout branch itself on hardware, because producing a genuinely wedged teardown on demand means deliberately wedging a board, and I did not want to do that to a working one to prove a negative. The graceful path above is what I can demonstrate. The claim behind the change — that SIGKILL leaves unreclaimable MI state — rests on the 2026-05-14 incident and on the encoder's own source comment, not on a reproduction in this PR. If you would rather see the failure branch exercised directly I am happy to do it on a spare board and post what the SIGKILL path actually leaves behind; say the word.