Skip to content

fix(deploy): surface container script stderr instead of generic failure messages#181

Merged
lee-reinhardt merged 1 commit into
mainfrom
lee-eng-2175
Jul 22, 2026
Merged

fix(deploy): surface container script stderr instead of generic failure messages#181
lee-reinhardt merged 1 commit into
mainfrom
lee-eng-2175

Conversation

@lee-reinhardt

Copy link
Copy Markdown
Member

Problem

When a script run inside the SDK container exits non-zero, the captured-output helpers collapse the failure to "no output" and discard stderr. Callers then invent their own explanation for the missing output, producing misleading errors:

  • avocado deploy reports Hash collection script produced no output when the script actually failed with an actionable diagnostic (for example, --connect-sign requiring a local signing key, or a runtime built without one). The real message was written to stderr and thrown away.
  • Deploy's stamp verification misreports any container failure (docker daemon down, entrypoint error) as a missing build stamp / runtime not built.
  • In TUI and --output json modes, failure prints were suppressed entirely, so those runs got no diagnostic at all.
  • --runs-on remote failures were fully silent.

Changes

  • New capture API (run_in_container_capture, plus a remote counterpart) returns exit status with both streams; the existing run_in_container_with_output becomes a compatibility wrapper with unchanged semantics for its ~22 call sites.
  • On failure, the wrapper now surfaces a bounded stderr tail (last 15 non-empty lines, 4 KiB cap, char-boundary safe) in every output mode: TUI runs queue it above the task region, JSON runs keep it on stderr so the NDJSON stdout stream stays clean, plain runs print it directly.
  • Deploy's stamp and hash phases produce truthful, self-explaining errors that carry the script's stderr in both the CLI error and the step_error JSON event, and container failures are no longer conflated with genuinely missing stamps.
  • Runtime build's TUF staging skip notice is now visible in TUI/JSON modes (previously suppressed).
  • Remote (--runs-on) failures report the container's stderr without embedding the docker command line (which carries -e KEY=value pairs) in error output.
  • --verbose prints the full captured stderr at failure sites; truncation annotations no longer promise output that a mode can't deliver.
  • Bounded allocations throughout: the tail formatter stays ~15 × 4 KiB peak regardless of stderr size, and captured streams avoid redundant full-buffer copies.

Example, deploying a runtime whose build has no update-authority baked:

Before:

Error: Hash collection script produced no output

After:

Error: Hash collection failed in the SDK container:
ERROR: No root.json found at /opt/_avocado/.../metadata/root.json
       The runtime has no update-authority (root.json) baked into its build.
       This usually means no signing key is configured. ...

Testing

  • Unit tests for the stderr-tail formatter (blank input, short output, tail selection, blank-line filtering, byte cap, UTF-8 boundaries); cargo test --lib — 1121 passed.
  • Live-verified against a project with an unbuilt/unsigned runtime: the script's ERROR: diagnostics now appear in human output and in the step_error NDJSON event.

Copilot AI review requested due to automatic review settings July 21, 2026 21:06

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves deploy/build diagnostics by preserving and surfacing container/remote stderr on non-zero exits, avoiding misleading “no output” failures across human, TUI, JSON, and --runs-on remote modes.

Changes:

  • Adds captured-execution APIs that return exit status plus both stdout/stderr for local and remote container runs.
  • Updates deploy/build phases to use captured results and emit self-explaining errors (including bounded stderr tails) instead of collapsing failures into “missing stamps” / “produced no output”.
  • Ensures important notices (e.g., TUF staging skip) and failure diagnostics are visible in TUI/JSON modes without corrupting NDJSON stdout.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/utils/runs_on.rs Surfaces remote container failures (bounded stderr) before returning None, and adds a captured-output API.
src/utils/remote.rs Introduces RemoteCommandOutput and a capture method that avoids leaking docker command lines in non-verbose errors.
src/utils/container.rs Adds captured container run output types, stderr-tail formatting, and output-mode-safe failure notice printing.
src/commands/runtime/deploy.rs Uses captured container results for stamp/hash phases and emits accurate, diagnostic-rich phase errors.
src/commands/runtime/build.rs Routes non-fatal staging skip notices through renderer/stderr in TUI/JSON, and uses captured container results for hash collection errors.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/utils/container.rs
Comment thread src/utils/remote.rs

@jetm jetm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review - essentially clean. Verified the risky parts: str_tail snaps forward to a char boundary (no multibyte panic; the accented-char repeat test covers it), JSON-mode failure notices go to stderr or serde-escaped step_error events so NDJSON stdout stays valid, and the non-verbose error paths never embed the docker command line or -e KEY=value (only stderr / the ssh target). One low-severity contract nit inline. Two pre-existing nits not worth blocking: verbose remote capture double-logs the command (matches the non-capture path), and deploy.rs's container_failure_message is a thin passthrough over the util fn build.rs calls directly.

Comment thread src/utils/container.rs Outdated
…re messages

captured container runs previously collapsed any non-zero exit to
Ok(None), discarding stderr. callers then invented explanations for the
missing output: deploy phase 1 reported "hash collection script
produced no output" while the script's actual diagnostic (e.g. the
signing-key prerequisite for --connect-sign) was thrown away, and stamp
verification misreported container failures as missing build stamps.

- split run_in_container_with_output into a full-result capture method
  (success + stdout + stderr) and a compatibility wrapper for the
  existing call sites
- on failure the wrapper now prints a stderr tail (last 15 non-empty
  lines) by default instead of hiding it behind --verbose; --verbose
  still prints full stderr
- migrate deploy's stamp-verification and hash-collection sites to the
  capture method so failures name the real problem, embed the stderr
  tail in both the cli error and the step_error json event (rendered by
  the desktop phase strip), and distinguish container failures from
  genuinely missing stamps
- print the underlying error in the remote (--runs-on) output path
  instead of swallowing it entirely
- fix the equivalent misleading context messages in runtime build's
  tuf staging steps

@jetm jetm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at ef4998e: my earlier stdout-trim-contract nit is fixed - both local and remote capture now bind let mut stdout and trim in place, so ContainerRunOutput.stdout has one uniform whitespace contract (stderr stays raw on both). The captured-execution rework is otherwise clean (char-boundary-safe stderr tail, NDJSON-safe JSON mode, no docker-command/-e leakage in non-verbose paths). LGTM.

@lee-reinhardt
lee-reinhardt merged commit 0a20f33 into main Jul 22, 2026
8 checks passed
@lee-reinhardt
lee-reinhardt deleted the lee-eng-2175 branch July 22, 2026 18:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants