Skip to content

validate: bound stage tests and kill leftovers on timeout - #266

Open
rycerzes wants to merge 1 commit into
huggingface:mainfrom
rycerzes:fix/validation-stage-timeout
Open

rycerzes wants to merge 1 commit into
huggingface:mainfrom
rycerzes:fix/validation-stage-timeout

Conversation

@rycerzes

Copy link
Copy Markdown
Contributor

Summary

  • Validation started the launcher without --timeout, so the stage limit only killed the host-side docker exec. The tests kept running in the shared container, and the next stage failed with precleanup_failed (moto: 193 stage logs alternate between a 600 s timeout and precleanup_failed). The launcher now gets the stage budget minus a margin for reset, rebuild and purge, so it stops and cleans up the tests itself.
  • If a stage still times out on the client side, the stage script and the test user's processes are killed before the next stage. This uses a short Python /proc sweep, because slim images have no pkill.

Test plan

  • New tests: launcher --timeout, the budget split, and the kill on a client-side timeout. They fail on main.
  • Real container: a 600 s test with a 30 s launcher limit reports timeout in about 34 s, and the next stage runs clean. Without a limit, the client-side timeout leaves no test-user process running.
  • test_validate_stage_script, test_pipeline_pr_runtime, test_ungraded_sites (106 passed), ruff check, ruff format --check.

Out of scope

A fresh container per stage (RFC per #253). Part of #253 (phase 0).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant