fix(config): fail fast with a clean error on a malformed head_server block

[sven-rankov]

What does this PR do?

Closes #2686.

A head_server value with the wrong type — typically a ++head_server.port=<not an int> override — was only validated when ServerClient.load_head_server_config built a BaseServerConfig, deep inside the command. BaseServerConfig is not a BaseNeMoGymCLIConfig, so _handle_pydantic_validation_error re-raised the pydantic error and the user got a 71-line traceback (gym eval run --no-serve), gym env start initialised Ray before failing, and gym env validate / gym env resolve accepted the value and exited 0.

This PR moves the parser's head_server handling to where the block is first read in GlobalConfigDictParser.parse() (previously it read port for the port allocator there and only filled the defaults ~20 lines later): fill the defaults, validate the block against BaseServerConfig, hand the validated port to the allocator, and raise a new HeadServerConfigMalformedError(ConfigError, ValueError) with an actionable message — the same pattern the parser already uses for config_paths (MalformedConfigPathsError). Every affected command loads the config first and is wrapped in exit_cleanly_on_config_error, so all of them now print one Error: line and exit 1 before any server (or Ray) starts.

Before / after

gym env validate ++head_server.port=notanint — before (main): exit 0

✓ Config is valid.

gym eval run --no-serve --agent simple_agent -i in.jsonl -o out ++head_server.port=notanint — before: exit 1, 71 lines on stderr (abridged):

Traceback (most recent call last):
  File "<repo>/.venv/bin/gym", line 10, in <module>
  ... 16 more frames ...
  File "<site-packages>/pydantic/main.py", line 732, in model_validate
pydantic_core._pydantic_core.ValidationError: 1 validation error for BaseServerConfig
port
  Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value='notanint', input_type=str]
    For further information visit https://errors.pydantic.dev/2.13/v/int_parsing

After — the same for every command that loads the config (env validate, env resolve, env start, eval run): exit 1, nothing on stderr

Error: 'head_server' is invalid: head_server.port (Input should be a valid integer, unable to parse string as an integer).
`head_server.host` must be a hostname or IP; `head_server.port` and, if set, `head_server.num_workers` must be integers. Set them in a config or as overrides, e.g.:
  ++head_server.host=127.0.0.1 ++head_server.port=11000

Notes:

• The issue's other finding (the override dropping the defaulted host) was already fixed by feat(orchestration): return and persist a record of every submission #3188; this PR covers the remaining traceback half.
• _handle_pydantic_validation_error's re-raise for non-CLI models is intentionally left as is: test_non_config_validation_error_is_reraised documents that a ValidationError from a non-CLI model may be a real bug that must propagate. Handling the user-facing block at the source keeps that contract.
• Validation only — no coerced values are written back, so gym env resolve output is unchanged for valid configs. The validated (coerced) port is what now reaches the port allocator, so a quoted port: "11000" is kept off it (before, the string '11000' never matched an allocated int).
• A scalar block (++head_server=foo) used to fail with an AttributeError at the parser's first .get; it now gets the same ConfigError treatment with its own message.
• Out of scope / possible follow-ups: a generic fallback for other non-CLI models validated mid-command (GlobalAIOHTTPAsyncClientConfig, TelemetryConfig, exporter configs).

Validation

Before (main dbd26a9) / after, from the repo root:

• gym eval run --no-serve --agent simple_agent -i in.jsonl -o out ++head_server.port=notanint — before: exit 1, 71-line traceback ending in ValidationError ... for BaseServerConfig; after: exit 1, the 5-line Error: message above on stdout, nothing on stderr.
• gym env validate ++head_server.port=notanint — before: ✓ Config is valid., exit 0; after: Error: ..., exit 1.
• gym env resolve ++head_server.port=notanint — before: dumps port: notanint, exit 0; after: Error: ..., exit 1.
• gym env validate ++head_server.port=12345 — still ✓ Config is valid., exit 0.
• gym env validate ++head_server.host=null — after: Error: naming head_server.host, exit 1.
• gym env validate ++head_server=foo — before: exit 1, 29-line traceback ending in AttributeError: 'str' object has no attribute 'get'; after: Error: 'head_server' must be a mapping of hostandport. Got: 'foo'., exit 1.
• gym env validate ++head_server.num_workers=abc — after: Error: naming head_server.num_workers, exit 1.

Tests:

• New TestConfigLoadErrors::test_parse_malformed_head_server_raises_actionable_error (parser level, parametrized over a non-integer port, a null host, and both at once; asserts every field path, the override hint, the ValueError compat contract, and that __cause__ is the pydantic error).
• New TestConfigLoadErrors::test_parse_scalar_head_server_raises_actionable_error (head_server: foo raises the ConfigError with the override hint instead of an AttributeError).
• New TestConfigLoadErrors::test_parse_keeps_a_head_server_value_pydantic_can_coerce (a quoted port: "8080" still parses and is left as written — the check is lax and validate-only — while disallowed_ports holds the validated int 8080).
• New TestFriendlyValidationError::test_malformed_head_server_override_is_a_config_error_not_a_traceback (through the real main(), parametrized over env validate, env resolve, eval run --no-serve; asserts exit 1, Error: on stdout, no "Config is valid", no traceback).
• New TestFriendlyValidationError::test_malformed_head_server_is_rejected_before_any_server_starts (gym env start through the real main() with RunHelper replaced by a sentinel that fails if constructed — proves the error fires before Ray or the head server come up).
• Verified each of these fails on main for the issue's reasons (exit 0 / raw ValidationError / the sentinel firing).
• pytest tests/unit_tests/test_global_config.py tests/unit_tests/test_cli_main.py tests/unit_tests/test_cli.py tests/unit_tests/test_server_utils.py tests/unit_tests/test_server_status.py tests/unit_tests/test_cli_eval.py tests/unit_tests/test_cli_eval_submit.py — 594 passed.
• pytest tests/unit_tests/ (macOS) — 5127 passed; the 53 failures are all pre-existing on main and unrelated (test_super_vllm_launcher.py bash launcher subtests, test_ci_environment.py, test_e2b_provider.py, test_enroot_provider.py).
• ruff check / ruff format --check, pre-commit run --all-files — clean.
• python3 tests/unit_tests/test_fern_docs_links.py — OK.

Rollouts

N/A — config parsing / CLI error handling only; no environment, agent, model, or verifier behavior changes.

Compatibility and benchmark impact

• A head_server block that was previously accepted at parse time despite a wrong-typed value (non-integer port, null host/port) now fails with a ConfigError for every command, instead of a traceback (or exit 0) later. Well-typed configs, including a string port that pydantic coerces (e.g. port: "8080"), are unaffected.
• No change to gym env resolve output for valid configs.
• HeadServerConfigMalformedError subclasses ValueError like the rest of the ConfigError family.
• No benchmark-result impact.

Proposed labels (I cannot apply labels): bug, area:config, complexity:low

Checklist

• I have read the contributing guidelines.
• The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
• Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
• Documentation added or updated, or N/A with justification.
• Pre-commit checks pass locally (pre-commit run --all-files) (so CI lint/format checks pass).
• New source files include the required Apache-2.0 SPDX header.
• All commits have DCO sign-off (git commit -s) (so the DCO check passes).

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[sven-rankov]

Friendly ping on this one. @yaoyu-33 Would you be able review?
I'm happy to rebase or split it if that would make review easier.