Rshahbazyan/add scicodepile benchmark

[rimashahbazyan]

What does this PR do?

Adds the executable stratum of SciCodePile as a NeMo Gym benchmark: 200
scientific code-generation tasks harvested from 37,737 public
computational-science repositories, each shipping its own test.

• Paper: https://arxiv.org/abs/2607.19104
• Dataset: https://huggingface.co/datasets/SciCodePile/SciCode-Runnable-Benchmark-Reviewed
• Upstream org: https://huggingface.co/SciCodePile

Each task's test defines check(candidate). The server looks up entry_point and
calls check with it. Reward is binary.

Not the same artifact as the existing scicode benchmark — different source
(SciCode1/SciCode vs SciCodePile), size (65 problems / 288 sub-steps vs 200
single-shot tasks), and structure (multi-step agent vs single-turn generation).
Checking coverage by name alone will read one as the other.

Verification design

setup_code, the model's code, and test are compiled and executed as three
separate units sharing one namespace — the __dict__ of a real module registered in
sys.modules. The separation is load-bearing, not stylistic: concatenated, a trailing
decorator in the model's output binds to the test's own def check and replaces the
assertions (a solution returning None then passed all 200 tasks), and
from __future__ imports are pushed off the top of the file.

Execution is out-of-process with a throwaway CWD per task (all 200 tasks carry the
upstream env_sensitive flag and 117 carry globals_patch, so tests mutate global
state), an RLIMIT_AS cap, pinned BLAS thread pools, and a process-group kill on
timeout. It is not a security sandbox — see the server README.

The dataset revision is pinned, so the benchmark is a fixed protocol rather than
whatever the Hub serves today.

Verification

The strongest correctness signal is model-independent:

• All 200 upstream canonical_solution values pass through the runner. This
  validates the harness with no model involved and cannot be confounded by model
  quality. Re-run it whenever the runner changes.
• Negative controls over all 200 tasks: a raising implementation fails 200/200;
  unparseable code errors 200/200; a stub returning None fails 197/200; a renamed
  function and an empty answer each score entry_point_missing 199/200.
• Two caveats the full-set controls surfaced, both documented in the server README:
  alignment/python/144, 178 and 273 pass with a stub returning None (their
  tests only assert the result is None or any object — worth ~1.5pp of any score),
  and alignment/python/76 defines its own entry point in setup_code.
• 84 tests, including mutation-checked regression tests for every issue raised in
  review.

Prompting

The upstream prompt is passed through unmodified with no system message. That is
the only prompting the paper specifies: zero-shot on "HumanEval-style task prompts,
each consisting of a function signature and its natural-language specification". The
paper publishes no system prompt, no decoding parameters, and no per-task sample count.

This costs some tasks where a chat model answers conversationally instead of writing
code. Note that the statuses do not cleanly isolate those non-attempts: with no
fence the extractor returns the whole response, so a refusal compiles and lands as
error/syntax_error, the same status as a real attempt with a syntax error.
Counting non-attempts means reading the text.

Comparison with the published figures

Upstream's Table 3 reports Pass@1 / Pass@5 for 15 models. GPT-5.4-mini — the strongest
model in that table — was run through this server on all 200 tasks with the shipped
config (upstream prompt verbatim, no system message, max_output_tokens 131072),
5 repeats:

Model	Metric	Upstream	This server	95% CI (ours)
GPT-5.4-mini	Pass@1	12.30%	9.30% (pass@1[avg-of-5])	[5.80%, 13.20%]
GPT-5.4-mini	Pass@5	15.50%	13.00%

The interval describes the precision of our estimate — a bootstrap resampling tasks
rather than rollouts, because repeats are clustered within a task. It is not a
comparison with upstream: it is wide enough to contain their figure, but containment
is not a test, and no test is available here.

harness_failure was 0.0% across all 1000 rollouts — no dataset-owned or
runner-internal fault fired, so the whole gap is model behaviour rather than harness
noise.

No test of "ours versus theirs" is offered, because the data to do one does not exist
publicly. The paper reports a rounded aggregate with no per-task outcomes and no
per-task sample count — 12.30% of 200 is 24.6, so the figure is not a count of tasks
and cannot be treated as one. Since both runs cover the same 200 tasks, a paired or
cluster-aware analysis would be appropriate if upstream's per-task outcomes were ever
published; an independent two-sample test on reconstructed counts would not be.

Exact reproduction is not possible from the paper as written: it publishes neither the
decoding parameters nor the number of samples per task, both of which the HumanEval
Pass@k estimator depends on. The server README therefore records only the
paper-reported figures and makes no reproduction claim.

Treat model ranking on this benchmark with caution at these score levels. The
paper's own table puts GPT-5 (5.70%) below GPT-4o (7.50%).

Notes

• example.jsonl ships 5 rows with example_metrics.json and example_rollouts.jsonl;
  gym env test --resources-server scicodepile +should_validate_data=true passes.
• code_extraction.py is duplicated from the bigcodebench server to hold score parity;
  nothing enforces the two stay identical, which the module docstring states.

Checklist

• I have read the contributing guidelines.
• The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
• Tests added or updated and pass locally (84 passed), plus a 200/200 canonical-solution validation.
• Pre-commit checks pass locally.
• All commits have DCO sign-off (git commit -s).

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[armarkos]

Thanks for a carefully built PR. The runner design, the upstream-data notes, and the negative-control mindset are all above the bar for code benchmarks in this repo. I reviewed at head efd0e4d, read the sibling code benchmarks (bigcodebench, evalplus/human_eval, code_fim, code_gen, scicode) and the core paths this server touches, and re-ran the model-independent checks over all 200 upstream tasks in a clean nemo-gym[dev] env. Two of the findings below are verifier-correctness issues that I think should block merge; the rest are ordered by impact.

Note on CI: because of the copy-pr-bot gate, only lint, copyright, secrets, and DCO ran on this PR. Unit tests and gym env test have not run in CI yet.

Blocking

1. One compile unit lets model output replace check. scp_runner.py:91-106 concatenates setup_code + code + test and compiles it once. A trailing decorator line in the model's code therefore decorates the test's def check(candidate):. This wrong solution passes 200/200 tasks:

def <entry_point>(*a, **k):
    return None
@(lambda f: (lambda candidate: None))

The @(lambda f: None) variant yields test_defines_no_check, which app.py:55 then files as a harness failure. Fix: exec setup_code and code, then compile and exec test as a separate unit in the same namespace, and take check from that unit only.

2. The same single unit mis-scores legitimate solutions and mis-attributes dataset faults. A correct solution that begins with from __future__ import annotations is scored error/syntax_error on all 105 tasks with non-empty setup_code ("from future imports must occur at the beginning of the file"). Failures raised by dataset-owned setup_code or the test's module body surface as exec_failed/syntax_error with failure_reason unset, i.e. charged to the model, and SyntaxError line numbers are offset by the setup length. Separate compile units with distinct filenames fix all three, and let setup/test-phase failures carry a harness FailureCode.

3. The verdict can still be forged. The fd-1 redirect (scp_runner.py:142-158) blocks only writes to fd 1. Task code that loops os.write(fd, forged_json) over fds 3..63 and then calls os._exit(0) produced {"status": "pass"} in the parent and reward 1.0; monkeypatching json.dumps at module level does the same. The docstring, commit 53e5dee and test_task_cannot_forge_a_pass_verdict claim forge resistance that does not hold. Either drop the claim and keep the redirect for its real value (fd-1 noise no longer corrupts honest verdicts), or add a parent-generated nonce passed on stdin and required in the result, and state the residual risk (frame introspection) explicitly.

4. gym env test data validation fails for this server. nemo_gym/cli/env.py:829-894 asserts exactly 5 rows in data/example.jsonl, plus data/example_metrics.json and data/example_rollouts.jsonl with 5 entries each; CI's changed-server job runs it with +should_validate_data=true (.github/workflows/unit-tests.yml:123). This PR ships 3 rows and neither companion file, and data/.gitignore (* / !example.jsonl) would exclude them if generated. 119 of the other 120 servers commit all three. Regenerate 5 examples, run gym dataset collate ... +mode=example_validation, collect the 5 example rollouts, and narrow the gitignore to the sibling pattern.

Should fix

5. failure_reason flags model-caused outcomes as harness faults. Through the real /verify endpoint: an infinite-loop solution gets failure_reason=timeout, os._exit(0) in the candidate gets unparseable_runner_output, and a compile() MemoryError gets runner_crashed. BaseVerifyResponse defines the field as "why reward may not reflect policy quality" (nemo_gym/base_resources_server.py:100-102), and the FailureCode docstring (app.py:36-43) tells readers to filter these rows out of accuracy. Doing so inflates pass@1 and makes hanging or exiting reward-neutral under RL. No code in nemo_gym consumes the field, and none of the six servers with a FailureCode enum flags a model-code timeout; code_gen scores TLE as a failure. Reserve failure_reason for setup/test-phase and runner-internal faults, and consider a harness_failure score in _score_fn so it gets its own metric line instead of a manual filter.

6. Runner exit path turns passes into timeouts. proc.communicate() (app.py:190) waits for EOF on stderr, which any process the task spawns inherits; a non-daemon thread or atexit hook left by task code also keeps the runner alive after the verdict is written. In both cases a written pass becomes timeout, a semaphore slot is held for the full 120 s, and children are orphaned because there is no process group. Six upstream rows already use subprocess/multiprocessing. Fix: os._exit(0) right after os.close(result_fd), redirect fd 2 in the runner, spawn with start_new_session=True and kill the group, and kill in a finally so a CancelledError on uvicorn shutdown (server_utils.py:949, 0.5 s grace) does not leave the child running while rmtree deletes its cwd.

7. The namespace is a bare dict, not a module. Dataclasses under PEP 563 (sys.modules[cls.__module__] is None), pickling or multiprocessing of task-defined functions ("import of module __scicodepile__ failed"), and __file__ all fail and are charged to the model. Exec in a types.ModuleType("__scicodepile__") registered in sys.modules with __file__ set.

8. system: "" sends an empty system message on every request. nemo_gym/prompt.py:66-67 emits a system turn whenever the key is present, and all three example rows carry {"role": "system", "content": ""}. Nothing downstream strips it; on the vLLM chat path it is template-dependent (Qwen templates drop their default system prompt, Llama templates render an empty block), and the completions path prepends a blank line pair. This is not the "unmodified upstream prompt" the README claims, and it perturbs the reported 9.50%. Of 41 prompt YAMLs only bigcodebench's uses system: "", for NeMo-Skills byte parity that does not apply here. Delete the key (or reuse benchmarks/prompts/generic/default.yaml) and regenerate example.jsonl.

9. Model-controlled exception text can 500 the endpoint. A lone surrogate in details.message (raise ValueError('\udcff'), or a surrogateescape-decoded filename) survives the runner's JSON round-trip and makes the response serialization raise, so /verify returns HTTP 500. With the default route_failures_to_sidecar=False that aborts the whole rollout run (rollout_collection.py:1890-1920). Sanitize details strings with .encode("utf-8", "replace").decode().

10. The 8 GiB RLIMIT_AS is untested where it applies. Every task imports numpy; with uncapped OpenBLAS threads, import numpy can fail under an 8 GiB address-space cap on many-core Linux nodes (OpenBLAS issue #4762 reproduces at exactly this limit), and it would be scored as the model's exec_failed. All tests pass max_as_limit=0 and the 200/200 validation ran on macOS, where the cap is skipped. Set OPENBLAS_NUM_THREADS=1/OMP_NUM_THREADS=1 in the child env, consider bigcodebench's 30 GiB, and add a Linux test with the cap on.

Minor

• python benchmarks/scicodepile/prepare.py --output x fails with Hydra's "unrecognized arguments: --output" because get_global_config_dict() re-parses argv; use maybe_get_global_config_dict() or drop the flag. The ast.literal_eval branch is dead (load_dataset yields dicts) and its comment is wrong. gym eval prepare works (200 rows).
• No test exercises verify(), _score_fn, or get_key_metrics (app.py 75% covered), and test_solution_stdout_does_not_corrupt_the_result drives the in-process run_task, so it stays green with the fd-1 protections deleted. Mirror bigcodebench's TestClient tests for pass/fail/empty_output/no_code_block.
• Three upstream tasks (alignment/python/144, 178, 273) pass with a stub returning None; their tests only assert the return is None or any object. The README's negative-control claim was measured on 30 tasks. Document or flag them, and run the controls over all 200 (none pass with an empty answer or a raising stub, which is good).
• The raw-vs-wrapped A/B is a single repeat where the gap is two tasks out of 200; either add repeats or soften "made from measurement".
• Prose answers land as error/syntax_error, not no_code_block (no fence means the whole text is returned), so the statuses do not cleanly isolate non-attempts.
• Task stdout/stderr are captured into never-read StringIO buffers even though fd 1 is already /dev/null; redirect to os.devnull instead.
• REVERIFY_MODE is left UNKNOWN though verify is a pure function of the persisted row; declaring STATELESS unlocks gym eval reverify without ++force.
• meta["test"]/meta["entry_point"] raise KeyError on a malformed row, which 500s and aborts the run by default; validate in the request model or in prepare.py.

What checks out

All 35 tests pass in a clean env; all 200 canonical solutions pass with only the shipped requirements; empty answers and raising stubs pass 0 tasks; the core benchmark-discovery tests pass with this tree; the dataset is public, Apache-2.0, 200 rows, all runnable and primary_score_eligible; the README row is byte-identical to the generator's output; ruff is clean; task_data.py follows the convention; the metric helpers are used correctly; and the execution model is the bigcodebench pattern with strictly more containment.