Codex/benchmark methodology doc

[ternaus]

Summary by Sourcery

Introduce a detailed benchmark methodology reference document and wire it into existing docs and tests so methodology, scope, and architecture remain consistent and discoverable.

Documentation:

• Add a comprehensive benchmark methodology document covering scopes, transform policy, timing regimes, environment isolation, and cloud execution.
• Clarify README and benchmark scope docs to distinguish micro vs DataLoader benchmarks and to reference the new methodology source.

Tests:

• Extend architecture docs tests to ensure the new methodology document is linked from README/scope docs and references core benchmark modules without table-format content.

[sourcery-ai]

Reviewer's Guide

Adds a new prose benchmark methodology document and wires it into tests and existing docs so methodology, scope, and architecture references stay consistent while clarifying DataLoader benchmark semantics.

Flow diagram for benchmark control flow and dispatch

flowchart LR
    A[Checked_in_YAML_config] --> B[benchmark_cli_py]
    B --> C[benchmark_config_resolve_py]
    C --> D[benchmark_config_models_py]
    D --> E[benchmark_config_plan_py]
    E --> F[benchmark_jobs_py]
    F --> G[benchmark_matrix_py]
    G --> H[benchmark_orchestrator_py]

    H --> I[benchmark_pyperf_micro_runner_py]
    H --> J[benchmark_pipeline_runner_py]
    H --> K[benchmark_dali_pipeline_worker_py]

Loading

File-Level Changes

Change	Details	Files
Add tests enforcing references to the new methodology document and core benchmark modules.	Introduce a test that ensures README and benchmark scope docs both link to docs/benchmark_methodology.md. Introduce a test that ensures the methodology doc mentions all core policy/matrix/modules and that those modules exist on disk. Add a guard that disallows table-style lines in the methodology doc to keep it prose-only.	tests/test_architecture_docs.py
Clarify README explanation of DataLoader benchmarks and point readers to the new methodology source document.	Rewrite the DataLoader benchmarks bullet to emphasize that they are full recipe measurements with fixed steps included in throughput. Replace the inline numbered methodology list with a short summary that delegates details to docs/benchmark_methodology.md.	README.md
Clarify benchmark scope documentation and reference the new methodology document for rationale and prose methodology.	Add a top-level note distinguishing the scope doc (operational reference) from the longer-form methodology doc. Explicitly state that fixed crop shape prep, normalization, and tensor conversion are included in DataLoader throughput. Add a cross-reference from the architecture scope section to the methodology doc for rationale behind scopes and timing decisions.	docs/benchmark_scope.md
Introduce a comprehensive benchmark methodology prose document that describes config flow, scenarios, timing regimes, policies, and interpretation rules.	Create docs/benchmark_methodology.md describing control flow from configs to jobs and orchestrators. Document scenario and library sets, transform selection policy, environment isolation, media loading, and cloud dataset staging. Describe micro timing via pyperf, DataLoader pipeline timing, GPU and DALI pipeline handling, slow-transform policy, unsupported/early-stopped rows, and result metadata. Provide guidance on interpreting claims, emphasizing separation of micro, CPU/GPU DataLoader, and DALI regimes.	docs/benchmark_methodology.md

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey - I've found 1 issue, and left some high level feedback:

• The test_methodology_doc_references_core_policy_modules test couples the methodology doc tightly to specific file paths, which may cause unnecessary test churn when files move or are renamed; consider only asserting the presence of higher-level concepts or a smaller, more stable subset of modules.
• The assert not any(line.startswith("|") for line in doc.splitlines()) check broadly forbids any line beginning with |, which may block legitimate future use of markdown tables or code examples; consider scoping this constraint to specific sections or using a more targeted pattern.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- The `test_methodology_doc_references_core_policy_modules` test couples the methodology doc tightly to specific file paths, which may cause unnecessary test churn when files move or are renamed; consider only asserting the presence of higher-level concepts or a smaller, more stable subset of modules.
- The `assert not any(line.startswith("|") for line in doc.splitlines())` check broadly forbids any line beginning with `|`, which may block legitimate future use of markdown tables or code examples; consider scoping this constraint to specific sections or using a more targeted pattern.

## Individual Comments

### Comment 1
<location path="tests/test_architecture_docs.py" line_range="83" />
<code_context>
+        assert module_path in doc
+        assert (REPO_ROOT / module_path).exists()
+
+    assert not any(line.startswith("|") for line in doc.splitlines())
+
+
</code_context>
<issue_to_address>
**issue:** The no-table assertion misses lines with leading whitespace before '|'

This won’t catch tables when the `|` is indented (e.g. `'  | col |'`). Consider stripping leading whitespace before the check, e.g. `any(line.lstrip().startswith("|") for line in doc.splitlines())`, so indented tables are also rejected.
</issue_to_address>

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[Copilot]

Pull request overview

This PR introduces a new prose “benchmark methodology” document intended to be the canonical narrative description of what the benchmark measures (and excludes), then wires it into existing documentation and adds tests to ensure the doc stays referenced and aligned with core modules.

Changes:

• Add docs/benchmark_methodology.md describing benchmark scopes (micro vs DataLoader vs GPU vs DALI), transform selection policy, isolation, and execution flow.
• Link the new methodology doc from README.md and docs/benchmark_scope.md (and tighten wording around DataLoader recipe measurement scope).
• Add tests that enforce the new doc is referenced and mentions key implementation modules.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
tests/test_architecture_docs.py	Adds assertions that the methodology doc is referenced and contains required module references.
README.md	Replaces the inline Methodology section with a link to the dedicated methodology doc; clarifies DataLoader benchmark scope.
docs/benchmark_scope.md	Adds a pointer to the new methodology doc and clarifies that fixed recipe steps are part of DataLoader throughput.
docs/benchmark_methodology.md	New long-form methodology source document describing benchmark measurement regimes and rationale.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.