Repository navigation
docs(readme): include kornia in benchmark tables - #70
Merged
Merged
Conversation
Add Kornia columns for RGB, 9-channel, and video README tables, generate 9-channel and video overview plots, and align docs with the published artifacts.
Reviewer's GuideAdds Kornia results to all README benchmark tables (RGB, 9-channel, video), introduces new 9-channel and video overview plots, and aligns documentation and synchronization logic with the newly published 9-channel and video benchmark artifacts. File-Level Changes
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've left some high level feedback:
- In
_scenario_overview_plot, the library axis is restricted toLIBRARY_ORDERand silently drops any library not listed there; consider either asserting that all libraries are covered or deriving the order from the data to avoid missing new or renamed libraries in future runs. - The new
_scenario_overview_plotuses several hard-coded plotting constants (figure size, x_min/x_max scaling factors, marker size, font sizes); pulling these into named constants or parameters would make it easier to tune the aesthetics consistently across RGB, 9‑channel, and video scenarios.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- In `_scenario_overview_plot`, the library axis is restricted to `LIBRARY_ORDER` and silently drops any library not listed there; consider either asserting that all libraries are covered or deriving the order from the data to avoid missing new or renamed libraries in future runs.
- The new `_scenario_overview_plot` uses several hard-coded plotting constants (figure size, x_min/x_max scaling factors, marker size, font sizes); pulling these into named constants or parameters would make it easier to tune the aesthetics consistently across RGB, 9‑channel, and video scenarios.Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2693a233bd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| | 9-channel CPU DataLoader Kornia | `configs/paper/prod_c4_highmem_9ch_dataloader_cpu_kornia.yaml` | Added after the standard C4 run killed Kornia while preloading 10k float32 9-channel tensors. | Run on `c4-highmem-16` to preserve the same `memory_dataloader_augment`, 10k-item protocol. If highmem quota is unavailable, run a reduced-n Kornia row and label it memory-limited/not same-n. | | ||
| | 9-channel GPU DataLoader | `configs/paper/prod_g2_9ch_dataloader_gpu.yaml` | Not confirmed complete locally. | Run or locate the G2 job after the current 1-GPU queue is clear. | | ||
| | Video CPU micro | `configs/paper/prod_c4_video_micro_cpu.yaml` | GCS run `ca332e6f5dd949fd85b8435c5b56346d` failed during Kornia `Rotate` with `SIGSEGV: 11`; GCS run `132f29d9503b49fead76c45ba25a6851` excluded `Rotate` and then failed during Kornia `Elastic` with `SIGSEGV: 11`. AlbumentationsX and TorchVision outputs were fetched locally from the first run. | Keep AlbumentationsX/TorchVision. Rerun Kornia CPU micro with `Rotate` and `Elastic` excluded by the scenario filter. | | ||
| | 9-channel GPU micro | `configs/paper/prod_g2_9ch_micro_gpu.yaml` | Published from `results/published/paper-9ch-micro-gpu-g2-standard-16-2026-05-10/`. Kornia has 34 ok rows and 4 unsupported CUDA rows; TorchVision has 21 ok rows. | Keep as the 9-channel GPU micro result set. | |
There was a problem hiding this comment.
The newly generated public overview data for this same scenario reports only 20/21 full TorchVision rows and 30/38 full Kornia rows (docs/benchmark_data/figure_image9ch_overview.csv), because Elastic and four Kornia transforms are early-stopped rather than full measured rows. Leaving this status row as “21 ok” and “34 ok” overstates the published measured coverage and contradicts the README-facing figure data that readers are directed to use for coverage interpretation.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Pull request overview
This PR updates the benchmark documentation and figure-generation pipeline to include Kornia coverage in README tables and add scenario overview plots for 9-channel images and video.
Changes:
- Adds 9-channel and video overview figures/CSV exports.
- Expands README benchmark tables with additional CPU/GPU library columns.
- Updates benchmark status and plotting guidance docs to reflect published artifacts.
Reviewed changes
Copilot reviewed 7 out of 16 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
scripts/paper/generate_figures_and_insights.py |
Adds scenario overview plot generation and expands README table columns. |
README.md |
Adds new overview figures and expanded benchmark result tables. |
docs/good_plots.md |
Updates recommended figure list and roles. |
docs/benchmark_scope.md |
Updates published benchmark status checklist. |
docs/benchmark_methodology.md |
Documents README-facing scenario overview plots. |
docs/benchmark_data/figure_image9ch_overview.csv |
Adds generated 9-channel overview summary data. |
docs/benchmark_data/figure_video16f_overview.csv |
Adds generated video overview summary data. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| | 9-channel CPU DataLoader Kornia | `configs/paper/prod_c4_highmem_9ch_dataloader_cpu_kornia.yaml` | Added after the standard C4 run killed Kornia while preloading 10k float32 9-channel tensors. | Run on `c4-highmem-16` to preserve the same `memory_dataloader_augment`, 10k-item protocol. If highmem quota is unavailable, run a reduced-n Kornia row and label it memory-limited/not same-n. | | ||
| | 9-channel GPU DataLoader | `configs/paper/prod_g2_9ch_dataloader_gpu.yaml` | Not confirmed complete locally. | Run or locate the G2 job after the current 1-GPU queue is clear. | | ||
| | Video CPU micro | `configs/paper/prod_c4_video_micro_cpu.yaml` | GCS run `ca332e6f5dd949fd85b8435c5b56346d` failed during Kornia `Rotate` with `SIGSEGV: 11`; GCS run `132f29d9503b49fead76c45ba25a6851` excluded `Rotate` and then failed during Kornia `Elastic` with `SIGSEGV: 11`. AlbumentationsX and TorchVision outputs were fetched locally from the first run. | Keep AlbumentationsX/TorchVision. Rerun Kornia CPU micro with `Rotate` and `Elastic` excluded by the scenario filter. | | ||
| | 9-channel GPU micro | `configs/paper/prod_g2_9ch_micro_gpu.yaml` | Published from `results/published/paper-9ch-micro-gpu-g2-standard-16-2026-05-10/`. Kornia has 34 ok rows and 4 unsupported CUDA rows; TorchVision has 21 ok rows. | Keep as the 9-channel GPU micro result set. | |
| | PlasmaContrast | 17.3 ± 0.0 | - | 1.4 ± 0.0 | **68.6 ± 0.0** | - | 11.7 ± 0.0 | - | 50.9 ± 0.0 | - | 50.3 ± 0.0 | - | - | | ||
| | PlasmaShadow | 58.8 ± 0.0 | - | 4.5 ± 0.0 | **139.2 ± 0.0** | - | 36.8 ± 0.0 | - | 61.4 ± 0.0 | - | 52.3 ± 0.0 | - | - | | ||
| | Posterize | 2910.0 ± 0.0 | 1053.1 ± 0.0 | - | 315.8 ± 0.0 | 225.7 ± 0.0 | - | **21706.1 ± 0.0** | - | 425.7 ± 0.0 | - | - | - | | ||
| | PyTorchVideoCanonical | - | - | - | - | - | - | - | - | - | - | - | **17.2 ± 0.0** | |
| ("video16f_dataloader_gpu", "torchvision"), | ||
| ("video16f_dataloader_gpu", "kornia"), | ||
| ("video16f_dataloader_gpu", "dali"), | ||
| ("video16f_dataloader_gpu", "pytorchvideo"), |
Comment on lines
+720
to
+726
| def _scenario_overview_plot( | ||
| df: pd.DataFrame, | ||
| *, | ||
| scenario_prefix: str, | ||
| output_stem: str, | ||
| item_unit: str, | ||
| ) -> pd.DataFrame: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add Kornia columns for RGB, 9-channel, and video README tables, generate 9-channel and video overview plots, and align docs with the published artifacts.
Summary by Sourcery
Update benchmark documentation and figure generation to incorporate Kornia across RGB, 9-channel, and video scenarios and align README figures and tables with the finalized published results.
New Features:
Enhancements: