Repository navigation
Conversation
99283b2 to
0778714
Compare
11558cb to
d0ae42e
Compare
Merging this PR will improve performance by ×2.32 warnings
Performance Changes
Tip Curious why performance improved? Comment Comparing |
798b112 to
3ed0d39
Compare
|
cc: @h-vetinari |
|
@travishathaway thank you for the quick review. I'm sorry i wans't able to "apply" your suggestion, i think Github changed their interface and I wasn't able to click "apply" to both. let me know if you prefer i squash these into 3 logical commits. |
| @@ -0,0 +1,19 @@ | |||
| ### Enhancements | |||
|
|
|||
| * Make `conda_build.variants.list_of_dicts_to_dict_of_lists` linear in the number of variants. Rendering a recipe against a large variant matrix, such as conda-forge's global pinning, spent most of its time rebuilding these lists. (#6267) | |||
There was a problem hiding this comment.
This is still quadratic. Not sure why you say it's linear.
There was a problem hiding this comment.
Basically O(m*n) where m is the number of variants and n the number of keys. Same as before
There was a problem hiding this comment.
I guess the difference is that lists and sets are not created and recreated over and over again.
There was a problem hiding this comment.
Happy to rephrase along this line
There was a problem hiding this comment.
I've rephrased this to remove the claim about linear vs quadratic. Hopefully that helps clarify the improvement.
There was a problem hiding this comment.
I wonder if we should keep only the top optimizations with allowing for simple ones too:
- conda-build 1
- conda-build 2
- conda-build 3
- rattler-build-conda-compat 1
- rattler-build-conda-compat 2
- conda-smithy 1
- conda-smithy 3
We do pay costs in complexity in some cases, like the async azure build ID fetching, for marginal gains.
This PR does the first 3 right? So that is ok right? |
810b880 to
4340c40
Compare
4340c40 to
34a4aa9
Compare
<details><summary>Claude's draft</summary> For every variant, each key's collected values were rebuilt from scratch (`list(old) + [v]`, then `list(set(...))`; `set(old) | set(v)`). Zipped keys keep one value per variant, duplicates included, because they are paired up again at the end, so their list grows to the number of variants and was copied in full for every variant: quadratic in the number of variants. The other keys only hold their few distinct values, but were still rebuilt as a new list and set every time. Values are now accumulated in place: zipped keys extend their list, and every other key collects into a set, turned into a list at the end for keys whose last value was a scalar, as before. Lists and dicts keep their previous rules, including the AttributeError a dict raises after a scalar. On conda-forge's opencv-feedstock (variant lists of up to 28,800 entries of ~520 keys) a rerender goes from 208 s to 96 s, byte-identical output. A 20,000-case random comparison with the previous implementation found no difference except for the order of values both collect through a set; the new tests pass before and after. Resume this Claude session: ``` cd /home/mark/git/ramona/python-owl/.claude/worktrees/bridge-cse_01BiGuuSU1wyGE5gzJ8oLsuD claude --resume 56499d72-0fb0-5cfd-a53c-c7dc17432c2f ``` </details>
e2a0345 to
03f6310
Compare
<details><summary>Claude's draft</summary> With values accumulated in place, the function still does a Python-level step, with its type checks, for every key of every variant. Each key is now squished on its own, a column at a time: when every variant holds the same keys (always, for an exploded matrix) the column is `list(map(itemgetter(k), variants))`; a column of hashable scalars reduces to one set() (or stays a list for zipped keys); any other column replays the original per-value rules in order, including the mixed scalar/list/dict cases and the AttributeError a dict raises after a scalar. The work is the same O(variants x keys); the gain is the constant factor. opencv-feedstock rerenders in 75 s instead of 96 s after the previous commit (208 s on main), byte-identical output. On a synthetic list of 16,000 variants of 520 keys the two are close (1.3 s against 1.4 s; 18.3 s on main), so most of the gain on opencv comes from the shape of its real variant lists. The same 20,000-case random comparison with the original implementation matched, except for the order of values both collect through a set. Scalars are recognised with isinstance, so subclasses of str, such as the ScalarString values ruamel.yaml returns, take the one-step path as well. This is the larger change of the two; the previous commit alone removes the quadratic part and rebuilds nothing. Resume this Claude session: ``` cd /home/mark/git/ramona/python-owl/.claude/worktrees/bridge-cse_01BiGuuSU1wyGE5gzJ8oLsuD claude --resume 56499d72-0fb0-5cfd-a53c-c7dc17432c2f ``` </details>
<details><summary>Claude's draft</summary> get_vars compared each variable across every variant in a Python generator; map(ne, repeat(first), map(itemgetter(var), others)) does the same comparisons, with the same short circuit and KeyError, in C. filter_by_key_value formatted a debug f-string (including the whole list of target values) for every variant it dropped, and called get_logger for each, although get_logger sets the level to INFO so the message is never shown unless a log config file asks for debug. The kept variants are now selected in one comprehension, and the messages are only built when debug is enabled. On conda-forge's opencv-feedstock (variant matrices of up to 28,800 entries) these two took a rerender from 96 s to 64 s, with byte-identical output. Resume this Claude session: ``` cd /home/mark/git/ramona/python-owl/.claude/worktrees/bridge-cse_01BiGuuSU1wyGE5gzJ8oLsuD claude --resume 56499d72-0fb0-5cfd-a53c-c7dc17432c2f ``` </details>
<details><summary>Claude's draft</summary> parse_config_file() used PyYAML's pure-Python BaseLoader. CBaseLoader is the same loader (BaseConstructor, BaseResolver) on libyaml's parser, used when PyYAML was built with libyaml. It parses conda-forge's global pinning about 7x faster (20 ms -> 3 ms), and returned identical data for the pinning and for every recipe conda_build_config.yaml and migration file of 23 feedstocks. Resume this Claude session: ``` cd /home/mark/git/ramona/python-owl/.claude/worktrees/bridge-cse_01BiGuuSU1wyGE5gzJ8oLsuD claude --resume 56499d72-0fb0-5cfd-a53c-c7dc17432c2f ``` </details>
03f6310 to
615cac5
Compare

This should really help rerendering things at conda forge.
Broken out into separate ideas in distinct commits. The biggest are the first two:
list_of_dicts_to_dict_of_listscopied every zipped key's growing list for every variant, which was quadratic in the number of variants; commit 1 accumulates in place, and commit 2 squishes a column at a time for a further constant-factor gain.Along with other speedups this should really help conda-forge with the growing number of global pinnings.
xref:
Claude
Description
Four commits, ordered from the smallest and most obviously safe to the larger ones. None changes a result. (An ASCII fast path for recipe file decoding, which does change how some unusual files are read, was split out into #6270.)
1. Accumulate in place. For every variant, each key's collected values were rebuilt from scratch (
list(old) + [v], thenlist(set(...));set(old) | set(v)). Zipped keys keep one value per variant, duplicates included, because they are paired up again at the end, so their list grows to the number of variants and was copied in full for every variant, which is quadratic in the number of variants; the other keys only hold their few distinct values but were still rebuilt every time. Zipped keys now extend their list and every other key collects into a set, turned into a list at the end where its last value was a scalar, as before. opencv-feedstock: 204 s → 99 s.2. Squish a column at a time. The same O(variants × keys) work with a much smaller constant: when every variant holds the same keys, a column is
list(map(itemgetter(k), variants)); a column of hashable scalars is oneset()(or stays a list for zipped keys); any other column replays the original per-value rules in order, including the mixed scalar/list/dict cases and theAttributeErrora dict raises after a scalar. Scalars are recognised withisinstance, so subclasses ofstrsuch as ruamel.yaml'sScalarStringtake the one-step path too. opencv-feedstock: 99 s → 76 s; on a synthetic list of 16,000 variants of 520 keys the two are close (1.3 s against 1.4 s). This is the larger change; commit 1 alone removes the quadratic part. New tests pin the behaviour and pass onmaintoo; a 20,000-case random comparison against the old implementation matched for both commits, except for the order of values both collect through aset.3.
get_varsandfilter_by_key_valuecomparisons in C.get_varscompared each variable across every variant in a generator;any(map(ne, repeat(first[var]), map(itemgetter(var), others)))makes the same comparisons, same short circuit, sameKeyError(andoperator.nestill reflects aNotImplementedfrom mixed types; tested).filter_by_key_valuecalledget_logger()and formatted a debug f-string with the whole list of target values for every variant it dropped, althoughget_loggersets INFO; the messages are now built only when debug is enabled. opencv-feedstock: 76 s → 67 s.4. libyaml for variant config files.
parse_config_fileusesCBaseLoaderwhen PyYAML has libyaml: the same constructor and resolver, 3 ms instead of 20 ms for the global pinning, identical data for the pinning and every config and migration file of 23 feedstocks.Locally with conda 26.10,
tests/test_variants.py,test_utils.py,test_render.py,test_metadata.py: the same three tests fail here as onmainin this environment (test_build_bootstrap_env_by_name,test_build_run_exports_act_on_host,test_inner_python_loop_with_output: unsatisfiable win-64 test packages); everything else passes.Why rerenders are slow, and the series this PR is part of
conda-build explodes every multi-valued key of the combined variant spec into the variant matrix before it learns which keys a recipe uses. conda-smithy renders against the whole global pinning, which holds the keys of every feedstock, so keys like
nodejs,r_base,hip_compiler_version,blas_impland theroot_base,libarrowandgeant4/clhepzips multiply the matrix of recipes that never mention them. That matrix is then walked by code that is quadratic, or Python-level per entry. For opencv-feedstock, 20 builds become 28,800 variants of ~520 keys, and a rerender takes 3.5 minutes and 3.2 GB.The fix is spread over three projects, one draft PR each. Inside each PR the commits are ordered from the most conservative (equivalent code, same result) to the least, and each commit stands alone:
list_of_dicts_to_dict_of_lists· 2. the same function a column at a time · 3.get_vars/filter_by_key_valuecomparisons in C · 4. libyaml for variant configsMedian rerender wall time after each commit, in the order of the chart: main, then +1 … +4 for this PR's commits, +5 … +6 for rattler-build-conda-compat, +7 … +8 for conda-smithy (lazy imports, then the collapse). One run for the slow feedstocks, 3–5 for the fast ones; same pinning for all, against conda-build
mainas of 2026-11-17; the rendered feedstock is byte-identical to main's at every step. Steps that change a slow feedstock by a few seconds are within run-to-run noise.v1 recipes (petsc, h5netcdf) gain mostly from rattler-build-conda-compat 1; v0 recipes from this PR's first three commits and, for the large ones, the conda-smithy collapse.
One recipe this series does not fix: ctng-compilers (20 outputs, 51
pin_subpackage) took 8.4 minutes with an earlier version of this series; on main it was killed after 2 h 10 min at 164 GB of memory without finishing. Its time is in conda-build re-rendering outputs for eachpin_subpackage, not in the variant matrix.Measured on one Linux machine (24 cores, each rerender single-threaded), rattler-build 0.76.1, pinning of 2026-11-17. Feedstocks that fail to rerender on main itself (cupy, gdal, h5py, numpy, scipy:
variant_addon their in-flight python 3.14t/3.15 migrations) are left out.Checklist - did you ...
newsdirectory (using the template) for the next release's release notes?Add / update outdated documentation?(no documented behaviour changes)Resume this Claude session: