Skip to content

Speed up variant handling on large variant matrices - #6267

Open
hmaarrfk wants to merge 4 commits into
conda:mainfrom
hmaarrfk:rerender-speedups
Open

hmaarrfk wants to merge 4 commits into
conda:mainfrom
hmaarrfk:rerender-speedups

Conversation

@hmaarrfk

@hmaarrfk hmaarrfk commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

This should really help rerendering things at conda forge.

Broken out into separate ideas in distinct commits. The biggest are the first two: list_of_dicts_to_dict_of_lists copied every zipped key's growing list for every variant, which was quadratic in the number of variants; commit 1 accumulates in place, and commit 2 squishes a column at a time for a further constant-factor gain.

Along with other speedups this should really help conda-forge with the growing number of global pinnings.

xref:

Claude

Description

Four commits, ordered from the smallest and most obviously safe to the larger ones. None changes a result. (An ASCII fast path for recipe file decoding, which does change how some unusual files are read, was split out into #6270.)

1. Accumulate in place. For every variant, each key's collected values were rebuilt from scratch (list(old) + [v], then list(set(...)); set(old) | set(v)). Zipped keys keep one value per variant, duplicates included, because they are paired up again at the end, so their list grows to the number of variants and was copied in full for every variant, which is quadratic in the number of variants; the other keys only hold their few distinct values but were still rebuilt every time. Zipped keys now extend their list and every other key collects into a set, turned into a list at the end where its last value was a scalar, as before. opencv-feedstock: 204 s → 99 s.

2. Squish a column at a time. The same O(variants × keys) work with a much smaller constant: when every variant holds the same keys, a column is list(map(itemgetter(k), variants)); a column of hashable scalars is one set() (or stays a list for zipped keys); any other column replays the original per-value rules in order, including the mixed scalar/list/dict cases and the AttributeError a dict raises after a scalar. Scalars are recognised with isinstance, so subclasses of str such as ruamel.yaml's ScalarString take the one-step path too. opencv-feedstock: 99 s → 76 s; on a synthetic list of 16,000 variants of 520 keys the two are close (1.3 s against 1.4 s). This is the larger change; commit 1 alone removes the quadratic part. New tests pin the behaviour and pass on main too; a 20,000-case random comparison against the old implementation matched for both commits, except for the order of values both collect through a set.

3. get_vars and filter_by_key_value comparisons in C. get_vars compared each variable across every variant in a generator; any(map(ne, repeat(first[var]), map(itemgetter(var), others))) makes the same comparisons, same short circuit, same KeyError (and operator.ne still reflects a NotImplemented from mixed types; tested). filter_by_key_value called get_logger() and formatted a debug f-string with the whole list of target values for every variant it dropped, although get_logger sets INFO; the messages are now built only when debug is enabled. opencv-feedstock: 76 s → 67 s.

4. libyaml for variant config files. parse_config_file uses CBaseLoader when PyYAML has libyaml: the same constructor and resolver, 3 ms instead of 20 ms for the global pinning, identical data for the pinning and every config and migration file of 23 feedstocks.

Locally with conda 26.10, tests/test_variants.py, test_utils.py, test_render.py, test_metadata.py: the same three tests fail here as on main in this environment (test_build_bootstrap_env_by_name, test_build_run_exports_act_on_host, test_inner_python_loop_with_output: unsatisfiable win-64 test packages); everything else passes.

Why rerenders are slow, and the series this PR is part of

conda-build explodes every multi-valued key of the combined variant spec into the variant matrix before it learns which keys a recipe uses. conda-smithy renders against the whole global pinning, which holds the keys of every feedstock, so keys like nodejs, r_base, hip_compiler_version, blas_impl and the root_base, libarrow and geant4/clhep zips multiply the matrix of recipes that never mention them. That matrix is then walked by code that is quadratic, or Python-level per entry. For opencv-feedstock, 20 builds become 28,800 variants of ~520 keys, and a rerender takes 3.5 minutes and 3.2 GB.

Where opencv's 28,800 variants come from

The fix is spread over three projects, one draft PR each. Inside each PR the commits are ordered from the most conservative (equivalent code, same result) to the least, and each commit stands alone:

PR commits, in order
#6267 1. values accumulated in place in list_of_dicts_to_dict_of_lists · 2. the same function a column at a time · 3. get_vars / filter_by_key_value comparisons in C · 4. libyaml for variant configs
conda-forge/rattler-build-conda-compat#146 1. an output that uses no variables no longer explodes the whole pinning · 2. variants passed to rattler-build as JSON
conda-forge/conda-smithy#2751 1. lazy imports of the GitHub and lint modules · 2. collapse variant dimensions the recipe cannot use (the one change that rests on a rule rather than on equivalent code)

What each commit removes from a rerender

Median rerender wall time after each commit, in the order of the chart: main, then +1 … +4 for this PR's commits, +5 … +6 for rattler-build-conda-compat, +7 … +8 for conda-smithy (lazy imports, then the collapse). One run for the slow feedstocks, 3–5 for the fast ones; same pinning for all, against conda-build main as of 2026-11-17; the rendered feedstock is byte-identical to main's at every step. Steps that change a slow feedstock by a few seconds are within run-to-run noise.

feedstock main +1 +2 +3 +4 +5 +6 +7 +8
ffmpeg 461 s 161 s 122 s 106 s 108 s 106 s 106 s 105 s 5.3 s
opencv 204 s 99 s 76 s 67 s 61 s 59 s 59 s 60 s 12 s
boost 106 s 86 s 78 s 76 s 78 s 81 s 75 s 75 s 16 s
python 59 s 48 s 42 s 40 s 39 s 40 s 39 s 38 s 18 s
pytorch-cpu 56 s 27 s 19 s 20 s 19 s 18 s 19 s 19 s 4.5 s
petsc 22 s 22 s 22 s 20 s 21 s 1.6 s 1.5 s 1.4 s 1.4 s
r-ggplot2 7.0 s 5.8 s 4.7 s 4.7 s 4.6 s 4.5 s 4.5 s 4.5 s 1.5 s
h5netcdf 1.1 s 1.0 s 1.1 s 1.1 s 1.0 s 1.0 s 1.0 s 0.9 s 0.8 s

v1 recipes (petsc, h5netcdf) gain mostly from rattler-build-conda-compat 1; v0 recipes from this PR's first three commits and, for the large ones, the conda-smithy collapse.

One recipe this series does not fix: ctng-compilers (20 outputs, 51 pin_subpackage) took 8.4 minutes with an earlier version of this series; on main it was killed after 2 h 10 min at 164 GB of memory without finishing. Its time is in conda-build re-rendering outputs for each pin_subpackage, not in the variant matrix.

Measured on one Linux machine (24 cores, each rerender single-threaded), rattler-build 0.76.1, pinning of 2026-11-17. Feedstocks that fail to rerender on main itself (cupy, gdal, h5py, numpy, scipy: variant_add on their in-flight python 3.14t/3.15 migrations) are left out.

Checklist - did you ...

  • Add a file to the news directory (using the template) for the next release's release notes?
  • Add / update necessary tests?
  • Add / update outdated documentation? (no documented behaviour changes)

Resume this Claude session:

cd /home/mark/git/ramona/python-owl/.claude/worktrees/bridge-cse_01BiGuuSU1wyGE5gzJ8oLsuD
claude --resume 56499d72-0fb0-5cfd-a53c-c7dc17432c2f

@codspeed

codspeed Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by ×2.3

2 warnings

⚠️ 2 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ 1 benchmark with a significant change ran on different hardware

Its base and head results were measured on machines with a different CPU, core count or memory, so the change might not come from the code. Run base and head on the same runner type.

Learn more about Macro Runners

See affected benchmarks

⚡ 1 improved benchmark
✅ 4 untouched benchmarks

Performance Changes

Benchmark BASE HEAD Efficiency
⚡ test_render_recipe 62.5 s 27.1 s ×2.3

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing hmaarrfk:rerender-speedups (615cac5) with main (9edede6)

Open in CodSpeed

@hmaarrfk
hmaarrfk force-pushed the rerender-speedups branch 2 times, most recently from 798b112 to 3ed0d39 Compare November 18, 2026 03:57
@hmaarrfk
hmaarrfk marked this pull request as ready for review November 18, 2026 12:43
@hmaarrfk
hmaarrfk requested a review from a team as a code owner November 18, 2026 12:43
@hmaarrfk

Copy link
Copy Markdown
Contributor Author

cc: @h-vetinari

Comment thread news/squish-variants-linear.md Resolved
Comment thread news/get-vars-in-c.md Resolved
@hmaarrfk

Copy link
Copy Markdown
Contributor Author

@travishathaway thank you for the quick review.

I'm sorry i wans't able to "apply" your suggestion, i think Github changed their interface and I wasn't able to click "apply" to both.

let me know if you prefer i squash these into 3 logical commits.

Comment thread news/squish-variants-linear.md Outdated
@@ -0,0 +1,19 @@
### Enhancements

* Make `conda_build.variants.list_of_dicts_to_dict_of_lists` linear in the number of variants. Rendering a recipe against a large variant matrix, such as conda-forge's global pinning, spent most of its time rebuilding these lists. (#6267)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is still quadratic. Not sure why you say it's linear.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically O(m*n) where m is the number of variants and n the number of keys. Same as before

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess the difference is that lists and sets are not created and recreated over and over again.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to rephrase along this line

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've rephrased this to remove the claim about linear vs quadratic. Hopefully that helps clarify the improvement.

@beckermr beckermr left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we should keep only the top optimizations with allowing for simple ones too:

  • conda-build 1
  • conda-build 2
  • conda-build 3
  • rattler-build-conda-compat 1
  • rattler-build-conda-compat 2
  • conda-smithy 1
  • conda-smithy 3

We do pay costs in complexity in some cases, like the async azure build ID fetching, for marginal gains.

@hmaarrfk

Copy link
Copy Markdown
Contributor Author

I wonder if we should keep only the top optimizations with allowing for simple ones too:

  • conda-build 1
  • conda-build 2
  • conda-build 3
  • rattler-build-conda-compat 1
  • rattler-build-conda-compat 2
  • conda-smithy 1
  • conda-smithy 3

We do pay costs in complexity in some cases, like the async azure build ID fetching, for marginal gains.

This PR does the first 3 right? So that is ok right?

@hmaarrfk
hmaarrfk force-pushed the rerender-speedups branch 2 times, most recently from 810b880 to 4340c40 Compare November 18, 2026 23:54
@hmaarrfk
hmaarrfk marked this pull request as draft November 18, 2026 23:55
@hmaarrfk

hmaarrfk commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

i marked this as draft since i am breaking up the 'big change' into 2 commits:

  1. one smaller change that shows a simpler line edit.
  2. One larger change
  3. get_vars
  4. libyaml

so 3 things instead of 3 and i'm regenerating the color plots that show the improvement breakdown.

Edit: Plots updated, now it looks like

image

@hmaarrfk
hmaarrfk marked this pull request as ready for review November 19, 2026 02:32
Comment thread conda_build/variants.py Resolved
@github-project-automation github-project-automation Bot moved this from 🆕 New to 🏗️ In Progress in 🔎 Review Nov 19, 2026
@hmaarrfk
hmaarrfk marked this pull request as draft November 19, 2026 12:44
<details><summary>Claude's draft</summary>

For every variant, each key's collected values were rebuilt from scratch
(`list(old) + [v]`, then `list(set(...))`; `set(old) | set(v)`). Zipped keys
keep one value per variant, duplicates included, because they are paired up
again at the end, so their list grows to the number of variants and was
copied in full for every variant: quadratic in the number of variants. The
other keys only hold their few distinct values, but were still rebuilt as a
new list and set every time.

Values are now accumulated in place: zipped keys extend their list, and
every other key collects into a set, turned into a list at the end for keys
whose last value was a scalar, as before. Lists and dicts keep their
previous rules, including the AttributeError a dict raises after a scalar.

On conda-forge's opencv-feedstock (variant lists of up to 28,800 entries of
~520 keys) a rerender goes from 208 s to 96 s, byte-identical output. A
20,000-case random comparison with the previous implementation found no
difference except for the order of values both collect through a set; the
new tests pass before and after.

Resume this Claude session:
```
cd /home/mark/git/ramona/python-owl/.claude/worktrees/bridge-cse_01BiGuuSU1wyGE5gzJ8oLsuD
claude --resume 56499d72-0fb0-5cfd-a53c-c7dc17432c2f
```
</details>
<details><summary>Claude's draft</summary>

With values accumulated in place, the function still does a Python-level
step, with its type checks, for every key of every variant. Each key is now squished on its own, a
column at a time: when every variant holds the same keys (always, for an
exploded matrix) the column is `list(map(itemgetter(k), variants))`; a column
of hashable scalars reduces to one set() (or stays a list for zipped keys);
any other column replays the original per-value rules in order, including
the mixed scalar/list/dict cases and the AttributeError a dict raises after
a scalar.

The work is the same O(variants x keys); the gain is the constant factor.
opencv-feedstock rerenders in 75 s instead of 96 s after the previous
commit (208 s on main), byte-identical output. On a synthetic list of
16,000 variants of 520 keys the two are close (1.3 s against 1.4 s; 18.3 s
on main), so most of the gain on opencv comes from the shape of its real
variant lists. The same 20,000-case random comparison with the original
implementation matched, except for the order of values both collect
through a set.

Scalars are recognised with isinstance, so subclasses of str, such as the
ScalarString values ruamel.yaml returns, take the one-step path as well.

This is the larger change of the two; the previous commit alone removes the
quadratic part and rebuilds nothing.

Resume this Claude session:
```
cd /home/mark/git/ramona/python-owl/.claude/worktrees/bridge-cse_01BiGuuSU1wyGE5gzJ8oLsuD
claude --resume 56499d72-0fb0-5cfd-a53c-c7dc17432c2f
```
</details>
<details><summary>Claude's draft</summary>

get_vars compared each variable across every variant in a Python generator;
map(ne, repeat(first), map(itemgetter(var), others)) does the same
comparisons, with the same short circuit and KeyError, in C.

filter_by_key_value formatted a debug f-string (including the whole list of
target values) for every variant it dropped, and called get_logger for each,
although get_logger sets the level to INFO so the message is never shown
unless a log config file asks for debug. The kept variants are now selected in
one comprehension, and the messages are only built when debug is enabled.

On conda-forge's opencv-feedstock (variant matrices of up to 28,800 entries)
these two took a rerender from 96 s to 64 s, with byte-identical output.

Resume this Claude session:
```
cd /home/mark/git/ramona/python-owl/.claude/worktrees/bridge-cse_01BiGuuSU1wyGE5gzJ8oLsuD
claude --resume 56499d72-0fb0-5cfd-a53c-c7dc17432c2f
```
</details>
<details><summary>Claude's draft</summary>

parse_config_file() used PyYAML's pure-Python BaseLoader. CBaseLoader is the
same loader (BaseConstructor, BaseResolver) on libyaml's parser, used when
PyYAML was built with libyaml. It parses conda-forge's global pinning about
7x faster (20 ms -> 3 ms), and returned identical data for the pinning and for
every recipe conda_build_config.yaml and migration file of 23 feedstocks.

Resume this Claude session:
```
cd /home/mark/git/ramona/python-owl/.claude/worktrees/bridge-cse_01BiGuuSU1wyGE5gzJ8oLsuD
claude --resume 56499d72-0fb0-5cfd-a53c-c7dc17432c2f
```
</details>
@hmaarrfk
hmaarrfk marked this pull request as ready for review November 21, 2026 03:00
@github-project-automation github-project-automation Bot moved this from 🏗️ In Progress to ✅ Approved in 🔎 Review Nov 21, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-signed [bot] added once the contributor has signed the CLA

Projects

Status: ✅ Approved

Development

Successfully merging this pull request may close these issues.

5 participants