Expand feed to 111 sources and stop the finder adding non-recipe repos

[dubadub]

Adds 50 GitHub recipe collections, retires 16 dead or near-empty ones, and fixes the finder script that had been adding repositories that contain no recipes.

The bug

scripts/find-cooklang-repos.py trusted GitHub code search directly: any repository containing a .cook file was added. That let in Cooklang tooling, parser test-fixture repos, and repos carrying a single sample file.

The worst case was not keyword noise. .cook is also the extension of Peter Miller's cook build tool, whose build scripts live in the etc/ directory of srecord, aegis, libexplain, the UCSD p-system tools and their many forks:

/*
 * srecord - manipulate eprom load files
 * Copyright (C) 1998-2000, 2003, 2004, 2006-2014 Peter Miller
 */
if [not [defined integration-build-targets]] then
    integration-build-targets = ;

Twelve such repositories passed a naive "does it contain @" check. This is the same family as mdhender/cook, which had already been disabled by hand.

The filter

A candidate must now be a fork-free, non-cooklang-org repository with no tooling keywords, at least --min-recipes recipes outside test/fixture directories, and sampled files carrying real Cooklang syntax — @flour{200%g}, #pan{}, ~{5%min} — rather than build-script constructs. Verified against nine known cases, accepting real cookbooks and rejecting every build-tool and tooling repo.

Of 180 candidates, 50 survived.

Other fixes in the script

• Code search returns only a stub repository object, so every entry the script ever wrote claimed ⭐ 0 and had no description. Repo metadata is now fetched per candidate.
• save_config() regenerated feeds.yaml, dropping comments and the disabled_at/disabled_by fields the config validator requires on every disabled feed — it would have broken startup. Entries are appended instead.
• The search ran one query, capped by the API at 1000 results out of ~23,000 .cook files on GitHub. It is now partitioned by file size.
• A failed download read as "not a recipe", rejecting good repos at random. Fetches retry, and a failure is reported as a failure.
• Titles came from GitHub descriptions, which are usually absent or carry typos and URLs. They are now built from the owner's display name; the description moves to notes.

Feed changes

Added 50 collections, taking the feed from 76 to 111 enabled sources — 7021 recipes. Largest: ianseddon/cookbook (816), johnhkchen/tabular-recipes (685), arnars/recipes (324).

Retired 16 feeds — disabled per config/README.md rather than deleted, so the entries stay as a record and the finder will not re-add them: 4 whose repositories now 404, and 12 holding only one or two recipes.

Fixed andreaskrath/recipes, which was enabled but pointed at a branch named main in a repository that only has master. Every crawl of it has failed since it was added in December. All 126 GitHub feeds were checked; it was the only live case.

Backfilled the metadata the old script could not collect. The fabricated ⭐ 0 was hiding some of the largest collections in the feed — nicholaswilde/recipes is 1230 recipes and 49 stars, Diegothx/CookBook is 953 recipes. 65 notes now carry counts measured on the branch each feed actually crawls.

Retitled every feed onto one convention, <Owner>'s <Collection>. The old titles were raw descriptions or invented strings — reipes in table form, javoulla, jackhumbert/recipes recipes, my website . Generated titles were corrected by hand where the generator does worse: one display name is written in Unicode homoglyphs, an org name carries a legal suffix, and two zarguell repos would have collided.

Languages were confirmed by reading the recipes rather than guessed from descriptions — which is how it2023115/javlakes turned out to be Greek cocktails rather than food. It is now tagged cocktails.

Tests

tests/feeds_config_test.rs is new: it loads the shipped config/feeds.yaml through the real FeedConfig::from_file validator, so a malformed entry fails CI instead of at startup. Confirmed it catches the failure mode — removing one disabled_at makes it fail with Feed #12 (algorist-ahmad/recipes): Disabled feed must have disabled_at field.

Full suite passes (97 lib tests + integration tests).

https://claude.ai/code/session_01XGqv3F4GVDc7kpqE2LVFmn

[claude]

Review

Solid piece of work — the root-cause analysis of the .cook extension collision with Peter Miller's cook build tool is excellent, and the fix (fork-free/org/keyword/test-dir/syntax/build-script checks, chained in is_recipe_collection) is well documented in both the code comments and scripts/README.md. A few notes below, nothing blocking.

Code quality / best practices

• FeedManager.save_config() (scripts/find-cooklang-repos.py:446) now appends before the # Validation configuration marker instead of regenerating the file — this correctly fixes the bug described in the PR (dropping comments and the disabled_at/disabled_by fields the validator requires). Nice touch validating the round-trip with yaml.safe_load + a duplicate-URL assertion right after writing.
• GitHubSearcher.enrich() (scripts/find-cooklang-repos.py:249-278) fetches and stores repo["archived"] from the GitHub API but nothing ever reads it — is_recipe_collection() doesn't filter on it. Either use it (an archived repo is a reasonable thing to flag/skip, or at least note in notes) or drop the fetch — right now it's dead data.
• Minor grammar nit: notes/messages always say "recipes" regardless of count — e.g. config/feeds.yaml ships several entries reading "Found via GitHub search (1 recipes, ⭐ 0)" (0xbenc, jlimcode, stusmall, shicks). Same pattern in the rejection message f"only {len(recipes)} recipe(s), need {min_recipes}" (line 325). Trivial, but easy to pluralize correctly (recipes.py:413).

Potential bugs / correctness

• Nothing that looks like a functional bug in the filtering/title logic I traced through (COLLECTION_NOUNS ordering, possessive apostrophe handling, sample-index selection for small n, the size-bucket split() recursion, dedup via the shared repos dict across both size ranges). The tests/feeds_config_test.rs addition is a good regression guard for exactly the failure mode called out in the PR description — confirmed it actually loads the shipped config/feeds.yaml through the real validator rather than a fixture.
• Worth double-checking whatjackhasmade/cookcli (config/feeds.yaml, added as "Jack Pritchard's Recipes") — the repo name shares "cookcli" with cooklang/cookcli, the org's own CLI tool. TOOLING_RE wouldn't false-positive here since there's no word boundary around "cli" inside "cookcli", but given the tooling-collision theme of this PR, it's worth a manual glance to confirm it's a genuine personal recipe repo and not an alias/mirror.

Performance

• enrich() makes three unthrottled GitHub API calls per candidate (repo metadata, owner metadata, recursive tree) with no time.sleep between them, unlike _count/_harvest which sleep 2s after each request. At the scale mentioned in the PR (180+ candidates), that's 500+ rapid-fire calls that could trip GitHub's secondary/abuse-detection limits even while staying under the 5000/hr core budget. The existing 403/429 backoff-and-retry in _make_request will recover, but a small delay in enrich() would avoid unnecessary backoff churn on large runs.

Security

• Reading the token from $GITHUB_TOKEN in addition to --token (main(), scripts/find-cooklang-repos.py) is a good improvement — keeps the token out of shell history/process listing.
• fetch_raw_file builds the raw.githubusercontent.com URL with quote() on both branch and path sourced from repo/tree metadata, which avoids URL-injection from untrusted repo content. Didn't spot other injection surfaces (no eval/shell=True/string-built SQL here).

Test coverage

• tests/feeds_config_test.rs is a good, minimal addition — validates the shipped config loads and has enabled feeds, which is exactly the class of bug this PR fixes (missing disabled_at/disabled_by). Confirmed in the PR description that it actually catches a removed disabled_at field.
• The new filtering/title-generation logic in find-cooklang-repos.py (looks_like_recipe, is_recipe_collection, _title_for, the TOOLING_RE/BUILDFILE_RE/COOKLANG_RE regexes) has no automated tests — understandable since it's an operational script rather than shipped code, and --dry-run gives a manual verification path, but this is exactly the kind of regex-heavy logic that regressed silently before (per the PR's own description of the previous script's bugs). A handful of pytest cases feeding known-good/known-bad .cook snippets through looks_like_recipe/is_recipe_collection would make future edits safer, if it's worth the investment for a script that's run manually.

Overall: well-scoped, well-explained, and the config/test changes look correct. The two performance/dead-code notes above are minor and not blocking.

[claude]

Review

Solid piece of work — the root-cause analysis of the .cook/cook-build-tool extension collision is well documented, the save_config() fix (append instead of regenerate) prevents a real startup-breaking class of bug, and tests/feeds_config_test.rs closes a legitimate gap (validated it actually catches a missing disabled_at, per the PR description).

A few things worth a look:

Correctness

code_files is computed but never used (scripts/find-cooklang-repos.py:285-298). enrich() walks the whole tree counting non-.cook source files into repo["code_files"], and the comment right above CODE_EXTENSIONS (line 66) says it's "used to tell a cookbook from a codebase that ships sample recipes" — but is_recipe_collection() never reads repo["code_files"]. So a repo that's mostly a Rust/JS/etc. codebase with a handful of legitimate-looking (non-fixture-directory) sample recipes can still pass, since nothing actually checks the code-vs-recipe ratio. Either wire it into the filter (e.g. reject when code_files heavily outnumbers recipes) or drop the computation — right now it's dead weight that also costs an extra pass over every tree entry for every candidate.

Similarly, archived is fetched in enrich() (line 272) but never consulted anywhere.

Minor / nits

• search_repos_with_cook_files's split() only recurses on the size:lo..hi bucket; the open-ended size:>10000 bucket (line 220) is harvested directly regardless of its total, so if it has more than 1000 hits, results past the API's per-query cap are silently dropped for that bucket — a small gap in the "partitioned so we see past the 1000 cap" claim in the docstring/README. Probably rare in practice since recipe files are small, but worth a comment if intentional.
• In main() (lines 562-567), the outer if len(accepted) >= args.limit and not args.randomize: is redundant — the inner len(accepted) >= args.limit * 3 already implies it. Reads a bit confusingly as-is; could collapse to one if not args.randomize and len(accepted) >= args.limit * 3: break.
• Style-only: new disabled_by entries in config/feeds.yaml use "dubadub" without the @ prefix that config/README.md's own documented example uses (disabled_by: "@admin"). Not a new inconsistency introduced by this PR (plenty of pre-existing added_by: "dubadub" entries already drop the @), so just flagging in case it's worth reconciling docs vs. convention at some point.

Test coverage

The Rust-side config validation test is a good addition. The actual core logic of this PR, though, is the Python heuristic filter (looks_like_recipe, TOOLING_RE, BUILDFILE_RE, is_recipe_collection, title generation) — none of it has automated tests. The PR description says it was "verified against nine known cases" by hand, which is good, but that verification isn't captured anywhere for future changes to regress against (e.g. if someone tweaks BUILDFILE_RE later and reintroduces the srecord/aegis false positives this PR specifically set out to fix). There's no existing pytest scaffolding in the repo, so this may be a deliberate scope call, but a small scripts/test_find_cooklang_repos.py with a few of the documented positive/negative fixtures (the cook-build-tool snippet from the PR description, a real Cooklang sample) would be cheap insurance for the trickiest part of this change.

Nice touches

• Retry-with-flag-not-boolean design in fetch_raw_file (returning (downloaded, text) rather than treating a failed download as "not a recipe") is a good catch — that's exactly the kind of bug that silently biases results.
• Backing off on 403/429 rather than dying, and appending to feeds.yaml instead of rewriting it, both directly fix real failure modes described in the PR body.