Skip to content

docs: the v20 scale run — 500 and 1,000 projects deployed, the sizing table re-measured - #24

Merged
prateekgsingh merged 1 commit into
mainfrom
docs/a6-scale-run
Nov 21, 2026
Merged

prateekgsingh merged 1 commit into
mainfrom
docs/a6-scale-run

Conversation

@prateekgsingh

Copy link
Copy Markdown
Collaborator

What

The A6 scale run of the v20 plan, recorded where deployers read it.

Deployed (the v20.0.0-rc.2 image as a separate Cloud Run service with the worker's settings — 2 GiB, --concurrency=4, --timeout=3600 — a Cloud Tasks queue at 25 concurrent dispatches, 150 ms simulated latency, 2% of projects denying every API):

Projects Duration Tasks Shards Merge Coverage
500 11 min 11 s 26, two waves project shards 5.0–5.8 min, scope shard 8.2 min 18 s 500/500, 0 errors, 0 retries
1,000 17 min 30 s 51, three waves median 5.7 min, scope shard 16.7 min 35 s 1,000/1,000, 0 errors

Cloud Run peaked at 13 instances; memory never passed 17% of 2 GiB, CPU 28%. Cloud Tasks delivered one shard of the 1,000-project run a second time while its first attempt still ran (at-least-once delivery); the second attempt cleaned up and rewrote the shard, and the report was identical table for table and row for row (57,451 rows) to the in-process run of the same organization.

In-process harness (tools/synthetic_scan.py --latency-ms 0) at 500 / 1,000 / 5,000 / 10,000: 178 API calls per project, ≤ 385 in flight, no retries, merge of the 10,000-project run in a minute.

Docs changed

  • docs/architecture.md › Sizing: the table's report sizes dated from the v16 check set and were ~10× too small — re-measured (12/11, 22/22, 54/110, 57/210 MB HTML/CSV; the 50,000 row marked not measured); durations as waves × 6 min with the two deployed rows marked measured; a new bullet on the scope shard, which grows with the organization (8 min at 500, 17 at 1,000) and is the first thing to size past a couple of thousand projects (SHARD_TIME_BUDGET_SECONDS, and what the report says when it is spent).
  • docs/operations.md › Load Testing: a deployed, measured for v20 paragraph.
  • docs/roadmap.md: the scope-shard split added at the head of Later.
  • RELEASE_NOTES.md: Verified at scale in the v20 section.

No code change. Suite 1,348 passed + 1 skipped.

… sharded path, the sizing table re-measured

The release image as a separate Cloud Run service with the worker's settings
(2 GiB, --concurrency=4, --timeout=3600), a Cloud Tasks queue at 25 concurrent
dispatches, 150 ms simulated latency, 2% of projects denying every API:
500 projects in 11 min 11 s (26 tasks, two waves, merge 18 s) and 1,000 in
17 min 30 s (51 tasks, three waves, the scope shard 16.7 min, merge 35 s);
every project covered, no shard error, memory under 17% of 2 GiB. One shard
of the 1,000-project run was delivered twice by Cloud Tasks while its first
attempt still ran; the second attempt cleaned up and rewrote it, and the
report was identical row for row (57,451) to the in-process run.

The sizing table's report sizes dated from the v16 check set and were about
ten times too small; re-measured for v20 at 500, 1,000, 5,000 and 10,000
projects (12/11, 22/22, 54/110, 57/210 MB HTML/CSV). The scope shard — the one
task that holds the organization-level checks — grows with the organization
(8 min at 500 projects, 17 at 1,000) and is now described as the first thing
to size past a couple of thousand projects; splitting it is on the roadmap.
Operations: a "deployed, measured for v20" paragraph. Notes: "Verified at
scale" in the v20 section.
@prateekgsingh
prateekgsingh merged commit a958d69 into main Nov 21, 2026
14 checks passed
@prateekgsingh
prateekgsingh deleted the docs/a6-scale-run branch November 21, 2026 15:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant