Repository navigation
docs: the v20 scale run — 500 and 1,000 projects deployed, the sizing table re-measured - #24
Merged
Merged
Conversation
… sharded path, the sizing table re-measured The release image as a separate Cloud Run service with the worker's settings (2 GiB, --concurrency=4, --timeout=3600), a Cloud Tasks queue at 25 concurrent dispatches, 150 ms simulated latency, 2% of projects denying every API: 500 projects in 11 min 11 s (26 tasks, two waves, merge 18 s) and 1,000 in 17 min 30 s (51 tasks, three waves, the scope shard 16.7 min, merge 35 s); every project covered, no shard error, memory under 17% of 2 GiB. One shard of the 1,000-project run was delivered twice by Cloud Tasks while its first attempt still ran; the second attempt cleaned up and rewrote it, and the report was identical row for row (57,451) to the in-process run. The sizing table's report sizes dated from the v16 check set and were about ten times too small; re-measured for v20 at 500, 1,000, 5,000 and 10,000 projects (12/11, 22/22, 54/110, 57/210 MB HTML/CSV). The scope shard — the one task that holds the organization-level checks — grows with the organization (8 min at 500 projects, 17 at 1,000) and is now described as the first thing to size past a couple of thousand projects; splitting it is on the roadmap. Operations: a "deployed, measured for v20" paragraph. Notes: "Verified at scale" in the v20 section.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The A6 scale run of the v20 plan, recorded where deployers read it.
Deployed (the
v20.0.0-rc.2image as a separate Cloud Run service with the worker's settings — 2 GiB,--concurrency=4,--timeout=3600— a Cloud Tasks queue at 25 concurrent dispatches, 150 ms simulated latency, 2% of projects denying every API):Cloud Run peaked at 13 instances; memory never passed 17% of 2 GiB, CPU 28%. Cloud Tasks delivered one shard of the 1,000-project run a second time while its first attempt still ran (at-least-once delivery); the second attempt cleaned up and rewrote the shard, and the report was identical table for table and row for row (57,451 rows) to the in-process run of the same organization.
In-process harness (
tools/synthetic_scan.py --latency-ms 0) at 500 / 1,000 / 5,000 / 10,000: 178 API calls per project, ≤ 385 in flight, no retries, merge of the 10,000-project run in a minute.Docs changed
docs/architecture.md› Sizing: the table's report sizes dated from the v16 check set and were ~10× too small — re-measured (12/11, 22/22, 54/110, 57/210 MB HTML/CSV; the 50,000 row marked not measured); durations as waves × 6 min with the two deployed rows marked measured; a new bullet on the scope shard, which grows with the organization (8 min at 500, 17 at 1,000) and is the first thing to size past a couple of thousand projects (SHARD_TIME_BUDGET_SECONDS, and what the report says when it is spent).docs/operations.md› Load Testing: a deployed, measured for v20 paragraph.docs/roadmap.md: the scope-shard split added at the head of Later.RELEASE_NOTES.md: Verified at scale in the v20 section.No code change. Suite 1,348 passed + 1 skipped.