Skip to content

test(cli): cover migration and seed ordering (CLI-2581) - #7521

Merged
7ttp merged 2 commits into
developfrom
7ttp/cli-2581-stack-audit-migration-and-seed-ordering-against-legacy
Nov 6, 2026
Merged

7ttp merged 2 commits into
developfrom
7ttp/cli-2581-stack-audit-migration-and-seed-ordering-against-legacy

Conversation

@7ttp

@7ttp 7ttp commented Nov 5, 2026

Copy link
Copy Markdown
Member

TL;DR

adds start and db reset ordering cases to the contract suite for migrations, seeds, config buckets and service schemas

the audit found no ordering difference between backends, only the already documented stack db reset warning on a rejected bucket upload

ref:

@7ttp 7ttp self-assigned this Nov 5, 2026
@7ttp
7ttp requested a review from a team as a code owner November 5, 2026 17:27
@7ttp
7ttp added this pull request to stack #7522 November 5, 2026 17:27
@7ttp
7ttp requested review from mdonnalley and removed request for a team November 5, 2026 17:27
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

CLI benchmark (Lightweight Commands): no changes found

Commit 9eebc9b against 7ttp/cli-2574-stack-add-a-consumer-contract-suite-across-legacy at 56468b7 · 10 runs each · report only, never blocks merging

Command Workload Before After Change
supabase --version version 235 ms 234 ms no real change
supabase --help help 266 ms 260 ms no real change
supabase init init 302 ms 275 ms no real change (-9%, within normal variation)
supabase migration new bench migration-new 291 ms 306 ms no real change
supabase gen types --lang typescript --project-id replayfixtureprojref gen-types 430 ms 416 ms no real change
supabase projects list projects-list 268 ms 250 ms no real change (-7%, within normal variation)
supabase functions list --project-ref replayfixtureprojref functions-list 267 ms 279 ms no real change

API calls: unchanged for all 7 commands.

Binary size: 134.7 MB, unchanged.

Details: timing ranges, API calls per command, slowest phases

Timing ranges

Command Workload Before range After range Change Verdict
supabase --version version 210-319 ms 199-347 ms -1 ms (-1%) within noise
supabase --help help 211-298 ms 214-276 ms -6 ms (-2%) within noise
supabase init init 251-403 ms 236-422 ms -27 ms (-9%) within noise
supabase migration new bench migration-new 242-487 ms 242-482 ms +15 ms (+5%) within noise
supabase gen types --lang typescript --project-id replayfixtureprojref gen-types 249-552 ms 331-505 ms -14 ms (-3%) within noise
supabase projects list projects-list 224-342 ms 228-273 ms -18 ms (-7%) within noise
supabase functions list --project-ref replayfixtureprojref functions-list 218-301 ms 245-322 ms +12 ms (+5%) within noise

API calls per command

Command Workload Before After Endpoint changes
supabase --version version 0 0
supabase --help help 0 0
supabase init init 0 0
supabase migration new bench migration-new 0 0
supabase gen types --lang typescript --project-id replayfixtureprojref gen-types 2 2
supabase projects list projects-list 1 1
supabase functions list --project-ref replayfixtureprojref functions-list 2 2

Slowest phases

No command moved beyond noise, so there are no phase deltas.

Workflow run

Comment /bench to run the lightweight benchmark (this comment) again on the latest commit, or /bench full to also run the integration-heavy benchmark, which posts in its own comment.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI Review

Confirmed Claude’s minor legacy-reset test readiness race and corrected its stale line reference. Codex reported no findings. Verification was static; runtime tests were not run.

Findings

Severity Location Category Sources Claim
🟡 MINOR apps/cli/tests/contract/cases.ts:367 test-flakiness claude The legacy reset case queries PostgREST immediately after reset without waiting for its reconnection and schema readiness. A transient PostgREST error can therefore fail the contract test even though the database reset succeeded.

Stats

Claude findings: 1 · Codex findings: 0 · Confirmed: 1 · Refuted: 0 · Uncertain: 0


Models: claude-opus-5-5 + gpt-6.1-sol · Trigger: auto · Workflow run

This review runs once per PR. A maintainer can request another with a /ai-review comment.

Comment thread apps/cli/tests/contract/cases.ts Resolved
Base automatically changed from 7ttp/cli-2574-stack-add-a-consumer-contract-suite-across-legacy to develop November 5, 2026 20:25
@7ttp
7ttp force-pushed the 7ttp/cli-2581-stack-audit-migration-and-seed-ordering-against-legacy branch from 9eebc9b to 3a21175 Compare November 5, 2026 20:25
@7ttp
7ttp requested a review from jgoux November 5, 2026 20:28
Comment thread apps/cli/tests/contract/target.ts Resolved
@7ttp
7ttp added this pull request to the merge queue Nov 6, 2026
Merged via the queue into develop with commit 7f8dae8 Nov 6, 2026
49 checks passed
@7ttp
7ttp deleted the 7ttp/cli-2581-stack-audit-migration-and-seed-ordering-against-legacy branch November 6, 2026 09:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants