Skip to content

Note that rerunning failed tests is web UI only - #1765

Closed
mattmenefee wants to merge 1 commit into
CircleCI-Public:mainfrom
mattmenefee:note-rerun-failed-tests-is-web-ui-only
Closed

mattmenefee wants to merge 1 commit into
CircleCI-Public:mainfrom
mattmenefee:note-rerun-failed-tests-is-web-ui-only

Conversation

@mattmenefee

Copy link
Copy Markdown

Summary

  • circleci workflow rerun --help now states that rerunning only the failed tests — the web UI's "Rerun failed tests" — has no CLI or public API equivalent, and links to the docs.
  • Compresses the opening paragraph of Long so the rendered help stays within the 40-line budget TestHelp enforces (now exactly 40/40, no overBudget entry added).
  • Documentation only: no behavior, flag, or output changes.

Refs #1659. This closes the documentation half of that issue — "There's nothing in the help output or the command tree that distinguishes 'not built yet' from 'not possible here'." The feature itself is left to the CircleCI team, for the reason below.

Why the flag isn't implemented here

The intended change was circleci workflow rerun <workflow-id> --failed-tests. The action is served by POST /private/tests-plugin-manager/rerun-failed-tests/{workflowId} on app.circleci.com, with eligibility at GET .../{workflowId}/job/{jobId} returning {"can_rerun_failed_tests": true}.

That endpoint is scoped to the web app's own session. Probing the same job-scoped URL the browser itself calls, on a workflow the UI reports as eligible:

Credential Response
None / bogus / empty Bearer 401 + www-authenticate
CLI OAuth token (valid against /api/v3) 401 Invalid token provided.
Personal API token 404 Not found — authenticates, but reads no data
Browser session cookie 200 {"can_rerun_failed_tests":true}

The 401 is an issuer mismatch. Per RFC 9728 metadata the resource advertises app.circleci.com as its authorization server, while the CLI runs its OAuth flow against cfg.EffectiveHost() — circleci.com (internal/cmd/cmdauth/login.go:110, internal/oauth/login.go:149-151). The web app also sets includeApiKeyWhenAvailable: false on this call, so it depends on the cookie by design.

Any one of these would unblock a CLI implementation:

  1. Authorize personal API tokens for /private/tests-plugin-manager/rerun-failed-tests/* — they already clear the gateway (404, not 401).
  2. Expose the capability under /api/v3 on circleci.com, where the CLI already authenticates. Preferred: needs no auth change and no app-host client.
  3. Decide whether the CLI should additionally hold an app.circleci.com-issued token. That AS advertises authorization_code + PKCE (S256) + PAR, which internal/oauth already implements — but a second token for a second issuer is an auth and security decision.

Test plan

  • go test ./... -count=1 — 32 packages pass, including acceptance
  • go test github.com/CircleCI-Public/circleci-cli/clikit/... -count=1 — 8 packages pass
  • golangci-lint run ./internal/cmd/workflow/... ./internal/cmd/root/... — 0 issues
  • Help goldens regenerated via go test ./internal/cmd/root/... -update, not hand-written
  • circleci workflow rerun --help renders at exactly 40 lines, within helpLineBudget
  • -race — pre-existing failure in TestDeprecationWarning_SunsetInOutput (data race at acceptance/deprecation_test.go:48). Reproduced on main at 2407ffd with this change absent, so it is unrelated and not addressed here.

The job page in the web UI offers four rerun actions; only two have CLI
equivalents. Neither the help text nor the command tree distinguished
"not built yet" from "not possible here", so anyone looking for the
narrower rerun had to go searching to learn it was not there.

That action is served by an endpoint scoped to the web app's own
session. A personal API token authenticates against it but reads no
data, and a token from circleci auth login is rejected outright: the
resource advertises app.circleci.com as its issuer, while the CLI
authenticates against circleci.com. No credential the CLI holds today
reaches it, so the help now says so and points at the docs rather than
leaving the reader to find out the hard way.

The opening paragraph is compressed to hold the rendered help inside
the 40-line budget TestHelp enforces.
@liamclarkedev

Copy link
Copy Markdown
Contributor

Hi @mattmenefee,

Thanks for opening this. This is now supported via the testsuite command, this PR can be closed.

I'll keep the issue open in case you run into any issues onboarding with the new testing tool.

#1659 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants