Skip to content

Resolve AWS signing name and endpoint host independently - #1596

Merged
kubukoz merged 1 commit into
series/0.19from
aws-signing-issues-wip
Sep 28, 2026
Merged

kubukoz merged 1 commit into
series/0.19from
aws-signing-issues-wip

Conversation

@kubukoz

@kubukoz kubukoz commented Sep 27, 2024 •

Copy link
Copy Markdown
Member

Closes #1568, closes #1532.

The SigV4 signing name and the endpoint host were both derived from endpointPrefix. That is only correct for services where the two happen to coincide, and it produced two distinct failures for services where they don't.

The two are independent

resolution order
host endpointPrefix → arnNamespace → service id
signing name sigv4.name → arnNamespace → endpointPrefix → service id

Neither falls back to the operation name any more, which is what produced ListRegions.us-east-1.amazonaws.com.

Rationale, taken from the trait definitions themselves:

  • aws.auth#sigv4's name is "the signature version 4 service signing name to use in the credential scope when signing requests" — authoritative for signing, so it wins.
  • endpointPrefix "identifies which endpoint in a given region should be used", resolving as {endpointPrefix}.{region}.{dnsSuffix} — authoritative for the host. It is also documented as non-unique and subject to change, and MUST NOT be used for anything else, which is why signing prefers arnNamespace over it.
  • arnNamespace is constrained to ^[a-z0-9.\-]{1,63}$ and defaults to the lowercased service shape name, making it a DNS-safe stand-in for the host when endpointPrefix is absent.

#1532 needs no endpoint-rules evaluation. The Account model's own smithy.rules#endpointTests assert that us-east-1 + no FIPS + no dual-stack → https://account.us-east-1.amazonaws.com, i.e. the plain pattern we already produce once the operation-name fallback is gone.

Also: skip aws.endpoints traits in codegen

While verifying #1532 end-to-end, the Account API turned out not to be usable at all, independently of the host bug: it carries @aws.endpoints#standardPartitionalEndpoints, and codegen rendered that as a hint referencing aws.endpoints.StandardPartitionalEndpoints, for which no artifact publishes Scala bindings. The generated service simply did not compile.

aws.endpoints is now skipped when generating hints, mirroring the existing treatment of smithy.rules (added in #1920). Reproduced on released 0.19.12, so this is not a regression — that codegen path was just never exercised in-tree, because the AWS specs we bootstrap don't carry the trait.

Tests

New fixture services in sampleSpecs/aws_example.smithy, one per combination seen in the wild, asserting both the host and the credential scope:

fixture modelled on host signs as
PrefixDiffersFromSigningName SES v2 email.… ses
DottedPrefix SageMaker (#1568) api.sagemaker.… sagemaker
NoEndpointPrefix Account (#1532) account.… account
NoSigv4 — endpointprefix.… arnnamespace
DynamoDB control — all three agree dynamodb.… dynamodb

4 of the 5 fail without this change. DynamoDB passes either way, which is the point of including it. The failures were: SES signed email, SageMaker signed api.sagemaker, Account got host DoThing.us-east-1.amazonaws.com and signed noendpointprefix.

aws-http4s/test is green (696 tests), including AwsSignatureTest, which property-checks our signatures against the real AWS SDK Aws4Signer. codegen/test and bootstrapped/compile also pass.

Verified against real AWS

Read-only calls, generated from the published models with no hand-editing:

call host credential scope
Account.ListRegions account.us-east-1.amazonaws.com …/us-east-1/account/aws4_request
Account.GetAccountInformation account.us-east-1.amazonaws.com …/us-east-1/account/aws4_request
SageMaker.ListEndpoints api.sagemaker.us-east-1.amazonaws.com …/us-east-1/sagemaker/aws4_request

All three returned real data. Separately, a downstream SES v2 test asserting /ses/aws4_request goes red → green against a local snapshot, with the host staying email.<region>.amazonaws.com.

Scope

Only the {prefix}.{region}.amazonaws.com pattern is handled. FIPS, dual-stack, non-standard partitions (amazonaws.com.cn, GovCloud) and operation- or tenant-level endpoints all require evaluating smithy.rules#endpointRuleSet, which smithy4s does not do. #2006 proposes measuring that gap against the models' own endpointTests.

X-Amz-Target is still set for every protocol, though it belongs to awsJson1_0/1_1 only. Deliberately left alone: the header is part of the canonical request, so changing it alters every signature (and the two compensating force-appends in AwsSignatureTest). That blast radius does not belong in a cherry-pickable correctness fix.

Also filed while working on this: #2007 (AwsCredentialsFile misreports an unparseable credentials file as a missing profile).

PR Checklist (not all items are relevant to all PRs)

  • Added unit-tests (for runtime code)
  • Added bootstrapped code + smoke tests (when the rendering logic is modified)
  • Added build-plugins integration tests (when reflection loading is required at codegen-time)
  • Added alloy compliance tests (when simpleRestJson protocol behaviour is expanded/updated)
  • Updated dynamic module to match generated-code behaviour
  • Added documentation
  • Updated changelog

@kubukoz kubukoz changed the title Quick attempt at resolving AWS signing problems (#1568, #1532) Quick attempt at resolving AWS signing problems Sep 27, 2024
Comment thread modules/aws-http4s/src/smithy4s/aws/AwsClient.scala Outdated
@kubukoz
kubukoz changed the base branch from series/0.18 to series/0.19 September 23, 2026 23:59
@kubukoz
kubukoz force-pushed the aws-signing-issues-wip branch from cac1af5 to 0e545f1 Compare September 24, 2026 00:06
@kubukoz kubukoz changed the title Quick attempt at resolving AWS signing problems Resolve AWS signing name and endpoint host independently Sep 24, 2026
The SigV4 signing name and the endpoint host were both derived from
`endpointPrefix`, which is only correct for services where the two happen
to coincide.

They are separate things. `aws.auth#sigv4`'s `name` is defined as "the
signature version 4 service signing name to use in the credential scope
when signing requests", so the signing name now comes from there, falling
back to `arnNamespace` and then `endpointPrefix`. The host keeps using
`endpointPrefix` -- the field documented for endpoint resolution -- and
falls back to `arnNamespace`, which is DNS-safe by construction, for the
services that omit it.

This fixes two failure modes:

- Services whose prefix and signing name differ were signed with the
  wrong credential scope and rejected by AWS with "Credential should be
  scoped to correct service". SES v2 is hosted at
  `email.<region>.amazonaws.com` but must be signed as `ses`; SageMaker
  is hosted at `api.sagemaker.<region>.amazonaws.com` but signs as
  `sagemaker` (#1568).

- Services with no `endpointPrefix` fell back to the *operation* name for
  the host, producing nonsense like `ListRegions.us-east-1.amazonaws.com`
  (#1532). The Account API's own endpoint tests confirm the expected host
  is `account.us-east-1.amazonaws.com`, so this needs no rule-set
  evaluation.

Adds fixture services to `sampleSpecs/aws_example.smithy` covering each
combination of `endpointPrefix`, `arnNamespace` and sigv4 name seen in
the wild, plus DynamoDB (where all three agree) as a no-regression case.
Four of the five new assertions fail without this change.

Verified against real AWS with read-only calls: Account.ListRegions and
Account.GetAccountInformation reach `account.us-east-1.amazonaws.com`
signed as `account`, and SageMaker.ListEndpoints reaches
`api.sagemaker.us-east-1.amazonaws.com` signed as `sagemaker`.

Also skips traits from the `aws.endpoints` namespace when generating
hints, mirroring the existing treatment of `smithy.rules`. Those traits
have no published Scala bindings, so any service carrying them -- the
Account API among them, via `aws.endpoints#standardPartitionalEndpoints`
-- generated a reference to `aws.endpoints.StandardPartitionalEndpoints`
that did not compile. That made the #1532 service unusable even with the
host fix in place.

Note: `X-Amz-Target` is still set for every protocol though it belongs to
awsJson1_0/1_1 only. Left alone deliberately -- the header is part of the
canonical request, so changing it alters every signature, which does not
belong in a correctness fix.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@kubukoz
kubukoz force-pushed the aws-signing-issues-wip branch from 0e545f1 to a123d85 Compare September 24, 2026 00:20
*/
private val hostPrefix: String =
awsService.endpointPrefix
.orElse(awsService.arnNamespace.map(_.value))

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth flagging where each of these rules actually comes from, since only some of it is documented.

Documented. The two primary rules are stated in the trait definitions themselves: endpointPrefix resolves as {endpointPrefix}.{region}.{dnsSuffix} and is authoritative for the host; aws.auth#sigv4's name is "the signature version 4 service signing name to use in the credential scope" and is authoritative for signing.

Inferred. The fallback chains are not. endpointPrefix has no documented default, so there is no spec text saying what to do when it is absent. arnNamespace is chosen here because it is constrained to ^[a-z0-9.\-]{1,63}$ and defaults to the lowercased service shape name, which makes it DNS-safe — and because for the Account API (#1532) it empirically produces the host that the model's own smithy.rules#endpointTests assert. That is a good signal, not a guarantee.

The authoritative algorithm is smithy.rules#endpointRuleSet, which smithy4s does not evaluate. So this is a best-effort reconstruction of it for the {prefix}.{region}.amazonaws.com case, and the fallbacks are where it is most likely to diverge for some service we have not looked at. #2006 proposes checking that systematically against endpointTests.

@kubukoz
kubukoz marked this pull request as ready for review September 24, 2026 00:42
@mergify

mergify Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

This pull request does not currently match the merge queue conditions, so it cannot be queued from here. The box comes back if it matches again.

@kubukoz
kubukoz merged commit b714b5c into series/0.19 Sep 28, 2026
52 checks passed
@kubukoz
kubukoz deleted the aws-signing-issues-wip branch September 28, 2026 22:48
kubukoz added a commit that referenced this pull request Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sagemaker: invalid signature (Credential should be scoped to correct service) AWS: Account API ListRegions not working (unknown host)

3 participants