Resolve AWS signing name and endpoint host independently - #1596
Conversation
cac1af5 to
0e545f1
Compare
The SigV4 signing name and the endpoint host were both derived from `endpointPrefix`, which is only correct for services where the two happen to coincide. They are separate things. `aws.auth#sigv4`'s `name` is defined as "the signature version 4 service signing name to use in the credential scope when signing requests", so the signing name now comes from there, falling back to `arnNamespace` and then `endpointPrefix`. The host keeps using `endpointPrefix` -- the field documented for endpoint resolution -- and falls back to `arnNamespace`, which is DNS-safe by construction, for the services that omit it. This fixes two failure modes: - Services whose prefix and signing name differ were signed with the wrong credential scope and rejected by AWS with "Credential should be scoped to correct service". SES v2 is hosted at `email.<region>.amazonaws.com` but must be signed as `ses`; SageMaker is hosted at `api.sagemaker.<region>.amazonaws.com` but signs as `sagemaker` (#1568). - Services with no `endpointPrefix` fell back to the *operation* name for the host, producing nonsense like `ListRegions.us-east-1.amazonaws.com` (#1532). The Account API's own endpoint tests confirm the expected host is `account.us-east-1.amazonaws.com`, so this needs no rule-set evaluation. Adds fixture services to `sampleSpecs/aws_example.smithy` covering each combination of `endpointPrefix`, `arnNamespace` and sigv4 name seen in the wild, plus DynamoDB (where all three agree) as a no-regression case. Four of the five new assertions fail without this change. Verified against real AWS with read-only calls: Account.ListRegions and Account.GetAccountInformation reach `account.us-east-1.amazonaws.com` signed as `account`, and SageMaker.ListEndpoints reaches `api.sagemaker.us-east-1.amazonaws.com` signed as `sagemaker`. Also skips traits from the `aws.endpoints` namespace when generating hints, mirroring the existing treatment of `smithy.rules`. Those traits have no published Scala bindings, so any service carrying them -- the Account API among them, via `aws.endpoints#standardPartitionalEndpoints` -- generated a reference to `aws.endpoints.StandardPartitionalEndpoints` that did not compile. That made the #1532 service unusable even with the host fix in place. Note: `X-Amz-Target` is still set for every protocol though it belongs to awsJson1_0/1_1 only. Left alone deliberately -- the header is part of the canonical request, so changing it alters every signature, which does not belong in a correctness fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
0e545f1 to
a123d85
Compare
| */ | ||
| private val hostPrefix: String = | ||
| awsService.endpointPrefix | ||
| .orElse(awsService.arnNamespace.map(_.value)) |
There was a problem hiding this comment.
Worth flagging where each of these rules actually comes from, since only some of it is documented.
Documented. The two primary rules are stated in the trait definitions themselves: endpointPrefix resolves as {endpointPrefix}.{region}.{dnsSuffix} and is authoritative for the host; aws.auth#sigv4's name is "the signature version 4 service signing name to use in the credential scope" and is authoritative for signing.
Inferred. The fallback chains are not. endpointPrefix has no documented default, so there is no spec text saying what to do when it is absent. arnNamespace is chosen here because it is constrained to ^[a-z0-9.\-]{1,63}$ and defaults to the lowercased service shape name, which makes it DNS-safe — and because for the Account API (#1532) it empirically produces the host that the model's own smithy.rules#endpointTests assert. That is a good signal, not a guarantee.
The authoritative algorithm is smithy.rules#endpointRuleSet, which smithy4s does not evaluate. So this is a best-effort reconstruction of it for the {prefix}.{region}.amazonaws.com case, and the fallbacks are where it is most likely to diverge for some service we have not looked at. #2006 proposes checking that systematically against endpointTests.
|
This pull request does not currently match the merge queue conditions, so it cannot be queued from here. The box comes back if it matches again. |
Closes #1568, closes #1532.
The SigV4 signing name and the endpoint host were both derived from
endpointPrefix. That is only correct for services where the two happen to coincide, and it produced two distinct failures for services where they don't.The two are independent
endpointPrefix→arnNamespace→ service idsigv4.name→arnNamespace→endpointPrefix→ service idNeither falls back to the operation name any more, which is what produced
ListRegions.us-east-1.amazonaws.com.Rationale, taken from the trait definitions themselves:
aws.auth#sigv4'snameis "the signature version 4 service signing name to use in the credential scope when signing requests" — authoritative for signing, so it wins.endpointPrefix"identifies which endpoint in a given region should be used", resolving as{endpointPrefix}.{region}.{dnsSuffix}— authoritative for the host. It is also documented as non-unique and subject to change, and MUST NOT be used for anything else, which is why signing prefersarnNamespaceover it.arnNamespaceis constrained to^[a-z0-9.\-]{1,63}$and defaults to the lowercased service shape name, making it a DNS-safe stand-in for the host whenendpointPrefixis absent.#1532 needs no endpoint-rules evaluation. The Account model's own
smithy.rules#endpointTestsassert thatus-east-1+ no FIPS + no dual-stack →https://account.us-east-1.amazonaws.com, i.e. the plain pattern we already produce once the operation-name fallback is gone.Also: skip
aws.endpointstraits in codegenWhile verifying #1532 end-to-end, the Account API turned out not to be usable at all, independently of the host bug: it carries
@aws.endpoints#standardPartitionalEndpoints, and codegen rendered that as a hint referencingaws.endpoints.StandardPartitionalEndpoints, for which no artifact publishes Scala bindings. The generated service simply did not compile.aws.endpointsis now skipped when generating hints, mirroring the existing treatment ofsmithy.rules(added in #1920). Reproduced on released 0.19.12, so this is not a regression — that codegen path was just never exercised in-tree, because the AWS specs we bootstrap don't carry the trait.Tests
New fixture services in
sampleSpecs/aws_example.smithy, one per combination seen in the wild, asserting both the host and the credential scope:PrefixDiffersFromSigningNameemail.…sesDottedPrefixapi.sagemaker.…sagemakerNoEndpointPrefixaccount.…accountNoSigv4endpointprefix.…arnnamespaceDynamoDBdynamodb.…dynamodb4 of the 5 fail without this change. DynamoDB passes either way, which is the point of including it. The failures were: SES signed
email, SageMaker signedapi.sagemaker, Account got hostDoThing.us-east-1.amazonaws.comand signednoendpointprefix.aws-http4s/testis green (696 tests), includingAwsSignatureTest, which property-checks our signatures against the real AWS SDKAws4Signer.codegen/testandbootstrapped/compilealso pass.Verified against real AWS
Read-only calls, generated from the published models with no hand-editing:
Account.ListRegionsaccount.us-east-1.amazonaws.com…/us-east-1/account/aws4_requestAccount.GetAccountInformationaccount.us-east-1.amazonaws.com…/us-east-1/account/aws4_requestSageMaker.ListEndpointsapi.sagemaker.us-east-1.amazonaws.com…/us-east-1/sagemaker/aws4_requestAll three returned real data. Separately, a downstream SES v2 test asserting
/ses/aws4_requestgoes red → green against a local snapshot, with the host stayingemail.<region>.amazonaws.com.Scope
Only the
{prefix}.{region}.amazonaws.compattern is handled. FIPS, dual-stack, non-standard partitions (amazonaws.com.cn, GovCloud) and operation- or tenant-level endpoints all require evaluatingsmithy.rules#endpointRuleSet, which smithy4s does not do. #2006 proposes measuring that gap against the models' ownendpointTests.X-Amz-Targetis still set for every protocol, though it belongs to awsJson1_0/1_1 only. Deliberately left alone: the header is part of the canonical request, so changing it alters every signature (and the two compensating force-appends inAwsSignatureTest). That blast radius does not belong in a cherry-pickable correctness fix.Also filed while working on this: #2007 (
AwsCredentialsFilemisreports an unparseable credentials file as a missing profile).PR Checklist (not all items are relevant to all PRs)