aws login credential errors: the fix stays, but the cause I wrote in my commits doesn't match the API docs

aws login is the sign-in method available from AWS CLI 2.32.0: you sign in to the console in a browser and the CLI gets temporary credentials, without the flow requiring a locally stored long-term access key. At least during this stretch of debugging in August 2026, two of my sites' deploy scripts ran on those credentials.

From August 22 the commits start recording an intermittent failure: CreateOAuth2Token returning "authorization grant is invalid, expired, revoked, or malformed", while the same command typed by hand a little later would succeed. Over two days I left a string of commits in two repos and changed hypotheses several times. This note is about which parts of that record hold up: how the hypotheses evolved, what AWS's docs say, what was recorded versus merely inferred, and what the current fix costs.

How the hypotheses changed over two days

All of this comes from the two repos' git logs (times in +08:00; the four 8/22 rows are in the website repo; both repos made the two 8/23 changes, shown at the other site's times; the two 9/16 rows are in the other site only; "hypothesis" means the explanation written in the commit at the time):

When	Hypothesis at the time	Change
8/22 10:48	A race with the token refresh	Identity check: up to 3 attempts, 3 s between failures
8/22 11:43	All three failed; not waiting long enough	Up to 5 attempts, waits doubling from 3 s, 45 s in all
8/22 14:21	Failures cluster in a "time window"	Up to 6 attempts, waits doubling from 10 s
8/22 17:07	Same	Comment added: probe by hand before deploying
8/23 14:57	Rotating refresh token: each call spends the old grant	Drop the retries; export once with export-credentials
8/23 19:45	The rotating call itself fails, but the new token is already written	If fetching credentials fails, retry once immediately
9/16 14:18	The new token takes a moment to be written back	Fetch credentials up to 4 times, 3 s apart (one site only)
9/16 16:13	Same	Make one read-only call before fetching (same site)

The turn came on the afternoon of August 23. The other site's commit records five uploads with five attempts each, all 25 failing, then every upload succeeding when run one at a time (that script runs under set -e, and the record doesn't say how those 25 attempts were run). The website's script had done the same that morning: the commit says the retry loop spent four minutes failing six times and re-running the whole script went straight through (going by the code, the waits between those six attempts add up to at least 310 seconds, so "four minutes" was a miscount at the time). The commit's explanation: aws login's refresh token rotates, each call spends the old grant and gets a new one, so in a quick sequence the later call is still holding a grant that's already been used; retrying doesn't help because every retry presents the same dead grant.

The fix was to call aws configure export-credentials --format env once at the start of the deploy, export the three variables, and have every later aws command use that fixed set. The commit records one end-to-end run after the change succeeding on the first attempt.

That's not what AWS's docs say

Writing this up, I went back to two documents. The AWS CLI user guide, "Login for AWS local development using console credentials", says:

• The CLI and SDKs refresh the cached credentials "every 15 minutes as needed"; the session as a whole lasts up to the IAM principal's configured duration, at most 12 hours.
• The temporary credentials and the metadata needed to refresh them are stored by default in ~/.aws/login/cache.

The API reference for CreateOAuth2Token (the AWS CLI reference page for signin create-o-auth2-token) is blunter: redeeming an authorization code is not idempotent, and each code can be used once; refreshing with a refresh token is "Idempotent within token validity window", meaning multiple calls with the same refresh token return consistent results within that window. The access token in the response is a 15-minute set of temporary credentials, and expiresIn is at most 900 seconds.

Each sentence knocks out half of my explanation. While the cached credentials are still usable, an aws command has no need to exchange a token, so "each call spends a grant" can't be right. And since refresh is idempotent within the validity window, "it was already used once" can't by itself explain why a second call was rejected. The mechanism I wrote into the commits and script comments isn't supported by the docs.

Recorded, inferred, unknown

Splitting the record apart:

• Recorded: the CLI reported the failing operation as CreateOAuth2Token; failures were seen arriving in clusters, with a lone call succeeding a while later; after the switch to a single export on August 23, each script has one recorded end-to-end success; on September 16 one of the sites added delayed retries and then a read-only call first, with a comment noting one first-try success afterwards and no measured success rate.
• Inferred: the failures may be tied to the moment the cached credentials near expiry and need refreshing, not to every call; after the export, no token is exchanged during the deploy, which may be why the fix works, though that isn't proven. September's "the new token takes a moment to be written back" belongs here too; it was never measured.
• Unknown: what the server actually rejected. The error message lists invalid, expired, revoked and malformed all at once, so the string can't pick one. Whether the window had passed, a cache file was read mid-write, or something else, the records I have contain no debug output from a failure to tell. The API reference also says the refresh token carries cnf.jkt (the thumbprint of a bound key); whether that plays a part, I don't know either.

Next time it fails, what I want to keep is the full command run with --debug (for example aws --debug sts get-caller-identity), the CLI version, the time and the cached credentials' expiry, keeping just the request and error lines with tokens and signatures masked, rather than another new explanation.

The cost of exporting once

I'm keeping the fix, because whatever the cause, it means no token exchange is needed during the deploy. It has a cost the original comment didn't mention, though: what gets exported is the current set of temporary credentials, which the API reference caps at 900 seconds for this login flow, and the time left at export can be shorter. The script already prints AWS_CREDENTIAL_EXPIRATION, but it only prints it; nothing checks it.

This isn't just theory. The website's commit of September 10 records that in August the CI "Sync to S3" step took 782 seconds, because it called put-object twice for each of 293 subdirectories, and that the same loop made a local deploy on September 10 stop partway through when the credentials expired. After that commit moved directory-index resolution into a CloudFront Function, the whole local deploy (build and checks included) measured 64 seconds. Whether export-credentials refreshes first when the credentials are about to expire isn't documented and I haven't tested it. Here's the guard I plan to add to the scripts; it isn't merged yet:

unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN AWS_SECURITY_TOKEN AWS_CREDENTIAL_EXPIRATION
creds=$(aws configure export-credentials --format env) || {
  echo "can't get credentials; run aws login first" >&2; exit 1; }
eval "$creds"
export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN AWS_CREDENTIAL_EXPIRATION
python3 - <<'PY' || exit 1
import os, sys, datetime as dt
exp = os.environ.get("AWS_CREDENTIAL_EXPIRATION")
if not exp:
    sys.exit("no expiry time; can't confirm the credentials will last")
left = (dt.datetime.fromisoformat(exp.replace("Z", "+00:00"))
        - dt.datetime.now(dt.timezone.utc)).total_seconds()
print(f"credentials have {left:.0f} s left")
if left < 300:
    sys.exit("less than 300 s left; get fresh credentials and rerun")
PY

The unset (including the legacy AWS_SECURITY_TOKEN) comes first because stale credential variables left in the environment take precedence for the CLI, and the guard would then wave things through on an old expiry; and the exit status of $(…) has to be caught before the eval, because eval "" succeeds anyway. The 300-second margin is sized from that one 64-second measurement and needs adjusting if the deploy grows. I tested it with a fake aws command in three cases (12 minutes left passes, 2 minutes left stops, a failed fetch stops); it hasn't run during a real failure.

If you remember one thing

A fix that works doesn't make the cause you wrote down correct. For the error in the opening, two days of commits cycled through several hypotheses, and the last one was the most confident and the least able to survive a read of the docs. Commit messages record what I believed at the time; the comments in the scripts should be relabelled as inference, with links to the two documents; the fix stays, plus the few lines that check the expiry.