Baobaobaolin.com
date
entry
013
topic
tooling
rev
1

Ask "who am I" before running any aws command

In a multi-account setup the expensive mistake is not typing the wrong command. It is typing the right one against the wrong account. Reads succeed and hand you the wrong answer; writes succeed and land in someone else's house. The CLI never asks whether you are sure.

A typical afternoon. You go looking for a client's resource:

aws ec2 describe-instances --region ap-northeast-1 \
  --filters "Name=tag:Name,Values=api-server"
# → empty

No error, no AccessDenied, just empty. The reasonable conclusion is "that instance does not exist", so you start checking whether CI failed to create it, or ask the client if they deleted it.

What actually happened: half an hour earlier you ran export AWS_PROFILE=other for something else, and that shell is still open. You queried a different account, and that account genuinely has no such machine.

"Empty" and "does not exist" are different facts, and the CLI prints them the same way.

How the credentials get chosen

The trap exists because which credentials to use is not decided by the line you just typed. It is decided by a precedence chain. Simplified, for AWS CLI v2:

  1. Command-line options (--profile)
  2. Environment variables (AWS_PROFILE, the AWS_ACCESS_KEY_ID family)
  3. The default entries in ~/.aws/credentials and ~/.aws/config
  4. Roles supplied by container or EC2 instance metadata

Item 2 causes most of the real-world damage, for a simple reason: environment variables outlive your memory of setting them. They follow that shell tab, that tmux session, the window you left open yesterday — and they appear in no prompt.

Item 4 is a different situation: running commands on an EC2 instance with no explicit credentials silently uses the instance role, which is usually nothing like your local identity.

The one-line check

aws sts get-caller-identity
{
  "UserId":  "AIDA...",
  "Account": "111111111111",
  "Arn":     "arn:aws:iam::111111111111:user/deploy"
}

It needs no permissions — every IAM identity may ask who it is — and it changes nothing. The Account field is the answer.

In a script, make it a gate:

EXPECTED_ACCOUNT=111111111111
ACTUAL=$(aws sts get-caller-identity --query Account --output text)
if [ "$ACTUAL" != "$EXPECTED_ACCOUNT" ]; then
  echo "WRONG ACCOUNT: $ACTUAL (expected $EXPECTED_ACCOUNT)" >&2
  exit 1
fi

Any script that mutates anything deserves those five lines at the top. They cost one API call and rule out things like running a teardown script for staging against production.

The danger is asymmetric

Running against the wrong account has two tiers of consequence, and they are far apart:

  • Reads → wrong conclusions. Nothing found means it does not exist, and the next step is built on that false premise — worst case, recreating something that already exists, so now the same resource lives in two accounts
  • Writes → real damage. Resources created in the wrong account (billing and permissions go with them), or worse, a delete whose target name happens to exist in both

The first tier is worth naming separately because it looks fine. It never appears in an incident report; it shows up as "that one took ages to figure out".

A command succeeding says nothing about where it succeeded.

Removing the need to remember

"Always check first" is discipline, and discipline fails under time pressure. Ways to make it structural:

  • Show the account in your shell prompt. Put the current AWS_PROFILE in there so you do not have to think of checking — and when it is unset, blank is information too
  • Have no default profile. With no default there is no "accidentally used the default account"; forgetting --profile becomes an error instead of a result. An error message beats a wrong outcome
  • Name profiles after what the account is for. prod, staging, client-a-prod beat abbreviations — the name has to convey danger at a glance
  • Put the account id in the project docs. "This project lives in 111111111111" is one line, and it gives that gate a constant to compare against

Same reasoning as using permissions rather than discipline to stop hand-edits to prod: a step that relies on remembering is skipped exactly when pressure is highest, which is the only moment it needed to work.

While we are here: region

There is a smaller version of the same problem. Unspecified, region falls back to a config default that you may have set three years ago. Same symptom — nothing found, assumed not to exist.

I write --region on every command. A few extra characters retire an entire class of "it was in another region" confusion.

Afterwards there is only CloudTrail

If something did happen in the wrong account, CloudTrail is what reconstructs it: who, when, under which identity, calling which API.

Worth knowing it is there — but it is not a defence. It is a tool for answering questions afterwards, not for preventing anything. By the time you are reading it, the thing is done.

If you remember one thing

An empty result is not an answer. It is two answers superimposed: "not in this account" and "you are not in that account." Spend one API call establishing which one you are looking at before you treat empty as a conclusion.

Revision history

  1. Gate on the ARN: the account-id comparison recommended here lets root through, see 023
  2. First published