Do not use set -e in a deploy script

First, why set -e is usually right. Most scripts are a sequence of self-contained computations, and carrying on after a failed step just feeds wrong input into the next one. Aborting is sensible there.

A deploy has a different shape. It is a sequence of modifications to the outside world, ordered and interdependent. Here, "stopped halfway" is not a safe state. It is a broken state nobody knows about.

The time it caught me

Deploying this site requires a closing step: writing every index.html to its directory key as well. Those keys are not files locally, so every sync --delete removes them, which fixes the order: sync, rewrite keys, invalidate, verify.

That run went:

1. sync --delete completed — every directory key from the previous run was now gone
2. the key-rewriting loop ran halfway and the AWS session expired
3. set -e did its job and aborted immediately
4. the invalidation never ran, and neither did the verification

The result was a batch of live 404s with no signal at all. CI did record a failure, but nobody was watching it — and the one step that would have identified which pages were wrong, the whole-site comparison, was precisely what set -e had skipped.

The steps you most need on a failure are usually the ones an abort skips.

Where the line is

Split the script in two at the first write to production:

• Before it — identity checks, build, argument validation. A failure here means production has not been touched, so aborting is the cleanest outcome. Exit directly
• After it — sync, key rewriting, invalidation, verification. Production is already mid-change, and aborting only leaves it there. Record the failure and keep going

Written out, roughly:

set -uo pipefail          # keep -u and pipefail, drop -e

# ---- production untouched: fail means stop ----
ACC=$(aws sts get-caller-identity --query Account --output text) || exit 1
[ "$ACC" = "$EXPECTED_ACCOUNT" ] || { echo "wrong account: $ACC" >&2; exit 1; }
npm run build || exit 1

# ---- writing to production: fail means count it, not stop ----
FAIL=0
while IFS= read -r f; do
  put_directory_key "$f" || { echo "!! FAILED $f" >&2; FAIL=$((FAIL+1)); }
done < <(find dist -mindepth 2 -name index.html)

[ "$FAIL" -gt 0 ] && echo "warning: $FAIL directory keys failed" >&2

aws cloudfront create-invalidation ...   # must run
verify_all_pages                          # must run
RC=$?
exit $(( RC != 0 || FAIL > 0 ))

The || exit 1 in the first half is written out explicitly. That reads better than a global set -e, because it turns "failing here means stopping" into a visible decision rather than an invisible default — a reader can see exactly where the author considered aborting safe.

Keep going, but exit honestly

"Do not abort" is not "pretend it worked". That FAIL counter exists so the script can still produce a correct exit code.

This is the easy part to get wrong: without set -e the script runs to the end, and with no tally at the end it returns 0 — CI goes green while the site serves 404s. That is worse than aborting, because an abort at least left a red light.

Two rules, then: never interrupt mid-flight, always tell the truth at the end.

Keep -u and pipefail

Only -e goes. The other two matter more in a deploy script, not less.

set -u — treating undefined variables as errors — catches things like:

aws s3 sync dist/ "s3://$BUCKET/$PREFIX" --delete

With $PREFIX unset, that line syncs to the bucket root — and with --delete, it removes everything in the bucket that is not in dist. The same mistake is more famous as rm -rf "$DIR/".

pipefail stops a mid-pipeline failure from being masked by the last command succeeding:

aws s3 ls "s3://$BUCKET" | grep -c index.html
# without pipefail: aws failed, grep still exits 0, and you take "0 files" as fact

Not only a shell problem

The same judgement applies to any ordered set of external modifications. A thread that fails halfway cannot be reposted from the top; a payment callback processed halfway cannot be un-received.

The shared property is that the outside world offers no transaction, so aborting does not undo what already happened. In that territory the goal of error handling is not to stop. It is to let the next run pick up, and to make sure a person knows where it stopped.

If you remember one thing

set -e is valuable while nothing external has changed; past that line, the thing it protects becomes the thing it damages. Find the first line in your script that writes to production. Abort before it, count after it.