Commas in AWS CLI shorthand: an error if you're lucky, two values if you're not

Many AWS CLI structured parameters can be written two ways: JSON, or the CLI's own shorthand syntax such as Key=value,Key2=value2. Shorthand is quicker to type, and plenty of AWS documentation examples use it.

But shorthand separates with commas, and values can contain commas too. This note is about what happens when an unprotected comma lands inside a value. I saw two outcomes: if the field is a string, the CLI errors and sends nothing; if the field is a list, it splits the value in two and gets ready to send it, with no error or warning about the split. It covers how shorthand is parsed, the case that errors, the case that doesn't, and how to reproduce both without touching any AWS account.

How shorthand is parsed

The AWS CLI user guide page "Using shorthand syntax in the AWS CLI" defines structure parameters as "a comma-separated list of key-value pairs". Top-level list parameters (such as --instance-ids id1 id2) are separated by spaces; list fields inside a structure, like Values=a,b later in this note, are separated by commas. The guide also warns that shorthand values are strings, so your terminal's quoting and escaping rules apply.

So there are two layers of parsing: the shell splits arguments, then the CLI looks for separators according to the shorthand grammar. Shell quotes are used up in the first layer. Wrap 'Comment=a, b' in single quotes and the shell leaves it alone, but the string the CLI receives still has an unprotected comma.

Case one: the field is a string, and the CLI errors

My deploy script for taiwanrep.com updates a CloudFront Function with update-function, and --function-config needs a Comment and a runtime. Here it is reproduced with create-function, which takes the same FunctionConfig, assuming the comment contains a comma:

aws cloudfront create-function --name t --function-code fileb://fn.js \
  --function-config 'Comment=www to apex, directory index,Runtime=cloudfront-js-2.0'

aws: [ERROR]: An error occurred (ParamValidation): Parameter validation failed:
Invalid type for parameter FunctionConfig.Comment, value: ['www to apex', 'directory index'], type: <class 'list'>, valid types: <class 'str'>

Exit code: 252.

The error prints the parse result: Comment became ['www to apex', 'directory index'], a two-element list. The shorthand itself parsed fine; what failed was type validation afterwards. Comment only accepts a string, so no request is sent. This is the good case.

Case two: the field is already a list, and the CLI sends it

Take a field that accepts several values. In ec2 describe-instances --filters, Values is a list of strings. Suppose a tag value is R&D, Taipei:

aws ec2 describe-instances --filters 'Name=tag:Team,Values=R&D, Taipei'

With --debug, the request parameters it was about to send:

'Filter.1.Name': 'tag:Team', 'Filter.1.Value.1': 'R&D', 'Filter.1.Value.2': 'Taipei'

One value became two, and the space after the comma was dropped. The CLI raised no error or warning about the split (the error at the end is the failed connection to the local endpoint, explained below). The CLI reference for describe-instances says multiple values for one filter "are joined with an OR" (its own example, Values=t2.micro,t3.micro, relies on the comma as a separator). By that rule this query matches instances tagged R&D or Taipei, and the one actually tagged R&D, Taipei doesn't match (that follows from the docs; I didn't run it against a real account). If the same split happened in a list field of a write command, the request would likewise carry an extra value; what the service does with it depends on the API, and I haven't verified that.

Type validation caught the first mistake. The second one happened to have the right type, so the CLI layer doesn't catch it.

Reproducing it without touching AWS

All of the above came from runs with no real credentials and no connection to AWS. Clear the inherited environment with env -i, use the example keys from the AWS docs, keep the CLI away from my own config files, and point the endpoint at a local port nothing answers on:

env -i PATH="$PATH" HOME="$HOME" \
  AWS_CONFIG_FILE=/dev/null AWS_SHARED_CREDENTIALS_FILE=/dev/null \
  AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE \
  AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY \
  AWS_DEFAULT_REGION=us-east-1 AWS_MAX_ATTEMPTS=1 \
  aws ec2 describe-instances --filters 'Name=tag:Team,Values=R&D, Taipei' \
  --endpoint-url http://127.0.0.1:9 --debug

The --debug output is long; I piped it through 2>&1 | grep -o "'body': {[^}]*}" to pull out the parameters. Drop the pipe when you want the exit code, or $? will be grep's.

A type-validation failure stops at ParamValidation with exit code 252. Anything that passes validation gets as far as sending and then fails with Could not connect to the endpoint URL: "http://127.0.0.1:9/...", exit code 255. So in this setup "could not connect" means the parameters made it through the CLI layer; the --debug output shows what it meant to send.

Version: aws-cli 2.34.23.

The fix: JSON for anything that goes into a script

All three of these keep Comment as one string (each tested with the method above):

# 1. JSON, wrapped in single quotes
--function-config '{"Comment":"www to apex, directory index","Runtime":"cloudfront-js-2.0"}'

# 2. Double quotes around the value inside the shorthand, single quotes outside
--function-config 'Comment="www to apex, directory index",Runtime=cloudfront-js-2.0'

# 3. Read it from a file
--function-config file://function-config.json

The same goes for the EC2 filter; in JSON it goes out as one value:

--filters '[{"Name":"tag:Team","Values":["R&D, Taipei"]}]'
# 'Filter.1.Name': 'tag:Team', 'Filter.1.Value.1': 'R&D, Taipei'

Number 2 works; the --debug body shows <Comment>www to apex, directory index</Comment>. Escaping the comma with a backslash inside shorthand (Values=R&D\, Taipei) also kept one value in my test. But both ask the reader to keep track of what each layer of quoting or escaping protects, and I didn't find either form described on that user guide page. The deploy.sh for taiwanrep.com uses number 1:

--function-config '{"Comment":"www to apex 301 and directory index","Runtime":"cloudfront-js-2.0"}'

If you remember one thing

In shorthand, every unprotected comma is a separator, whatever you meant by it. Back to the two outcomes at the top: the one that errors is the lucky one. The case to guard against is the one where the type happens to fit, and the most direct place to see it is the request parameters in --debug.