The nginx error_page that overwrites your error body

Registration broke in a mobile app. Here is the client log, copied out (host and email changed):

[API Request] POST https://api.example.com/api/v1/customer-accounts/register
[API Params] {company_name: ..., email: someone@example.com, ...}
[ErrorUtils] Status: 500
[ErrorUtils] Response: {"status": "ok", "message": "Request processed"}

Those two lines contradict each other. The status code says the request failed; the body says it succeeded. The app renders errors from the body's message field, so what the user saw was a meaningless sentence instead of “that email is already registered”.

The backend logs looked fine — it really had returned a 500 with a useful body. The body was replaced somewhere between the backend and the phone.

Two lines of nginx do this

SSH onto the box, open the site config for that domain, and it is at the bottom:

proxy_intercept_errors on;
error_page 400 401 403 404 500 502 503 504 /custom_error.html;

location = /custom_error.html {
    internal;
    return 200 "{\"status\": \"ok\", \"message\": \"Request processed\"}";
}

proxy_intercept_errors on means error statuses from upstream are handled by nginx itself. On its own that is harmless — you get nginx's default error pages. The damage comes from the second line, which routes eight status codes into one location that returns a hardcoded string.

So whether the backend returns {"error":"email already registered"} or a GORM duplicate key value violates unique constraint, the client receives the same sentence.

Why it survives so long

Three things keep this config invisible:

1. The status code is correct. Because error_page has no =200, nginx keeps the original status and swaps only the body. Health checks, APM and alert rules that key on status codes all keep working.
2. The body has the right shape. It is valid JSON with a status and a message, so the client parser never throws. It just quietly displays the wrong thing.
3. The success path is untouched. A 2xx never enters error_page, so any smoke test passes. Only failing requests lie, and nobody watches failing requests on a normal day.

The failures your monitoring catches are never the expensive ones.

This belongs to the same family as CloudFront turning a 404 into a 200 homepage: a middle layer rewrites the response for a reason that made sense at the time, and the rewrite leaves no trace in the logs on either side of it.

Finding the responsible layer in five minutes

Peel from the outside in, replaying the same request at each layer. First, is there a CDN in front?

dig +short api.example.com
# CDN IP      → there is an edge layer, one more layer to peel
# your own IP → straight to origin, skip the next step

With a CDN, pin the hostname to the origin IP with --resolve and bypass the edge:

curl -ki --resolve "api.example.com:443:203.0.113.10" \
  https://api.example.com/api/v1/customer-accounts/register \
  -X POST -H 'Content-Type: application/json' -d '{}'

Compare the two. Same body means the edge is not doing it; keep going inward. Now get onto the box and hit the backend port directly, skipping nginx:

curl -i http://localhost:8007/api/v1/customer-accounts/register \
  -X POST -H 'Content-Type: application/json' -d '{}'

This is the decisive step. If localhost returns a real error message while the public hostname returns Request processed, the rewrite happens in nginx and you have narrowed it down to a single file.

The change

The smallest fix is to cut that status list down to infrastructure-level failures only:

error_page 502 503 504 /custom_error.html;

502, 503 and 504 mean the backend never answered, so nginx producing a generic page is reasonable — nothing is being covered up. 4xx and 500 are the cases where the backend has something to say, and those should pass through.

Three steps, in this order:

1. sudo cp site.conf /root/nginx-backup/site.conf.$(date +%s) — keep the backup outside sites-enabled/
2. sudo nginx -t — do not continue until the syntax check passes
3. sudo systemctl reload nginx — reload, not restart, so open connections survive

Then replay the original request unchanged and confirm you get the backend's real message.

Why someone writes this

The config was not random. The intent is visible: do not leak backend error detail to the outside. Stack traces, SQL statements, internal paths — none of that belongs in a public API response.

The mistake is doing the masking at the wrong layer. nginx cannot see meaning, only status codes, so it has no way to tell a leaked stack trace apart from “that email is already in use”, which the user needs. Mask in the application: let the backend decide which messages are safe to expose, keep the detail in the logs, and return a stable error code.

Put it in nginx and what you get is not safety. It is a mute switch nobody knows exists.

If you remember one thing

“The status code is right” and “the response is right” are different claims. Verifying an API's error path means replaying failing requests too, and comparing the content of the body, not just its shape. A test suite that only exercises the success path will never notice this config exists.