Three ways MQTT fails to connect, all with the same symptom

Draw the chain first. For an app to reach a self-hosted MQTT broker, the hops are:

Flutter client  →  CDN (TLS 443)  →  nginx (/mqtt location)
  →  EMQX websocket listener  →  HTTP authn callback  →  backend /mqtt/auth

Six of them. Break any one and the phone shows NoConnectionException or Missing CONNACK, followed by a reconnect loop. The symptom carries no information about which hop failed.

I have hit all three of the following. Their causes sit at three different points on that chain.

One: the client library overwrites your settings

Flutter's mqtt_client v10.x has an order-sensitive behaviour: set both secure = true and useWebSocket = true and an internal if (secure) branch flips useWebSocket back to false. You believe you are speaking WSS; you are actually on raw TCP + TLS, and that /mqtt location in nginx only accepts a WebSocket upgrade.

The fix is to leave secure alone entirely:

client.useWebSocket = true;
client.useAlternateWebSocketImplementation = true;  // do not set secure = true
client.websocketProtocols = ['mqtt'];

The same library has a second trap: the WS2 handshake builds its GET request line from uri.path. Give it a host with no path and it sends GET ? HTTP/1.1 — an invalid request line, and the CDN simply closes the connection. So pass a complete URI as the first argument:

MqttServerClient.withPort('wss://$host/mqtt', clientId, port);

To find out which path you are actually on, turn the library's own logging on:

client.logging(on: true);
// lines that matter:
//   "WS URL is wss://..."                          → is the path right
//   "alternate websocket implementation selected"  → you are on WS
//   any MqttSecureConnection                       → you fell back to raw TCP+TLS

Two: the broker reads "200 with no body" as a refusal

EMQX 5's HTTP authentication callback expects JSON carrying a result field:

{"result": "allow"}   // or "deny" / "ignore"

A bare 200 OK with an empty body — the intuitive way to write that handler — is read as a denial, and the client receives CONNACK code 5, notAuthorized.

What makes this one nasty is that the backend log looks perfectly healthy. The access line reads POST /api/v1/mqtt/auth 200, no error, no warning. Viewed from the backend, that authentication succeeded.

// Go / Gin
c.JSON(http.StatusOK, gin.H{"result": "allow"})

This is the mirror image of a correct status code with a lying body: there the body was replaced by a middle layer, here it was never written at all — and both times every log involved says "200, all good".

Three: Docker's default network does not resolve service names

This was the actual root cause that time. EMQX's auth config pointed at http://backend:8007/api/v1/mqtt/auth, which looks entirely reasonable — it is exactly what a compose file would say.

But those containers had each been started with docker run, so they all sat on Docker's default bridge network. The default bridge does not do DNS resolution by service name — that is a user-defined network feature. So backend never resolved and the auth request was never sent at all.

The quick fix routes through the host gateway:

docker run --add-host=host.docker.internal:host-gateway ...
# auth URL becomes http://host.docker.internal:8007/api/v1/mqtt/auth

The durable fix is to start them with compose, which creates a user-defined network where service-name DNS just works:

emqx:
  extra_hosts:
    - "host.docker.internal:host-gateway"

Where to start

When the symptom carries no information, the right move is to establish how far the request got, not to guess which hop misbehaved. Work inward; each hop has a direct question you can ask it.

Can the container reach the backend at all (a 000 means no, which is trap three):

docker exec my-emqx curl -s -o /dev/null -w "%{http_code}\n" \
  -X POST http://host.docker.internal:8007/api/v1/mqtt/auth \
  -H "Content-Type: application/json" -d '{"username":"x","password":"y"}'
# 200 or 401 both mean reachable; 000 means no route

Is the broker seeing connections at all:

docker exec my-emqx /opt/emqx/bin/emqx ctl listeners
# current_conn: 0        → nobody is arriving; the problem is upstream of the broker
# shutdown_count rising  → they arrive and get kicked; the problem is authentication

Does the whole external path work (CDN → nginx → EMQX WebSocket upgrade):

curl -ki --http1.1 \
  -H "Connection: Upgrade" -H "Upgrade: websocket" \
  -H "Sec-WebSocket-Version: 13" -H "Sec-WebSocket-Protocol: mqtt" \
  -H "Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==" \
  https://api.example.com/mqtt
# expect HTTP/1.1 101 Switching Protocols + sec-websocket-protocol: mqtt

That last one earns its keep by splitting "the phone is wrong" from "the server is wrong". A 101 comes back and trap one is on the client side; it does not and you keep working inward.

What the three have in common

They sit at three unrelated points with three unrelated causes, and share exactly one property: the hop that broke reports itself healthy.

• the library believes it connected the way you configured it (it had already rewritten the config)
• the backend believes it answered the auth request (it just never wrote a body)
• EMQX believes it sent the auth request (it just could not resolve the hostname)

Not one of those logs is wrong. Each is honest, and each stops at the boundary of its own hop — which is precisely where the broken thing lives.

If you remember one thing

The first question in a multi-layer system is not "which hop is wrong", it is "how far did the request get". The first has to be guessed; the second can be measured. Find one command per hop that answers it directly and work inward — far faster than reading six logs, because logs tell you what each hop did, never whether the request arrived.