Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 12 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,16 @@ EMQX_AUTH_FILE := .emqx/auth-bootstrap.csv
# Generate the EMQX built-in-database bootstrap file from the broker credentials in .env.
# EMQX ships with anonymous MQTT enabled; this seeds the one account the API and the bots
# use so the broker can reject everything else. The file holds a plaintext password, so it
# is written 0600 and gitignored.
# is gitignored.
#
# The secret is kept off other host users by the MODE OF THE DIRECTORY (0700), not of the
# file (0644). The file is bind-mounted into the broker, which runs as its own `emqx` user
# (uid 1000): on Linux a bind mount carries the host uid through unmapped, so a 0600 file
# owned by the deploying user is unreadable inside the container. EMQX then logs a
# `Permission denied` for the bootstrap file, skips the import, comes up healthy with NO
# accounts, and rejects the API's correct credentials as `Not authorized` (issue #224).
# Directory mode is enough because Docker resolves the bind-mount path as root once, at
# mount time — the container never traverses .emqx/ to reach the file.
#
# is_superuser is deliberately false: EMQX superusers bypass authorization entirely, which
# would make emqx/acl.conf dead config.
Expand All @@ -73,8 +82,9 @@ emqx-auth:
exit 1; \
fi; \
mkdir -p $(dir $(EMQX_AUTH_FILE)); \
chmod 700 $(dir $(EMQX_AUTH_FILE)); \
printf 'user_id,password,is_superuser\n%s,%s,false\n' "$$user" "$$pass" > $(EMQX_AUTH_FILE); \
chmod 600 $(EMQX_AUTH_FILE); \
chmod 644 $(EMQX_AUTH_FILE); \
echo "[INFO] Wrote $(EMQX_AUTH_FILE) for broker user $$user"

# Broker container to audit. Override to check another deployment:
Expand Down
21 changes: 19 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -288,8 +288,25 @@ make doctor
Checks dependencies, `.env` (including credentials still left at well-known
defaults), the `hummingbot-api` / `hummingbot-broker` / `hummingbot-postgres`
containers, which ports are on a public interface, Tailscale's tailnet *and*
serve status, and whether the API actually answers an authenticated request.
Read-only, and it names the fix for whatever it finds.
serve status, whether the API actually answers an authenticated request, and
whether it is connected to the MQTT broker. Read-only, and it names the fix for
whatever it finds.

**Bots deploy but report nothing — no controllers, no logs, no performance?**

The API is up and REST works, but it is not connected to the broker: bot status
arrives over MQTT, so a bot with no broker is a bot with nothing to say. Confirm
it, then re-seed:
```bash
make doctor # "Broker connection" says whether the API is connected
docker compose logs emqx | grep -i "auth-bootstrap"
make emqx-auth-reset # rewrites the bootstrap file and re-seeds the broker
```
A `Permission denied` on `/opt/emqx/etc/auth-bootstrap.csv` means the broker
could not read the file it seeds the account from, so no account was ever
created and the API's correct credentials came back `Not authorized`. The broker
still comes up healthy either way, which is why this shows up as missing bots
rather than as a broker error.

**API won't start?**
```bash
Expand Down
28 changes: 23 additions & 5 deletions doctor.sh
Original file line number Diff line number Diff line change
Expand Up @@ -372,28 +372,46 @@ if ! has_cmd curl; then
elif [ "$ENV_OK" != true ]; then
row warn "Reachability" "skipped — no .env to read credentials from"
else
# / is a deliberately public liveness endpoint (no auth_user dependency in main.py) --
# it returns 200 to everyone, correct credentials, wrong credentials, or none at all.
# Checking it here would validate nothing: use an actual protected route instead, so a
# non-200 here means what it says (bad credentials) rather than being unreachable in
# principle.
code="$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 3 --max-time 8 \
-u "${HB_USERNAME}:${HB_PASSWORD}" http://localhost:8000/ 2>/dev/null)"
-u "${HB_USERNAME}:${HB_PASSWORD}" http://localhost:8000/system/resources 2>/dev/null)"
case "$code" in
200)
row ok "Authenticated request" "200 from http://localhost:8000/"
row ok "Authenticated request" "200 from http://localhost:8000/system/resources"
# A 200 without credentials would mean auth is not being enforced.
anon="$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 3 --max-time 8 \
http://localhost:8000/ 2>/dev/null)"
http://localhost:8000/system/resources 2>/dev/null)"
if [ "$anon" = "200" ]; then
row fail "Auth enforcement" "an unauthenticated request also returned 200 — the API is answering anyone who can reach the port"
else
row ok "Auth enforcement" "unauthenticated request rejected ($anon)"
fi
# REST answering is not the same as the stack working: bot status, controller
# reports and logs all arrive over MQTT. With the broker refusing the API, the
# API stays 200-healthy while every deployed bot reports nothing at all, which
# is a much harder symptom to trace back to the broker than a named check.
mqtt="$(curl -s --connect-timeout 3 --max-time 8 \
-u "${HB_USERNAME}:${HB_PASSWORD}" http://localhost:8000/bot-orchestration/mqtt 2>/dev/null)"
if printf '%s' "$mqtt" | grep -q '"mqtt_connected"[[:space:]]*:[[:space:]]*true'; then
row ok "Broker connection" "the API is connected to the MQTT broker"
elif printf '%s' "$mqtt" | grep -q '"mqtt_connected"'; then
row fail "Broker connection" "the API is NOT connected to the broker — deployed bots will report no controllers, no logs and no performance. Check \`docker compose logs emqx\` for a \`Permission denied\` on /opt/emqx/etc/auth-bootstrap.csv (the account was never seeded; \`make emqx-auth-reset\` rewrites the file and re-seeds), then that BROKER_USERNAME/BROKER_PASSWORD in .env match what the broker was seeded with"
else
row warn "Broker connection" "could not read /bot-orchestration/mqtt — check \`docker compose logs hummingbot-api\`"
fi
;;
401|403)
row fail "Authenticated request" "$code — the API is up but USERNAME/PASSWORD in .env do not match what it is running with. Restart it after changing them: \`make deploy\`"
;;
000|"")
row fail "Authenticated request" "no response from http://localhost:8000/ — the API is not listening. \`make deploy\` (Docker) or \`make run\` (source)"
row fail "Authenticated request" "no response from http://localhost:8000/system/resources — the API is not listening. \`make deploy\` (Docker) or \`make run\` (source)"
;;
*)
row warn "Authenticated request" "unexpected HTTP $code from http://localhost:8000/ — check \`docker compose logs hummingbot-api\`"
row warn "Authenticated request" "unexpected HTTP $code from http://localhost:8000/system/resources — check \`docker compose logs hummingbot-api\`"
;;
esac
fi
Expand Down
Loading