fix(concourse): ship worker journald units to Loki

[Ardiea]

What are the relevant tickets?

N/A

Description (What does it do?)

Concourse worker AMIs never installed the vector journald config, so nothing logged on a worker reached Loki. That covers spot-watch, lifecycle-hook, preflight and the drain they invoke. The ASG churn behind #6412 had to be reconstructed from CloudTrail instead.

• Install concourse_logs.yaml on both node types. It was only added inside the web-node branch of deploy.py. concourse_metrics.yaml stays web-only.
• Set environment (and labels) on every line, not only on lines whose .message parses as JSON. Plain-text lines (the helper scripts, the reaper, some concourse.service output) shipped without it.
• Add node_application (concourse-web / concourse-worker, from APPLICATION in /etc/default/vector). Both node types share the application="concourse" stream, so this field is what tells a web line from a worker line.

Implementation details

Vector itself was already running on workers with credentials and ENVIRONMENT resolved. The worker user data writes /etc/default/vector, and QA Mimir has vector host metrics labelled application="concourse-worker", environment="operations-qa". Only the log source was missing.

The include_units names are correct as written: vector appends .service to a unit name that has no suffix. The helper scripts log via systemd-cat, which attributes each line to the calling unit; the reaper on the web nodes uses the same pattern and its lines arrive. The drain script runs as a child of spot-watch and lifecycle-hook, so its lines land under those two units.

Before this change, sum by (unit) (count_over_time({application="concourse"} | json unit="_SYSTEMD_UNIT" [7d])) on QA (and [24h] on Production) returned only concourse.service and concourse-worker-reaper.service.

How can this be tested?

• Ran vector validate --no-environment on the new config and vector vrl against a plain-text event and a JSON event, both in timberio/vector:0.47.0-debian. Both events come out with environment, labels and node_application set.
• The change only takes effect on workers launched from a rebuilt AMI. After the QA roll, run sum by (unit, env) (count_over_time({application="concourse"} | json unit="_SYSTEMD_UNIT", env="environment" [24h])). It should list concourse-worker-spot-watch.service, concourse-worker-lifecycle-hook.service and concourse-worker-preflight.service, each with env set.

Additional Context

Workers now also ship their own concourse.service lines. The existing INFO drop only applies to lines that parse as JSON, so plain-text level=info stderr from the worker (e.g. containerd) would pass through, all on one shared stream. That volume can't be measured before a worker runs this AMI. After the QA roll I'll check ingest split by node_application, and add a plain-text filter if it's significant.

🤖 Generated with Claude Code

[Copilot]

🟢 Approval recommended

The changes are narrowly scoped, both node types supply the required environment variables, and no blocking issues were identified.

0 open findings

What changed in this PR

Enables Concourse worker journald logs to reach Loki, improving visibility into worker lifecycle events.

Changes:

• Installs the log configuration on both node types while keeping metrics web-only.
• Adds environment metadata to plain-text and JSON logs.
• Adds node_application to distinguish web and worker logs.

File	Description
src/​bilder/​images/​concourse/​templates/​vector/​concourse_logs.yaml	Applies metadata consistently and identifies the node type.
src/​bilder/​images/​concourse/​deploy.py	Installs the shared log configuration on workers and web nodes.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.