Release builds omit hashicorpmetrics tag and hide dependency metrics

[eirriel]

Overview of the Issue

The official Consul 2.0.3 release binary does not expose Raft, Serf, and Memberlist metrics through /v1/agent/metrics, even though the same workload exposes them in 1.22.7. Representative missing metrics include consul_raft_fsm_apply, consul_raft_peers, consul_serf_member_join, and consul_memberlist_gossip.

This appears to be a release-build regression introduced by the hashicorp/go-metrics migration in #23635. Consul now initializes the github.com/hashicorp/go-metrics global sink, while Raft, Serf, and Memberlist use its compat package. The compatibility package requires the hashicorpmetrics build tag to select that same global sink; without it, dependency metrics are sent to the separate armon/go-metrics global and never reach Consul's configured sinks.

The Makefile and test/distribution workflows set hashicorpmetrics, but both direct artifact builds in .github/workflows/build.yml invoke go build without it. The same untagged commands remain on main and release/2.0.x.

This is consistent with the earlier work in #22080, which explicitly added the tag so Raft, Serf, and Memberlist metrics would flow through hashicorp/go-metrics.

Reproduction Steps

1. Start 1.22.7 and 2.0.3 dev agents with identical telemetry configuration:

podman run -d --name consul-metrics-v1 -p 18500:8500 \
  -e 'CONSUL_LOCAL_CONFIG={"telemetry":{"prometheus_retention_time":"10m","disable_hostname":true}}' \
  docker.io/hashicorp/consul:1.22.7 agent -dev -client=0.0.0.0 -node=metrics-test

podman run -d --name consul-metrics-v2 -p 28500:8500 \
  -e 'CONSUL_LOCAL_CONFIG={"telemetry":{"prometheus_retention_time":"10m","disable_hostname":true}}' \
  docker.io/hashicorp/consul:2.0.3 agent -dev -client=0.0.0.0 -node=metrics-test

2. Generate identical Raft activity:

for port in 18500 28500; do
  for i in $(seq 1 20); do
    curl -fsS -X PUT --data "value-${i}" \
      "http://127.0.0.1:${port}/v1/kv/metrics/key-${i}" >/dev/null
  done
done

3. Compare dependency metric names:

comm -23 \
  <(curl -fsS 'http://127.0.0.1:18500/v1/agent/metrics?format=prometheus' | \
    perl -ne 'print "$1\n" if /^(consul_(?:raft|serf|memberlist)_[A-Za-z0-9_:]+)/' | sort -u) \
  <(curl -fsS 'http://127.0.0.1:28500/v1/agent/metrics?format=prometheus' | \
    perl -ne 'print "$1\n" if /^(consul_(?:raft|serf|memberlist)_[A-Za-z0-9_:]+)/' | sort -u)

The 2.0.3 endpoint omits the affected dependency metrics. The exact set varies with timing and activity, but the reproduction consistently includes Raft, Serf, and Memberlist families.

The official 2.0.3 binary's go version -m output contains no -tags build setting. Rebuilding the same v2.0.3 source with the following command records -tags=hashicorpmetrics and restores the missing metrics in the same Podman test:

go build -tags hashicorpmetrics -o consul .

Consul info for both Client and Server

Both reproductions use a single dev-mode server agent.

Consul v1.22.7
Revision c18bcb9d

Consul v2.0.3
Revision d0f2be93

Equivalent telemetry configuration for both agents:

telemetry {
  prometheus_retention_time = "10m"
  disable_hostname          = true
}

Operating system and Environment details

Linux amd64
Podman 5.8.4
Official docker.io/hashicorp/consul images

Log Fragments

Both agents start successfully and process the test writes. No metric-related error is logged because the dependency metrics are accepted by the wrong compile-time-selected global sink.

[alexdulin]

Hi, any update on this issue? This is leading to missing metrics that the official documentation even describes as key metrics to monitor: https://developer.hashicorp.com/consul/docs/monitor/telemetry/agent#key-metrics

It seems pretty irresponsible to have a community contribution in place for three weeks unmerged to fix missing critical metrics for monitoring consul health.