Skip to content

Send GA4 Ping from Node with leader election - #498

Open
tiagolobocastro wants to merge 2 commits into
developfrom
ping-ld
Open

tiagolobocastro wants to merge 2 commits into
developfrom
ping-ld

Conversation

@tiagolobocastro

Copy link
Copy Markdown
Member
test(analytics): cover the leader election and ping races

Add unit tests for the analytics lease election, using an in-memory API
server which enforces the resourceVersion, and for the ping loop, using
an in-memory configmap:

- concurrent acquires of a free lease: one wins, the other gets a 409
- an unexpired lease is not taken, an expired one is taken over
- a failing leader steps down right away, the others take over only
  once the lease expires
- a paused leader (stale renewal) neither leads nor pings, while the
  new leader carries on the schedule without a new install event
- a ping is skipped and rescheduled if another leader pinged meanwhile
- a regained leader resumes from the other leader's last ping

fix(analytics): send ping from the node plugin lease holder

The install/ping events were sent from the controller, so disabling the
components which need it (eg: resize) stopped them altogether. Sending
them from every node plugin instead would show up as duplicates.

Run the analytics from the node plugins, where a Lease based leader
election (rawfile-analytics) picks a single sender. The lease lasts 180s
and is renewed every 60s; on any error leadership is dropped right away.
Followers take over once the lease expires, resuming the ping schedule
from the timestamps in the rawfile-config configmap, so no new install
event is sent.

To avoid duplicate pings, before sending one we also check that the
lease was renewed recently (a paused leader may resume still believing
it leads), and re-read the last ping timestamp from the configmap in
case another leader pinged meanwhile.

The analytics ClusterRole gains create/get/update on leases.

The install/ping events were sent from the controller, so disabling the
components which need it (eg: resize) stopped them altogether. Sending
them from every node plugin instead would show up as duplicates.

Run the analytics from the node plugins, where a Lease based leader
election (rawfile-analytics) picks a single sender. The lease lasts 180s
and is renewed every 60s; on any error leadership is dropped right away.
Followers take over once the lease expires, resuming the ping schedule
from the timestamps in the rawfile-config configmap, so no new install
event is sent.

To avoid duplicate pings, before sending one we also check that the
lease was renewed recently (a paused leader may resume still believing
it leads), and re-read the last ping timestamp from the configmap in
case another leader pinged meanwhile.

The analytics ClusterRole gains create/get/update on leases.

Signed-off-by: Tiago Castro <tiagolobocastro@gmail.com>
Add unit tests for the analytics lease election, using an in-memory API
server which enforces the resourceVersion, and for the ping loop, using
an in-memory configmap:

- concurrent acquires of a free lease: one wins, the other gets a 409
- an unexpired lease is not taken, an expired one is taken over
- a failing leader steps down right away, the others take over only
  once the lease expires
- a paused leader (stale renewal) neither leads nor pings, while the
  new leader carries on the schedule without a new install event
- a ping is skipped and rescheduled if another leader pinged meanwhile
- a regained leader resumes from the other leader's last ping

Signed-off-by: Tiago Castro <tiagolobocastro@gmail.com>
@tiagolobocastro
tiagolobocastro requested a review from a team as a code owner November 6, 2026 19:55
@mergify

mergify Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant