Skip to content

fix(engine): migration deadlocks - #5357

Merged
mrkaye97 merged 1 commit into
mainfrom
mk/fix-migration-deadlock
Oct 28, 2026
Merged

mrkaye97 merged 1 commit into
mainfrom
mk/fix-migration-deadlock

Conversation

@mrkaye97

Copy link
Copy Markdown
Contributor

Description

Running this without a tx because the trigger creations can deadlock

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Checklist

Changes have been:

  • Documented (where applicable)
  • Added to CHANGELOG (where applicable) -- see Keep a Changelog

🤖 AI Disclosure

No AI

@vercel

vercel Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hatchet-docs Ready Ready Preview Oct 28, 2026 5:55pm UTC

Request Review

@github-actions github-actions Bot added the engine Related to the core Hatchet engine label Oct 28, 2026
@greptile-apps

greptile-apps Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 4/5

[Medium impact] The change reaches OLAP aggregate setup and the counts gathered from run, task-event, and event writes.

Keep the three status triggers atomic before merging so live upgrades cannot leave status counts wrong.

Summary

This PR changes the aggregate migration to release locks between trigger creations and makes object creation safe to repeat.

  • The OLAP aggregate migration creates its triggers outside a transaction.

Reviews (1) · Last reviewed commit: "fix: migration deadlocks" · Reviewed by Greptile

Comment on lines +238 to +244
CREATE OR REPLACE TRIGGER v1_agg_task_statuses_insert_trigger
AFTER INSERT ON v1_runs_olap
REFERENCING NEW TABLE AS new_rows
FOR EACH STATEMENT
EXECUTE FUNCTION v1_agg_task_statuses_insert_fn();

CREATE TRIGGER v1_agg_task_statuses_update_trigger
CREATE OR REPLACE TRIGGER v1_agg_task_statuses_update_trigger

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Status counts miss updates

If an existing engine keeps writing during migration, installing the three v1_runs_olap triggers in separate transactions can leave counts wrong. After the insert trigger commits but before the update trigger commits, a run can be inserted as QUEUED and then finish. The insert adds a queued count, but the update produces no matching change, so the run stays counted as queued. A failure while creating the update trigger extends this window.

The compactor only adds recorded changes; it cannot repair the missed update. Keep these three triggers in one transaction while installing triggers for other tables separately.

@mrkaye97
mrkaye97 merged commit d99a946 into main Oct 28, 2026
41 checks passed
@mrkaye97
mrkaye97 deleted the mk/fix-migration-deadlock branch October 28, 2026 18:18

This branch was successfully deployed

1 active deployment
Preview — b861468c Deployed Oct 28, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

engine Related to the core Hatchet engine

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants