Skip to content

fix(functions): reconcile an ambiguous async execution instead of retrying it (CLOUD-3RCB) - #31

Open
claudear wants to merge 1 commit into
mainfrom
fix/nats-parked-connection-cascade
Open

claudear wants to merge 1 commit into
mainfrom
fix/nats-parked-connection-cascade

Conversation

@claudear

Copy link
Copy Markdown
Contributor

Cascade of appwrite/appwrite#15510 (CLOUD-3RCB).

What changed upstream

The fix is entirely inside the server's NATS client (packages/nats/): a pooled connection to the broker that nothing had read since the broker closed it — during a rolling restart, where each node sends INFO {"ldm":true} and then drops the sockets parked on it — was written into before anything read it, so the first publish after a restart threw ConnectionException: Connection closed by server. Connection::settle() now processes whatever the broker already sent before any write-then-read path writes, and reconnects while that is still safe.

No SDK API changed. Nothing in packages/nats/ maps to a generated service, model or enum in appwrite/appwrite (the composer package, which is the PHP SDK). I diffed the SDK between 32.2.0 and 32.3.0 to confirm: the only additions are the account-devices, Vipps OAuth, proxy-rule and WAF routes, none of which this playground calls.

What reaches a caller

The failure the fix removes, which upstream's own "Known limitations" keeps possible in a narrower window. Queueing an async function run is two server-side steps — Create.php writes the execution row with status waiting, then $publisherForFunctions->enqueue(...) publishes the job. That enqueue is uncaught in the route, so a throw folds into general.php's default branch: 500, type general_unknown, message Server Error. The caller cannot tell "nothing happened" from "the row exists and the job was lost".

Retrying that blindly is what hurts: when the row was written, the retry queues a second run and the function does its work twice.

createExecution() takes no caller-chosen execution ID, so there is nothing to retry as. It does take the path the function is called on, and the server stores it — query string included — as the execution's requestPath, which listExecutions() returns. So createAsyncExecution() tags its attempt there and asks what was recorded before deciding:

  • nothing carries the tag → the attempt left no trace, retry it (with a short backoff, up to ENQUEUE_ATTEMPTS);
  • something does → the run is registered; report it rather than queueing another. One still waiting long afterwards lost its publish, and replacing it is a decision for the caller, not a retry.

Anything below 500 is an answer, not an ambiguity, and is still raised unchanged.

Changes

  • composer.json: appwrite/appwrite ^32.2.0 → ^32.3.0.
  • playground.php: createAsyncExecution() tags and reconciles as above; new findTaggedExecution() helper over listExecutions().
  • tests/queue-restart.php: runs the playground against a stub told to fail one async execution, once with the row recorded first and once without, asserting the request sequence each produces — create async, list, create async, get when nothing was recorded, and create async, list (no second run) when something was.
  • tests/stub-server.php: STUB_EXECUTIONS_FAIL / STUB_EXECUTIONS_RECORD_ON_FAILURE, and the execution routes now record what they create so a list can be read back. counters() renamed to state() now that it holds more than counts.
  • tests/stub.php: Stub::start() takes extra environment for the server.
  • README.md: a "Queued work" section.

Verified

In a php:8.5-cli container (local PHP is 8.4), with appwrite/appwrite resolving to 32.3.0:

  • composer install clean, php -l over every file, composer test green — 21 playground calls over 26 requests, the 13 abuse-limit checks, and the 8 new ones.
  • docker build . succeeds.
  • Mutation check: stubbing findTaggedExecution() to always return null turns the second case into create async, list, create async, get and fails four assertions — i.e. the test does catch the double-queue it guards against.

🤖 Generated with Claude Code

…rying it

appwrite/appwrite#15510 (CLOUD-3RCB) fixed the server's NATS client writing
into a pooled connection the broker had closed while it sat idle, which made
the first publish after a broker rolling restart throw. The fix is entirely
server-internal and changes no API this SDK exposes.

What reaches a caller is the failure it caused, and still can in the narrower
window upstream documents: queueing an async function run writes the execution
row and only then publishes the job, so a failed publish answers
`500 general_unknown` without saying whether the row was written. Retrying that
blindly queues a second run of the function.

createExecution() takes no caller-chosen execution ID, so createAsyncExecution()
tags its attempt in the execution's path — which the server stores as
`requestPath` — and asks listExecutions() what was recorded before deciding:
retry when nothing carries the tag, and hand back the execution left behind when
something does.

- bump appwrite/appwrite to ^32.3.0
- tests/queue-restart.php drives both halves against the stub, which grew
  STUB_EXECUTIONS_FAIL and STUB_EXECUTIONS_RECORD_ON_FAILURE plus a recorded
  execution list to answer them

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@appwrite

appwrite Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

🟢 Tier S · Ready to merge

This change upgrades the PHP SDK and updates async execution handling to tag an attempt, inspect recorded executions after an ambiguous server error, and retry only when no matching execution is found. It also adds stub-server support and tests for both recorded and unrecorded failures, and documents the queued-work behavior.

Verdict New comments Fixed Still open
✅ Approved 0 0 0
📂 Walkthrough · 6
File Change
README.md Documents ambiguous async queue failures and the new reconciliation behavior.
composer.json Updates the Appwrite SDK requirement and registers the queue-restart test.
playground.php Tags async execution requests, searches recent executions after 5xx responses, and retries missing attempts with backoff.
tests/queue-restart.php Exercises the recorded and unrecorded failure cases against the stub server.
tests/stub-server.php Adds stateful execution creation/list routes and configurable async failures.
tests/stub.php Allows tests to pass extra environment settings to the stub server.
🔇 Filtered out · 1

Findings Hansi considered but did not post.

Finding Why
The 25-item lookup can miss the execution and queue a duplicate Verifier: The lookup runs right after the failed call, so the tagged row would have to be pushed past 25 newer executions within that short window. This is a playground script, and the finding offers no concrete scenario where that happens. I'm treating it as a speculative edge case, not a reachable bug.

Reviewed 3675366 · Details · Comment @appwrite review to re-run, or mention @appwrite with a question.

@appwrite appwrite Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Tier S · Looks good to merge. Summary

@ArnabChatterjee20k

Copy link
Copy Markdown
Member

the change doesn't make sense here since its an internal issue. how playground php is related ? @claudear

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants