Skip to content

operations-runbook.md

Latest commit

 

History

History
360 lines (279 loc) · 35.6 KB

operations-runbook.md

File metadata and controls

360 lines (279 loc) · 35.6 KB
Raw

Operations runbook

This runbook covers a self-hosted Anchorage deployment on Cloudflare. Review the Deployment reference and Security threat model before production.

Operating roles

Define people and escalation paths for:

  • application owner: business outcome and policy;
  • platform operator: Worker, D1, Durable Objects, R2, Queue, secrets, and incidents;
  • connector owner: manifest accuracy, vendor behavior, credentials, and idempotency;
  • reviewer: approval queue and separation of duties;
  • security responder: identity, provisioning isolation, audit, and vulnerability response.

These operating roles are not the same as breakwater's five application role labels.

Validate a release

From the repository:

corepack enable
pnpm install --frozen-lockfile
pnpm docs:check
pnpm docs:api
pnpm lint
pnpm typecheck
pnpm test
pnpm build
pnpm test:packed-breakwater
pnpm test:packed-flowsafe-agent-host
pnpm test:packed-flowsafe-provisioning
pnpm --filter @proofoftech/flowsafe test:signals-client-export
pnpm --filter @proofoftech/flowsafe typecheck:react18
pnpm --filter @proofoftech/flowsafe spike:verify
pnpm --filter anchorage-agent-starter test
pnpm --filter anchorage-agent-starter typecheck

The deterministic spike starts workerd, suspends runs, kills the process, restarts on the same D1 state, and proves resume plus fail-closed security cases. Do not replace it with an in-process-only test.

Run the optional live-model proof before changing model providers or durable-agent integration:

SPIKE_LLM_MODEL_ID='provider/model' \
SPIKE_LLM_API_KEY='secret' \
pnpm --filter @proofoftech/flowsafe spike:verify:llm

Deploy

The baseline Worker is in packages/flowsafe/deploy/. The advanced agent host is in packages/agent-starter/.

Local baseline:

pnpm --filter @proofoftech/flowsafe deploy:dev

Production baseline:

pnpm --filter @proofoftech/flowsafe deploy:cf

Before deployment:

  1. Confirm the Worker script name and D1 database name are unique to the deployment tag, and confirm all append-only Durable Object migrations. Never reuse a legacy pooled script name: it retains the old Durable Object namespaces.
  2. Confirm DEPLOYMENT_TENANT matches the singleton sentinel in every bound D1 database.
  3. Confirm DEPLOYMENT_IDENTITY_SECRET is distinct per deployment and reaches every Worker-to-Durable-Object topology.
  4. Confirm every enabled route has its required namespace or storage domain.
  5. Confirm the MAINTENANCE singleton binding and append-only migration exist, then authenticate POST /admin/ensure-maintenance.
  6. Confirm the identity verifier fails closed with missing configuration.
  7. Confirm internal-object, maintenance-admin, stream-ticket, OAuth, model, webhook, connector, and SIEM secrets are distinct.
  8. Confirm R2 artifact storage is passed to runtime, retention, and the deployment decommission inventory.
  9. Confirm audit Queue dead-letter policy and collector authorization.
  10. Confirm a rollback bundle remains compatible with the deployed database and migration tags; for a rollback to an earlier flowsafe release, also follow the rollback notes in this section.

Before the first guarded agent-host deployment, drain approvals created by the legacy raw-agent target. Approve or reject them through the existing API so the state machine and audit record every transition, then verify that no open records remain:

pnpm --dir packages/agent-starter exec wrangler d1 execute anchorage-agent-starter-acme \
  --remote \
  --config wrangler.jsonc \
  --command "SELECT COUNT(*) AS open_agent_approvals FROM flowsafe_approvals WHERE workflow_id = 'durable-agentic-loop' AND status IN ('pending', 'claimed', 'escalated');"

Deploy only when open_agent_approvals is zero. Do not close these records through direct SQL mutation.

During a gradual deployment from a flowsafe version earlier than 0.24.0, a Worker still on the earlier version answers a keyed start replay of an INTERRUPTED run with 503 persisted start is not readable, because that version does not recognize the INTERRUPTED error code. Retry the replay once every Worker version runs 0.24.0 or later; the run itself is unaffected.

A deploy from a flowsafe version earlier than 0.25.0 leaves outgoing agent legs that do not touch their run row, so the thread object repairs an agent start that goes more than six minutes without a write as StartOutcomeUnknown; Durable agents describes the wait and the stamp. Deploy when no agent start is running, and check the effects of any agent run that reads StartOutcomeUnknown before you start another. From a version earlier than 0.24.0, drain agent starts first: the outgoing leg writes without the settled-row guard, so its write can land over the repair.

Rolling back to 0.24.x from 0.25.0 or later brings back the 0.24.x agent start recovery: the thread object repairs a pending agent start about a minute after admission, without waiting for its leg to stop and without the stamp. An agent leg still running on the outgoing instance reads that repair as live, and its final write lands over it after the run's journal, run record and ownership are cleared. Roll back only when no agent start is pending, with the deployment's tablePrefix in front of the table name:

wrangler d1 execute <database> --remote --config wrangler.jsonc \
  --command "SELECT run_id FROM <prefix>mastra_workflow_snapshot WHERE workflow_name = 'durable-agentic-loop' AND CASE WHEN json_valid(snapshot) THEN json_extract(snapshot, '$.status') = 'pending' ELSE 0 END"

After a rollback, check each agent run started in the hour before it: one that read StartOutcomeUnknown and later reads another status lost its repair; terminate it if it is suspended, and check the agent's external effects.

A flowsafe version earlier than 0.26.0 stored run snapshots nested past SQLite's JSON depth limit. The deadline sweep, retention and the drain inventory cannot read such a row, so the run has no deadline, is never purged, and keeps a drain proof non-empty. List them in each database after the upgrade, with the deployment's tablePrefix in front of the table name (for example flsmastra_workflow_snapshot for the prefix fls):

wrangler d1 execute <database> --remote --config wrangler.jsonc \
  --command "SELECT workflow_name, run_id FROM <prefix>mastra_workflow_snapshot WHERE NOT json_valid(snapshot)"

Terminate each listed run that is not terminal. A terminate of such a run whose leg writes continuously answers 503 and records nothing; retry it once the leg stops writing. Retention never removes these rows: remove a terminal one by hand from the tables RUN_TTL_PURGE_TABLES and RUN_TTL_FLOWSAFE_PURGE_TABLES name, after deleting its artifacts. During a gradual deployment from a version earlier than 0.26.0, a Worker still on the earlier version answers a keyed start replay of a run that failed with RUN_STATE_NOT_STORABLE with 503 persisted start is not readable; retry it once every Worker runs 0.26.0 or later.

Before 0.26.1, a schedule target value nested about 1,000 levels deep could leave a deferred fire whose metadata SQLite cannot parse; every later tick then logged schedule-tick-error with malformed JSON, and no schedule in the deployment fired. The upgraded tick resolves such a fire from the status lookup without sending it again (0.26.1 and 0.26.2 settle it without the lookup), settles it as failed with reason invalid-deferred-dispatch when the lookup throws, and fails each later fire of a target nested more than 256 levels deep with reason input-too-deep. A workflow or agent fire settled with reason invalid-deferred-dispatch may still have started its run, and its statusError names why the lookup failed: check the fire's external effects before running its work again. During a gradual deployment, a Worker on an earlier version still accepts such a target and still stops on its fires, so finish the rollout first. Before a rollback to an earlier version, update or delete each schedule whose target an earlier tick would copy into a fire record SQLite cannot parse, whether or not it has fired since the upgrade. The query nests each stored target four levels deeper, as a fire record does with a margin, and lists those SQLite cannot parse:

wrangler d1 execute <database> --remote --config wrangler.jsonc \
  --command "SELECT id FROM <prefix>mastra_schedules WHERE NOT json_valid('[[[[' || target || ']]]]')"

Provision a deployment

Choose one stable deployment tag per organization. It must match ^[a-z0-9]{3,32}$ and is provisioning material, not a display name or request claim.

pnpm --dir packages/flowsafe provision:deployment -- \
  --database anchorage-flowsafe-acme \
  --tag acme \
  --remote \
  --config deploy/wrangler.jsonc
pnpm --dir packages/flowsafe exec wrangler secret put \
  DEPLOYMENT_IDENTITY_SECRET \
  --config deploy/wrangler.jsonc
pnpm --dir packages/flowsafe exec wrangler secret put \
  MAINTENANCE_ADMIN_SECRET \
  --config deploy/wrangler.jsonc

Run the provisioning command immediately after D1 creation and before application migrations. It verifies the strict singleton schema, is idempotent for the same tag, refuses to re-stamp a database owned by another deployment, and refuses an unowned database with application tables. Do not expose it as a public route. Generate distinct 32–256 visible ASCII deployment-identity and maintenance-admin credentials for each deployment.

Then:

  1. Allocate a dedicated tag-suffixed Worker script name, D1 database, and set of Durable Object namespaces. Never deploy over a retained script from another organization or a legacy pooled deployment. Allocate dedicated R2, Queue, and secret resources where those domains are enabled.
  2. Seed the new D1 database, then apply application migrations.
  3. Set DEPLOYMENT_TENANT to the seeded tag and bind the internal credential only to that deployment's Worker and Durable Objects.
  4. Record the one-to-one organization-to-resource mapping in the control plane.
  5. Deploy without public traffic, authenticate POST /admin/ensure-maintenance, and confirm GET /admin/maintenance-status has a non-null alarmAt.
  6. Call a protected route and confirm the sentinel check succeeds before issuing user credentials.
  7. Issue verified credentials that carry only the actor id, role, and any application-specific identity claims.
  8. Start a smoke run and confirm the host returns opaque run and thread ids without an organization prefix.
  9. In a scratch deployment, bind the Worker to a database with a different sentinel and confirm protected routes return 503 without reading application data.
  10. Bind a scratch Worker namespace to another deployment's Durable Objects and confirm the wrong internal credential is rejected before object storage is read.
  11. Restore the correct bindings and repeat the smoke run.

Never bind two organizations to one data-plane resource set. Never reuse a tag while retained resources or audit records could make its ownership ambiguous.

Decommission a deployment

  1. Stop new traffic, scheduled work, and credential issuance for the deployment.
  2. Revoke active credentials and wait for short-lived credentials and stream tickets to expire.
  3. Disable provider subscriptions and external webhooks when the vendor requires explicit cleanup.
  4. Record the decommissioning authorization and immutable resource inventory.
  5. Export records that policy requires you to retain, including audit evidence.
  6. Remove routes and delete the Worker so no code can open the bound stores.
  7. Delete the deployment's D1 databases, Durable Object namespaces and storage, R2 buckets, Queue bindings or dedicated queues, and secrets.
  8. Verify the former hostname and every recorded resource identifier are absent or inaccessible.
  9. Retain a control-plane tombstone that records the tag, authorization, completion time, and deletion results.

There is no in-database organization purge. Abandoned live runs and approvals disappear only when the physical database is deleted, so revoke traffic before deletion and treat the resource inventory as the completion checklist.

Retire legacy Cloudflare resources

Use this procedure when the original fleet record or supported decommission workflow cannot authorize a legacy resource set. Treat the deletion plan, verifier, runtime, and captured provider evidence as one reviewed authority package.

  1. Record the exact account, Worker service and version identities, database and namespace IDs, storage names, ingress, automation, and explicit exclusions. Use immutable provider IDs where the API supports them. For a name-addressed mutation, freeze provider writes and recheck the name-to-ID mapping immediately before the mutation.
  2. Freeze repository pushes, provider edits, Builds triggers, credential changes, and every other writer for the deletion window. Run reviewed tooling from an isolated, hash-pinned runtime instead of a mutable shared checkout or dependency tree.
  3. Export every state domain that retention policy requires. Store exports outside Git, restrict their permissions, record their size and digest, and prove that each export restores before deleting anything.
  4. Capture a complete reverse-dependency and ingress inventory. Include all deployable retained Worker versions, Pages production and preview configurations and deployments, dispatch scripts, service and Durable Object bindings, routes, custom domains, workers.dev, preview URLs, cron triggers, Queue consumers, Email Routing rules, tail consumers, Builds triggers, and conventional Workers Sites KV namespaces. Inventory application R2 buckets and every Durable Object namespace and stored object that the deletion can remove.
  5. Fail closed on incomplete, malformed, duplicated, or ambiguous provider response fields. A disabled route, email action, consumer, or build trigger still counts as a reference because an operator can re-enable it.
  6. Review the exact authority package and preflight evidence before mutation. Record independent digests for the authority and evidence, and verify both at every destructive boundary. Regenerating any preflight artifact starts a new review cycle and requires a new approved evidence digest.
  7. Remove all ingress and background execution, then prove their absence. Recheck application-owned R2 buckets after traffic quiesces; stop while any bucket remains attached or nonempty. Evacuate or explicitly purge application data through an application-owned procedure, never a generic automatic purge.
  8. Delete the Worker without bypassing provider dependency checks. If Worker deletion also removes Durable Object or Workers Sites storage, enumerate and export those exact resources first. Treat a timeout or lost response as ambiguous: read back the exact identities before deciding whether to retry.
  9. Prove the Worker and every authorized cascading resource are absent. Repeat the complete reverse-dependency scan before deleting independent storage.
  10. Delete D1 and other independent storage by immutable provider ID. Keep interactive target readback enabled for irreversible commands; do not combine mutable-name resolution with a confirmation bypass.
  11. Revalidate the replacement deployment after each destructive boundary and at completion. Check its exact version, bindings, ingress, automation, health, protected-route behavior, assets, and zero-write sentinel. Retain the export, approved digests, provider evidence, and deletion results according to policy.

Do not commit provider tokens, account-specific resource IDs, captured provider responses, data exports, or one-off deletion tooling. Commit only reusable procedures and invariant-checking code that belongs to the supported control plane. Cloudflare Worker versions preserve code and configuration but not storage state, so retained versions remain part of the reverse-dependency proof. See Workers versions and deployments, D1 Wrangler commands, and Durable Object class exports for current provider behavior.

Alarm-driven duties

The maintenance singleton runs separate alarm invocations for:

  • run deadline expiry;
  • approval SLA sweep;
  • retention purge;
  • schedule fire tick, when schedules are enabled;
  • notification dispatch, when the host wires it.

A Workers CPU-limit termination kills the isolate and bypasses JavaScript catch. The object persists the next alarm before each duty and runs one duty per invocation, so one runaway class cannot permanently starve another.

The purge invocation contains independent failure boundaries for each configured domain:

  • terminal workflow snapshots and their artifacts;
  • approved/rejected approvals;
  • idle threads and their messages;
  • terminal notifications;
  • thread state and goals;
  • schedule trigger history;
  • terminal background tasks;
  • deployment-owned extra data.

Provider polling runs through the singleton provider-host Durable Object's alarms. Subscription mutations should call the provider-host topology's reconcilePolling() after committing D1. A reconciliation failure means the mutation is already durable; retry reconciliation rather than trying to undo the row.

Deferred schedule fires

A schedule fire stays deferred while the tick cannot learn its outcome. Each reconcile pass asks its target again, the drain inventory counts it under schedule-deferred-dispatches, and deleting its schedule answers 202 with pending: true until the tick settles the schedule's last deferred fire. Add schedules describes when the tick settles a threadless agent start. A fire recorded failed with reason dispatch-unresolved, including one an earlier version left deferred, may have started its run, which was then cancelled, timed out or purged before a pass saw it, or may still be running with its ownership commit stuck behind a start recovery fault, so check the agent's external effects before running its work again. A deferred workflow fire is recorded failed with reason start-error-confirmed once the status lookup finds no run. A pass that runs more than RUN_RETENTION_DAYS after the fire, as any pass can with 0, finds none for a run that completed and that retention removed, so check the workflow's effects before running its work again. List the fires still deferred, with the deployment's tablePrefix in front of the table name:

wrangler d1 execute <database> --remote --config wrangler.jsonc \
  --command "SELECT id, scheduleId, runId, datetime(actualFireAt / 1000, 'unixepoch') AS firedAt, error, CASE WHEN json_valid(metadata) THEN json_extract(metadata, '$.dispatchRef.target') END AS target, CASE WHEN json_valid(metadata) THEN json_extract(metadata, '$.dispatchRef.mode') END AS mode, CASE WHEN json_valid(metadata) THEN json_extract(metadata, '$.statusError') END AS statusError FROM <prefix>mastra_schedule_triggers WHERE outcome = 'deferred' ORDER BY actualFireAt"

statusError names why the last pass could not decide the fire. Fix what it names, and the next pass settles the fire and completes a pending deletion. Do not settle a fire through direct SQL: the store completes a pending deletion only when the tick settles the fire.

A fire whose claim is stored without a dispatch record, as when the run-cap check throws after the claim, or whose dispatch record does not parse, cannot be sent or looked up. The next pass records it failed with reason invalid-deferred-dispatch and audits a schedule.fire event with its schedule's target as the store reads it. When its schedule cannot be read or is no longer stored (as when it is pending deletion), the pass logs schedule-tick-bookkeeping-error in place of the event. When the fire was the last deferred fire of a schedule pending deletion, the settlement also completes the deletion, which removes the fire's trigger history.

Retention policy

Data Eligible
Workflow snapshots Terminal statuses older than run TTL
Approval records Approved or rejected older than approval TTL
Threads and messages Thread idle longer than optional thread TTL
Notifications Terminal rows older than optional notification TTL
Thread state and goals Rows older than optional state TTL
Schedule triggers Fires older than optional trigger TTL
Background tasks Terminal rows older than task TTL
Schedules and subscriptions Never by TTL; authorized deletion or deployment decommissioning
Resources and working memory Never by TTL; explicit host teardown or deployment decommissioning
flowsafe_resource_owners Run retention and schedule deletion release their claims. Thread and resource claims require explicit host teardown or deployment decommissioning

Open approval requests and suspended/running runs remain at any age.

Set TTLs longer than the operational recovery window. A terminal approval can disappear before a delayed resume redrive; the next reconciliation may need a new human decision. Idle-thread retention removes memory rows but keeps thread and resource ownership because standing wake sources can recreate the memory. TTL retention does not replace an authorized deletion workflow or the resource inventory used during deployment decommissioning.

Health checks after deploy

curl https://worker.example/healthz

curl https://worker.example/workflows \
  -H 'authorization: Bearer operator-token'

curl https://worker.example/api/approvals \
  -H 'authorization: Bearer reviewer-token'

Exercise:

  1. start a workflow;
  2. observe an approval;
  3. attempt self-decision and expect the configured result;
  4. approve as an independent reviewer;
  5. observe run completion;
  6. start a guarded agent and observe its suspension;
  7. confirm every public agent resume path returns 404;
  8. restart the Worker, approve as a different reviewer, and confirm the original requester remains the execution principal;
  9. disconnect the live socket and confirm polling catches up;
  10. evict agent stream replay and confirm status remains authoritative after the stream returns 409;
  11. if enabled, send a signal, fire a schedule, run a task, and reconcile a provider subscription.

Audit and alerting

At minimum, alert on:

  • authentication/configuration failures;
  • deployment sentinel or internal Durable Object credential missing, malformed, or mismatched;
  • control-plane drift from the one-to-one organization/resource mapping;
  • connector denied or error decisions;
  • approval SLA escalation;
  • approval resume failures;
  • audit sink and Queue retry/dead-letter activity;
  • maintenance bootstrap, stale-alarm recovery, and duty failures;
  • any purge duty failure;
  • provider alarm/reconciliation failure;
  • schedule skip due to run cap or invalid stored data;
  • foreign run/thread probes;
  • repeated signal or webhook rate-limit denial;
  • idempotency store degradation after a successful side effect;
  • Agent CLI timeout, process-tree termination failure, spawn failure, or non-zero exit;
  • demo global run budget near exhaustion.

Audit records are security evidence. Queue depth and SIEM ingestion status must be monitored separately from application health because sink failure is intentionally contained.

Common incidents

Symptom Check Action
Every protected route returns 401 Identity secret and verifier parsing Restore valid verifier configuration; do not add an unauthenticated fallback
Protected routes return 503 before routing DEPLOYMENT_TENANT, DEPLOYMENT_IDENTITY_SECRET, D1 sentinel, or a wrong database/object binding Restore the verified one-to-one bindings and secret. Never overwrite the sentinel to adopt a database
A foreign resource returns 403 instead of 404 Route ownership ordering Treat as an information-oracle regression and fix the shared boundary
Start returns duplicate-run conflict Client retried after a response loss Query the server-minted run id; do not mint a replacement blindly
Run stays suspended after approval Decision result and resume outcome Read stored decision, status, and audit; redrive through trusted resume. A resume outcome with code: 'SUSPENSION_CHANGED' is refused on every redrive: decide the approval of the step's current suspension instead
A reviewer is refused the approval of a step's current suspension as having advanced the run at an earlier gate Whether that reviewer's earlier approval of the step answered SUSPENSION_CHANGED on resume Expected: the separation-of-duties check counts that approval although it advanced nothing. Have another reviewer decide, or use a configured exemption
Run reads failed with errorEnvelope.code: 'INTERRUPTED' run-leg-interrupted log line (workflow, run, leg trigger) and the step that was in flight Its leg stopped mid-step and nothing re-ran it. Check the step's external effects before starting a new run; split a step that runs past one invocation
Run reads failed with errorEnvelope.code: 'RUN_STATE_NOT_STORABLE' The step that ran last and the data it returned, or for an agent the tool results and the thread's recalled messages A step or tool returned data nested past the depth SQLite parses; the run stopped at that write and is not re-executed. Check the step's external effects, then bound or flatten the data before starting again; for an agent thread, see durable agents
Run stays running with no progress Run row updatedAt, run-leg-reset log lines, and the workflow storage The run object settles it about six minutes after its last write. On workflow storage other than FencedWorkflowsStorageD1, or for a leg run by an older flowsafe version, terminate the run
An agent run's status or terminate route answers 503 RUN_START_PENDING and the thread logs agent-start-recovery-pending lines The runId in the lines, and its run row's status and updatedAt Expected for about seven minutes after the start's last touch: the thread object then repairs the start as StartOutcomeUnknown and the lines stop. If the same runId is still logged after that, its journal does not repair on its own. Check that the thread's runtime has the execution fence the start was prepared under: restore the wiring and the next wake repairs it. A journal prepared on an explicitly unfenced runtime never repairs a pending start, so check the agent's external effects and apply a tested, version-specific repair only as Recovery rules allow
The thread logs agent-start-recovery-failed lines for one runId The error in the lines; that the agent is still in the catalog with the same workflow; that startIdempotency and the execution fence are wired; that a scheduled agent's host provides discardScheduleDispatch; whether the run has a row in the workflow snapshot table Fix the configuration the error names. The line carries only the outermost error, and run start recovery is unresolved names none: an unwired startIdempotency, a prepared start whose run row is absent and a failure inside the runtime's recovery all read that way, so check each item in the previous column. The next wake retries the journal. A prepared start whose run row is absent is the next row
Every agent start on one thread answers 409 thread is blocked by run '<runId>', and that run's status and terminate routes answer 404 Whether <runId> has a row in the workflow snapshot table, and the thread's agent-start-recovery-failed lines for it The start was prepared but its run row was never written (an isolate eviction, or an admission error that is not a definitive refusal), or a purge by an earlier flowsafe version, or by a direct caller that passes no resourceOwnerTable, removed the row before the thread recorded that it read it. A start journal an earlier flowsafe version wrote records no read, so the same holds when retention removes its run before the thread's first wake on this version, and for a thread already blocked this way when the deployment upgraded. The thread keeps the start's journal and run record so that an unkeyed retry cannot execute the run twice, and stays blocked; flowsafe does not clear such a journal. Start the work on another thread; do not delete the journal or the run record by hand
A run's dispatch-status route answers 503 EXECUTION_FENCE_UNREADABLE with run start recovery is unresolved, a schedule fire of the run stays deferred, and the run object wakes every minute Whether the run has a row in the workflow snapshot table The run object prepared the start, and the run's row was never written or was removed before the object recorded that it read the start's outcome, by the same causes as the previous row. The object keeps the start's journal so that a start whose admission may still land never runs twice, and flowsafe does not clear it. Check the run's external effects before running its work again; do not delete the journal by hand
A suspended run has no open approval bound to its current gate's suspension, and one of the step's approvals reads rejected with superseded: stale suspension fingerprint The run's approvals and its suspendedAt and resumeCount A status read taken before a timeout resume superseded that gate's approval on a version earlier than 0.26.0, and the gate cannot be filed again. Terminate the run, or resume it through the raw resume route, which carries no approval grant, so a connector-protected step denies. Do not reopen the record through direct SQL
Durable agent resume fails after eviction Prepare/observe registration and memory binding Use resumeViaRuntime() through the thread topology; never raw inherited resume
Agent stream returns 409 In-memory replay cache was evicted or the isolate restarted Read the authoritative status route; reconnect only for events still present in the configured cache
Connector says approval missing after an approved record Fingerprint, connector id, workflow, run, and deployment store Confirm the record matches current step, suspendedAt, resumeCount, and exact connector id
Connector repeats an external write Idempotency key, shared store, pending TTL Fix the business key/store reach/TTL; inspect vendor idempotency evidence
Connector denies idempotency-key-migration Legacy writer versions, inspectLegacyConnectorIdempotency() outcome, external tuple evidence, and the exact legacy record Stop and drain every old writer sharing the store. Prove one tuple owns the row, set the drained-writer acknowledgement, and use migrateLegacyConnectorIdempotency(); leave pending, changed, invalid, conflicting, or unproven rows fail closed
One workload throttles another Deployment-wide connector budget Tune the deployment limit or split workloads into separate deployments when they require independent capacity
Egress policy allowed an unexpected socket Connector bypassed runtime fetch Route the transport through guarded fetch and add infrastructure egress control
Live updates stop but HTTP works Hub binding, ticket secret/expiry, socket liveness Let client poll; repair stream configuration without disabling authorization
Subscription changed but polling did not Reconcile response/audit and provider alarm Retry reconcilePolling() against committed subscriptions
Schedule did not fire Maintenance alarm status, active status, CAS winner, run cap, stored metadata, and schedule-tick-error and schedule-tick-bookkeeping-error lines Re-arm maintenance or correct the stored row; do not bypass the tick guard. A fire recorded failed with reason input-too-deep has a stored target value nested more than 256 levels deep: update the schedule with a shallower value or delete it
A fleet audit reports maintenance-stale for the deadline duty deadline-sweep-error lines, the maintenance status's lastDeadlineError, and running runs past their deadline A pass expires every due run it selects and fails when any one of them fails, so one run whose timeout keeps failing fails every pass, and the audit's re-arm does not clear it. Fix what the error names or terminate that run; until the duty succeeds, runs past their deadline keep running
A fleet audit reports maintenance-stale for the notification duty, or due notifications are not delivered notification-dispatch-error lines, a config-error line naming maintenance.notification, and the maintenance status's lastNotificationError Restore the notificationTick wiring a config-error line names, or fix what the error names, then check that the next notification-dispatch line reports the notifications that were due
Deleting a schedule answers 202 with pending: true and the schedule stays, or the drain inventory's schedule-deferred-dispatches category does not empty The fires still deferred and each one's statusError: see Deferred schedule fires Act as that section says
Purge removed a snapshot but left R2 Artifact store omitted or delete failed Restore row/key evidence if available, repair paired purge, scan known prefix
Audit is absent while requests succeed Sink, Queue, consumer, SIEM Restore export, preserve local ring/Logs, assess evidence gap
Agent CLI error lacks output Expected safe diagnostics Inspect the isolated workspace/vendor logs under appropriate access; do not weaken public error safety
Agent CLI reports runtime-unavailable before Windows launch SystemRoot/WINDIR presence and whether it is a drive-absolute local path Restore the trusted Windows system-root environment; UNC, device, root-relative, and relative paths are rejected. Never add the workspace or another writable directory as a taskkill lookup fallback
Agent CLI reports termination-failed Sanitized terminationMethod, systemCode, and numeric exit metadata; host process inventory Isolate the workspace and kill remaining descendants through the container or host supervisor before retrying. Repair POSIX signal permission or the absolute Windows system taskkill executable; do not treat the attempt as a clean timeout

Recovery rules

  • Do not edit a D1 snapshot manually unless you have a tested, version-specific repair and no supported redrive.
  • Do not copy an approval grant into a resume request.
  • Do not roll a terminal approval back to open.
  • Do not delete a pending idempotency record while its owner may still run.
  • Do not replay or copy an ambiguous legacy idempotency row until external evidence proves exactly one scoped tuple owns it. Use the connector-bound atomic D1 migration for proven rows; leave all others fail closed.
  • Do not change a deployed Durable Object migration tag.
  • Do not overwrite or bypass a deployment sentinel to recover a mis-bound database.
  • Do not expose system stores or raw namespace access as an emergency endpoint.

Prefer replaying a trusted orchestration step from authoritative persisted state.

Capacity tuning

Tune from measured workload:

  • approval list page size and poll interval;
  • SLA sweep frequency;
  • purge batch sizes and TTLs;
  • signal body and rate caps;
  • schedule count, fire rate, and unattended-run cap;
  • notification dispatch chunk size;
  • provider poll interval and webhook rate cap;
  • connector fixed-window count;
  • idempotency pending TTL;
  • Agent CLI timeout and retained output bytes;
  • Queue batch and retry/dead-letter settings;
  • public-demo per-session and global daily run budgets.

Do not shorten a safety timeout below the maximum real operation merely to increase throughput. Scale execution and stores instead.