Skip to content

integrations.md

Latest commit

 

History

History
92 lines (59 loc) · 19 KB

integrations.md

File metadata and controls

92 lines (59 loc) · 19 KB
Raw

Integrations: catalog, generic inbound receptor & mappers

Integrations that are "tools the agent uses" present themselves at the data/config layer, not in source, whenever possible. Chatwoot is the justified exception (core, high volume, deep domain wiring; its own dedicated module). Asaas, Calendar and similar are not modules: they are catalog entries + (outbound) declarative toolpacks + one (inbound) pure mapper over the generic substrate built here.

Catalog (src/modules/integrations/catalog.ts)

A catalog entry is data, one of three kinds:

  • TOOLPACK — declarative HTTP tools (the ToolDefinition mechanism). The path for Asaas outbound (payment_link_create, status). Not MCP, not a module. n8n-style model (src/graph/tools/http.ts): each inputSchema field has a source — ai (the model fills it; in the tool schema) or fixed (a constant / {{context}} template sent without the model). Context (conversation_id, contact_name, …) and the credential ({{secret}}) are interpolated into fixed values, headers, the URL path/query and a raw body, all server-side — the secret never enters the model schema/return or a trace (the origin pin keeps {{secret}} out of the host). ToolDefinition.body is one of {mode:"kv", rows} (flat JSON from explicit key/value rows), {mode:"raw", raw} (the template as written — the way a NESTED payload is built) or legacy/absent {mode:"fields"} (JSON assembled from the declared input fields). Any other shape is refused at the write: parseBody executes only those three and falls back to the fields assembly for anything else, so a body authored as a plain JSON object used to be stored and then silently discarded, with the request going out assembled from field names (issue #150). The check is unsupportedBodyShape in src/modules/tool-definitions/body-shape.ts; the bundle import warns and drops instead of refusing, since a legacy bundle's body was already inert on the instance that exported it. When an ack is configured the tool exposes an optional __wait_message the model may fill (consumed for the ack, never sent). The structured editor lives in resources/ToolsPanel.tsx.
  • MCP — an external MCP server we consume (McpServerConnection + @langchain/mcp-adapters).
  • NATIVE — a source module. Only when justified (Chatwoot; deep-domain integrations).

IntegrationInstance.catalogType is a plain String validated against this in-code registry (isKnownCatalogType), not a DB enum — a new integration needs no enum migration. Seeded entries: GENERIC, ASAAS, GOOGLE_CALENDAR. An entry may exist before its mapper ships; until then inbound for it fails closed (durable FAILED record + log).

Generic inbound receptor (src/modules/webhooks/inbound/)

The n8n-style webhook node, generalized — but the mapping lands in a CLOSED set of our domain events (conversion, status_update, agent_nudge), never arbitrary action. We are an app; n8n-export covers arbitrary automation.

Endpoint: POST /api/v1/integrations/inbound/:routeToken (integrations.controller.ts). JWT-less by design — not behind tenancyPlugin/requireAuth; the route token resolves the tenant and the per-instance strategy authenticates. POST only, so the GET 404 guards and the SPA catch-all never apply (smoke-tested in tests/api/v1/carveout.test.ts).

Flow (receiveInbound → ack; processInboundDelivery → detached async):

  1. Resolve tenant by route token. The token is an opaque 256-bit value. Its SHA-256 hash is stored in IntegrationInstance.routeTokenHash (unique), so lookup is a constant-time B-tree probe — no linear scan, no timing oracle. Resolution is cross-tenant → asSuperAdmin. See "The webhook URL is re-readable" below for why the token itself is also stored, encrypted.
  2. Verify auth AFTER resolving the tenant (so the secret read is tenant-scoped). Strategies: NONE | STATIC_HEADER | HMAC_SHA256 (secret from the vault by inboundSecretRef; header names overridable per instance via config, because providers dictate their own). Unknown token, disabled instance, and bad auth all return the same uniform 401 (no oracle for which tokens are live), and the server logs WHY at warn (route_unknown, route_disabled, secret_not_configured, secret_ref_unresolved, secret_pending, secret_unusable, header_name_unusable, header_missing, credential_mismatch) with ids and the vault ref, never the header value or the body. Uniformity is a property of the RESPONSE; without the log an unresolvable ref, an unfilled secret and a wrong token were the same event everywhere. inboundSecretRef (like every other vault:<id> column) is checked against the vault on write, so a bare entry NAME is a 400 instead of a value no resolver can ever answer. The two header NAMES in config (authHeader, signatureHeader) are checked on write the same way: a value outside RFC 7230's token alphabet is a 400 naming the key, because Headers.get throws on it and the delivery would answer 500 — a status no other refusal gives, and therefore an oracle. The read refuses it too (header_name_unusable, never a fallback to the default name), because a row written before that check still carries whatever it carries.
  3. Ack fast (<5s), process async. A slow/non-2xx ack makes some providers (Chatwoot) auto-escalate pending→open. receiveInbound does the cheap work (resolve, auth, normalize, persist) and returns; the controller fires processInboundDelivery detached.
  4. Normalize via the integration's pure mapper (below). { ok: false, reason: "unhandled" } (lifecycle noise we deliberately skip) → ignored, no persistence. { ok: false, reason: "invalid" } (the payload SHOULD have matched and did not — schema drift) → logger.warn (catalogType/instance/zod issue paths only, never the body) + a durable FAILED delivery with payload: { reason: "invalid", issues } and a body-digest dedupeKey, same fail-closed mold as no-mapper; the response carries outcome: "invalid" so the provider's delivery log shows the distinction.
  5. Persist an idempotent InboundDelivery — unique (tenantId, integrationInstanceId, dedupeKey). The stored payload is the mapper's allowlisted projection (PII-bearing; never the raw external JSON, never echoed in logs/audit). A duplicate returns the existing id.
  6. Dispatch (processInboundDelivery) under a status CAS (PENDING→PROCESSING→PROCESSED, re-entry is a no-op). conversion → correlate externalId→thread via IntegrationExternalRef then record ConversionEvent idempotently (ON CONFLICT DO NOTHING, so the surrounding tx stays alive); an uncorrelated/duplicate conversion is dropped with a log, never invented. status_update/agent_nudge are declared seams (Fases 3/4), no-op for now.

The webhook URL is re-readable (and rotatable)

The operator pastes …/api/v1/integrations/inbound/<routeToken> into the provider's dashboard (Asaas). That is an address, not the authenticator — inbound calls are authenticated by inboundAuthStrategy + the vault secret — and an address you cannot look up again is a dead end: configure Asaas from another machine, or simply lose the tab, and the integration is unrecoverable.

So the token is persisted twice, each copy serving a different reader:

Column Form Read by
routeTokenHash SHA-256 the hot inbound path (resolveInboundRouteByToken), a unique-index probe
routeToken encryptJson() blob in a plain String column the editor, via GET /v1/integrations/instances/:id

Consequences worth knowing:

  • Only the single-instance read returns it. GET /instances leaves routeToken null — that screen shows no URL, so shipping N tokens to the browser would be pure exposure. The list/update/delete DTOs never carry it either.
  • Three states, never two. The DTO carries routeTokenStatus: present (readable), absent (nothing stored — an instance older than this column, or an outbound-only entry), unreadable (a blob the key can no longer decrypt). A decrypt failure degrades instead of throwing — the one place that bends the "throw, don't fall back" rule of the Encryption section, deliberately: nothing downstream makes a security decision on this value, and throwing would 500 the edit modal of every integration. But it must not collapse into absent either, or the editor tells the operator "this predates the feature" when the real cause is the key. Both non-present states recover by rotating; the UI just names the right one. The logger.warn on failure is the ops signal.
  • GET /instances/:id sets Cache-Control: no-store. It is the only read that returns the decrypted token and nothing global sets the header (same treatment as branding.controller.ts).
  • POST /instances/:id/route-token rotates. It mints a new token, replaces both columns, and returns the plaintext. The old URL stops resolving the moment it commits, so the caller must tell the operator to update the provider — the editor does this with a danger-styled confirm. Rotation covers the two cases the stored copy cannot: a row created before the column existed (routeToken null — hash-only, plaintext gone for good) and a leaked URL. Outbound-only catalog entries (Calendar/Drive) have no inbound surface, so rotating one is a 400.

Mappers (src/modules/integrations/mappers.ts)

A mapper is a pure translator: (raw: unknown) => MapResult. No DB, no network, no LLM (correlation/idempotency/dispatch happen in the receptor around it). It returns { ok: true, event: { kind, externalId, dedupeKey, occurredAt?, ...allowlisted fields } }, or { ok: false, reason: "unhandled" } for payloads it deliberately skips, or { ok: false, reason: "invalid", detail } when the shape should have matched and did not (detail = zod issue paths only, never values). Shape validated with zod per integration; optional provider fields that arrive as explicit null (Asaas does this) use .nullish(), not bare .optional().

  • Correlation is by primary key, not LLM. externalId (created at outbound time in IntegrationExternalRef) is looked up to recover the thread. An agent-as-mapper was rejected: hallucinated correlation of financial data is a privacy incident, breaks idempotency, taxes the <5s ack, and turns an untrusted external payload → LLM → tools into a prompt-injection vector. The LLM is never in the correlation critical path.
  • Why code, not a runtime DSL: inbound processes untrusted external bytes (larger attack surface than outbound); the long tail of providers resists a DSL (inner-platform); and the set of real mapper authors is small and is us. ~15 lines, type-safe, unit-testable, injection-bounded. A future sandboxed declarativeMapper(config) can slot in behind the same InboundMapper interface if a non-us author ever needs it — not built now.
  • Registry, keyed by catalogType. Adding an integration = a catalog entry + a mapper + a zod schema + a registerMapper line; the core (route-token, auth, ack/async, idempotency, correlation, dispatch) is untouched. GENERIC ships now (accepts our normalized shape); Asaas/Calendar mappers ship when those integrations land.

Known limitations

  • No inbound reprocessor yet. Dispatch is detached after the ack; if the process crashes before it runs, or processInboundDelivery throws (the scoped tx rolls back to PENDING, logged via the controller's .catch), the InboundDelivery row stays PENDING with nothing retrying it. The row is durable; a sweep that re-dispatches stranded PENDING/PROCESSING rows (owned by the scheduler, or a tick mirroring the outbound reaper) is the planned fix. Not silent — it logs — but not yet self-healing.
  • One ConversionEvent per (tenant, threadId, source). This is the plan's idempotency key. A second genuine conversion on the same thread from the same source (e.g. a renewal) is dropped by skipDuplicates. Acceptable for the MVP; revisit (add a discriminator) if multi-conversion threads become a real funnel case.
  • Auth timing oracle (low). An unknown/disabled token returns after one lookup; a bad-auth attempt does an extra scoped read + HMAC, so response timing differs. The 256-bit token space makes candidate enumeration infeasible, so the oracle's value is marginal; uniform-work hardening is deferred.

agentNudge (declared seam)

After the deterministic core (resolve → PK correlation → status CAS → mechanical effects), a domain event of kind agent_nudge will inject a normalized, sanitized system event into the conversation's LangGraph thread, where the agent decides whether to follow up. Guardrails to be enforced when wired: assignment gate (human handling ⇒ private note, not a customer message), idempotent on the transition (CAS before the nudge), the LLM sees our normalized enum/value (never raw external JSON), a suspended interrupt() defers the nudge, and proactive output passes follow-up + business-hours policy. The injection must happen outside the processing transaction (network/async).

Google Calendar toolpack — per-customer isolation (src/modules/integrations/toolpacks/google-calendar.ts)

One clinic calendar serves MANY WhatsApp contacts, so every event the agent creates is stamped with extendedProperties.private.secv4Contact = "<tenantId>:<contactDbId>" — injected from the trusted context, never a model arg. Reads/writes are isolated three ways, fail-closed:

The secv4* key names predate the brand rename and are frozen on purpose: they live on real events in customers' calendars, and the list fence takes exactly one privateExtendedProperty filter per request, so a second name would double every listing forever or need a backfill we cannot run. Renaming them is part of the 2.0 cut.

  • List filters server-side by privateExtendedProperty=secv4Contact=<stamp> AND re-verifies each returned event's stamp client-side (eventStamp(e) === stamp). The re-verify is defense in depth; when it drops an event the server fence already should have excluded, it logs gcal: list re-verify dropped events… (a non-zero dropped count is a signal the fence is leaking). With no contact in scope (playground) the per-contact tools refuse outright.
  • Update / cancel re-fetch the event (?fields=extendedProperties) and refuse with FOREIGN_EVENT unless the stamp matches, so an id-guess can never touch another contact's (or a staff-created) appointment.
  • Availability is freeBusy (busy windows only, zero details) → another contact's bookings count as busy without leaking anything; it returns EVERY bookable slot in a range capped to 24h.
  • Availability spans SEVERAL calendars when the instance allows several and the model omits calendarId (issue #100). Each calendar is computed on its OWN busy list and the results are merged chronologically, every slot tagged calendarId/calendarLabel. Busy windows are never pooled: pooling answers when they are ALL free at once, which is a meeting room, not a clinic. Booking, rescheduling and cancelling still act on ONE explicit calendar.
    • Requests. Batched at 10 calendars per freeBusy request, which is what satisfies Google's calendarExpansionMax (a PER-REQUEST cap of 50) and keeps a batch's response far under MAX_RESPONSE_CHARS (gcalFetch truncates a body BEFORE parsing it, so one oversized answer parses to null and would read as the whole clinic being unreachable). Collected with allSettled, so a timeout on one batch costs only its own calendars.
    • Size. An aggregate query covers at most 50 calendars and is REFUSED above that; calendarIds is an arbitrary-length array nothing validates, and aggregation is where it turns into outbound requests, slot entries and an always-emitted first start time at once, so the bound sits on the count rather than on each consequence. An explicit calendarId is unaffected at any allowlist size.
    • Truncation. The merged list is bounded at 250 entries when aggregating (a single-calendar query stays uncapped: the multiplication the bound exists for is not there). It drops WHOLE start times, never part of the calendars offered at one time, and reports coveredUntil so the caller resumes instead of believing it saw the whole day.
    • Unreadable calendars. One that freeBusy reports an error for is dropped and named in unavailableCalendars, never carried as free: an empty busy list means "free all day", and offering a professional whose bookings we cannot see is a double booking. If none could be read the tool refuses, because an empty slot list reads as "fully booked".
    • Overlap with blocking. A blocking calendar that is ALSO an operable one is read and applied to its siblings but not to itself, decided from the calendars that ANSWERED rather than the ones asked for. freeBusy ignores the transparent/all-day events a closure is made of, so skipping that read would make the closure invisible to everyone; with no readable sibling left it is pure risk and is skipped.

Meet room on create (createMeetLink, default ON): calendar_create_event asks Google for a hangoutsMeet conference and the event projection exposes it as meetLink — THE link to hand the customer (htmlLink is only the event's calendar page, useless to a lead). Two API traps, pinned by tests: the request must carry conferenceDataVersion=1 (otherwise conferenceData is silently ignored, no error) and createRequest.requestId must be unique per event (a reused id returns the SAME room, putting different leads in one meeting). If the response still has the room pending, one cheap re-read fetches the link; an account without Meet keeps the event, just without a conference. Operators using the calendar purely as a busy-block set createMeetLink: false in the instance config (toggle in the editor's Calendars tab).

Legacy/staff events without a stamp are invisible to the agent by design (the list fence and the re-verify both exclude them; update/cancel refuse them) — not a bug, the cost of fail-closed.

Deploy gotcha: registerToolpack runs at import time, so a hot-reload may keep an old toolpack build in memory. A stale process (pre-isolation code still running) is what masked the isolation in an early live test — restart the process on deploy, do not rely on hot-reload for toolpack changes (see docs/deploy.md).

Positioning: an external CRM is a bridge, never a second funnel

Chatwoot's Kanban is the native sales funnel and the source of truth. An external CRM is the bridge to the system of record the customer already runs, and must not become a second funnel competing with the Kanban. Where the operator has no external CRM, the Kanban IS the CRM; where they already run one, the agent syncs to it and the funnel stays here.

So the idiomatic way to integrate a real external CRM is the one this document already describes: a CATALOG entry plus a toolpack and a mapper, under Components → Integrations. catalogType is a validated String with no migration behind it — a CRM integration is not a new screen.