LLM provider management: write tools (BYOM)

[mocha06]

Closes #415. Final phase of milestone 19 (AI Agents: Knowledge Bases, Capabilities & Model Selection); the write counterpart to the provider discovery surface (#411).

Motivation

The discovery phase (#411) made LLM providers readable but not manageable. Bringing your own model (BYOM) still required the Pipefy UI: there was no way to create a custom provider, rotate its credentials, toggle it, or set the organization default from an agent or script. This phase closes that gap while keeping provider secrets out of arguments, logs, errors, and responses.

What ships

Six write tools across SDK → MCP → CLI, plus docs and the AI-agents skill:

MCP	CLI
create_llm_provider	pipefy ai-provider create
update_llm_provider	pipefy ai-provider update
delete_llm_provider	pipefy ai-provider delete
set_llm_provider_active_status	pipefy ai-provider set-active-status
set_default_llm_provider	pipefy ai-provider default set
reset_default_llm_provider	pipefy ai-provider default reset

Proven contract

Verified against the live GraphQL schema and traced through the full delegation chain (GraphQL entry → downstream credential-validating service), per-argument:

• Configuration is secret-bearing and file-only. MCP (configuration_file_path) and CLI (--config-file) accept the provider configuration only via a local JSON file, never inline — so it never becomes a logged tool argument. The provider key selects the vendor; vendor/model membership and credential validity are validated server-side (a live credential test call), so the toolkit treats the object as opaque rather than coupling to the vendor schema.
• Secrets are never returned. create/update deliberately omit configuration from their selection sets even though the payload exposes it; the response carries only id/name/type/active/organizationDefault. File/parse errors report only the path and a structural reason, never file contents.
• Update is a full replacement, and the redaction round-trip is safe. configuration is required on every update. A caller can fetch the (redacted) configuration, edit the non-secret fields, and resubmit: any value left as the __REDACTED__ placeholder is stripped upstream and the stored secret is preserved — confirmed live (an update with the placeholder succeeds; the backend used the stored key, not the literal placeholder). Supplying a new real value rotates the secret.
• Identifier forms (not uniform, following the API): create/update/delete take the organization UUID; set-default/reset take the numeric organization id; set-active-status takes no organization argument; set-default requires exactly one of provider_id / system_provider_id (enforced client-side and downstream).
• Auth: writes require the manage_ai_providers organization permission and an eligible plan — distinct from manage_ai_agents. The read-access probe proves read only, so a green probe does not imply write entitlement.
• Owner-scoped operations need a service-account credential. set-active-status, set-default, reset, and the organization-owner get_default_llm_provider authorize against the credential's own organization context. A service account is bound to one organization and supplies that context; a bare personal token is denied regardless of the organization passed — even when the same token can create/update/delete providers there (those authorize against the passed UUID). Documented in the tool docstrings and docs/mcp/tools/llm-providers.md.

Behavior boundaries

• Default set/reset are organization-scoped (owner is always the organization); assistant/behavior owner scopes are out of scope for the write surface.
• Create/update stay excluded from the remote profile (they read a local file); all six writes are unmarked, so the default-deny remote seed is unchanged.
• CLI create/update are gated on the read-access probe, treated as clean only when it is ok and carries no problem (a present problem is partial denial, never full access); the 2. LLM provider discovery: read tools and access probe #411 probe docs now state this clean-gate contract explicitly.
• Deletes require confirmation (MCP two-step confirm; CLI prompt unless --yes).

Known upstream issue

Provider credential-validation errors are forwarded from the backend and can echo a masked copy of the submitted credential plus an internal error_code blob. This originates upstream; the toolkit surfaces the classified message verbatim (no client-side scrub, by decision) and the upstream fix is tracked separately (here)

Docs and skills touched

docs/mcp/tools/llm-providers.md (writes, secret handling, the update flow, the probe clean-gate contract, service-account requirement), docs/parity.md, README / packages/mcp/README tool counts (reconciled to 182), the AI-agents skill (BYOM provider management flow), and the registry.

Testing

Unit tests assert configuration never appears in responses, logs, or errors; the XOR rule; full-replacement update; the redaction placeholder passthrough; confirm-delete; and the probe clean-gate. Live end-to-end against a lab organization:

#	Scenario	Observed	Verdict
1	Read-access probe	ok, no problem, providers visible	pass
2	Read redaction	secrets return as __REDACTED__	pass
3	create (valid key)	provider created; response carries no configuration	pass
4	Created provider reads redacted	token __REDACTED__	pass
5	update redaction round-trip	success with __REDACTED__; model changed; stored secret preserved (no re-auth failure)	pass
6	create — non-object config	rejected; secret not echoed	pass
7	create — malformed JSON	rejected with position only, no content	pass
8	set-default — both ids / neither	rejected "exactly one"	pass
9	create — invalid key	backend rejects; nothing created	pass
10	set-active-status (service account)	deactivate then reactivate	pass
11	set-default — provider id / system id (service account)	both paths succeed	pass
12	reset-default (service account)	default cleared	pass
13	Owner-scoped ops with a personal token	PERMISSION_DENIED (documented auth boundary)	pass (expected rejection)