feat: research agent runner, trigger endpoint, and report UI - #5158
Open
daviddanialy wants to merge 10 commits into
Open
feat: research agent runner, trigger endpoint, and report UI#5158daviddanialy wants to merge 10 commits into
daviddanialy wants to merge 10 commits into
Conversation
Adds the MCP research agent behind the mcp_approval feature: a bounded tool-calling runner (web search + page fetch over OpenRouter) executed in a Temporal activity, a decide-scoped startResearch endpoint with single-flight semantics and interruption compensation, and the report UI on the approval review page — run trigger with token-cost warning, live polling while a run is active, and tiered, cited report rendering capped at the five most relevant claims. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
🦋 Changeset detectedLatest commit: 912b1a5 The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
cubic analysis
All reported issues were addressed across 61 files
Linked issue analysis
Linked issue: AIS-475: feat: MCP research agent
| Status | Acceptance criteria | Notes |
|---|---|---|
| ✅ | Trigger endpoint that starts a single-flight research run (creates a report row, enqueues a Temporal workflow keyed by report, starting while one runs returns the running report). | Design + server wiring, client SDK, and background workflow use a report-keyed workflow (USE_EXISTING) and there are DB queries for running reports; start handler and tests exercise enqueue behavior. |
| ✅ | Runner: bounded tool-calling loop over the platform web-search and page-fetch tools with a forced tool-less wrap-up turn (so transcripts always end in prose). | researchagent implements a bounded turn loop and tests assert the wrap-up turn and tool usage shape. |
| ✅ | Strict structured extraction into a versioned report: summary, independent-coverage level, at most 5 tiered claims (independently_reported vs vendor_claim), drops uncited web claims, rejects degenerate extractions, and fails if every tool call errored. | Report shape/versioning and claim tiering are implemented; extraction pass and validation logic that drops uncited claims and fails on degenerate/exhausted-tool runs are present and covered by tests. |
| Fetched content treated as untrusted (prompt-injection posture) and prompt-injection shaped material surfaced as findings (i.e., do not let fetched content act as instructions). | The system prompt and agent prompt explicitly enforce the untrusted-data posture and there are changes to the PI judge plumbing; however, it is not unambiguously visible in the diff that every fetched page is subjected to the standalone PI judge and then surfaced as its own finding. The posture is implemented, but explicit per-page PI-judge invocation/surfacing is not clearly shown in the provided snippets. | |
| ✅ | Agent never adjudicates (no scores/verdicts); report gathers and cites only. | Report model/tiering contains no scoring or verdict fields; code and prompt explicitly state 'never adjudicates.' |
| ✅ | UI: Run Research control on the approval review page, spinner/polling while active, latest report rendered with coverage callout, tier chips, per-claim citation links, and older runs collapsed behind a toggle. | Dashboard components add startResearch SDK hook, UI elements for the Run Research button and polling, and rendering changes for coverage and per-claim display. |
| ✅ | Tests and wire-shape checks: scripted completions/unit tests for runner behavior, OpenRouter payload/annotation tests, and end-to-end/local run exercised. | There are unit tests for the runner/extraction behavior, added OpenRouter decode/marshal tests, and test harness pieces asserting expected behavior; the PR notes an end-to-end local run and includes test coverage for core behaviors. |
Tip: instead of fixing issues one by one fix them all with cubic
Re-trigger cubic
This was referenced Aug 12, 2026
…ddanialy/ais-475-feat-mcp-research-agent
Citation URLs are written by a model that has just read untrusted pages, and the review page turns them into links — so a javascript: or data: URL surviving extraction would put attacker-authored script one admin click away. Nothing checked the scheme: validate() only asked whether a web-tier claim had any citations at all, so an empty or unopenable URL also counted as "cited" and satisfied the rule it was meant to enforce. Citations are now kept only when they parse as http(s) with a host, a claim with no surviving citation is dropped like an uncited one, and an empty claim text — which the panel rendered as a blank row inside the top five — is dropped too. The extraction schema asks for the same shape, and the panel filters again on render, because reports stored before this are still on file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four ways the row outlived the run that owns it, all with the same symptom: the approval page polls a report that never resolves and its Run button stays disabled forever. - The workflow's run timeout bounded execution but not queue time, so a saturated task queue expired the workflow while the activity waited — taking the compensation with it. The activity now carries a ScheduleToClose inside the run timeout, and the relationship between the four budgets is asserted in a test rather than described in a comment. - A failed enqueue compensated on the request's context, which is canceled precisely when it is needed: the client has already gone. - A completion that could not be written returned without resolving the row, leaving the workflow's compensation to report an interruption rather than what actually happened. - Completion could resurrect a terminal report, so a late result flipped a failure an admin had already been shown back to completed. It now only touches rows still in running, mirroring the failure update, and a late result is discarded deliberately. Also drops HasRunningResearchReport and GetResearchReport, which nothing called. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 7 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Fix all with cubic | Re-trigger cubic
AIS-475 asks for this directly: run the PI judge over fetched content and surface injection-shaped material as a finding in its own right, because a page that tries to manipulate the reviewer is among the strongest signals about the server it describes. The runner fetched pages and handed them straight to the model — the prompt posture treated them as untrusted, but nothing ever detected or reported an attempt. Every fetched page now goes through the same prompt-injection scanner the risk pipeline uses, wrapped behind a narrow judge interface so this package never grows its own second opinion about what an attack is. A flagged page is not withheld: the research may still need what it says, so it arrives labelled as material that tried to instruct its reader, and the finding is attached to the document after extraction — the model writing the report has just read the page doing the steering, so what that page attempted cannot be left to it to report. The judge scans strictly here, unlike the gating paths that fail open: a page nobody could judge is counted as unjudged rather than passed over, so an empty findings list is not read as "nothing tried". Also closes the start race: the in-flight check and the insert now share a transaction behind a row lock on the request, so two clicks that land together buy one run. The durable form is a partial unique index on (mcp_approval_request_id) WHERE status = 'running', which needs its own migration. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four ways untrusted material or unbounded cost reached somewhere it should not, all on the same run. - The briefing handed the evidence document to the model as plain task context, but the reviewed server wrote much of what is in it — tool names, tool descriptions, package blurbs, registry copy. It is now fenced and labelled as untrusted data, so a field shaped like an instruction is read as the reviewed party talking. - The extraction pass received the transcript, fetched pages and all, under a prompt about report structure and nothing about who wrote what it was reading. It now opens with the same posture the agent loop carries. - A long run's transcript was truncated from the front, which discards the tail — and the tail is the wrap-up turn, which exists precisely so the transcript ends on the agent's findings. Both ends are kept now. - Searches are billed completions the run pays for, and the runner counted only its own turns. Tools that spend on a run's behalf report it, and the stored token totals describe the whole run. Also: the compensation activity gets its own ScheduleToClose, because queue time bounds nothing otherwise and the invariant test was resting on a retry budget that a saturated queue ignores; the search result loop bounds with >= so a non-positive cap yields nothing rather than everything; Run Research checks its scope against the current project, matching the endpoint behind it; and the plugins array both behaviours ride on gets a wire-shape test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 17 files (changes from recent commits).
Tip: instead of fixing issues one by one fix them all with cubic
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
…ddanialy/ais-475-feat-mcp-research-agent # Conflicts: # server/internal/platformtools/research/tool_web_search.go
Follow-ups from review of the injection-finding change. - An injection finding's URL was stored unfiltered, though it becomes a link and comes from a page the judge just called hostile. It now goes through the same http(s)-with-a-host check as a citation, which moved into one shared helper rather than being written twice. - The same page can be fetched twice, and a repeat is not a second attempt: findings are recorded once per URL, which the rendering also depends on since it keys by URL. - Citations and injection links now open through the shared safe-external path — uniquely named tab, cleared opener, no referrer, blocked popups reported — instead of a plain anchor. These URLs are the least trustworthy in the product; they should not be the one place that skips it. - A search cap of zero returned one result, because the bound was tested after the append. - The transcript's truncation marker was added on top of the budget rather than counted inside it. - The extraction prompt changed meaning, so PromptVersion is 4: stored reports have to be distinguishable across it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ddanialy/ais-475-feat-mcp-research-agent
Both branches added the same check, so the merge kept two copies and the second returned the wrong arity. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements AIS-475 and AIS-476: the research agent itself, the endpoint that launches it, and the report surface on the approval review page.
Stacked on #5147 (research web toolset). Merges after it.
Important
Cost caveat: research runs execute on our OpenRouter chat key — the customer is NOT billed for this spend today, and the UI's token warning deliberately makes no billing claim. We are actively working on monetizing this OpenRouter spend, alongside the other internal-key inference surfaces (risk policy scanning, judges, etc.). Until that lands, every research run is real money out of our key; per-run token usage is recorded in the report's run meta so the spend is attributable when monetization arrives.
Runner (
mcpapproval/researchagent)independently_reportedvsvendor_claim) and cited. Deterministic briefing facts are excluded — the evidence panel already shows them.Trigger endpoint
mcpApproval.startResearch, gated onmcp_approval:decide. Single-flight per request: re-clicking while a run is active returns the running report; completed runs are additive history.v1:mcp-research/<reportID>(USE_EXISTING). Activity runs with MaximumAttempts 1 + heartbeats; on failure or worker crash a compensation activity marks the report failed (guardedAND status = 'running'so late compensation can't retro-fail a completed report). Enqueue failure fails the row immediately — no stranded "running" rows.mcp_research_reportstable that shipped in mig: add MCP approval workflow tables #5035; no new migration.Report UI
Verification
🤖 Generated with Claude Code
Summary by cubic
Runs a cited MCP research agent end to end and exposes it via a trigger endpoint and report UI so admins can launch web research from the approval page. Previously research was manual; now a single‑flight, bounded agent run produces a strictly validated, cited report with prompt‑injection findings, safe external links, and spend accounting.
Highlights
mcpApproval.startResearch(decide + project scoped) row‑locks to single‑flight; opens a report and enqueues Temporal workflowv1:mcp-research/<reportID>with run/compensation budgets; compensates on enqueue/work errors; only completes rows stillrunning; exposed via OpenAPI/@gram/client/React Query/CLI; no migration.Implements AIS-475.
Written for commit 912b1a5. Summary will update on new commits.