Skip to content

feat: two-phase Copilot SDK driver for threat detection job#37133

Open
Copilot wants to merge 17 commits into
mainfrom
copilot/add-dedicated-driver-detection-job
Open

feat: two-phase Copilot SDK driver for threat detection job#37133
Copilot wants to merge 17 commits into
mainfrom
copilot/add-dedicated-driver-detection-job

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Jun 5, 2026

Threat detection runs the same large model as the main agent, making every run expensive even when content is obviously safe. This adds a dedicated two-phase driver that pre-screens with a cheap model and only escalates to full analysis when needed.

New driver: detection_job_driver.cjs

Two-phase execution:

  1. Phase 1 (small model) — triage prompt asking for exactly safe or unsafe. On safe, writes an all-false THREAT_DETECTION_RESULT: to stdout and exits without running Phase 2.
  2. Phase 2 (large model) — only runs on unsafe or Phase 1 failure. Sends the full detection prompt; response is parsed by the existing parse_threat_detection_results.cjs logic unchanged.

Model names are overridable via GH_AW_DETECTION_SMALL_MODEL / GH_AW_DETECTION_LARGE_MODEL.

Log messages use small: and large: as prefixes to identify which phase is running (e.g. small: running triage (model=...), large: running full analysis (model=...)).

New utility files

detection_triage_helpers.cjs

The pure helper functions (buildTriagePrompt, classifyTriageResponse) and their related constants (SAFE_VERDICT, THREAT_DETECTION_RESULT_PREFIX) are extracted into a dedicated file. The driver and its tests both import from there.

copilot_sdk_runner.cjs

runWithCopilotSDK and its supporting helpers (buildCopilotSDKPermissionHandler, summarizePermissionRequest, logPermissionDenied) are extracted from copilot_sdk_driver.cjs into a dedicated file. This makes the function agnostic to where it runs, allowing safe reuse by both copilot_sdk_driver.cjs (which re-exports it for backward compatibility) and detection_job_driver.cjs.

New copilot-sdk-driver frontmatter field

safe-outputs.threat-detection now accepts a copilot-sdk-driver key:

safe-outputs:
  threat-detection:
    copilot-sdk-driver: detection_job_driver.cjs

Path resolution: bare filenames (no /) resolve from the built-in setup action directory; paths containing / (e.g. .github/drivers/my.cjs) remain workspace-relative. Backward-compatible — all existing custom driver usages contain /.

Go changes

  • ThreatDetectionConfig gets a CopilotSDKDriver field; when set, buildDetectionEngineExecutionStep auto-enables CopilotSDK: true on the detection engine config and getThreatDetectionEngineID implicitly returns "copilot".
  • isCopilotSDKEnabledForActiveEngine now returns true when CopilotSDKDriver is set on the detection config, ensuring the SDK package is installed even when the main engine doesn't use SDK mode.
  • customSDKDriverConfigured path resolution (in copilot_engine_execution.go) now uses a / check rather than a non-empty check.

Workflows configured

Six workflows are updated to use the new driver: architecture-guardian, breaking-change-checker, code-scanning-fixer, daily-compiler-threat-spec-optimizer, daily-safe-output-integrator, and smoke-copilot. Lock files recompiled.


pr-sous-chef: updated branch during https://github.com/github/gh-aw/actions/runs/27027532322

Generated by 👨‍🍳 PR Sous Chef · 63.8 AIC ·



✨ PR Review Safe Output Test - Run 27036908295

[!WARNING]

💥 THE END] — Illustrated by [Smoke Claude · opus48 435K · agent 130.5 AIC · threat-detection 30.4 AIC ·


Changeset

  • Type: patch
  • Description: Added a dedicated two-phase threat-detection driver with Copilot SDK configuration support.

Generated by 📋 Changeset Generator for issue #37133 ·



✨ PR Review Safe Output Test - Run 27044319757

[!WARNING]

💥 THE END] — Illustrated by [Smoke Claude · opus48 293.4K · agent 88 AIC · threat-detection 30.6 AIC ·

Copilot AI and others added 2 commits June 5, 2026 15:07
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title feat: add copilot SDK detection job driver with two-phase triage feat: two-phase Copilot SDK driver for threat detection job Jun 5, 2026
Copilot AI requested a review from pelikhan June 5, 2026 15:13
@pelikhan
Copy link
Copy Markdown
Collaborator

pelikhan commented Jun 5, 2026

@copilot enable it on smoke-copilot. Make sure the copilot-sdk is installed

Comment thread actions/setup/js/detection_job_driver.cjs Outdated
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Jun 5, 2026

@copilot enable it on smoke-copilot. Make sure the copilot-sdk is installed

Done in 17e7bcd. Added threat-detection: copilot-sdk-driver: detection_job_driver.cjs to smoke-copilot.md's safe-outputs and recompiled the lock file. The compiled lock now includes "copilot-sdk":"1.0.0" in engine_versions, confirming the SDK package will be installed for the detection job.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
@pelikhan
Copy link
Copy Markdown
Collaborator

pelikhan commented Jun 5, 2026

@copilot The runWithCopilotSDK function is a helper function and should be refactored into its own file to ensure it is agnostic to where it runs, allowing it to be safely reused in both the detection driver and the copilot SDK driver.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Jun 5, 2026

@copilot The runWithCopilotSDK function is a helper function and should be refactored into its own file to ensure it is agnostic to where it runs, allowing it to be safely reused in both the detection...

Done in the latest commit. runWithCopilotSDK and its supporting helpers (buildCopilotSDKPermissionHandler, summarizePermissionRequest, logPermissionDenied) now live in copilot_sdk_runner.cjs. Both copilot_sdk_driver.cjs (which re-exports it for backward compatibility) and detection_job_driver.cjs import directly from the new file.

@pelikhan pelikhan marked this pull request as ready for review June 5, 2026 16:33
Copilot AI review requested due to automatic review settings June 5, 2026 16:33
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a dedicated two-phase Copilot SDK driver for the safe-outputs threat-detection job to reduce cost by pre-screening with a smaller model and only escalating to full analysis when needed, plus the compiler/schema/workflow plumbing to select and run that driver.

Changes:

  • Added detection_job_driver.cjs (two-phase triage + full analysis) and extracted shared helpers into detection_triage_helpers.cjs.
  • Extracted runWithCopilotSDK into a reusable copilot_sdk_runner.cjs and updated existing Copilot SDK driver/tests to use it.
  • Added safe-outputs.threat-detection.copilot-sdk-driver support end-to-end (Go compiler, JSON schema, TS types) and updated multiple workflows to use the new driver.
Show a summary per file
File Description
pkg/workflow/threat_detection.go Adds copilot-sdk-driver to threat detection config; enables SDK mode/engine selection for detection job when set.
pkg/workflow/copilot_engine_execution.go Adjusts SDK driver path resolution to distinguish built-in (bare filename) vs workspace-relative (contains /).
pkg/workflow/compiler_yaml_lookups.go Ensures Copilot SDK installation is triggered when threat detection sets a SDK driver.
pkg/parser/schemas/main_workflow_schema.json Extends schema to allow safe-outputs.threat-detection.copilot-sdk-driver.
actions/setup/js/types/safe-outputs-config.d.ts Adds TS typing for copilot-sdk-driver in threat detection config.
actions/setup/js/detection_triage_helpers.cjs New pure helper utilities/constants for triage prompt + response classification.
actions/setup/js/detection_job_driver.test.cjs Adds unit tests for triage helper behavior.
actions/setup/js/detection_job_driver.cjs New two-phase Copilot SDK driver (triage + full analysis) for threat-detection job.
actions/setup/js/copilot_sdk_runner.cjs New extracted SDK runner used by both SDK drivers and detection driver.
actions/setup/js/copilot_sdk_driver.test.cjs Updates tests to target the extracted runner module.
actions/setup/js/copilot_sdk_driver.cjs Refactors driver to re-export/use the extracted SDK runner.
.github/workflows/smoke-copilot.md Configures threat detection to use detection_job_driver.cjs.
.github/workflows/smoke-copilot.lock.yml Recompiled lock output reflecting new driver and SDK install/runtime wiring.
.github/workflows/daily-safe-output-integrator.md Configures threat detection to use detection_job_driver.cjs.
.github/workflows/daily-safe-output-integrator.lock.yml Recompiled lock output reflecting new driver and SDK install/runtime wiring.
.github/workflows/daily-compiler-threat-spec-optimizer.md Configures threat detection to use detection_job_driver.cjs.
.github/workflows/daily-compiler-threat-spec-optimizer.lock.yml Recompiled lock output reflecting new driver and SDK install/runtime wiring.
.github/workflows/code-scanning-fixer.md Configures threat detection to use detection_job_driver.cjs.
.github/workflows/code-scanning-fixer.lock.yml Recompiled lock output reflecting new driver and SDK install/runtime wiring.
.github/workflows/breaking-change-checker.md Configures threat detection to use detection_job_driver.cjs.
.github/workflows/breaking-change-checker.lock.yml Recompiled lock output reflecting new driver and SDK install/runtime wiring.
.github/workflows/architecture-guardian.md Configures threat detection to use detection_job_driver.cjs.
.github/workflows/architecture-guardian.lock.yml Recompiled lock output reflecting new driver and SDK install/runtime wiring.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 22/23 changed files
  • Comments generated: 2

Comment thread actions/setup/js/copilot_sdk_runner.cjs
Comment on lines +240 to +246
// Parse copilot-sdk-driver field
if sdkDriver, exists := configMap["copilot-sdk-driver"]; exists {
if sdkDriverStr, ok := sdkDriver.(string); ok && sdkDriverStr != "" {
threatConfig.CopilotSDKDriver = sdkDriverStr
threatLog.Printf("Threat detection copilot-sdk-driver set to: %s", sdkDriverStr)
}
}
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 5, 2026

@copilot refresh the branch, rerun checks, and summarize any remaining blockers.

Generated by 👨‍🍳 PR Sous Chef · 63.8 AIC ·

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 5, 2026

Agent Container Tool Check

Tool Status Version
bash 5.2.21
sh available
git 2.54.0
jq 1.7
yq v4.53.2
curl 8.5.0
gh 2.92.0
node 24.16.0
python3 3.13.13
go 1.24.13
java 21.0.11
dotnet 10.0.300

Result: 12/12 tools available ✅

Overall Status: PASS

🔧 Tool validation by Agent Container Smoke Test · sonnet46 134.1K · agent 40.2 AIC · threat-detection 12.2 AIC ·

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 5, 2026

Smoke Test Results

  • GitHub MCP Testing: ✅
  • Web Fetch Testing: ✅
  • File Writing Testing: ✅
  • Bash Tool Testing: ✅
  • Build gh-aw: ❌

Overall status: FAIL

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • localhost

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "localhost"

See Network Configuration for more information.

Smoke Gemini — Powered by Gemini ·

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 5, 2026

PR: feat: two-phase Copilot SDK driver for threat detection job

  • GitHub MCP: ✅
  • Serena: ✅
  • Playwright: ✅
  • Web Fetch: ❌
  • Overall: FAIL

Warning

Firewall blocked 6 domains

The following domains were blocked by the firewall during workflow execution:

  • accounts.google.com
  • android.clients.google.com
  • clients2.google.com
  • contentautofill.googleapis.com
  • safebrowsingohttpgateway.googleapis.com
  • www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex · gpt54mini 171.8K · agent 51.6 AIC · threat-detection 3.46 AIC ·

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 5, 2026

🧪 Smoke Test: Claude — Run 27044319757

Core #1-13: 1✅ 2✅ 3❌(Serena MCP unavailable) 4✅ 5✅ 6✅ 7✅ 8✅ 9✅ 10✅ 11✅ 12✅ 13✅
PR review #14-20: 14✅ 15✅ 16✅ 17✅ 18✅ 19✅ 20⚠️(no safe PR to close)

Overall: FAIL — Serena MCP server not exposed as a CLI bridge.
Run

Warning

Firewall blocked 6 domains

The following domains were blocked by the firewall during workflow execution:

  • accounts.google.com
  • android.clients.google.com
  • clients2.google.com
  • contentautofill.googleapis.com
  • safebrowsingohttpgateway.googleapis.com
  • www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

💥 [THE END] — Illustrated by Smoke Claude · opus48 293.4K · agent 88 AIC · threat-detection 30.6 AIC ·

Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💥 Automated smoke test review - all systems nominal!

Warning

Firewall blocked 6 domains

The following domains were blocked by the firewall during workflow execution:

  • accounts.google.com
  • android.clients.google.com
  • clients2.google.com
  • contentautofill.googleapis.com
  • safebrowsingohttpgateway.googleapis.com
  • www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

💥 [THE END] — Illustrated by Smoke Claude · opus48 293.4K · agent 88 AIC · threat-detection 30.6 AIC

@@ -30,6 +30,8 @@ tools:
bash:
- "cat:*"
safe-outputs:
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice — wiring the two-phase copilot-sdk-driver here keeps threat detection cheap. Consider a brief comment noting the haiku→sonnet escalation behavior.

@@ -37,6 +37,8 @@ imports:
assignees: [copilot]
- shared/otlp.md
safe-outputs:
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same driver config applied consistently across workflows — good. Worth confirming the model override env vars are documented in the reference.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 5, 2026

Smoke Copilot: FAIL
PR: feat: two-phase Copilot SDK driver for threat detection job
Author: app/copilot-swe-agent
Assignees: pelikhan, Copilot
✅ 1,2,3,4,5,6,7,8,9,11,12,13,14,15
❌ 10 (create_discussion lacks requested label/temporary_id)

Warning

Firewall blocked 6 domains

The following domains were blocked by the firewall during workflow execution:

  • accounts.google.com
  • android.clients.google.com
  • clients2.google.com
  • contentautofill.googleapis.com
  • safebrowsingohttpgateway.googleapis.com
  • www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

📰 BREAKING: Report filed by Smoke Copilot · gpt54 1.3M · 402.1 AIC ·

@pelikhan
Copy link
Copy Markdown
Collaborator

pelikhan commented Jun 5, 2026

@copilot Merge main and recompile.

Copilot AI and others added 2 commits June 5, 2026 23:56
…-driver-detection-job

# Conflicts:
#	.github/workflows/architecture-guardian.lock.yml
#	.github/workflows/breaking-change-checker.lock.yml
#	.github/workflows/code-scanning-fixer.lock.yml
#	.github/workflows/daily-agent-of-the-day-blog-writer.lock.yml
#	.github/workflows/daily-cache-strategy-analyzer.lock.yml
#	.github/workflows/daily-compiler-threat-spec-optimizer.lock.yml
#	.github/workflows/daily-fact.lock.yml
#	.github/workflows/daily-observability-report.lock.yml
#	.github/workflows/daily-safe-output-integrator.lock.yml
#	.github/workflows/dev.lock.yml
#	.github/workflows/duplicate-code-detector.lock.yml
#	.github/workflows/grumpy-reviewer.lock.yml
#	.github/workflows/issue-arborist.lock.yml
#	.github/workflows/necromancer.lock.yml
#	.github/workflows/schema-feature-coverage.lock.yml
#	.github/workflows/smoke-call-workflow.lock.yml
#	.github/workflows/smoke-codex.lock.yml
#	.github/workflows/smoke-copilot.lock.yml

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Jun 5, 2026

@copilot Merge main and recompile.

Done. Merged main, resolved conflicts, and recompiled all 241 workflows (0 errors).

@pelikhan
Copy link
Copy Markdown
Collaborator

pelikhan commented Jun 6, 2026

@copilot use "small" for first LLM, "large" for second LLM

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Jun 6, 2026

@copilot use "small" for first LLM, "large" for second LLM

Done in the latest commit. Log messages and section labels now use small: and large: as the primary identifiers (e.g. small: running triage (model=...), large: running full analysis (model=...)). The JSDoc header also no longer references specific model names.

@pelikhan
Copy link
Copy Markdown
Collaborator

pelikhan commented Jun 6, 2026

@copilot merge main and recompile

Copilot AI and others added 2 commits June 6, 2026 02:18
…-driver-detection-job

# Conflicts:
#	.github/workflows/smoke-copilot.lock.yml

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Jun 6, 2026

@copilot merge main and recompile

Done. Merged main, resolved conflicts, and recompiled all 241 workflows (0 errors).

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 6, 2026

@copilot review all comments and address unresolved review feedback.

Generated by 👨‍🍳 PR Sous Chef · 97.4 AIC · ⌖ 0.977 AIC ·

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 6, 2026

@copilot refresh the branch and rerun checks, then summarize any remaining blockers.

Generated by 👨‍🍳 PR Sous Chef · 97.4 AIC · ⌖ 0.977 AIC ·

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants