feat: two-phase Copilot SDK driver for threat detection job#37133
feat: two-phase Copilot SDK driver for threat detection job#37133Copilot wants to merge 17 commits into
Conversation
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
|
@copilot enable it on smoke-copilot. Make sure the copilot-sdk is installed |
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Done in 17e7bcd. Added |
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
|
@copilot The runWithCopilotSDK function is a helper function and should be refactored into its own file to ensure it is agnostic to where it runs, allowing it to be safely reused in both the detection driver and the copilot SDK driver. |
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Done in the latest commit. |
There was a problem hiding this comment.
Pull request overview
This PR introduces a dedicated two-phase Copilot SDK driver for the safe-outputs threat-detection job to reduce cost by pre-screening with a smaller model and only escalating to full analysis when needed, plus the compiler/schema/workflow plumbing to select and run that driver.
Changes:
- Added
detection_job_driver.cjs(two-phase triage + full analysis) and extracted shared helpers intodetection_triage_helpers.cjs. - Extracted
runWithCopilotSDKinto a reusablecopilot_sdk_runner.cjsand updated existing Copilot SDK driver/tests to use it. - Added
safe-outputs.threat-detection.copilot-sdk-driversupport end-to-end (Go compiler, JSON schema, TS types) and updated multiple workflows to use the new driver.
Show a summary per file
| File | Description |
|---|---|
| pkg/workflow/threat_detection.go | Adds copilot-sdk-driver to threat detection config; enables SDK mode/engine selection for detection job when set. |
| pkg/workflow/copilot_engine_execution.go | Adjusts SDK driver path resolution to distinguish built-in (bare filename) vs workspace-relative (contains /). |
| pkg/workflow/compiler_yaml_lookups.go | Ensures Copilot SDK installation is triggered when threat detection sets a SDK driver. |
| pkg/parser/schemas/main_workflow_schema.json | Extends schema to allow safe-outputs.threat-detection.copilot-sdk-driver. |
| actions/setup/js/types/safe-outputs-config.d.ts | Adds TS typing for copilot-sdk-driver in threat detection config. |
| actions/setup/js/detection_triage_helpers.cjs | New pure helper utilities/constants for triage prompt + response classification. |
| actions/setup/js/detection_job_driver.test.cjs | Adds unit tests for triage helper behavior. |
| actions/setup/js/detection_job_driver.cjs | New two-phase Copilot SDK driver (triage + full analysis) for threat-detection job. |
| actions/setup/js/copilot_sdk_runner.cjs | New extracted SDK runner used by both SDK drivers and detection driver. |
| actions/setup/js/copilot_sdk_driver.test.cjs | Updates tests to target the extracted runner module. |
| actions/setup/js/copilot_sdk_driver.cjs | Refactors driver to re-export/use the extracted SDK runner. |
| .github/workflows/smoke-copilot.md | Configures threat detection to use detection_job_driver.cjs. |
| .github/workflows/smoke-copilot.lock.yml | Recompiled lock output reflecting new driver and SDK install/runtime wiring. |
| .github/workflows/daily-safe-output-integrator.md | Configures threat detection to use detection_job_driver.cjs. |
| .github/workflows/daily-safe-output-integrator.lock.yml | Recompiled lock output reflecting new driver and SDK install/runtime wiring. |
| .github/workflows/daily-compiler-threat-spec-optimizer.md | Configures threat detection to use detection_job_driver.cjs. |
| .github/workflows/daily-compiler-threat-spec-optimizer.lock.yml | Recompiled lock output reflecting new driver and SDK install/runtime wiring. |
| .github/workflows/code-scanning-fixer.md | Configures threat detection to use detection_job_driver.cjs. |
| .github/workflows/code-scanning-fixer.lock.yml | Recompiled lock output reflecting new driver and SDK install/runtime wiring. |
| .github/workflows/breaking-change-checker.md | Configures threat detection to use detection_job_driver.cjs. |
| .github/workflows/breaking-change-checker.lock.yml | Recompiled lock output reflecting new driver and SDK install/runtime wiring. |
| .github/workflows/architecture-guardian.md | Configures threat detection to use detection_job_driver.cjs. |
| .github/workflows/architecture-guardian.lock.yml | Recompiled lock output reflecting new driver and SDK install/runtime wiring. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 22/23 changed files
- Comments generated: 2
| // Parse copilot-sdk-driver field | ||
| if sdkDriver, exists := configMap["copilot-sdk-driver"]; exists { | ||
| if sdkDriverStr, ok := sdkDriver.(string); ok && sdkDriverStr != "" { | ||
| threatConfig.CopilotSDKDriver = sdkDriverStr | ||
| threatLog.Printf("Threat detection copilot-sdk-driver set to: %s", sdkDriverStr) | ||
| } | ||
| } |
|
|
…-driver-detection-job
Agent Container Tool Check
Result: 12/12 tools available ✅ Overall Status: PASS
|
Smoke Test Results
Overall status: FAIL Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "localhost"See Network Configuration for more information.
|
|
PR: feat: two-phase Copilot SDK driver for threat detection job
Warning Firewall blocked 6 domainsThe following domains were blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "accounts.google.com"
- "android.clients.google.com"
- "clients2.google.com"
- "contentautofill.googleapis.com"
- "safebrowsingohttpgateway.googleapis.com"
- "www.google.com"See Network Configuration for more information.
|
🧪 Smoke Test: Claude — Run 27044319757Core #1-13: 1✅ 2✅ 3❌(Serena MCP unavailable) 4✅ 5✅ 6✅ 7✅ 8✅ 9✅ 10✅ 11✅ 12✅ 13✅ Overall: FAIL — Serena MCP server not exposed as a CLI bridge. Warning Firewall blocked 6 domainsThe following domains were blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "accounts.google.com"
- "android.clients.google.com"
- "clients2.google.com"
- "contentautofill.googleapis.com"
- "safebrowsingohttpgateway.googleapis.com"
- "www.google.com"See Network Configuration for more information.
|
There was a problem hiding this comment.
💥 Automated smoke test review - all systems nominal!
Warning
Firewall blocked 6 domains
The following domains were blocked by the firewall during workflow execution:
accounts.google.comandroid.clients.google.comclients2.google.comcontentautofill.googleapis.comsafebrowsingohttpgateway.googleapis.comwww.google.com
To allow these domains, add them to the
network.allowedlist in your workflow frontmatter:
network:
allowed:
- defaults
- "accounts.google.com"
- "android.clients.google.com"
- "clients2.google.com"
- "contentautofill.googleapis.com"
- "safebrowsingohttpgateway.googleapis.com"
- "www.google.com"See Network Configuration for more information.
💥 [THE END] — Illustrated by Smoke Claude · opus48 293.4K · agent 88 AIC · threat-detection 30.6 AIC
| @@ -30,6 +30,8 @@ tools: | |||
| bash: | |||
| - "cat:*" | |||
| safe-outputs: | |||
There was a problem hiding this comment.
Nice — wiring the two-phase copilot-sdk-driver here keeps threat detection cheap. Consider a brief comment noting the haiku→sonnet escalation behavior.
| @@ -37,6 +37,8 @@ imports: | |||
| assignees: [copilot] | |||
| - shared/otlp.md | |||
| safe-outputs: | |||
There was a problem hiding this comment.
Same driver config applied consistently across workflows — good. Worth confirming the model override env vars are documented in the reference.
|
Smoke Copilot: FAIL Warning Firewall blocked 6 domainsThe following domains were blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "accounts.google.com"
- "android.clients.google.com"
- "clients2.google.com"
- "contentautofill.googleapis.com"
- "safebrowsingohttpgateway.googleapis.com"
- "www.google.com"See Network Configuration for more information.
|
|
@copilot Merge main and recompile. |
…-driver-detection-job # Conflicts: # .github/workflows/architecture-guardian.lock.yml # .github/workflows/breaking-change-checker.lock.yml # .github/workflows/code-scanning-fixer.lock.yml # .github/workflows/daily-agent-of-the-day-blog-writer.lock.yml # .github/workflows/daily-cache-strategy-analyzer.lock.yml # .github/workflows/daily-compiler-threat-spec-optimizer.lock.yml # .github/workflows/daily-fact.lock.yml # .github/workflows/daily-observability-report.lock.yml # .github/workflows/daily-safe-output-integrator.lock.yml # .github/workflows/dev.lock.yml # .github/workflows/duplicate-code-detector.lock.yml # .github/workflows/grumpy-reviewer.lock.yml # .github/workflows/issue-arborist.lock.yml # .github/workflows/necromancer.lock.yml # .github/workflows/schema-feature-coverage.lock.yml # .github/workflows/smoke-call-workflow.lock.yml # .github/workflows/smoke-codex.lock.yml # .github/workflows/smoke-copilot.lock.yml Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Done. Merged main, resolved conflicts, and recompiled all 241 workflows (0 errors). |
|
@copilot use "small" for first LLM, "large" for second LLM |
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Done in the latest commit. Log messages and section labels now use |
|
@copilot merge main and recompile |
…-driver-detection-job # Conflicts: # .github/workflows/smoke-copilot.lock.yml Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Done. Merged main, resolved conflicts, and recompiled all 241 workflows (0 errors). |
|
@copilot review all comments and address unresolved review feedback.
|
|
@copilot refresh the branch and rerun checks, then summarize any remaining blockers.
|
Threat detection runs the same large model as the main agent, making every run expensive even when content is obviously safe. This adds a dedicated two-phase driver that pre-screens with a cheap model and only escalates to full analysis when needed.
New driver:
detection_job_driver.cjsTwo-phase execution:
safeorunsafe. Onsafe, writes an all-falseTHREAT_DETECTION_RESULT:to stdout and exits without running Phase 2.unsafeor Phase 1 failure. Sends the full detection prompt; response is parsed by the existingparse_threat_detection_results.cjslogic unchanged.Model names are overridable via
GH_AW_DETECTION_SMALL_MODEL/GH_AW_DETECTION_LARGE_MODEL.Log messages use
small:andlarge:as prefixes to identify which phase is running (e.g.small: running triage (model=...),large: running full analysis (model=...)).New utility files
detection_triage_helpers.cjsThe pure helper functions (
buildTriagePrompt,classifyTriageResponse) and their related constants (SAFE_VERDICT,THREAT_DETECTION_RESULT_PREFIX) are extracted into a dedicated file. The driver and its tests both import from there.copilot_sdk_runner.cjsrunWithCopilotSDKand its supporting helpers (buildCopilotSDKPermissionHandler,summarizePermissionRequest,logPermissionDenied) are extracted fromcopilot_sdk_driver.cjsinto a dedicated file. This makes the function agnostic to where it runs, allowing safe reuse by bothcopilot_sdk_driver.cjs(which re-exports it for backward compatibility) anddetection_job_driver.cjs.New
copilot-sdk-driverfrontmatter fieldsafe-outputs.threat-detectionnow accepts acopilot-sdk-driverkey:Path resolution: bare filenames (no
/) resolve from the built-in setup action directory; paths containing/(e.g..github/drivers/my.cjs) remain workspace-relative. Backward-compatible — all existing custom driver usages contain/.Go changes
ThreatDetectionConfiggets aCopilotSDKDriverfield; when set,buildDetectionEngineExecutionStepauto-enablesCopilotSDK: trueon the detection engine config andgetThreatDetectionEngineIDimplicitly returns"copilot".isCopilotSDKEnabledForActiveEnginenow returnstruewhenCopilotSDKDriveris set on the detection config, ensuring the SDK package is installed even when the main engine doesn't use SDK mode.customSDKDriverConfiguredpath resolution (incopilot_engine_execution.go) now uses a/check rather than a non-empty check.Workflows configured
Six workflows are updated to use the new driver:
architecture-guardian,breaking-change-checker,code-scanning-fixer,daily-compiler-threat-spec-optimizer,daily-safe-output-integrator, andsmoke-copilot. Lock files recompiled.pr-sous-chef: updated branch during https://github.com/github/gh-aw/actions/runs/27027532322
✨ PR Review Safe Output Test - Run 27036908295
Changeset
✨ PR Review Safe Output Test - Run 27044319757