[copilot-cli-research] Copilot CLI Deep Research - 2026-03-30 #23591
Closed
Replies: 1 comment
-
|
This discussion has been marked as outdated by Copilot CLI Deep Research Agent. A newer discussion is available at Discussion #23780. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
📊 Executive Summary
Analysis Date: 2026-03-30 | Repository: github/gh-aw | Scope: 178 total workflows, 85 using Copilot engine (48%)
This is the first comprehensive analysis of Copilot CLI feature adoption in this repository. The findings reveal a significant gap between the feature-rich Copilot engine and actual usage patterns. Most workflows use a minimal subset of available capabilities — primarily
github:tool +network:config — while powerful differentiators likemax-continuations, custom agent files, and sandbox isolation are almost completely absent from the workflow library.The most striking finding:
max-continuations— the only feature exclusive to Copilot that enables autonomous multi-step operation — is used by exactly 1 workflow (1%), despite being the key mechanism that distinguishes Copilot from other engines for long-running tasks.🔴 Critical Findings (High Priority)
1.
max-continuationsis nearly unused (1/85 = 1%)This is Copilot's unique superpower — enabling autopilot mode for complex, multi-step work. Only
smoke-copilot.mduses it. Workflows likehourly-ci-cleaner,dead-code-remover,code-simplifier, anddaily-workflow-updaterwould benefit enormously.2. Sandbox adoption is too low (12/85 = 14%) for code-modifying workflows
79 workflows use the
edit:tool to modify files, but only 14 workflows total have sandbox configuration. Code-modifying workflows running without AWF isolation are a security gap — especially for scheduled jobs that run autonomously.🟡 Medium Priority Opportunities
3. Custom agent files barely used (3/85 = 4%)
9 agent files exist in
.github/agents/but only 3 workflows (glossary-maintainer, hourly-ci-cleaner, technical-doc-writer) useengine.agent. Workflows with specialized roles (security-review, code-scanning-fixer, etc.) could benefit from domain-specific agent personas.4.
mcp-scriptsadoption is minimal (3 total)mcp-scripts enable structured tool outputs with strong typing. Only 3 workflows use this capability despite it being available across the codebase.
5. Overly broad GitHub toolsets —
[default]used 46 times46 workflows use
toolsets: [default]which grants broad permissions (context, repos, issues, pull_requests). Workflows that only read could use[repos]or[issues]— reducing attack surface.1️⃣ Copilot CLI Capabilities Inventory
View Full Capabilities Inventory
CLI Flags (Auto-configured by gh-aw)
--add-dir--disable-builtin-mcps--allow-all-toolsbash: ["*"]or["*"]--allow-all-pathsedit:tool enabled--allow-tool shell(cmd)bash:tool list--allow-tool github(fn)github.allowed:list--allow-tool web_fetchweb-fetch:enabled--autopilotmax-continuations > 1--max-autopilot-continues Nmax-continuations > 1--agent agent-idengine.agent:set--log-level all--log-dirExtended Engine Configuration Options
engine.id: copilotengine.versionengine.modelCOPILOT_MODEL)engine.argsengine.envengine.agentengine.api-targetSandbox Options
sandbox.agent: awfsandbox.agent: srtsandbox.agent.mountssandbox.agent: falseExecution Environment Variables Set
GH_AW_PROMPT— the compiled promptGH_AW_WORKFLOW_NAME— workflow nameGH_AW_ENGINE— engine identifierGH_AW_VERSION— compiler versionGH_AW_PHASE— "agent" or "detection"COPILOT_MODEL— model override (whenengine.modelset)2️⃣ Feature Usage Matrix
network:configsafe-outputs:strict:modegithub:toolbash:tooledit:toolcache-memory:features.copilot-requestsweb-fetch:toolplaywright:toolsandbox:engine.agentmcp-scripts:max-continuations:engine.version(copilot)engine.envengine.argsengine.api-target3️⃣ Missed Opportunities
View High Priority Opportunities (🔴)
🔴 Opportunity 1: Enable
max-continuationsfor Complex WorkflowsWhat:
max-continuationsenables Copilot's autopilot mode, allowing it to complete work across multiple consecutive runs without human intervention. This is the only Copilot-exclusive feature not available in Claude/Codex.Why It Matters: Long-running tasks like CI fixing, doc updates, and code refactoring often can't complete in a single run.
max-continuationsis the mechanism that enables true autonomous completion.Where: These workflows would benefit most:
hourly-ci-cleaner.md— Already uses extended engine config but doesn't set max-continuationsdead-code-remover.md— Code removal requires multiple iterationscode-simplifier.md— Simplification across large codebasesdaily-workflow-updater.md— Updating many workflows in sequencedaily-testify-uber-super-expert.md— Test refactoringHow to Implement:
Note: The comment in
hourly-ci-cleaner.mdeven acknowledges this gap:# Note: max-turns not available for Copilot engine (Claude only)— but misses thatmax-continuationsIS available for Copilot!🔴 Opportunity 2: Add Sandbox to Code-Modifying Workflows
What: The AWF sandbox (Agent Workspace Firewall) isolates the agent from the host environment, preventing unintended side effects and enforcing network restrictions.
Why It Matters: Workflows that modify code (
edit:enabled) running without sandbox isolation have unrestricted access to the GitHub Actions runner. For scheduled workflows running autonomously, this is a security risk.Where: The following copilot workflows have
edit:but nosandbox:(sample):code-simplifier.mddead-code-remover.mddaily-file-diet.mddaily-compiler-quality.mddaily-testify-uber-super-expert.mdbreaking-change-checker.mdjsweep.mdHow to Implement:
View Medium Priority Opportunities (🟡)
🟡 Opportunity 3: Leverage Custom Agent Files for Specialized Workflows
What:
.github/agents/contains 9 agent files with domain-specific personas and instructions. Only 3 workflows useengine.agent.Available Unused Agents:
agentic-workflows.agent.md— Expert in writing gh-aw workflowscontribution-checker.agent.md— PR review specialistgrumpy-reviewer.agent.md— Critical code reviewercreate-safe-output-type.agent.md— Safe output specialistinteractive-agent-designer.agent.md— UX-focused agent designerw3c-specification-writer.agent.md— Spec writingcustom-engine-implementation.agent.md— Engine implementation expertWhere: Workflows that would benefit:
contribution-check.md→ usecontribution-checkeragentcode-scanning-fixer.md→ usegrumpy-reviewerfor careful analysiscraft.md/workflow-generator.md→ useagentic-workflowsagentHow to Implement:
🟡 Opportunity 4: Narrow GitHub Toolsets for Least-Privilege Access
What: 46 workflows use
toolsets: [default]which grants access to context, repos, issues, and pull_requests APIs. Many workflows only need a subset.Example: A workflow that only creates discussions could use:
Mapping:
[default][repos, issues]or[discussions][default][pull_requests, repos][default][issues, labels][default, actions][actions]🟡 Opportunity 5: Use
engine.modelfor Cost OptimizationWhat: The Copilot engine passes the model via
COPILOT_MODEL. Only 1 workflow (poem-bot.md) explicitly sets a model. Lightweight workflows (reports, summaries) could use faster, cheaper models.How to Implement:
Candidates for lighter models:
daily-fact.md— Simple daily fact generationdaily-community-attribution.md— Structured data processingView Low Priority Opportunities (🟢)
🟢 Opportunity 6: Pin
engine.versionfor Production-Critical WorkflowsWhat: No copilot workflows pin a specific CLI version.
latestinstalls whatever is current, which can cause unexpected behavior when new versions ship.How to Implement:
Best candidates: Workflows with complex tool configurations or that have historically been sensitive to CLI behavior changes.
🟢 Opportunity 7: Use
engine.envfor Workflow-Specific ConfigurationWhat:
engine.envcan inject custom environment variables into the agent execution. No workflow currently uses this.Use cases:
How to Implement:
🟢 Opportunity 8: Expand
mcp-scriptsAdoptionWhat: Only 3 workflows use mcp-scripts. This feature enables custom MCP server scripts for specialized tool outputs with structured data.
Where: Workflows that need structured outputs beyond what safe-outputs provides could benefit.
4️⃣ Specific Workflow Recommendations
View Workflow-Specific Recommendations
hourly-ci-cleaner.mdengine.agent: ci-cleaner, but the comment explicitly notes max-turns isn't available for Copilot — missing thatmax-continuationsIS availablemax-continuations: 5to allow the CI cleaner to complete complex fixes across multiple roundscontribution-check.mdengine: copilotwithout agent customizationengine.agent: contribution-checkerto leverage the dedicated agent filecode-simplifier.md/dead-code-remover.mdedit:tool without sandbox isolationsandbox:with appropriate mounts for the build tools useddaily-malicious-code-scan.md[repos, code_security])research.md5️⃣ Trends & Insights
View Trend Analysis
This is the first run of this research workflow — baseline established for future comparisons.
Key baseline metrics (2026-03-30):
max-continuationsadoption: 1%engine.agentadoption: 4%Notable Patterns:
strict:mode (67%) — security posture is generally goodsafe-outputs:adoption (96%) — almost all workflows properly gate writes through safe-outputscache-memorywell adopted (41%) — good state persistence practices existFuture tracking targets:
max-continuationsadoption after this reportengine.agentusage growth as new agent personas are developed6️⃣ Best Practice Guidelines
Based on this analysis, here are the recommended best practices for Copilot workflows:
Code-modifying workflows → Always add
sandbox:Any workflow with
edit:should include sandbox isolation.hourly-ci-cleaner.mdis the gold standard.Long-running autonomous tasks → Use
max-continuations:When a task genuinely requires multiple rounds of iteration (fix → test → fix), set
max-continuationsinstead of hoping one pass is enough.Use least-privilege toolsets
Instead of
[default]for every workflow, map the actual GitHub APIs needed.daily-malicious-code-scan.mdis the model:[repos, code_security].Match agent files to workflow specialization
The
.github/agents/directory has 9 specialized personas. Useengine.agentfor specialized workflows.Pin versions for stability-sensitive workflows
Workflows with complex tool configurations should pin
engine.versionto avoid surprises on CLI updates.7️⃣ Action Items
Immediate (high-impact, low-effort):
max-continuations: 3-5tohourly-ci-cleaner.md(the comment literally says it's missing!)engine.agent: contribution-checkertocontribution-check.mdengine.agent: agentic-workflowstocraft.mdandworkflow-generator.mdShort-term (this month):
edit:and nosandbox:— add sandbox to top 10 highest-risktoolsets: [default]with precise toolsets in read-only reporting workflowsmax-continuationsfordead-code-remover.mdandcode-simplifier.mdLong-term (this quarter):
engine.modeloptimization for lightweight reporting workflowsmcp-scriptsusage for workflows needing structured tool outputsView Research Methodology
Research Methodology
Data Sources:
pkg/workflow/copilot_engine*.go,copilot_mcp.go,copilot_engine_tools.godocs/src/content/docs/reference/engines.md*.mdfiles in.github/workflows/.github/agents/*.agent.mdAnalysis Approach:
Limitations:
Tooling
grepfor pattern matchingPrevious Research
References:
copilot_engine_execution.goBeta Was this translation helpful? Give feedback.
All reactions