From b713648338a9d294756a774896bc2b4d50f7fa49 Mon Sep 17 00:00:00 2001 From: Hong Yi Chen Date: Sat, 15 Aug 2026 16:54:57 -0700 Subject: [PATCH 1/5] docs(factories): document the control room Co-Authored-By: Warp Agent --- src/content/docs/factories/control-room.mdx | 106 +++++++++++++++++++- 1 file changed, 103 insertions(+), 3 deletions(-) diff --git a/src/content/docs/factories/control-room.mdx b/src/content/docs/factories/control-room.mdx index d6a1f5552..cdc4d46a2 100644 --- a/src/content/docs/factories/control-room.mdx +++ b/src/content/docs/factories/control-room.mdx @@ -1,9 +1,109 @@ --- -title: Control room +title: The Factory control room description: >- - Factory control room documentation will cover monitoring and intervention in a follow-up PR. + Track work items, inspect agent runs, read factory metrics, and make the + configuration changes the control room supports. sidebar: label: "Control room" --- -Factory control room documentation will land in a follow-up PR. +The control room is the web surface for one factory. It shows what your agents are working on, what they produced, the recorded attributed cost of that work, and how the factory is configured. Select a factory in the sidebar to expand its pages. + +## Where each surface lives + +Open a factory in the sidebar to reach its pages. **Runs**, **MCPs and apps**, and **Secrets** sit above the factory list and cover your whole team. + +| Page | What it shows | What you do there | +| --- | --- | --- | +| **Dashboard** | Autonomy, time saved, PR latency, cost per PR, pull requests, run breakdown, scorer results, and Self-improvement PRs | Compare periods and find work worth investigating | +| **Activity** | Work items grouped by stage | Search, filter, open a work item, and stop one | +| **Agents** | The factory's agent roster | Create and edit agents, instructions, models, runners, hosts, secrets, and MCP servers | +| **Automations** | Triggers that start runs | Create, edit, and delete automations | +| **Runs** | Individual agent runs for this factory | Search, filter, start, cancel, and score runs, or create a benchmark task from one | +| **Scorers** | Scorer definitions and results | Define rubrics and review classifications | +| **Benchmarks** | Benchmark suites and their runs | Compare harness, model, and runner configurations on fixed tasks | +| **Code** | The factory's definition files | Browse and edit a Warp-managed definition | +| **Settings** | Factory identity, repos, runners, Self-improvement, integrations, and deletion | Change configuration the factory owns | + +## Track work items on Activity + +**Activity** is the stage view for the factory's work items, rendered as one grouped list ordered by stage: Triage, Planning, Building, and Reviewing. + +* **Search** - Enter text in **Search tasks...** to match work item titles. +* **Filter** - Click the filter control to add a **Stage**, **Created by**, or **Created on** filter. Active filters appear as chips with a running result count and a **Clear** action. +* **Sort** - Choose **Created** or **Updated**, in either direction. + +Two defaults shape what you see. Activity filters to work items you created, so a teammate's work stays hidden until you change the **Created by** filter. It also shows only the four active stages. To see finished work, add a **Stage** filter and select **Complete** or **Cancelled**. + +Filters and the open work item are stored in the URL, so a filtered view survives a reload and is shareable. Search text and sort order are not. + +### Open and stop a work item + +Click a row to open the detail pane beside the list. The pane shows the work item title, current agent activity, stage, originating prompt rendered as Markdown, external references, pull request artifacts, environment, cost, and run time. + +Click **Event history** to open a reverse-chronological timeline of the runs behind the work item. Click the short identifier in the pane header to copy the full work item ID. + +Use **Stop task** in the pane header to cancel the work item's current run. + +:::caution +**Stop task** takes effect immediately with no confirmation prompt. It appears only when the current run is cancellable and is disabled once that run reaches a terminal state. +::: + +## Inspect runs + +A run is one agent execution. The team-level **Runs** page lists every run you have access to; a factory's **Runs** page lists only runs executed by that factory's agents. Both support search, sorting, and filters for source, status, execution location, creator, creation date, and agent. + +Click **New** on a factory's **Runs** page to send a prompt, with optional attachments, to that factory's foreman agent. Click a run to open its detail pane, which shows the run timeline, the triggering prompt, cost, environment, harness, artifacts, and, for an orchestrator run, a **Sub-agents** tab listing its child runs. **View session** opens the agent's session in a new tab. + +Where a run supports it, the pane offers **Stop run** and **Score run**. You can also create a benchmark task from an existing run. + +:::note +The control room reports run progress; it does not stream it. Pages refresh on a polling interval, so new events appear within seconds rather than instantly. There is no in-page conversation, approval, or redirection control: apart from stopping and scoring a run, you observe here and steer the work through your integrations and your repository's review process. +::: + +## Read dashboard metrics + +**Dashboard** covers a date range you choose with the presets or the date picker. Three cards head the page: **Autonomy** (the share of merged PRs that reached merge with no human touch beyond an approval review), **Time saved** (merged-PR line changes converted to approximate engineer-hours), and **PR latency** (median time from run kickoff through PR, review, and merge). + +Below them, **Cost per PR** headlines the median cost of the PRs opened in the range, with **By complexity** and **By size** views and an expander for the most expensive PRs by attributed cost. It is a lower-bound estimate that can omit some run usage and does not match billing; see [Measure and improve a factory](/factories/measure-and-improve/) for its limitations. The **Pull requests** chart compares opened and merged PRs, and **Run breakdown** charts total runs with a drill-down by agent type, status, source, root versus subruns, model, and harness. + +The **Insights** panel lists **Self-improvement PRs** opened by your foreman, linking each one out to the code host. When scorers are available, scorer cards summarize recent classification results underneath the charts. + +## Manage agents and automations + +**Agents** lists the factory's roster with search and a type filter covering Foreman, Triage, Spec, Code, Review, Monitor, and Custom. Click **New** to add an agent, or click a card to edit its name, description, instructions, model, runner, host, attached secrets, and MCP servers, plus the automations that target it. + +**Automations** lists each automation with its trigger, target agent, and creation date. The editor builds triggers from a schedule (hourly, daily, weekly, or a custom cron expression) or from a GitHub, Linear, Slack, or Jira event, with per-provider filters. + +Two limits apply. The agent editor sets the model, runner, and host but not a harness or environment, and the automation editor overrides no execution settings; set what these omit in the factory definition. When a factory's definition lives in an external repository, both pages become read-only and link to the source of truth. + +## Edit definitions in the Code tab + +**Code** reflects how the factory's definition is managed. For a Warp-managed definition, it opens a file tree and editor: browse the definition, edit files, and save a batch of changes as one atomic commit. Saving validates the whole tree first, and problems come back as diagnostics located to a file, line, and column in the problems drawer. An unsaved edit prompts you before you navigate away. + +For a definition managed in GitHub, **Code** is read-only and offers **Open repository** instead, because pull requests there are the edit path. A live-managed factory has no source repository and says so. + +## Score and benchmark + +**Scorers** defines how an LLM judge classifies completed conversations: rubric and labels, threshold, agent scope, sampling rate, judge model, and whether the scorer is active or paused. Each scorer has a **Self-improvement** section you enable or disable. + +Benchmark suites define an agent and its tasks, each with success criteria. The launch dialog chooses the harness, model, and runner configurations to compare, the scorers, and how many repetitions to run. Results include a scoreboard, pass rates, and per-task comparisons across configurations. Because benchmark quality depends on scorer classifications, read results alongside [Measure and improve a factory](/factories/measure-and-improve/). + +## Change factory settings + +**Settings** holds the configuration the factory owns: + +* **General info** - Change the factory name, avatar, and connected repos, and choose whether pull requests are authored by the agent or the run creator. The factory alias is read-only. +* **Self-improvement** - Choose the **Analysis model** used to analyze failed runs and group recurring issues before fixes enter the factory. +* **Runners** - Review, create, and edit the factory's runners for a Warp-managed factory; a file-managed factory shows that `runners/*.yaml` is the source of truth and links to the repository. This section shares the Code tab's availability. +* **Factory repo** - See where the factory's definition is hosted. +* **Factory integrations** - Review the connected Slack app. When your permissions and the factory's source mode allow changes, set up, edit, disconnect, or remove cloud provider configurations. +* **Danger zone** - Delete the factory. Deletion cannot be undone. + +Settings respects your permissions. When you lack edit access, or the configuration is managed in an external repository, the affected controls are disabled and the page explains where to make the change. + +## Next steps + +* [How Warp Factories work](/factories/how-factories-work/) - Understand why a work item moves between the stages you just read, and where the human decisions sit. +* [Factory definitions as code](/factories/factory-as-code/) - Define agents, automations, runners, and source ownership. +* [Measure and improve a factory](/factories/measure-and-improve/) - Configure the scorers and benchmarks behind the Dashboard. From 31a0fcc6faa45562df0e588fb3613811aa4080bc Mon Sep 17 00:00:00 2001 From: hongyi-chen Date: Sun, 16 Aug 2026 05:30:00 +0000 Subject: [PATCH 2/5] docs(factories): clarify control room copy Editorial pass on the control room page for external readers: - Replace internal jargon (web surface, source mode, terminal state, polling interval, atomic commit) with plain language - State Activity's default filters directly instead of "two defaults shape what you see" - Restructure dense comma lists into scannable bullets (dashboard metrics, run pane actions, Code tab source modes, editor boundaries) - Explain what the scorer Self-improvement toggle does, per the UI copy - Fix inverted phrasing ("the automation editor overrides no execution settings") and cut duplicated sidebar orientation No factual changes; all labels and behavior stay as verified. Co-Authored-By: Warp --- src/content/docs/factories/control-room.mdx | 108 ++++++++++++-------- 1 file changed, 65 insertions(+), 43 deletions(-) diff --git a/src/content/docs/factories/control-room.mdx b/src/content/docs/factories/control-room.mdx index cdc4d46a2..fae4056bd 100644 --- a/src/content/docs/factories/control-room.mdx +++ b/src/content/docs/factories/control-room.mdx @@ -1,109 +1,131 @@ --- title: The Factory control room description: >- - Track work items, inspect agent runs, read factory metrics, and make the - configuration changes the control room supports. + Track work items, inspect runs, read factory metrics, and manage agents, + automations, and settings from the control room. sidebar: label: "Control room" --- -The control room is the web surface for one factory. It shows what your agents are working on, what they produced, the recorded attributed cost of that work, and how the factory is configured. Select a factory in the sidebar to expand its pages. +The control room is the web app for operating a single factory. Use it to track the work your agents are doing, inspect the runs and pull requests they produce, and manage the agents, automations, and settings the factory owns. -## Where each surface lives +## Pages at a glance -Open a factory in the sidebar to reach its pages. **Runs**, **MCPs and apps**, and **Secrets** sit above the factory list and cover your whole team. +Select a factory in the sidebar to open its pages. **Runs**, **MCPs and apps**, and **Secrets** sit above the factory list and cover your whole team, not a single factory. | Page | What it shows | What you do there | | --- | --- | --- | -| **Dashboard** | Autonomy, time saved, PR latency, cost per PR, pull requests, run breakdown, scorer results, and Self-improvement PRs | Compare periods and find work worth investigating | -| **Activity** | Work items grouped by stage | Search, filter, open a work item, and stop one | -| **Agents** | The factory's agent roster | Create and edit agents, instructions, models, runners, hosts, secrets, and MCP servers | +| **Dashboard** | Factory metrics: autonomy, time saved, PR latency, cost, and run volume | Compare periods and find work worth investigating | +| **Activity** | Work items grouped by stage | Search, filter, open, and stop work items | +| **Agents** | The factory's agent roster | Create and edit agents | | **Automations** | Triggers that start runs | Create, edit, and delete automations | -| **Runs** | Individual agent runs for this factory | Search, filter, start, cancel, and score runs, or create a benchmark task from one | +| **Runs** | The factory's individual agent runs | Start, stop, and score runs | | **Scorers** | Scorer definitions and results | Define rubrics and review classifications | -| **Benchmarks** | Benchmark suites and their runs | Compare harness, model, and runner configurations on fixed tasks | +| **Benchmarks** | Benchmark suites and their runs | Compare harness, model, and runner configurations | | **Code** | The factory's definition files | Browse and edit a Warp-managed definition | -| **Settings** | Factory identity, repos, runners, Self-improvement, integrations, and deletion | Change configuration the factory owns | +| **Settings** | Configuration the factory owns | Change identity, repos, runners, and integrations | ## Track work items on Activity -**Activity** is the stage view for the factory's work items, rendered as one grouped list ordered by stage: Triage, Planning, Building, and Reviewing. +**Activity** shows the factory's work items grouped by stage: Triage, Planning, Building, and Reviewing. -* **Search** - Enter text in **Search tasks...** to match work item titles. -* **Filter** - Click the filter control to add a **Stage**, **Created by**, or **Created on** filter. Active filters appear as chips with a running result count and a **Clear** action. -* **Sort** - Choose **Created** or **Updated**, in either direction. +* **Search** - Type in **Search tasks...** to match work item titles. +* **Filter** - Add a **Stage**, **Created by**, or **Created on** filter. Active filters appear as chips with a result count and a **Clear** action. +* **Sort** - Order work items by **Created** or **Updated**, in either direction. -Two defaults shape what you see. Activity filters to work items you created, so a teammate's work stays hidden until you change the **Created by** filter. It also shows only the four active stages. To see finished work, add a **Stage** filter and select **Complete** or **Cancelled**. +By default, Activity shows only work items you created, and only the four active stages. To see a teammate's work, change the **Created by** filter. To see finished work, add a **Stage** filter and select **Complete** or **Cancelled**. -Filters and the open work item are stored in the URL, so a filtered view survives a reload and is shareable. Search text and sort order are not. +Filters and the open work item are stored in the URL, so you can reload or share a filtered view. Search text and sort order reset on reload. ### Open and stop a work item -Click a row to open the detail pane beside the list. The pane shows the work item title, current agent activity, stage, originating prompt rendered as Markdown, external references, pull request artifacts, environment, cost, and run time. +Click a work item to open its detail pane. The pane shows the work item's stage, current agent activity, the prompt that started it, external references, pull requests, environment, cost, and run time. Click the short ID in the pane header to copy the full work item ID. -Click **Event history** to open a reverse-chronological timeline of the runs behind the work item. Click the short identifier in the pane header to copy the full work item ID. - -Use **Stop task** in the pane header to cancel the work item's current run. +* **Event history** - A timeline of the runs behind the work item, newest first. +* **Stop task** - Cancels the work item's current run. Available only while a run is in progress. :::caution -**Stop task** takes effect immediately with no confirmation prompt. It appears only when the current run is cancellable and is disabled once that run reaches a terminal state. +**Stop task** takes effect immediately, with no confirmation prompt. ::: ## Inspect runs -A run is one agent execution. The team-level **Runs** page lists every run you have access to; a factory's **Runs** page lists only runs executed by that factory's agents. Both support search, sorting, and filters for source, status, execution location, creator, creation date, and agent. +A run is a single agent execution. Runs appear in two places: + +* The team-level **Runs** page lists every run you have access to, across all factories. +* A factory's **Runs** page lists only runs executed by that factory's agents. -Click **New** on a factory's **Runs** page to send a prompt, with optional attachments, to that factory's foreman agent. Click a run to open its detail pane, which shows the run timeline, the triggering prompt, cost, environment, harness, artifacts, and, for an orchestrator run, a **Sub-agents** tab listing its child runs. **View session** opens the agent's session in a new tab. +Both pages support search, sorting, and filters for status, source, agent, creator, creation date, and execution location. -Where a run supports it, the pane offers **Stop run** and **Score run**. You can also create a benchmark task from an existing run. +On a factory's **Runs** page, click **New** to send a prompt, with optional attachments, to the factory's foreman agent. + +Click a run to open its detail pane, which shows the run timeline, triggering prompt, cost, environment, and harness. An orchestrator run adds a **Sub-agents** tab listing its child runs. From the pane, click **View session** to open the agent's session in a new tab. Where the run supports it, you can also **Stop run**, **Score run**, or create a benchmark task from the run. :::note -The control room reports run progress; it does not stream it. Pages refresh on a polling interval, so new events appear within seconds rather than instantly. There is no in-page conversation, approval, or redirection control: apart from stopping and scoring a run, you observe here and steer the work through your integrations and your repository's review process. +The control room is for watching work, not steering it. Pages refresh every few seconds, and there is no in-page way to chat with an agent or approve its actions. Direct the work itself through your integrations and your repository's review process. ::: ## Read dashboard metrics -**Dashboard** covers a date range you choose with the presets or the date picker. Three cards head the page: **Autonomy** (the share of merged PRs that reached merge with no human touch beyond an approval review), **Time saved** (merged-PR line changes converted to approximate engineer-hours), and **PR latency** (median time from run kickoff through PR, review, and merge). +**Dashboard** summarizes the factory over a date range you choose with the presets or the date picker. Three cards lead the page: + +* **Autonomy** - The share of merged PRs that needed no human input beyond an approving review. +* **Time saved** - Approximate engineer-hours saved, estimated from the line changes in merged PRs. +* **PR latency** - The median time from run kickoff through PR, review, and merge. -Below them, **Cost per PR** headlines the median cost of the PRs opened in the range, with **By complexity** and **By size** views and an expander for the most expensive PRs by attributed cost. It is a lower-bound estimate that can omit some run usage and does not match billing; see [Measure and improve a factory](/factories/measure-and-improve/) for its limitations. The **Pull requests** chart compares opened and merged PRs, and **Run breakdown** charts total runs with a drill-down by agent type, status, source, root versus subruns, model, and harness. +Below the cards: -The **Insights** panel lists **Self-improvement PRs** opened by your foreman, linking each one out to the code host. When scorers are available, scorer cards summarize recent classification results underneath the charts. +* **Cost per PR** - The median cost of PRs opened in the range, with **By complexity** and **By size** views and an expandable list of the most expensive PRs. Treat it as a lower-bound estimate: it can miss some run usage and does not match billing. See [Measure and improve a factory](/factories/measure-and-improve/) for its limitations. +* **Pull requests** - PRs opened versus merged over the range. +* **Run breakdown** - Total runs, with drill-downs by agent type, status, source, root runs versus subruns, model, and harness. +* **Insights** - Self-improvement PRs opened by your foreman agent, each linked to your code host. + +When scorers are set up, scorer cards below the charts summarize recent classification results. ## Manage agents and automations -**Agents** lists the factory's roster with search and a type filter covering Foreman, Triage, Spec, Code, Review, Monitor, and Custom. Click **New** to add an agent, or click a card to edit its name, description, instructions, model, runner, host, attached secrets, and MCP servers, plus the automations that target it. +**Agents** lists the factory's roster, with search and a type filter (Foreman, Triage, Spec, Code, Review, Monitor, or Custom). Click **New** to create an agent, or click a card to edit its name, description, instructions, model, runner, host, secrets, and MCP servers, along with the automations that target it. + +**Automations** lists the triggers that start runs, showing each automation's trigger, target agent, and creation date. An automation runs on a schedule (hourly, daily, weekly, or a custom cron expression) or on a GitHub, Linear, Slack, or Jira event, with per-provider filters. -**Automations** lists each automation with its trigger, target agent, and creation date. The editor builds triggers from a schedule (hourly, daily, weekly, or a custom cron expression) or from a GitHub, Linear, Slack, or Jira event, with per-provider filters. +Two things live outside these editors: -Two limits apply. The agent editor sets the model, runner, and host but not a harness or environment, and the automation editor overrides no execution settings; set what these omit in the factory definition. When a factory's definition lives in an external repository, both pages become read-only and link to the source of truth. +* **Harness and environment** - The agent editor sets the model, runner, and host; set the harness and environment in the factory definition. Automations never override execution settings. +* **Externally managed factories** - When the factory's definition lives in an external repository, both pages are read-only and link to the repository. Make changes there through pull requests. ## Edit definitions in the Code tab -**Code** reflects how the factory's definition is managed. For a Warp-managed definition, it opens a file tree and editor: browse the definition, edit files, and save a batch of changes as one atomic commit. Saving validates the whole tree first, and problems come back as diagnostics located to a file, line, and column in the problems drawer. An unsaved edit prompts you before you navigate away. +What the **Code** tab offers depends on where the factory's definition lives: -For a definition managed in GitHub, **Code** is read-only and offers **Open repository** instead, because pull requests there are the edit path. A live-managed factory has no source repository and says so. +* **Warp-managed** - A file tree and editor. Browse the definition, edit files, and save your changes as a single commit. Warp validates the whole definition before saving and lists any problems with their file and line. If you try to navigate away with unsaved edits, the page prompts you first. +* **Managed in GitHub** - Read-only, with an **Open repository** button. Edit the definition through pull requests in the repository. +* **Managed through the API** - No definition files exist, so there is nothing to browse. ## Score and benchmark -**Scorers** defines how an LLM judge classifies completed conversations: rubric and labels, threshold, agent scope, sampling rate, judge model, and whether the scorer is active or paused. Each scorer has a **Self-improvement** section you enable or disable. +A scorer is an LLM judge that classifies completed runs against a rubric you write. On **Scorers**, define each scorer's rubric, labels, and pass threshold, choose which agents it covers, and set its sampling rate and judge model. You can pause and resume a scorer at any time. + +Each scorer also has a **Self-improvement** toggle. When it's on, the factory periodically triages the runs that scorer marks as failing and files fixes for recurring issues. + +**Benchmarks** compares configurations on a fixed set of tasks. A suite defines an agent and its tasks, each with success criteria. When you launch a suite, choose the harness, model, and runner combinations to compare, the scorers to apply, and the number of repetitions. Results include a scoreboard, pass rates, and per-task comparisons across configurations. -Benchmark suites define an agent and its tasks, each with success criteria. The launch dialog chooses the harness, model, and runner configurations to compare, the scorers, and how many repetitions to run. Results include a scoreboard, pass rates, and per-task comparisons across configurations. Because benchmark quality depends on scorer classifications, read results alongside [Measure and improve a factory](/factories/measure-and-improve/). +Benchmark results are only as reliable as the scorers behind them, so read them alongside [Measure and improve a factory](/factories/measure-and-improve/). ## Change factory settings **Settings** holds the configuration the factory owns: -* **General info** - Change the factory name, avatar, and connected repos, and choose whether pull requests are authored by the agent or the run creator. The factory alias is read-only. -* **Self-improvement** - Choose the **Analysis model** used to analyze failed runs and group recurring issues before fixes enter the factory. -* **Runners** - Review, create, and edit the factory's runners for a Warp-managed factory; a file-managed factory shows that `runners/*.yaml` is the source of truth and links to the repository. This section shares the Code tab's availability. +* **General info** - Rename the factory, change its avatar, connect repos, and choose whether pull requests are authored by the agent or by the run creator. The factory alias is read-only. +* **Self-improvement** - Choose the **Analysis model** that analyzes failed runs and groups recurring issues before fixes enter the factory. +* **Runners** - Create and edit the factory's runners. For a file-managed factory, `runners/*.yaml` in the repository is the source of truth, and this section links there instead. * **Factory repo** - See where the factory's definition is hosted. -* **Factory integrations** - Review the connected Slack app. When your permissions and the factory's source mode allow changes, set up, edit, disconnect, or remove cloud provider configurations. +* **Factory integrations** - Review the connected Slack app and set up, edit, or remove cloud provider configurations. * **Danger zone** - Delete the factory. Deletion cannot be undone. -Settings respects your permissions. When you lack edit access, or the configuration is managed in an external repository, the affected controls are disabled and the page explains where to make the change. +If you lack edit access, or a setting is managed in an external repository, the control is disabled and the page points you to where the change belongs. ## Next steps -* [How Warp Factories work](/factories/how-factories-work/) - Understand why a work item moves between the stages you just read, and where the human decisions sit. -* [Factory definitions as code](/factories/factory-as-code/) - Define agents, automations, runners, and source ownership. +* [How Warp Factories work](/factories/how-factories-work/) - The lifecycle behind Activity's stages and where humans stay in the loop. +* [Factory definitions as code](/factories/factory-as-code/) - Define agents, automations, runners, and source ownership in code. * [Measure and improve a factory](/factories/measure-and-improve/) - Configure the scorers and benchmarks behind the Dashboard. From 1f2c243c527a569ccc2ca573dee832747a860ff6 Mon Sep 17 00:00:00 2001 From: hongyi-chen Date: Sun, 16 Aug 2026 05:36:09 +0000 Subject: [PATCH 3/5] docs(factories): trim UI narration from control room page Cut prose that describes what UI elements do rather than documenting behavior: search/filter/sort control walkthroughs, filter chip and URL state mechanics, pane field inventories, refresh cadence, and editor form-field enumerations. Kept the semantics readers can't infer from the UI: default Activity filters, metric definitions and cost caveats, the Stop task warning, source-mode boundaries, and read-only rules. Co-Authored-By: Warp --- src/content/docs/factories/control-room.mdx | 76 +++++---------------- 1 file changed, 18 insertions(+), 58 deletions(-) diff --git a/src/content/docs/factories/control-room.mdx b/src/content/docs/factories/control-room.mdx index fae4056bd..a98d311c2 100644 --- a/src/content/docs/factories/control-room.mdx +++ b/src/content/docs/factories/control-room.mdx @@ -29,20 +29,9 @@ Select a factory in the sidebar to open its pages. **Runs**, **MCPs and apps**, **Activity** shows the factory's work items grouped by stage: Triage, Planning, Building, and Reviewing. -* **Search** - Type in **Search tasks...** to match work item titles. -* **Filter** - Add a **Stage**, **Created by**, or **Created on** filter. Active filters appear as chips with a result count and a **Clear** action. -* **Sort** - Order work items by **Created** or **Updated**, in either direction. +By default, Activity shows only work items you created, and only the four active stages. Change the **Created by** filter to see a teammate's work, and add a **Stage** filter for **Complete** or **Cancelled** to see finished work. -By default, Activity shows only work items you created, and only the four active stages. To see a teammate's work, change the **Created by** filter. To see finished work, add a **Stage** filter and select **Complete** or **Cancelled**. - -Filters and the open work item are stored in the URL, so you can reload or share a filtered view. Search text and sort order reset on reload. - -### Open and stop a work item - -Click a work item to open its detail pane. The pane shows the work item's stage, current agent activity, the prompt that started it, external references, pull requests, environment, cost, and run time. Click the short ID in the pane header to copy the full work item ID. - -* **Event history** - A timeline of the runs behind the work item, newest first. -* **Stop task** - Cancels the work item's current run. Available only while a run is in progress. +Click a work item to open its detail pane, which includes the prompt that started it, the pull requests it produced, and its cost. **Event history** lists the runs behind the work item, and **Stop task** cancels the current one. :::caution **Stop task** takes effect immediately, with no confirmation prompt. @@ -50,79 +39,50 @@ Click a work item to open its detail pane. The pane shows the work item's stage, ## Inspect runs -A run is a single agent execution. Runs appear in two places: - -* The team-level **Runs** page lists every run you have access to, across all factories. -* A factory's **Runs** page lists only runs executed by that factory's agents. - -Both pages support search, sorting, and filters for status, source, agent, creator, creation date, and execution location. - -On a factory's **Runs** page, click **New** to send a prompt, with optional attachments, to the factory's foreman agent. +A run is a single agent execution. The team-level **Runs** page lists every run you have access to; a factory's **Runs** page lists only runs from that factory's agents. -Click a run to open its detail pane, which shows the run timeline, triggering prompt, cost, environment, and harness. An orchestrator run adds a **Sub-agents** tab listing its child runs. From the pane, click **View session** to open the agent's session in a new tab. Where the run supports it, you can also **Stop run**, **Score run**, or create a benchmark task from the run. +Click **New** on a factory's **Runs** page to send a prompt to the factory's foreman agent. Open a run to see its timeline and cost, plus a **Sub-agents** tab for an orchestrator run's child runs. From there you can view the agent's full session, stop or score the run, or turn it into a benchmark task. :::note -The control room is for watching work, not steering it. Pages refresh every few seconds, and there is no in-page way to chat with an agent or approve its actions. Direct the work itself through your integrations and your repository's review process. +The control room is for observing runs, not steering them. There is no in-page way to chat with an agent or approve its actions. Direct the work through your integrations and your repository's review process. ::: ## Read dashboard metrics -**Dashboard** summarizes the factory over a date range you choose with the presets or the date picker. Three cards lead the page: +**Dashboard** summarizes the factory over a date range you choose: * **Autonomy** - The share of merged PRs that needed no human input beyond an approving review. * **Time saved** - Approximate engineer-hours saved, estimated from the line changes in merged PRs. * **PR latency** - The median time from run kickoff through PR, review, and merge. +* **Cost per PR** - The median cost of PRs opened in the range. Treat it as a lower-bound estimate: it can miss some run usage and does not match billing. See [Measure and improve a factory](/factories/measure-and-improve/) for its limitations. -Below the cards: - -* **Cost per PR** - The median cost of PRs opened in the range, with **By complexity** and **By size** views and an expandable list of the most expensive PRs. Treat it as a lower-bound estimate: it can miss some run usage and does not match billing. See [Measure and improve a factory](/factories/measure-and-improve/) for its limitations. -* **Pull requests** - PRs opened versus merged over the range. -* **Run breakdown** - Total runs, with drill-downs by agent type, status, source, root runs versus subruns, model, and harness. -* **Insights** - Self-improvement PRs opened by your foreman agent, each linked to your code host. - -When scorers are set up, scorer cards below the charts summarize recent classification results. +The page also charts opened versus merged PRs and a breakdown of runs, and lists the Self-improvement PRs your foreman agent has opened. When scorers are set up, scorer cards summarize recent classification results. ## Manage agents and automations -**Agents** lists the factory's roster, with search and a type filter (Foreman, Triage, Spec, Code, Review, Monitor, or Custom). Click **New** to create an agent, or click a card to edit its name, description, instructions, model, runner, host, secrets, and MCP servers, along with the automations that target it. - -**Automations** lists the triggers that start runs, showing each automation's trigger, target agent, and creation date. An automation runs on a schedule (hourly, daily, weekly, or a custom cron expression) or on a GitHub, Linear, Slack, or Jira event, with per-provider filters. - -Two things live outside these editors: +**Agents** is the factory's roster. Create agents and edit their instructions, model, runner, host, secrets, and MCP servers. **Automations** defines the triggers that start runs: a schedule (including custom cron expressions) or a GitHub, Linear, Slack, or Jira event. -* **Harness and environment** - The agent editor sets the model, runner, and host; set the harness and environment in the factory definition. Automations never override execution settings. -* **Externally managed factories** - When the factory's definition lives in an external repository, both pages are read-only and link to the repository. Make changes there through pull requests. +Harness and environment are set in the factory definition, not the agent editor, and automations never override execution settings. When the factory's definition lives in an external repository, both pages are read-only; make changes there through pull requests. ## Edit definitions in the Code tab -What the **Code** tab offers depends on where the factory's definition lives: +What **Code** offers depends on where the factory's definition lives: -* **Warp-managed** - A file tree and editor. Browse the definition, edit files, and save your changes as a single commit. Warp validates the whole definition before saving and lists any problems with their file and line. If you try to navigate away with unsaved edits, the page prompts you first. -* **Managed in GitHub** - Read-only, with an **Open repository** button. Edit the definition through pull requests in the repository. -* **Managed through the API** - No definition files exist, so there is nothing to browse. +* **Warp-managed** - Browse and edit the definition files. Saving validates the definition and commits all changes together. +* **Managed in GitHub** - Read-only. Edit the definition through pull requests in the repository. +* **Managed through the API** - There are no definition files to browse. ## Score and benchmark -A scorer is an LLM judge that classifies completed runs against a rubric you write. On **Scorers**, define each scorer's rubric, labels, and pass threshold, choose which agents it covers, and set its sampling rate and judge model. You can pause and resume a scorer at any time. +A scorer is an LLM judge that classifies completed runs against a rubric you define, scoped to the agents you choose and sampled at a rate you set. Each scorer has a **Self-improvement** toggle: when it's on, the factory periodically triages the runs that scorer marks as failing and files fixes for recurring issues. -Each scorer also has a **Self-improvement** toggle. When it's on, the factory periodically triages the runs that scorer marks as failing and files fixes for recurring issues. - -**Benchmarks** compares configurations on a fixed set of tasks. A suite defines an agent and its tasks, each with success criteria. When you launch a suite, choose the harness, model, and runner combinations to compare, the scorers to apply, and the number of repetitions. Results include a scoreboard, pass rates, and per-task comparisons across configurations. - -Benchmark results are only as reliable as the scorers behind them, so read them alongside [Measure and improve a factory](/factories/measure-and-improve/). +**Benchmarks** compares harness, model, and runner configurations against a fixed set of tasks with success criteria. Benchmark results are only as reliable as the scorers behind them, so read them alongside [Measure and improve a factory](/factories/measure-and-improve/). ## Change factory settings -**Settings** holds the configuration the factory owns: - -* **General info** - Rename the factory, change its avatar, connect repos, and choose whether pull requests are authored by the agent or by the run creator. The factory alias is read-only. -* **Self-improvement** - Choose the **Analysis model** that analyzes failed runs and groups recurring issues before fixes enter the factory. -* **Runners** - Create and edit the factory's runners. For a file-managed factory, `runners/*.yaml` in the repository is the source of truth, and this section links there instead. -* **Factory repo** - See where the factory's definition is hosted. -* **Factory integrations** - Review the connected Slack app and set up, edit, or remove cloud provider configurations. -* **Danger zone** - Delete the factory. Deletion cannot be undone. +**Settings** holds the configuration the factory owns: the factory's name and connected repos, whether pull requests are authored by the agent or the run creator, the **Analysis model** self-improvement uses to analyze failed runs, runners, integrations, and deletion. Deleting a factory cannot be undone. -If you lack edit access, or a setting is managed in an external repository, the control is disabled and the page points you to where the change belongs. +For a file-managed factory, `runners/*.yaml` in the repository is the source of truth. Anything managed in an external repository is read-only in Settings. ## Next steps From ef7d547920703b894cca635db9589b0975374491 Mon Sep 17 00:00:00 2001 From: hongyi-chen Date: Sun, 16 Aug 2026 23:17:53 +0000 Subject: [PATCH 4/5] =?UTF-8?q?docs(factories):=20control=20room=20feedbac?= =?UTF-8?q?k=20=E2=80=94=20session=20steering,=20self-improvement,=20code?= =?UTF-8?q?=20tab?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Replace the 'no way to steer' note: View session opens the run's shared agent session (cloud agent session sharing) for real-time follow-ups. - Add the Self-improvement page to the pages table and scorer section; drop the stale claim that the Dashboard lists self-improvement PRs. - Link the Code tab section to Factory definitions as code, align its source modes with the current UI (GitHub links out; live-managed), and document the in-room branch review (Request changes / Approve & merge). - Refresh details: harness is editable in the agent editor, Scorers are read-only for file-managed factories, work-item pane shows origin and View agent, Cost per PR expands into most expensive PRs. Co-Authored-By: Warp --- src/content/docs/factories/control-room.mdx | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/src/content/docs/factories/control-room.mdx b/src/content/docs/factories/control-room.mdx index a98d311c2..d546aee31 100644 --- a/src/content/docs/factories/control-room.mdx +++ b/src/content/docs/factories/control-room.mdx @@ -21,6 +21,7 @@ Select a factory in the sidebar to open its pages. **Runs**, **MCPs and apps**, | **Automations** | Triggers that start runs | Create, edit, and delete automations | | **Runs** | The factory's individual agent runs | Start, stop, and score runs | | **Scorers** | Scorer definitions and results | Define rubrics and review classifications | +| **Self-improvement** | Pull requests filed to fix scorer-detected failures | Review fix PRs and open the runs behind them | | **Benchmarks** | Benchmark suites and their runs | Compare harness, model, and runner configurations | | **Code** | The factory's definition files | Browse and edit a Warp-managed definition | | **Settings** | Configuration the factory owns | Change identity, repos, runners, and integrations | @@ -31,7 +32,7 @@ Select a factory in the sidebar to open its pages. **Runs**, **MCPs and apps**, By default, Activity shows only work items you created, and only the four active stages. Change the **Created by** filter to see a teammate's work, and add a **Stage** filter for **Complete** or **Cancelled** to see finished work. -Click a work item to open its detail pane, which includes the prompt that started it, the pull requests it produced, and its cost. **Event history** lists the runs behind the work item, and **Stop task** cancels the current one. +Click a work item to open its detail pane, which includes the prompt that started it, the ticket or thread it came from, the pull requests it produced, and its cost. **View agent** opens the agent's session, **Event history** lists the runs behind the work item, and **Stop task** cancels the current run. :::caution **Stop task** takes effect immediately, with no confirmation prompt. @@ -44,7 +45,7 @@ A run is a single agent execution. The team-level **Runs** page lists every run Click **New** on a factory's **Runs** page to send a prompt to the factory's foreman agent. Open a run to see its timeline and cost, plus a **Sub-agents** tab for an orchestrator run's child runs. From there you can view the agent's full session, stop or score the run, or turn it into a benchmark task. :::note -The control room is for observing runs, not steering them. There is no in-page way to chat with an agent or approve its actions. Direct the work through your integrations and your repository's review process. +Run pages don't include a chat input, but you can still steer a run: **View session** opens its [shared agent session](/platform/viewing-cloud-agent-runs/), where you follow the agent in real time and send follow-up instructions while the run's environment is active. After the environment shuts down, the same button opens the conversation transcript. ::: ## Read dashboard metrics @@ -56,25 +57,27 @@ The control room is for observing runs, not steering them. There is no in-page w * **PR latency** - The median time from run kickoff through PR, review, and merge. * **Cost per PR** - The median cost of PRs opened in the range. Treat it as a lower-bound estimate: it can miss some run usage and does not match billing. See [Measure and improve a factory](/factories/measure-and-improve/) for its limitations. -The page also charts opened versus merged PRs and a breakdown of runs, and lists the Self-improvement PRs your foreman agent has opened. When scorers are set up, scorer cards summarize recent classification results. +The page also charts opened versus merged PRs and a breakdown of runs, and the **Cost per PR** card expands to list the most expensive PRs in the range. When scorers are set up, scorer cards summarize recent classification results. ## Manage agents and automations -**Agents** is the factory's roster. Create agents and edit their instructions, model, runner, host, secrets, and MCP servers. **Automations** defines the triggers that start runs: a schedule (including custom cron expressions) or a GitHub, Linear, Slack, or Jira event. +**Agents** is the factory's roster. Create agents and edit their instructions, model or harness, runner, host, secrets, and MCP servers. **Automations** defines the triggers that start runs: a schedule (including custom cron expressions) or a GitHub, Linear, Slack, or Jira event. -Harness and environment are set in the factory definition, not the agent editor, and automations never override execution settings. When the factory's definition lives in an external repository, both pages are read-only; make changes there through pull requests. +Environment is set in the factory definition, not the agent editor, and automations never override execution settings. When the factory's definition lives in an external repository, Agents, Automations, and Scorers are read-only; make changes there through pull requests. ## Edit definitions in the Code tab -What **Code** offers depends on where the factory's definition lives: +**Code** is the control room's view of the factory's definition files, which [Factory definitions as code](/factories/factory-as-code/) describes in full. What the tab offers depends on where the definition lives: * **Warp-managed** - Browse and edit the definition files. Saving validates the definition and commits all changes together. -* **Managed in GitHub** - Read-only. Edit the definition through pull requests in the repository. -* **Managed through the API** - There are no definition files to browse. +* **Managed in GitHub** - Links to the repository; edit the definition through pull requests there. +* **Live-managed** - The factory is edited directly in the control room, so there are no definition files to browse. + +When an agent proposes a change to a Warp-managed definition, its work item on **Activity** links to a review of the branch inside the control room. From there, comment on the diff, use **Request changes** to send feedback back to the agent, or **Approve & merge**. ## Score and benchmark -A scorer is an LLM judge that classifies completed runs against a rubric you define, scoped to the agents you choose and sampled at a rate you set. Each scorer has a **Self-improvement** toggle: when it's on, the factory periodically triages the runs that scorer marks as failing and files fixes for recurring issues. +A scorer is an LLM judge that classifies completed runs against a rubric you define, scoped to the agents you choose and sampled at a rate you set. Each scorer has a **Self-improvement** toggle: when it's on, the factory periodically triages the runs that scorer marks as failing and files fixes for recurring issues. The **Self-improvement** page tracks the pull requests those fixes open, linking each one to the run that produced it. **Benchmarks** compares harness, model, and runner configurations against a fixed set of tasks with success criteria. Benchmark results are only as reliable as the scorers behind them, so read them alongside [Measure and improve a factory](/factories/measure-and-improve/). From ccb560db634ca38c4c29ca2c3a141baa6ee34278 Mon Sep 17 00:00:00 2001 From: Warp Agent Date: Mon, 17 Aug 2026 17:02:58 +0000 Subject: [PATCH 5/5] docs(factories): address control room review feedback - List Integrations among the team-level sidebar pages - Clarify Settings manages the integrations the factory can access - Mention the Complete and Cancelled terminal stages on Activity - Contrast runs with Activity work items Co-Authored-By: Warp --- src/content/docs/factories/control-room.mdx | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/src/content/docs/factories/control-room.mdx b/src/content/docs/factories/control-room.mdx index d546aee31..394511130 100644 --- a/src/content/docs/factories/control-room.mdx +++ b/src/content/docs/factories/control-room.mdx @@ -11,7 +11,7 @@ The control room is the web app for operating a single factory. Use it to track ## Pages at a glance -Select a factory in the sidebar to open its pages. **Runs**, **MCPs and apps**, and **Secrets** sit above the factory list and cover your whole team, not a single factory. +Select a factory in the sidebar to open its pages. **Runs**, **MCPs and apps**, **Secrets**, and **Integrations** sit above the factory list and cover your whole team, not a single factory. | Page | What it shows | What you do there | | --- | --- | --- | @@ -24,11 +24,11 @@ Select a factory in the sidebar to open its pages. **Runs**, **MCPs and apps**, | **Self-improvement** | Pull requests filed to fix scorer-detected failures | Review fix PRs and open the runs behind them | | **Benchmarks** | Benchmark suites and their runs | Compare harness, model, and runner configurations | | **Code** | The factory's definition files | Browse and edit a Warp-managed definition | -| **Settings** | Configuration the factory owns | Change identity, repos, runners, and integrations | +| **Settings** | Configuration the factory owns | Change identity, repos, runners, and the integrations the factory can access | ## Track work items on Activity -**Activity** shows the factory's work items grouped by stage: Triage, Planning, Building, and Reviewing. +**Activity** shows the factory's work items grouped by stage: Triage, Planning, Building, and Reviewing. Finished work items move to two terminal stages, Complete and Cancelled. By default, Activity shows only work items you created, and only the four active stages. Change the **Created by** filter to see a teammate's work, and add a **Stage** filter for **Complete** or **Cancelled** to see finished work. @@ -40,7 +40,7 @@ Click a work item to open its detail pane, which includes the prompt that starte ## Inspect runs -A run is a single agent execution. The team-level **Runs** page lists every run you have access to; a factory's **Runs** page lists only runs from that factory's agents. +A run is a single agent execution. A work item on **Activity** tracks one piece of work through the factory's stages and can span several runs as different agents pick it up. The team-level **Runs** page lists every run you have access to; a factory's **Runs** page lists only runs from that factory's agents. Click **New** on a factory's **Runs** page to send a prompt to the factory's foreman agent. Open a run to see its timeline and cost, plus a **Sub-agents** tab for an orchestrator run's child runs. From there you can view the agent's full session, stop or score the run, or turn it into a benchmark task. @@ -83,7 +83,7 @@ A scorer is an LLM judge that classifies completed runs against a rubric you def ## Change factory settings -**Settings** holds the configuration the factory owns: the factory's name and connected repos, whether pull requests are authored by the agent or the run creator, the **Analysis model** self-improvement uses to analyze failed runs, runners, integrations, and deletion. Deleting a factory cannot be undone. +**Settings** holds the configuration the factory owns: the factory's name and connected repos, whether pull requests are authored by the agent or the run creator, the **Analysis model** self-improvement uses to analyze failed runs, runners, the integrations accessible to this factory, and deletion. Deleting a factory cannot be undone. For a file-managed factory, `runners/*.yaml` in the repository is the source of truth. Anything managed in an external repository is read-only in Settings.