Skip to content

run-workflow: long runs, run records, and trace reading #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mzxchandra wants to merge 1 commit into
base: main
Choose a base branch
from
Open

run-workflow: long runs, run records, and trace reading #5

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions skills/run-workflow/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,20 @@ sim --output json workflows run <workflowId> \
Do not guess a source run or synthesize upstream outputs. Confirm that the source run belongs to the
workflow and contains the state the selected block needs.

## Runs longer than about a minute

A manual run holds its HTTP connection open for the entire execution, and `--manual` refuses
`--async`. On hosted deployments the fronting load balancer drops idle connections after roughly a
minute (observed; re-verify on the current deployment), and a dropped connection cancels the run.
The failure surfaces as a transport error such as `Could not reach <origin>: fetch failed`, not as
a timeout, so it reads like network flakiness. It is not: retrying the same synchronous run fails
the same way, and a sequence of such attempts corrupts the evidence - a deterministic workflow
starts looking nondeterministic because most of its recorded attempts are transport casualties.

When a workflow can plausibly exceed a minute, deploy it and run `--async`, then wait with
`workflows runs wait` or poll `workflows runs get` with a stopping bound. Reserve synchronous
manual runs for graphs that finish quickly.

## Keep output focused

- Use repeated `--select-output <blockName.field>` values when only specific outputs matter.
Expand All @@ -61,6 +75,23 @@ workflow and contains the state the selected block needs.
4. Correct the graph with the build skill. Do not hide a deterministic failure behind retries or a
different execution mode.

Four properties of runs and run records that mislead diagnosis when unknown:

- A run record has a lifecycle. `logs get` returns NOT_FOUND for a run that is still in flight and
for one that was cancelled, and a completed run can report a `redacting` status for a minute or
two before its content is readable. Poll with a bounded retry before concluding the run vanished.
- Firing a run immediately after `workflows deploy` reports the new version active can still
execute the previous deployment, silently. When a run exists to verify a deploy, confirm the
executed behavior from its outputs rather than trusting the deploy response, and re-run if the
outputs match the old version.
- Run traces nest: a child workflow's blocks appear inside its parent span, not at the top level.
Reading only the top level of the span tree silently drops every block a child workflow ran,
which can be most of the run. Walk the tree recursively before counting blocks or hunting the
failing one, and judge the block-level trace rather than the wrapper status.
- Cost comes in two units. The CLI's `logs get` reports `cost.total` in dollars, while the
in-workflow logs block reports the same run's cost in credits. Never compare or store the two as
one number.

Report which mode ran, the terminal status, and the relevant output or error. Include the run id when
the selected execution mode returns one; `--follow` streams omit it. Never print profile credentials
or raw secrets from block inputs.
Loading