Support Harbor multi-step tasks

[hallerite]

Summary

• route Harbor [[steps]] tasks through a bundled taskset-level environment while preserving ordinary task behavior
• execute ordered steps in one shared runtime with fresh or resumed agent context, step setup/healthchecks/verifiers/artifacts/early stopping, and Harbor-compatible reward aggregation
• allow owner-scoped sequential reuse of restricted runtimes and inherit default environments through HarborTaskset subclasses
• document supported semantics and remaining parity gaps

Why

Canonical Harbor multi-step tasks have no root instruction.md and were skipped by the loader. The existing single-rollout adapter also had no control-flow surface for per-step setup, verification, early stopping, or trial reward aggregation.

Validation

• uv run --extra harbor pytest tests/ — 916 passed, 67 skipped
• uv run --extra harbor pre-commit run --all-files
• push hook — markdownlint, Ruff, and ty CI parity
• configs/harbor.toml validates to HarborEnvConfig / HarborConfig

Live Docker E2E was not run because the Docker CLI is unavailable locally; restricted reuse and proxy refresh have deterministic test coverage.

Parity

This reaches close parity with Harbor's shared-verifier multi-step contract. Remaining gaps are phase-policy overrides, per-command users, and separate verifier containers.

Note

Add multi-step task support to Harbor taskset and environment

• Introduces HarborStep and StepHealthcheck models in taskset.py to represent per-step configuration (prompt, timeouts, env, collect, artifacts, healthcheck, min_reward).
• Adds HarborEnv in env.py with two execution modes: fresh (each step runs as a separate agent.run() call sharing one provisioned runtime) and resumed (steps run within a single agent interaction session with per-step budgets).
• Reward aggregation supports mean and final strategies across steps; min_reward can short-circuit execution when a step's reward falls below a threshold.
• Adds Runtime.reuse() context manager in base.py to allow a single owner to reuse a restricted runtime across sequential steps without reopening network policy.
• HarborTaskset.ENV defaults to HarborEnv, and loaders.environment_class falls back to Taskset.ENV when no explicit environment plugin is exported.
• Risk: HarborTask.solved now returns float | dict[str, float] instead of always float, which may affect callers that assumed a scalar reward.

📊 Macroscope summarized 016ba37. 8 files reviewed, 0 issues evaluated, 0 issues filtered, 0 comments posted

🗂️ Filtered Issues

No issues evaluated.

[macroscopeapp]

🟡 Medium harbor/env.py:77

In _run_resumed, the step_task.setup(...) call for steps after the first is not wrapped in asyncio.wait_for with step.timeout.setup, so a slow stage_step_workdir or healthcheck in a later step can exceed the step's declared setup budget or hang indefinitely. The first step's setup timeout is enforced by the rollout lifecycle, but subsequent steps bypass it. Consider wrapping the setup call in asyncio.wait_for(..., step.timeout.setup).

Suggested change

	await step_task.setup(interaction.trace, runtime)
	if index:
	await asyncio.wait_for(step_task.setup(interaction.trace, runtime), step.timeout.setup)

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/tasksets/harbor/env.py around line 77:

In `_run_resumed`, the `step_task.setup(...)` call for steps after the first is not wrapped in `asyncio.wait_for` with `step.timeout.setup`, so a slow `stage_step_workdir` or healthcheck in a later step can exceed the step's declared setup budget or hang indefinitely. The first step's setup timeout is enforced by the rollout lifecycle, but subsequent steps bypass it. Consider wrapping the `setup` call in `asyncio.wait_for(..., step.timeout.setup)`.

[macroscopeapp]

🟠 High harbor/taskset.py:728

read_rewards accepts non-finite scores like NaN and Infinity. Python's json.loads and float() produce these as valid floats, and the numeric type check doesn't reject them, so a verifier emitting NaN or Infinity records it as a reward. This contaminates aggregation and training: NaN also bypasses min_reward thresholds because all NaN comparisons are false, and downstream strict JSON serialization (e.g. allow_nan=False) will fail. Validate every parsed score with math.isfinite in both the reward.json and reward.txt paths.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/tasksets/harbor/taskset.py around line 728:

`read_rewards` accepts non-finite scores like `NaN` and `Infinity`. Python's `json.loads` and `float()` produce these as valid floats, and the numeric type check doesn't reject them, so a verifier emitting `NaN` or `Infinity` records it as a reward. This contaminates aggregation and training: `NaN` also bypasses `min_reward` thresholds because all `NaN` comparisons are false, and downstream strict JSON serialization (e.g. `allow_nan=False`) will fail. Validate every parsed score with `math.isfinite` in both the `reward.json` and `reward.txt` paths.

[macroscopeapp]

🟠 High harbor/env.py:77

In _run_resumed, later steps' setup runs without the step.timeout.setup deadline, so a slow workdir staging or healthcheck can exceed that step's declared setup budget. The first step is covered by the rollout lifecycle timeout and the fresh-step path goes through agents.agent.run, but await step_task.setup(...) on line 77 has no asyncio.wait_for guard. Wrap it in asyncio.wait_for(..., step.timeout.setup) to enforce the same budget.

Suggested change

	await step_task.setup(interaction.trace, runtime)
	if index:
	await asyncio.wait_for(step_task.setup(interaction.trace, runtime), step.timeout.setup)

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/tasksets/harbor/env.py around line 77:

In `_run_resumed`, later steps' `setup` runs without the `step.timeout.setup` deadline, so a slow workdir staging or healthcheck can exceed that step's declared setup budget. The first step is covered by the rollout lifecycle timeout and the fresh-step path goes through `agents.agent.run`, but `await step_task.setup(...)` on line 77 has no `asyncio.wait_for` guard. Wrap it in `asyncio.wait_for(..., step.timeout.setup)` to enforce the same budget.

[macroscopeapp]

🟠 High runtimes/base.py:315

reuse() authorizes concurrent reuse of a restricted runtime by setting a runtime-wide _setup_reusable flag that persists for the duration of the async context. While the context body awaits, any concurrent task with a reference to the same Runtime can call prepare_setup() and pass the single-rollout guard, because the flag neither identifies the trusted owner nor enforces sequential access. This defeats the isolation check that is meant to prevent a second borrower from reusing the restricted filesystem/process namespace. Consider tracking the active owner (e.g., via a task-scoped token) or serializing access so the guard cannot be bypassed while a rollout is in flight.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/runtimes/base.py around line 315:

`reuse()` authorizes concurrent reuse of a restricted runtime by setting a runtime-wide `_setup_reusable` flag that persists for the duration of the async context. While the context body awaits, any concurrent task with a reference to the same `Runtime` can call `prepare_setup()` and pass the single-rollout guard, because the flag neither identifies the trusted owner nor enforces sequential access. This defeats the isolation check that is meant to prevent a second borrower from reusing the restricted filesystem/process namespace. Consider tracking the active owner (e.g., via a task-scoped token) or serializing access so the guard cannot be bypassed while a rollout is in flight.

[macroscopeapp]

🟠 High harbor/env.py:86

In _run_resumed, step_task.collect_step(...) is called without wrapping it in step.timeout.finalize, so collect hooks and artifact collection can exceed the declared per-step finalize budget or hang until the episode-wide timeout. In fresh mode the same finalization runs through the normal bounded rollout lifecycle, so resumed steps silently lose Harbor's independent finalize deadline. Consider wrapping the collect_step call in asyncio.wait_for(..., step.timeout.finalize), the same way verify_step is already bounded by step.timeout.scoring.

Suggested change

	await step_task.collect_step(interaction.trace, runtime)
	await asyncio.wait_for(
	step_task.collect_step(interaction.trace, runtime),
	step.timeout.finalize,
	)

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/tasksets/harbor/env.py around line 86:

In `_run_resumed`, `step_task.collect_step(...)` is called without wrapping it in `step.timeout.finalize`, so collect hooks and artifact collection can exceed the declared per-step finalize budget or hang until the episode-wide timeout. In fresh mode the same finalization runs through the normal bounded rollout lifecycle, so resumed steps silently lose Harbor's independent finalize deadline. Consider wrapping the `collect_step` call in `asyncio.wait_for(..., step.timeout.finalize)`, the same way `verify_step` is already bounded by `step.timeout.scoring`.