Add NeMo Gym V1 integration

[xeophon]

Overview

Adds a NeMo Gym resources-server integration for Verifiers V1. Existing Verifiers harnesses drive the rollout while NeMo Gym remains responsible for resource-server state, native tools, and scoring.

Details

• Loads NeMo Gym JSONL requests into typed V1 tasks and preserves the source row for session setup and verification.
• Carries per-rollout cookies and signed MCP session headers through V1 state while exposing upstream tool schemas to MCP-capable harnesses.
• Converts completed single-branch traces into NeMo Responses objects and records /verify rewards and numeric metrics.
• Adds a Python 3.12+ nemo-gym optional dependency and reuses resource-server classes from the published package, with V1 owning startup and cleanup.
• Uses uv dependency metadata to install only the resource-server import slice instead of the full NeMo agent, model, and evaluation stack.
• Adds environments/nemo_gym_weather_v1 as a thin, self-starting example package; external resource-server URLs remain supported for custom deployments.

Fixes RES-1090

---

Note

Medium Risk
New subprocess lifecycle, HTTP/MCP session bridging, and a large optional dependency graph (Ray, wandb, etc.) on 3.12+; core 3.11 installs are gated but lockfile/resolution changes affect all developers.

Overview
Adds NeMo Gym as a V1 taskset path so harnesses can run rollouts against Gym resource servers while scoring stays on Gym’s /verify API.

The new verifiers/v1/tasksets/nemo_gym package loads JSONL rows into tasks, seeds per-rollout sessions (cookies + optional MCP metadata), bridges tools via NeMoGymToolset (upstream MCP or direct HTTP), and converts a single-branch trace to a NeMo Responses payload for verification. NeMoGymEnv can spawn a published resource-server class in a subprocess when no resources_url is set.

Packaging: optional nemo-gym extra (Python 3.12+) with uv dependency-metadata to trim NeMo’s declared stack to resource-server needs; lockfile picks up transitive deps. Example package environments/nemo_gym_weather_v1 pins the example MCP weather server and dataset. MCP server registration is renamed from _register to register on the base server/toolset.

CI smoke evals skip nemo_gym_weather_v1 because it is not in the default install.

^{Reviewed by Cursor Bugbot for commit 0e1a385. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add NeMo Gym V1 integration for agent evaluation against managed resource servers

• Adds NeMoGymTaskset and NeMoGymEnv in verifiers/v1/tasksets/nemo_gym/ to load tasks from a JSONL dataset, seed sessions with a NeMo Gym resource server, and score rollouts via the server's /verify endpoint.
• NeMoGymEnv.start automatically spawns a per-worker resource server subprocess, polls for its port, and wires the task config to use it; NeMoGymEnv.stop tears it down cleanly.
• NeMoGymToolset proxies list_tools and call_tool to the upstream MCP server when available, or falls back to direct HTTP POSTs for datasets that declare tools explicitly.
• trace_to_nemo_response converts a completed Verifiers trace into a NeMo Gym-compatible response payload for submission to /verify.
• Adds the nemo-gym-weather-v1 example environment package in environments/nemo_gym_weather_v1/ demonstrating how to wire a concrete taskset to a resource server.

Changes since #2212 opened

• Changed NeMoGymTask.setup method to remove transport validation and default values when processing MCP metadata [3865703]
• Refactored NeMoGymTask.nemo_gym reward method to change verification result processing [3865703]
• Fixed server.main function module name parsing to handle modules without dots [3865703]
• Removed NeMoGymState and NeMoGymToolset exports from nemo_gym.__init__ module [3865703]
• Removed from __future__ import annotations imports and inlined port file path construction [3865703]
• Modified test infrastructure to skip nemo_gym_weather_v1 taskset and remove editable overlay support from eval commands [0e1a385]
• Changed Python version environment marker for nemo-gym optional dependency [0e1a385]

^{Macroscope summarized 9cbd5c2.}

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[xeophon]

Migrated review context

This PR carries the exact head commit from the prior fork-backed review. Outstanding point retained for follow-up:

• Normalize Codex-qualified MCP tool names back to the original NeMo Gym tool name before sending the trace to /verify.

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR introduces a new NeMo Gym V1 integration with substantial new capabilities including subprocess management for resource servers, HTTP/MCP communication with external services, and new scoring mechanisms. The scope and complexity of this new feature warrants human review.

^{You can customize Macroscope's approvability policy. Learn more.}

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit d102c94. Configure here.}