WIP feat: swe bench scorer

[tianmu-li]

What does this PR do?

Type of change

• Bug fix
• New feature
• Documentation update
• Refactor/cleanup

Related issues

Testing

• Tests added/updated
• All tests pass locally
• Manual testing completed

Checklist

• Code follows project style
• Pre-commit hooks pass
• Documentation updated (if needed)

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[gemini-code-assist]

Code Review

This pull request introduces support for SWE-bench accuracy evaluation by adding a new accuracy-only SWEBench dataset, a SWEBenchScorer that runs evaluations using mini-swe-agent in an isolated environment, and associated configuration templates, tests, and runbooks. Feedback on the changes focuses on improving the robustness of the SWEBenchScorer implementation, specifically by safely handling missing or null values when parsing configuration templates, benchmark configurations, and evaluation results, as well as gracefully handling cases where the Docker binary is missing from the system's PATH during preflight checks.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

If docker is not installed or not present on the system's PATH, calling subprocess.run(["docker", ...]) will raise a FileNotFoundError rather than returning a non-zero exit code. This will cause the preflight check to crash with an unhandled traceback. Checking if docker is on PATH using shutil.which and wrapping the execution in a try-except block ensures a clean SetupError is raised.

Suggested change

	docker_result = subprocess.run(
	["docker", "version"],
	stdout=subprocess.DEVNULL,
	stderr=subprocess.PIPE,
	timeout=10,
	)
	if docker_result.returncode != 0:
	raise SetupError(
	"Docker daemon is not running or docker is not on PATH. "
	"Start Docker and retry."
	)
	if shutil.which("docker") is None:
	raise SetupError(
	"docker is not on PATH. Install Docker and retry."
	)
	try:
	docker_result = subprocess.run(
	["docker", "version"],
	stdout=subprocess.DEVNULL,
	stderr=subprocess.PIPE,
	timeout=10,
	)
	except Exception as e:
	raise SetupError(f"Failed to execute docker command: {e}")
	if docker_result.returncode != 0:
	raise SetupError(
	"Docker daemon is not running. Start Docker and retry."
	)

[gemini-code-assist]

The validation of the swebench_config_template structure is fragile and can raise an AttributeError or TypeError if _tmpl is parsed as None (e.g., an empty file), or if "model" is not a dictionary (e.g., a string or None). Using a safer dictionary retrieval pattern prevents these potential runtime crashes.

Suggested change

	with self.swebench_config_template.open() as _f:
	_tmpl = yaml.safe_load(_f)
	if not isinstance((_tmpl or {}).get("model", {}).get("model_kwargs"), dict):
	raise ValueError(
	f"swebench template {self.swebench_config_template} must have a "
	"'model.model_kwargs' dict; check the template structure."
	)
	with self.swebench_config_template.open() as _f:
	_tmpl = yaml.safe_load(_f) or {}
	model_cfg = _tmpl.get("model")
	if not isinstance(model_cfg, dict) or not isinstance(model_cfg.get("model_kwargs"), dict):
	raise ValueError(
	f"swebench template {self.swebench_config_template} must have a "
	"'model.model_kwargs' dict; check the template structure."
	)

[gemini-code-assist]

If endpoint_config or model_params is defined as empty or null in the YAML configuration, benchmark_config_dict.get(...) can return None. Calling .get() on None will raise an AttributeError. Using the or {} pattern ensures safe dictionary access.

        model_params = benchmark_config_dict.get("model_params") or {}
        endpoint_cfg = benchmark_config_dict.get("endpoint_config") or {}
        endpoints = endpoint_cfg.get("endpoints", [])

[gemini-code-assist]

If the SWE-bench evaluation results JSON contains null values for submitted_instances or resolved_instances, .get(..., 0) will return None instead of 0. This will subsequently cause a TypeError when calculating the resolved rate. Using or 0 ensures that any None values are safely defaulted to 0.

Suggested change

	submitted = result.get("submitted_instances", 0)
	resolved = result.get("resolved_instances", 0)
	submitted = result.get("submitted_instances") or 0
	resolved = result.get("resolved_instances") or 0