Interactive Visual Gym

Evaluating Visual Interaction Models with Simulated Users in Interactive Worlds

University of Washington   ·   Salesforce AI Research

Get started   /   Evaluate your model   /   Submit via pull request   /   Citation

Visual guidance changes what happens next. Interactive Visual Gym evaluates and trains visual interaction models in a closed loop: a simulated user plans and acts, while the model observes the activity and provides guidance. Scheduled out-of-plan events and user personas let us measure how that guidance affects task execution and interaction quality.

Cooking	Household activities	Mobile interfaces
CookSim Prepare and serve recipes in an Overcooked-style kitchen.	VH-Home Complete household routines in furnished apartments.	ScreenSim Configure phone settings to satisfy user goals.

• Executable environments. Visual observations, semantic actions, and scene-graph states support task execution and progress tracking.
• Controllable users. Schedule errors, questions, proactive requests, and plan changes at task-progress checkpoints; vary skill, trust in AI, execution speed, and interruption preference.
• Closed-loop evaluation. Measure task success, error detection, interaction quality, and question answering from the resulting trajectories.

Tip

Have a model to evaluate? Submit it to IVG-Bench via a pull request. Run the benchmark, upload your trajectories, and add a submission request. Validation is automatic; official scoring and leaderboard publication follow maintainer review.

Get started

Clone the repository, then follow the quickstart for your environment. Install from the checkout: each engine uses assets and benchmark definitions alongside its Python packages.

git clone https://github.com/SalesforceAIResearch/Interactive-Visual-Gym.git
cd Interactive-Visual-Gym

Environment	Install and run one episode	Evaluate your model	Create cases
CookSim	Quickstart	Evaluation	Case generation
VH-Home	Quickstart	Evaluation	Case generation
ScreenSim	Quickstart	Evaluation	Case generation

Each engine is independently installable. The shared workflow is inspect cases → run an episode → inspect the record → score → run the benchmark; simulator dependencies, action vocabularies, and record formats are documented in the guides above.

Evaluation protocol

IVG-Bench contains 105 cases, evaluated with three personas per case: 150 cooking, 75 household, and 90 mobile-interface episodes per model per run. A case specifies a task and scheduled events; an episode is one execution with a particular user persona and model.

Evaluation setting	CookSim and VH-Home	ScreenSim
Fixed timing · main comparison	Fixed two-tick delay: --latency 2	Fixed zero-tick action–response setting
Measured latency · latency comparison	Measured inference delay: --latency measured	Same fixed setting

See the complete run commands, benchmark protocol, and metric definitions. Published results are available in the paper and on the project leaderboard.

Submit via pull request

Submit your model to IVG-Bench by opening a pull request in this repository.

1. Run and package. Run the benchmark and package your trajectories with ivg-submit.
2. Upload the archive. Use a GitHub Release asset and generate a submission request containing its URL and checksum. Include a pinned inference-code repository and runnable instructions in your submission metadata.
3. Open a PR. Add submissions/requests/<submission-id>.json—not the trajectory archive or inference code itself.

PR validation checks the submitted records automatically. Maintainers then grade and review the results before approving leaderboard publication.

Submission guide: commands, required files, and review process →

Documentation

Guide	Contents
Getting started	Requirements, installation, and your first run
Evaluating models	Model interfaces, run commands, and outputs
Benchmark protocol	Cases, personas, repeated runs, and configuration
Metrics	Engine-derived scores and model-judged interaction quality
Submitting results	Trajectory packaging, official scoring, and inference-code audit
Development	Repository structure, tests, and compatibility checks
Validation · Release notes	Validation checks, limitations, and configuration differences

Repository structure

cooksim/                Cooking engine, assets, benchmark harness, and tests
vhhome/                 Household engine, simulator setup, benchmark harness, and tests
screensim/              Phone engine, assets, benchmark harness, and tests
docs/                   Shared workflow and benchmark documentation
scripts/                Repository-wide validation
ivg_submission/         Submission validation, grading handoff, and leaderboard export
submissions/            Submission requests and maintainer-reviewed results

Citation

If you use Interactive Visual Gym in your research, please cite our paper:

@misc{zheng2026interactivevisualgymevaluating,
      title={Interactive Visual Gym: Evaluating Visual Interaction Models with Simulated Users in Interactive Worlds},
      author={Chenhao Zheng and Zixian Ma and Kai Li and Kanchana Ranasinghe and Jieyu Zhang and Artemis Panagopoulou and Weikai Huang and Jianxu Shangguan and Yuanyuan Tan and Silvio Savarese and Juan Carlos Niebles and Ranjay Krishna and Ran Xu and Honglu Zhou},
      year={2026},
      eprint={2611.08286},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2611.08286},
}

Download citation.bib, or use CITATION.cff.

Contributing and licenses

See CONTRIBUTING.md. This repository is released under the Apache-2.0 license. CookSim and VH-Home retain their included MIT licenses; ScreenSim carries an Apache-2.0 license. External simulators, model weights, and other third-party assets retain their own terms.