Skip to content

fix(vllm): account for pipeline-parallel size in per_gpu_throughput#258

Open
atnair-amd wants to merge 5 commits into
dev/dtnifrom
atnair/vllm-p1-metrics
Open

fix(vllm): account for pipeline-parallel size in per_gpu_throughput#258
atnair-amd wants to merge 5 commits into
dev/dtnifrom
atnair/vllm-p1-metrics

Conversation

@atnair-amd

@atnair-amd atnair-amd commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • per_gpu_throughput in to_client_metrics divided total token throughput by tp alone, undercounting GPU count on any pipeline-parallel run (actual GPUs = tp * pp). pp is now a required keyword-only argument.
  • Widened the vLLM Ray-backend regression test to assert pp explicitly (subTest over pp=1,2) so a broken pp passthrough can't silently regress again — the prior version only asserted tp/isl.
  • Scope is vLLM only. InferenceX ATOM's to_client_metrics call sites are not updated for the new required pp kwarg and are left broken by this change (out of scope here).
  • Added GPU metrics polling to the vLLM suite: test_vllm_inference now takes pre/post-model-load VRAM snapshots and polls amd-smi in the background while the client runs; a new test_gpu_metric test reports one HTML row per GPU metric per sweep cell (peak_gpu_memory_mb, model_load_memory_mb, model_load_s, gpu_bandwidth_util_pct, gpu_compute_util_pct). Threshold-coverage validation (vllm_config_loader.py) now gates gpu.* metrics the same way it already gated client.* metrics. amd-smi is invoked via the container-routed orch.exec/orch.exec_on_head — GPU device files (/dev/kfd, /dev/dri) are passed through into the container, so no host-vs-container bypass routing is needed.

Test plan

  • python3 -m unittest discover -s cvs/lib/inference/unittests -p "test_*.py" — 188 tests, only known-out-of-scope failures: test_inferencing_config_loader.py (2 pre-existing, unrelated failures confirmed via git stash against commit 5134be1) and test_inferencex_atom_orch_parse.py (3 failures caused by this PR's pp requirement — ATOM is not being updated here, tracked separately)
  • python3 -m unittest discover -s cvs/lib/utils/unittests -p "test_*.py" — 126/126 pass (gpu.py polling/aggregation logic)
  • python3 -m unittest discover -s cvs/core/orchestrators/unittests -p "test_*.py" — 57/57 pass
  • Mutation-tested: reverting the pp fix in vllm_job.py causes the regression test to fail, confirming it actually guards the fix
  • Hardware validation on demo cluster: full vLLM suite run against DeepSeek-R1-0528 in a real tp8pp2 config (8-way tensor-parallel, 2-way pipeline-parallel, 2-node Ray cluster) — 43 passed, 1 skipped, rc=0. Confirms per_gpu_throughput and the rest of the metrics pipeline compute and report correctly with pp=2 on live hardware (total throughput 823.47 tok/s, TTFT/TPOT/ITL all populated in the results table).
  • pytest --collect-only against cvs/tests/inference/vllm/vllm.py with a real vLLM config confirms test_gpu_metric collects correctly (5 GPU metric rows per cell) alongside the existing 44 tests, with no collection errors.

Known out-of-scope breakage

InferenceX ATOM's to_client_metrics call sites do not pass pp and will fail after this change (ATOM has no pipeline-parallel concept). This is expected and intentionally left unfixed here — scope is vLLM only.

@atnair-amd
atnair-amd changed the base branch from main to dev/dtni July 16, 2026 18:24
@atnair-amd
atnair-amd force-pushed the atnair/vllm-p1-metrics branch from 0431712 to cda0107 Compare July 16, 2026 21:13
@atnair-amd
atnair-amd marked this pull request as ready for review July 16, 2026 21:47
…required

ATOM has no pipeline-parallel concept (single-node TP only), so pp=1
preserves the prior per_gpu_throughput numeric behavior. Also widen the
Ray-backend regression test to assert pp explicitly across pp=1/2 so a
broken pp passthrough can't slip through silently again.
@atnair-amd
atnair-amd force-pushed the atnair/vllm-p1-metrics branch from cda0107 to 005b5e4 Compare July 17, 2026 19:01
Adds test_gpu_metric (one HTML row per GPU metric per sweep cell),
GPU pre/post-load VRAM snapshots and background amd-smi polling
during test_vllm_inference, and extends threshold-coverage validation
to gate gpu.* metrics alongside client.* metrics.

amd-smi runs fine from inside the benchmark container (GPU device
files are passed through), so gpu.py calls orch.exec/exec_on_head
directly like every other command in the suite -- no host-vs-container
bypass routing needed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant