Hnimrama/ix atom multinode#247
Open
hnimra-amd wants to merge 25 commits into
Open
Conversation
hnimra-amd
force-pushed
the
hnimrama/ix-atom-multinode
branch
6 times, most recently
from
July 8, 2026 18:33
1ca54e2 to
3117e6d
Compare
hnimra-amd
force-pushed
the
hnimrama/ix-atom-multinode
branch
from
July 13, 2026 18:24
3117e6d to
33b0e4a
Compare
…tric Extend the variant schema with nnodes, pipeline parallelism, rendezvous addresses, and optional single-node throughput baseline for scaling.efficiency_pct. Multinode sweep cells include PP and NNODES in the threshold cell key.
When params.nnodes is greater than one, launch one server per host with per-rank distributed executor flags, poll all ranks for failures, and run the benchmark client on the head node only. Single-node runs are unchanged.
…riant Ship example cluster JSON, config, and seeded thresholds for the W1 DeepSeek R1 FP8 multinode reference cell. Thresholds stay record-only until lab confirmation.
Extend FakeOrch with exec_on_head and per-host exec tracking; add unit tests for distributed server launch, head-only client, scaling metric, and perf_multi variant schema.
Add lab run instructions for the 2-node W1 perf_multi stem and describe nnodes, pipeline parallelism, rendezvous, and scaling baseline fields.
Multinode container runs need in-container sshd on port 2224; some ROCm images ship without openssh-server. Fall back to apt install before start.
…cycle The function body was left orphaned after the du_bytes import, causing an IndentationError when pytest collected inferencex_atom_single.
… PP flags Replace unrecognized vLLM distributed CLI args with ATOM SPMD env and -dp when TP allows; TP8 two-node runs independent replicas per host. Harden multinode sshd setup (iproute2, pgrep fallback) and health-check all ranks.
…and report Enable threshold enforcement for the 2-node perf variant, gate scaling.efficiency_pct as a dedicated metric tier, and surface it in the IX Run Deck preset.
Restore the newline between the docstring and timing call that was lost during rebase conflict resolution.
Collection failures emit CollectReport objects without user_properties; avoid AttributeError so pytest-html surfaces the real collection error.
Drop the misleading _single suffix so multinode runs use cvs run inferencex_atom; keep inferencex_atom_single as a deprecated config framework alias.
Clarify lab setup order so Makefile can manage .cvs_venv without an active shell inside it.
Apply ruff formatting and resolve lint issues in the 12 multinode-touched Python files so fmt-check and ruff check pass on this branch delta.
…riant Add mi355x_atom_multi cluster file, perf_multi config/threshold scaffold, README runbook, and config-loader unit test mirroring the MI300X 2-node path.
hnimra-amd
force-pushed
the
hnimrama/ix-atom-multinode
branch
from
July 15, 2026 17:54
480b192 to
d4ef345
Compare
DeepSeek R1 FP8 sweep over 1K/1K and 8K/1K at concurrency 4 through 256 with portable gated thresholds.
Same 1K/1K and 8K/1K concurrency matrix as MI300X with record-only threshold seeds until lab calibration.
Assert 14-cell matrix, max_model_length, and threshold coverage for MI300X and MI355X baseline sweep variants.
Add variant table entries and MI300X lab commands for the 14-cell 1K/1K and 8K/1K concurrency sweep.
Set HF hub cache to /home/models and bind-mount it into ATOM containers across all shipped variant configs.
Document the shared model cache path on GPU nodes in MI300X and MI355X atom cluster examples.
Update runbook, reference docs, and config loader test for the shared model path and container bind mount.
Replace four GPU/topology-specific cluster files with one inferencex_atom_cluster.json and document matching node_dict length to variant nnodes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add multinode ATOM inference support for InferenceX atom suites, including distributed serve, scaling-efficiency gates, and 2-node W1 perf sweep variants for MI300X and MI355X.
Motivation
Single-node
inferencex_atomcoverage exists ondev/dtni, but lab validation for multi-GPU / multi-node DeepSeek R1 runs requires a DTNI path that can launch ATOM across cluster ranks, assert scaling efficiency, and run a realistic multinode sweep matrix.This PR adds that multinode layer: distributed ATOM serve, multinode config/schema fields, scaling metrics in gates and reports, 2-node cluster examples for MI300X and MI355X, and matching
perf_multivariants with expanded sweep thresholds.Merge order: Intended to land after
hnimrama/refactor-ix-atom(moves shared suite helpers underinference/utils/). Expect a short rebase/conflict-resolution pass once that refactor merges.Technical Details
Suite rename
inferencex_atom_single→inferencex_atom(test module, config directory, docs, CLI references)cvs/input/config_file/inference/inferencex_atom_single/→inferencex_atom/inferencex_atom_single.pyremoved (auto-register usesinferencex_atomstem)Multinode orchestration (
InferenceXAtomJob)ATOM_DP_*,-dp), not vLLM--node-rank/ PP flags_atom_multinode_argv()isolates ATOM CLI tokens; vLLM distributed flags stripped fromatom_argswhen presentparse_resultsaccepts optionalscaling_baseline_output_throughput+nnodesfor efficiency derivationConfig & metrics
nnodes,node_rank,master_addr,master_port,pipeline_parallel_size,scaling_baseline_output_throughputscalingwithscaling.efficiency_pct(= actual / (single-node baseline × nnodes))scaling.efficiency_pctthresholds when baseline is setPP=/NN=suffixes when multinode dimensions applyCluster & variants
cvs/input/cluster_file/mi300x_atom_multi.jsonmi300x_inferencex-atom-single_deepseek-r1_fp8_perf_multicvs/input/cluster_file/mi355x_atom_multi.jsonmi355x_inferencex-atom-single_deepseek-r1_fp8_perf_multiBoth variants use the same sweep matrix: 5 sequence shapes × concurrencies 16/64/128 (15 cells),
nnodes=2,PP=2.MI300X:
rocm/atom-dev:latest,scaling_baseline_output_throughput=1500,enforce_thresholds: true.MI355X:
rocm/atom-dev:nightly_202606211542,scaling_baseline_output_throughput=4000(seeded from single-node CI),enforce_thresholds: falseuntil 2-node lab confirms.Infrastructure
container.py: installopenssh-serverwhen image lackssshd(multinode SSH setup)CollectReporttolerance in HTML hooks,test_launch_containersyntax repairKey commits
perf_multivariantinferencex_atom_single→inferencex_atomperf_multivariant + config-loader testTest Plan
test_inferencex_atom_config_loader,test_inferencex_atom_orch_parse,test_inferencex_atom_parsing)test_load_w1_mi355x_multinode_variantcvs run inferencex_atomwith multinode cluster +perf_multi(2-node MI325X, W1)Test Result
test_inferencex_atom_config_loader,test_inferencex_atom_orch_parse)test_load_w1_mi355x_multinode_variant