[NVBug: 6563509] Harden meta-device skeleton build; drop Phi-3-vision / Phi-4-multimodal PTQ support - #2115
[NVBug: 6563509] Harden meta-device skeleton build; drop Phi-3-vision / Phi-4-multimodal PTQ support#2115cjluo-nv wants to merge 5 commits into
Conversation
… fails
get_model() builds a throwaway skeleton under
init_empty_weights(include_buffers=True) purely to size the model for
infer_auto_device_map. include_buffers=True makes accelerate push a global
torch.device("meta") context, so every tensor constructed in __init__ lands on
meta -- not just parameters and buffers. Remote-code checkpoints written before
Transformers v5 routinely derive scalar hyperparameters from real tensors there
(Phi-4-multimodal's conformer subsampling does int(torch.tensor(...))), which
raises "Tensor.item() cannot be called on meta tensors" and killed the whole
run before from_pretrained was ever reached.
Retry the skeleton without the global meta context, and if that also fails,
warn and skip the memory estimate instead of aborting -- from_pretrained can
still map the model on its own. Losing the estimate only costs the automatic
max_memory shrink, which --use_seq_device_map and --gpu_max_memory_percentage
already cover.
Note this does not by itself make Phi-4-multimodal-instruct loadable: its
remote code additionally needs transformers <4.52 (Phi4MMModel relies on
PreTrainedModel inheriting GenerationMixin, which peft's get_peft_model calls
into) and declares _tied_weights_keys as a list, which Transformers 5.x
rejects. Both are outside ModelOpt.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Chenjie Luo <chenjiel@nvidia.com>
Both ship remote code that predates Transformers v5 and no longer loads on any version this repo supports (transformers>=4.57,<5.15): - Phi-4-multimodal needs transformers<4.52. Its __init__ calls peft.get_peft_model on Phi4MMModel, which reads prepare_inputs_for_generation -- present only while PreTrainedModel still inherited GenerationMixin. Verified loading at 4.48.2 / 4.49.0 / 4.50.0 / 4.51.3, failing at 4.53.3 / 4.56.2 / 4.57.1 with AttributeError. - Both declare _tied_weights_keys as a list; Transformers 5.x calls .keys() on it in post_init and raises AttributeError. - Phi-4-multimodal additionally computes int(torch.tensor(...)) in __init__, which Transformers 5.x's meta-device from_pretrained cannot evaluate. The model card pins transformers==4.48.2 / peft==0.13.2, so there is no overlap with our floor and nothing on our side can bridge it. Removes the support-matrix row, the phi4mm model type, the multimodal-detection heuristics that only ever matched these two (vision_lora, audio_processor, embd_layer.image_embd_layer), the Phi3Image/PhiImage embedding-export exclusions, and modelopt_recipes/huggingface/phi4mm/. Text-only Phi-3/Phi-4 and Phi-3.5-MoE are natively supported by transformers and are untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Chenjie Luo <chenjiel@nvidia.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (3)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughThe change removes PTQ support for Phi-3-vision and Phi-4-multimodal. It removes related detection, export handling, documentation, and recipes. It also skips device-map memory estimation when model skeleton construction fails. ChangesPTQ support removal
HF PTQ loading fallback
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant HFPTQ as HF PTQ
participant Skeleton as Meta skeleton construction
participant DeviceMap as Device-map inference
HFPTQ->>Skeleton: Build skeleton without buffers
alt Construction succeeds
HFPTQ->>DeviceMap: Infer device map
else Construction fails
Skeleton-->>HFPTQ: Return None and warning
HFPTQ->>HFPTQ: Skip memory estimation
end
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 6✅ Passed checks (6 passed)
✨ Finishing Touches 💡 1⚔️ Resolve merge conflicts 💡
📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Warning
CodeRabbit couldn't request changes on this pull request because it doesn't have sufficient GitHub permissions.
Please grant CodeRabbit Pull requests: Read and write permission and re-run the review.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@examples/hf_ptq/example_utils.py`:
- Around line 898-903: Update the fallback path after skeleton construction
fails, before the final from_pretrained load, to populate GPU entries in
model_kwargs["max_memory"] using the configured GPU memory percentage when model
is None. Preserve any limit already established by sequential device mapping,
and keep the existing fallback loading behavior unchanged.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: c15bf192-fc6f-44b2-86ab-c408eea1be07
📒 Files selected for processing (10)
CHANGELOG.rstexamples/hf_ptq/README.mdexamples/hf_ptq/example_utils.pyexamples/hf_ptq/hf_ptq.pymodelopt/torch/export/layer_utils.pymodelopt/torch/export/model_utils.pymodelopt_recipes/huggingface/phi4mm/ptq/README.mdmodelopt_recipes/huggingface/phi4mm/ptq/disabled_quantizers.yamlmodelopt_recipes/huggingface/phi4mm/ptq/nvfp4-kv_fp8_cast.yamlmodelopt_recipes/ptq.md
💤 Files with no reviewable changes (5)
- examples/hf_ptq/hf_ptq.py
- modelopt_recipes/huggingface/phi4mm/ptq/nvfp4-kv_fp8_cast.yaml
- modelopt_recipes/huggingface/phi4mm/ptq/README.md
- modelopt_recipes/huggingface/phi4mm/ptq/disabled_quantizers.yaml
- modelopt/torch/export/model_utils.py
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2115 +/- ##
==========================================
- Coverage 78.60% 76.92% -1.68%
==========================================
Files 522 522
Lines 60167 62441 +2274
==========================================
+ Hits 47294 48033 +739
- Misses 12873 14408 +1535
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| config_dtype = _get_config_dtype(config_for_init) | ||
| model_kwargs2 = _apply_dtype_to_config( | ||
| model_kwargs, config_dtype, architecture, apply_config_dtype=True | ||
| # When computing the device_map, assuming bfloat16 precision by default, |
There was a problem hiding this comment.
how about we make this a util function?
There was a problem hiding this comment.
Done in 8444c11 — extracted as _build_meta_skeleton(), placed next to the other get_model helpers (_resolve_init_config / _get_config_dtype). get_model is now a single call, and the fallback is unit-testable on its own: verified all three paths (tier-1 succeeds / tier-1 fails on meta and tier-2 succeeds / both fail returning None with one warning).
There was a problem hiding this comment.
Follow-up: _build_meta_skeleton() simplified further in 1b062da. The two-tier retry is gone — it is now a single include_buffers=False build, because that is what Transformers' own from_pretrained uses and the old include_buffers=True probe was stricter than the loader it was predicting. Details in the design thread.
|
|
||
| > *This is a subset of the models supported. For the full list please check the [TensorRT-LLM support matrix](https://nvidia.github.io/TensorRT-LLM/reference/precision.html#support-matrix)* | ||
|
|
||
| > *Phi-3-vision and Phi-4-multimodal were dropped from this matrix: their bundled |
There was a problem hiding this comment.
no need to explain here.
There was a problem hiding this comment.
Done in 8444c11 — note removed. The CHANGELOG entry under 0.46 Backward Breaking Changes carries the explanation.
meenchen
left a comment
meenchen
left a comment
There was a problem hiding this comment.
Bot review (gpt-5.6-sol) — DM the bot to share feedback.
The fallback direction is reasonable, but there is a correctness issue in the failure path and the breaking Phi-3-vision removal is not supported by the compatibility evidence presented. No tests were added for the new two-attempt/fallback behavior.
Design review: the problem is to prevent an optional meta skeleton used for device-map sizing from aborting an otherwise viable from_pretrained load. Existing alternatives are (1) the repo's build_meta_causal_lm pattern in modelopt/torch/utils/plugins/model_load_utils.py, which uses init_empty_weights(include_buffers=False) directly, (2) relying on Accelerate/Transformers' existing device_map="auto" plus an explicit max_memory, or (3) extending/using Transformers' safe meta creation patches. The PR body explains why include_buffers=True is hazardous and why it retries, but does not compare why retrying is preferable to always using the existing include_buffers=False pattern or delegating directly to from_pretrained. Please document that choice, as required by the design-review gate.
Additional comments (outside the PR diff):
examples/hf_ptq/example_utils.py:909— > Bot comment.
The suggested --gpu_max_memory_percentage mitigation has no effect when both skeleton attempts fail. In this branch model is None, so the later elif model is not None block never scales max_memory or places it in model_kwargs; from_pretrained(device_map="auto") therefore uses its default memory budget regardless of this flag. Please either pass an appropriately scaled max_memory in the fallback path or remove this recommendation. Add mocked tests for first-attempt failure/second-attempt success and both-attempts failure; those would also pin whether infer_auto_device_map is skipped and whether the advertised memory limit reaches from_pretrained.
|
|
||
| **Deprecations** | ||
|
|
||
| - Drop PTQ support for **Phi-3-vision** and **Phi-4-multimodal**. Their bundled remote code predates Transformers v5 and no longer loads on the versions this repo requires (``transformers>=4.57``): Phi-4-multimodal needs ``transformers<4.52`` because it reaches ``prepare_inputs_for_generation`` through ``peft``, which requires ``PreTrainedModel`` to still inherit ``GenerationMixin``, and both models declare ``_tied_weights_keys`` as a list, which Transformers 5.x rejects. Removes the ``phi4mm`` model type, its multimodal-detection heuristics (``vision_lora`` / ``audio_processor`` / ``embd_layer.image_embd_layer``), the ``Phi3Image`` / ``PhiImage`` embedding-export exclusions, and the ``modelopt_recipes/huggingface/phi4mm/`` recipes. |
There was a problem hiding this comment.
Bot comment.
The supplied evidence does not establish that Phi-3-vision is unloadable across the supported range. The dependency floor and CI matrix explicitly include Transformers 4.57, while the listed Phi-3 blocker (_tied_weights_keys as a list) is described as a Transformers 5.x failure; the version matrix in the PR body only tests Phi-4-MM. Since this is a breaking support removal, please provide/test the Phi-3-vision failure on 4.57 (and identify its blocker), or retain Phi-3-vision support and scope the deletion/documentation to Phi-4-MM.
There was a problem hiding this comment.
Fair reading of the evidence, but we are keeping Phi-3-vision in the removal.
You are right about what was and was not proven. Phi-3-vision has no get_peft_model call, so it does not have the transformers-4.52 blocker — that one is Phi-4-MM only. The defect I verified for it is _tied_weights_keys declared as a list (modeling_phi3_v.py:1214), which is a Transformers 5.x failure, and I did not run it end-to-end on 4.57.
We are dropping it anyway, for product reasons rather than a 4.57 repro: Phi-3-vision is the older, superseded model, and Phi-4-multimodal is its replacement. If we cannot support the successor there is no reason to carry the predecessor — it is strictly the weaker model and nobody starting today would pick it over Phi-4-MM.
The technical direction agrees. Phi-3-vision is confirmed broken on Transformers 5.x, and per the 0.46 changelog we already bumped the floor to 4.57 with "Transformers 4.x support will be dropped in a future release". So the best case for keeping it is a support window that closes on its own within a release or two, on a model whose remote code Microsoft is not updating. The same reasoning already applied to VILA / NVILA in this release.
Happy to reopen if someone has an active Phi-3-vision workflow on 4.57.
0.46 is the release these land in, not the still-open 0.47 section. The Phi drop goes under Backward Breaking Changes next to the VILA / NVILA entry, which removed model support for the same reason (bundled remote code pinned below our transformers floor); the skeleton fallback goes under Bug Fixes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Chenjie Luo <chenjiel@nvidia.com>
… note - Extract the two-attempt skeleton build into `_build_meta_skeleton()` next to the other `get_model` helpers (@cjluo-nv). `get_model` now reads as one line, and the fallback logic is unit-testable in isolation. - Fix the OOM hint in the fallback warning. It named `--gpu_max_memory_percentage`, which does not exist -- the flag is `--gpu_max_mem_percentage` -- and on its own that flag has no effect when the skeleton fails: `model_kwargs["max_memory"]` is only set by the `_disk_offload` and `use_seq_device_map` paths, so the skipped `infer_auto_device_map` branch never applies it. Point at `--use_seq_device_map` (which does apply the percentage) and `--batch_size` instead (@coderabbitai). Not adopting the suggested unconditional cap: the original code only shrinks `max_memory` when `infer_auto_device_map` reports a CPU spill, so capping whenever the skeleton fails would force CPU offload onto models that currently fit entirely on GPU. - Drop the support-matrix explanation note; the CHANGELOG covers it (@cjluo-nv). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Chenjie Luo <chenjiel@nvidia.com>
meenchen
left a comment
meenchen
left a comment
There was a problem hiding this comment.
Bot review (gpt-5.6-sol) — DM the bot to share feedback.
Re-review: the prior max-memory failure-path concern is addressed by the revised warning, and extracting _build_meta_skeleton makes the fallback testable. However, two critical items remain: no tests were committed for the new retry/skip behavior, and the Phi-3-vision compatibility claim still overstates the evidence (the author explicitly confirmed it was not tested on supported Transformers 4.57 and is being removed for product reasons). The changelog also retains the obsolete CLI spelling. Design review remains incomplete: the problem is preventing an optional sizing skeleton from aborting the real load; alternatives include the repo's direct include_buffers=False approach in modelopt/torch/utils/plugins/model_load_utils.py, delegating to Transformers/Accelerate device_map="auto" with max_memory, and Transformers' safe meta-creation patches. The PR discusses the last two and why global-meta creation is hazardous, but still does not explain why the two-tier retry is preferable to always using the existing include_buffers=False pattern. Please document that tradeoff before approval.
| return hf_config | ||
|
|
||
|
|
||
| def _build_meta_skeleton(from_config, config_for_init, model_kwargs, architecture): |
There was a problem hiding this comment.
Bot comment.
This addresses the implementation shape, but the prior critical test request is still unresolved: none of the changed files adds tests for this helper. Please commit mocked coverage for (1) include_buffers=True succeeding, (2) the first attempt failing and include_buffers=False succeeding, and (3) both attempts failing with one warning. The last case should also exercise get_model to pin that infer_auto_device_map is skipped while from_pretrained is still called. Existing test_get_model_* tests only exercise a successful skeleton.
There was a problem hiding this comment.
Both points addressed in 1b062da — and you were right that the retry needed justifying. It did not survive the justification, so it is gone.
Design. The probe answers exactly one boolean: will the model spill to CPU, so should max_memory get the gpu_max_mem_percentage haircut. (inferred_device_map has two references in the file — it is built, tested with "cpu" in .values(), and discarded.) infer_auto_device_map reaches that via compute_module_sizes, which is tensor.numel() * dtype_byte_size(tensor.dtype) — shapes and dtypes, never storage.
So the probe only has to reproduce the module tree, and the binding constraint is that it must be no stricter than the loader it predicts:
how from_pretrained builds the model |
|
|---|---|
| transformers 4.57.6 | init_contexts = [no_init_weights(), init_empty_weights()] — include_buffers defaults to False (modeling_utils.py:4378) |
| transformers 5.5.4 | torch.device("meta") + meta_device_safe_creation_ops() (redirects torch.linspace to CPU) |
| old probe | bare torch.device("meta"), no patch |
include_buffers=True was stricter than both. accelerate implements it as a bare global device context, so it captures scratch arithmetic in __init__ that has nothing to do with weights. Transformers added meta_device_safe_creation_ops precisely because pre-v5 remote code derives scalars that way. A model doing int(torch.tensor(...)) in __init__ loads fine via from_pretrained on 4.x and 5.x, but died in our probe — an optimization killing runs it exists only to speed up.
So the answer to "why not just use the existing include_buffers=False pattern" is: we should, and now do. Single build, no retry loop, no noqa: PERF203, no error accumulation. Measured cost on real checkpoints is nil — Qwen3-8B and DeepSeek-R1-Distill-Llama-70B both retain 0.0 MiB and produce identical sized totals (15.256 / 131.417 GiB) either way, for +0.09s and +0.19s.
One note on build_meta_causal_lm: it uses include_buffers=False for a different reason, not as a general preference. Its skeleton is kept and has real weights loaded into it under FSDP2, and inv_freq / original_inv_freq are persistent=False (verified absent from state_dict()), so they are computed at init and never loaded — on meta they would stay meta. It needs real buffers; this probe needs none. Same call, opposite motivation.
Tests. Added in tests/examples/hf_ptq/test_example_utils.py: the probe uses permissive patching, it survives a meta-hostile __init__ (int(torch.tensor(...)), the Phi-4-MM shape), it returns None with a warning on failure, and get_model skips infer_auto_device_map while still calling from_pretrained — asserting no max_memory cap is invented, per the CodeRabbit thread. 32 passed in the file.
| - Remove the deprecated ``examples/llm_qad`` Megatron-LM QAD example (deprecated in 0.45). Use the `megatron_bridge QAD example <https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/megatron_bridge#quantization-aware-distillation-qad>`_ instead, which provides a simpler Python-based interface and better model coverage. | ||
| - Dropped VILA / NVILA vision-language model support in ``examples/hf_ptq``. VILA's modeling code requires ``transformers<=4.50.0``, which conflicts with ModelOpt's minimum supported ``transformers`` version. The VILA-specific bootstrap (repo clone, ``requirements-vila.txt``) and loading paths in ``example_utils.py`` have been removed. | ||
| - Dropped **Phi-3-vision** and **Phi-4-multimodal** PTQ support in ``examples/hf_ptq`` (NVBug 6563509). Their bundled remote code predates Transformers v5 and no longer loads on any version ModelOpt supports (``transformers>=4.57``): Phi-4-multimodal requires ``transformers<4.52`` because it reaches ``prepare_inputs_for_generation`` through ``peft``, which needs ``PreTrainedModel`` to still inherit ``GenerationMixin``, and both models declare ``_tied_weights_keys`` as a list, which Transformers 5.x rejects. The support-matrix row, the ``phi4mm`` model type, the multimodal-detection heuristics that only ever matched these two (``vision_lora`` / ``audio_processor`` / ``embd_layer.image_embd_layer``), the ``Phi3Image`` / ``PhiImage`` embedding-export exclusions, and the ``modelopt_recipes/huggingface/phi4mm/`` recipes have been removed. Text-only Phi-3/Phi-4 and Phi-3.5-MoE are unaffected. | ||
|
|
There was a problem hiding this comment.
Bot comment.
This still says Phi-3-vision “no longer loads on any version ModelOpt supports,” but the author’s reply confirms it was not tested on supported Transformers 4.57 and that its identified _tied_weights_keys blocker is a 5.x failure. If the removal is a product decision because the model is superseded, document that rationale instead of making an unverified compatibility claim (or provide the 4.57 repro/blocker).
| - Fix ``examples/vllm_serve`` serving shared experts uncalibrated: their ``gate_proj``/``up_proj`` quantizer keys were not merged into ``gate_up_proj`` on reload, so they matched no module and were dropped. | ||
| - Fix Qwen3-VL MoE PTQ failing on ``transformers>=5.12`` with ``AttributeError: 'QuantQwen3VLMoeTextExperts' object has no attribute 'hidden_size'`` (NVBug 6518551). transformers 5.12 moved ``Qwen3VLMoeTextExperts`` onto the standard ``@use_experts_implementation`` fused layout (``hidden_size``/``expert_dim`` renamed to ``hidden_dim``/``intermediate_dim``, ``gate_up_proj`` transposed to ``(num_experts, 2*intermediate_dim, hidden_dim)``, two ``F.linear`` calls per expert), but the legacy ``_QuantQwen3VLMoeTextExperts`` wrapper stayed statically registered and shadowed on-the-fly detection. The new layout is now left to ``register_fused_experts_on_the_fly``, which claims it with the generic ``_QuantFusedExperts``; the legacy wrapper is still registered on ``transformers<5.12``, whose ``torch.bmm``-based forward the generic wrapper cannot intercept. | ||
| - Fix ``examples/hf_ptq`` multi-node FSDP2 export (``--use_fsdp2``) failing with ``RuntimeError: Cannot set version_counter for inference tensor``. ``export_quantized`` wrapped its whole body in ``torch.inference_mode()``, so the full params gathered by ``get_model_state_dict(full_state_dict=True)`` were inference tensors and the subsequent ``state_dict()`` -> ``param.detach()`` could not set their version counter. The export context is now ``torch.no_grad()``, which still disables autograd but keeps the gathered params as normal tensors. | ||
| - Fix ``examples/hf_ptq`` aborting the whole model load when the throwaway meta-device skeleton it builds to size ``infer_auto_device_map`` cannot be constructed (NVBug 6563509). ``init_empty_weights(include_buffers=True)`` pushes a global ``torch.device("meta")`` context, so remote-code checkpoints that derive scalar hyperparameters from real tensors in ``__init__`` failed with ``Tensor.item() cannot be called on meta tensors`` before ``from_pretrained`` was ever reached. The skeleton is now retried without the global meta context, and if that also fails the memory estimate is skipped with a warning instead of failing the run; ``--use_seq_device_map`` and ``--gpu_max_memory_percentage`` cover the lost heuristic. |
There was a problem hiding this comment.
Bot comment.
The fallback warning was corrected, but this changelog entry still uses the nonexistent --gpu_max_memory_percentage spelling and implies that flag independently covers the lost estimate. The actual option is --gpu_max_mem_percentage, and in this failure path it only takes effect with --use_seq_device_map; please mirror the accurate warning text here.
The probe exists only to answer one boolean -- will this model spill to CPU, so
should max_memory get the gpu_max_mem_percentage haircut. `infer_auto_device_map`
gets there via `compute_module_sizes`, which reads `tensor.numel()` and
`tensor.dtype` and never touches storage. So the probe only has to reproduce the
module tree, and it must be no stricter than the loader it predicts.
`include_buffers=True` was stricter than both loaders:
- Transformers 4.x `from_pretrained` builds under `init_empty_weights()`, whose
`include_buffers` defaults to False (modeling_utils.py:4378).
- Transformers 5.x builds under `torch.device("meta")` plus
`meta_device_safe_creation_ops()`, which redirects `torch.linspace` to CPU
precisely so pre-v5 remote code that derives scalars in `__init__` keeps
working.
accelerate implements `include_buffers=True` as a bare global `torch.device("meta")`
context, which also captures scratch arithmetic in `__init__` that has nothing to
do with weights. A model doing `int(torch.tensor(...))` there loads fine through
`from_pretrained` on both 4.x and 5.x but died in our probe -- an optimization
killing runs it was only meant to speed up.
Dropping to a single `include_buffers=False` build removes the retry loop, the
PERF203 waiver, and the two-attempt error accumulation. Measured cost on real
checkpoints is nil: Qwen3-8B and DeepSeek-R1-Distill-Llama-70B both report 0.0 MiB
retained and identical sized totals (15.256 / 131.417 GiB) either way, for +0.09s
and +0.19s respectively.
Adds the coverage requested in review: the probe uses permissive patching, it
survives a meta-hostile `__init__`, it returns None and warns on failure, and
`get_model` skips `infer_auto_device_map` while still calling `from_pretrained`
without inventing a max_memory cap.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Chenjie Luo <chenjiel@nvidia.com>
2 participants
What does this PR do?
Type of change: Bug fix + deprecation
Fixes NVBug 6563509, where
hf_ptq.pyon Phi-4-multimodal-instruct died withRuntimeError: Tensor.item() cannot be called on meta tensors.1. Don't abort the load when the meta-device skeleton fails.
get_model()builds a throwaway skeleton underinit_empty_weights(include_buffers=True)purely to size the model forinfer_auto_device_map. That flag makes accelerate push a globaltorch.device("meta")context, so every tensor constructed in__init__lands on meta — not just parameters and buffers. Remote-code checkpoints
written before Transformers v5 routinely derive scalar hyperparameters from
real tensors there, and Transformers documents this exact hazard in
initialization.py(meta_device_safe_creation_ops, which only patchestorch.linspace).The skeleton is now retried without the global meta context; if that also
fails, the memory estimate is skipped with a warning instead of killing the
run.
--use_seq_device_mapand--gpu_max_memory_percentagealready coverthe lost heuristic.
2. Drop Phi-3-vision / Phi-4-multimodal.
Fixing (1) does not make these loadable — it just moves the failure. Their
bundled remote code predates Transformers v5 and does not load on any
version in our supported range (
transformers>=4.57,<5.15):peft.get_peft_modelreadsprepare_inputs_for_generation, gone since transformers 4.52 droppedGenerationMixinfromPreTrainedModelmodeling_phi4mm.py:1959_tied_weights_keysdeclared as a list; Transformers 5.x calls.keys()on it inpost_initmodeling_phi4mm.py:1937,modeling_phi3_v.py:1214int(torch.tensor(...))in__init__under 5.x's meta-devicefrom_pretrainedspeech_conformer_encoder.py:1435The Phi-4-MM model card pins
transformers==4.48.2/peft==0.13.2, so thereis no overlap with our floor and nothing on our side can bridge it.
Removed: the support-matrix row, the
phi4mmmodel type, themultimodal-detection heuristics that only ever matched these two
(
vision_lora,audio_processor,embd_layer.image_embd_layer), thePhi3Image/PhiImageembedding-export exclusions, andmodelopt_recipes/huggingface/phi4mm/.Text-only Phi-3/Phi-4 and Phi-3.5-MoE are natively supported by
transformers and are untouched.
Testing
Measured on 2xH200,
nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc23(torch 2.12, transformers 5.5.4), against the real checkpoint:
4.48.2 / 4.49.0 / 4.50.0 / 4.51.3 and fails at 4.53.3 / 4.56.2 / 4.57.1
(
AttributeError) and 5.5.4 (meta-init, then tied-keys).naming both attempts and proceeds to
from_pretrained(which then surfacesthe real blocker) instead of hard-crashing in the skeleton.
get_model()on Qwen3-8B still builds the skeleton on thefirst try and maps across both GPUs, with no new warnings.
pytest tests/unit/torch/export— 139 passed, 8 skipped. The 1 failure(
test_build_reverse_rules_from_mixtral_conversion_mapping_cpu) is apre-existing broken-import in my local env and fails identically on the base
commit.
pre-commitclean on all changed files, including recipe validation.Before your PR is "Ready for review"
Phi-4-multimodal is removed, along with the
huggingface/phi4mm/ptq/*recipes. Both models are already unloadable on every supported transformers
version, so no working workflow regresses.
CONTRIBUTING.md: N/Afallback on an examples-only code path; covered by the existing export suite.
Additional Information
Two related references were left in place deliberately; say the word and I'll
fold them in:
tests/examples/hf_ptq/test_deploy.pystill deploys the already-publishednvidia/Phi-4-multimodal-instruct-{NVFP4,FP8}checkpoints. Those artifactsexist and serve fine; this PR only removes the ability to produce them.
examples/torch_onnx/README.mdstill lists Phi-4-multimodal-instruct. Thatis a separate ONNX pipeline that does not go through
get_model()and wasnot tested here.
Summary by CodeRabbit
Breaking Changes
Bug Fixes