Plan for MoE #5456
Description
Is there a plan for adding Mixture of experts for GPT-style models? I've found this PR https://github.com/NVIDIA/NeMo/pull/5409/files but that seems to be for T5-like models. Thanks!
{{ message }}
Is there a plan for adding Mixture of experts for GPT-style models? I've found this PR https://github.com/NVIDIA/NeMo/pull/5409/files but that seems to be for T5-like models. Thanks!