Add Muse Glimmer 30B CUDA INT4 recipe

[Justin Chu (justinchuby)]

Summary

Adds a CUDA recipe for meta-models/Muse-Glimmer-30B:

• exports the native BF16 checkpoint as decoder, vision encoder, and embedding ONNX components with Olive MobiusBuilder;
• quantizes eligible weights with OnnxKQuantQuantization (bits=4, block_size=32) for a CUDA-targeted package under 20 GB;
• emits the full ORT GenAI package, including tokenizer and image processor assets;
• enables CUDA Graph capture for the decoder while keeping one-shot vision and embedding sessions uncaptured;
• emits Muse normalization with fused RMSNormalization and residual SkipSimplifiedLayerNormalization operators;
• includes text-only and image+text streaming inference, recipe metadata, model license, and deployment documentation;
• pins mobius-onnx to the exact Muse implementation validated in Add Muse Glimmer 30B support onnxruntime/mobius#475 until that support is released.

Validation

• complete BF16 export and INT4 quantization finished on NVIDIA H200 in 1,436 seconds;
• the generated three-model ORT GenAI package occupies approximately 18 GB;
• real text generation produced a coherent Rayleigh-scattering answer;
• real image generation identified the test mountain scene after enabling decoder CUDA Graph capture;
• repository lintrunner and run-script checks pass;
• the pinned Mobius implementation has real-checkpoint BF16 CUDA L4/L5 coverage and full multimodal tiny parity.

Native Muse loading currently requires microsoft/onnxruntime-genai#2397.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[Copilot]

Pull request overview

Adds a new Olive CUDA INT4 recipe folder for meta-models/Muse-Glimmer-30B, including build config, metadata, and a small streaming inference script to generate an ORT GenAI package via Mobius export + K-Quant.

Changes:

• Add CUDA INT4 Olive pipeline config (MobiusBuilder BF16 export → OnnxKQuantQuantization INT4) and output ignore rules.
• Add end-user documentation (build/inference/validation) and an inference entrypoint using onnxruntime_genai.
• Register the recipe via info.yml and add a recipe-local requirements.txt including a pinned Mobius git dependency.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
meta-models-Muse-Glimmer-30B/requirements.txt	Adds recipe dependencies, including a pinned Mobius git ref and Olive GPU extras.
meta-models-Muse-Glimmer-30B/README.md	Documents conversion pipeline, prerequisites, build, inference, and validation references.
meta-models-Muse-Glimmer-30B/LICENSE	Adds Apache 2.0 license text for the recipe/model distribution context.
meta-models-Muse-Glimmer-30B/info.yml	Registers the recipe for indexing (arch/device/EP + config path).
meta-models-Muse-Glimmer-30B/inference.py	Provides a minimal streaming inference script for the generated ORT GenAI package.
meta-models-Muse-Glimmer-30B/cuda/int4/config.json	Defines the Olive pipeline and CUDA target/output directory.
meta-models-Muse-Glimmer-30B/.gitignore	Ignores Olive cache and generated model artifacts.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Justin Chu (justinchuby)]

CI note: the earlier build failures were the same pre-existing repository-wide pre-commit issue. end-of-file-fixer modified meta-llama-Llama-3.1-8B-Instruct/OpenVINO/info.yaml, which is unchanged by this PR. All files added by this PR pass the repository pre-commit hooks locally; run-script, CLA, and CodeQL passed.

[Justin Chu (justinchuby)]

Full H200 E2E is now complete. The exact recipe finished BF16 export + INT4 K-Quant in 1,436 seconds and produced an 18 GB three-model ORT GenAI package (decoder 15.4 GB, embedding 2.7 GB, vision 1.1 GB external data).

Runtime smoke results with microsoft/onnxruntime-genai#2397:

• native muse_glimmer CUDA text generation completed coherently
• CUDA image generation correctly identified the test image as a Chinatown street scene with a traditional archway

The run exposed and fixed three integration gaps: the distribution name is mobius-onnx, torchvision is required to load the HF image processor, and Muse needs packed Qwen-style image transforms while retaining its own <|patch|> prompt tokens and standard 1D text RoPE. The recipe now pins the corresponding Mobius fix and documents the runtime dependency.

[Justin Chu (justinchuby)]

Final recipe validation for 2bd7d28 completed on NVIDIA H200:

• BF16 Mobius export: 736.8s; INT4 K-Quant: 718.5s; complete package: 18 GB.
• ORT GenAI text and image E2E passed with CUDA Graph capture and shared KV buffer.
• Optimized decoder has 2,458 nodes, no decomposed ReduceMean/Pow RMSNorm paths.
• Same-runtime decode benchmark: 61.76 tok/s median (61.74-61.82), +11.29% over the decomposed graph.
• Published package: https://huggingface.co/justinchuby/Muse-Glimmer-30B-ONNX-INT4-CUDA (including ONNX GenAI inference_metadata.yaml; remote SHA-256 verification passed).