Conversation
New cookbooks/cosmos3/quantization/ ships a self-contained ModelOpt static-scale FP8 recipe: - src/: the Cosmos3 DiT model + full recipe (load, calibrate on real denoising, export a drop-in diffusers checkpoint). - notebooks/: quantize Nano/Super, Super Text-to-Image, and Super Image-to-Video (base + 4-step distilled). - Reproduces the shipped FP8 checkpoints bit-for-bit. Also add a "Quantized Checkpoints" section to the vLLM-Omni generator notebooks (audiovisual, action, transfer) and the vLLM reasoner notebook: serving the FP8 build is just --revision fp8. Signed-off-by: Wojciech Kutak <wkutak@nvidia.com>
wkutak
changed the title
|
@wkutak , who is the tester validate the inference function and user guide in this PR? The person should be reviewer of this PR as well, approve once tested working with log test result in google doc. |
Signed-off-by: Wojciech Kutak <wkutak@nvidia.com>
|
@KrzysztofTomala - please review and approve this PR once you completed testing. |
|
vLLM cookbook The model download and server startup works fine. I had to remove from the video example as the requests were failing, but the issue is happening also on the non-FP8 model. |
|
vLLM-Omni cookbooks |
|
Quantization cookbooks Everything works, there are a couple of warning, but nothing crucial. |
KrzysztofTomala
left a comment
KrzysztofTomala
left a comment
There was a problem hiding this comment.
Overall everything works as expected, all of the serious problems are not FP8 related
…d99f1254459ef3a4f1bd4263' Signed-off-by: Wojciech Kutak <wkutak@nvidia.com>
… README, small fixes Signed-off-by: Wojciech Kutak <wkutak@nvidia.com>
|
The transfer example issue @KrzysztofTomala mentioned will be fixed here: #299 |
|
@wkutak Can you please fix the merge conflict? |
Signed-off-by: Wojciech Kutak <wkutak@nvidia.com>
|
Merge conflicts fixed, @rickzw please review the PR |
Remove vendored Cosmos3 model code. Replace model, tokenizer, vae, and scheduler loading with cosmos-framework's API.
* Update dependency. * Use legacy calibration for all notebooks.
Migrate everything under `cookbooks/cosmos3/quantization/src` to `cosmos-framework`. Import paths in the notebooks are modified accordingly. Signed-off-by: Mu-Ti Chung <mutic@nvidia.com>
|
The recent commits are to achieve the following goals:
Blocked by NVIDIA/cosmos-framework#233. |
Moves all python helpers under `src/` from the quantization cookbooks in NVIDIA/cosmos#298 into `cosmos-framework`. The cookbooks will become thin wrappers that calls the functions to quantize checkpoints. Blocks NVIDIA/cosmos#298. --------- Signed-off-by: Mu-Ti Chung <mutic@nvidia.com> Co-authored-by: lfengad <liangf@nvidia.com> Co-authored-by: pengcuo <pzeren@nvidia.com>
|
Does it make sense to not have a folder containing 3 notebooks, but take those 3 notebook out of folder and have 3 notebook parallel to readme.md? |
…de to cosmos-framework repo. Signed-off-by: Wojciech Kutak <wkutak@nvidia.com>
## Problem The original quantization pipeline did not create the corresponding `modelopt_state.pth` that is presented in the released FP8 checkpoints. The file is required for HuggingFace `transformers`/`diffusers` to load the and run inference on the quantized checkpoints. ## Changes * Export the missing `modelopt_state.pth` files for `transformers` and `diffusers`. * Fix updated functions imported by legacy mode. * Add mixed precision W8A16 policy configuration. Blocks NVIDIA/cosmos#298 --------- Signed-off-by: Mu-Ti Chung <mutic@nvidia.com> Signed-off-by: Wojciech Kutak <wkutak@nvidia.com> Co-authored-by: Wojciech Kutak <wkutak@nvidia.com>
Signed-off-by: Wojciech Kutak <wkutak@nvidia.com>
5 participants
New cookbooks/cosmos3/quantization/ ships a self-contained ModelOpt static-scale FP8 recipe:
Also add a "Quantized Checkpoints" section to the vLLM-Omni generator notebooks (audiovisual, action, transfer) and the vLLM reasoner notebook: serving the FP8 build is just --revision fp8.