D3D decouples a diffusion draft model from the target verifier on separate GPUs. While the target verifies a draft block, the draft prepares continuations for several possible verification outcomes. The target's result then selects the matching continuation for the next decoding step.
How it works: a target-selected anchor starts a parallel diffusion draft; the target verifies the pending block while the draft GPU prepares possible continuations; verification feedback selects the continuation to reuse. See the architecture guide for the execution flow.
The D3D collection
contains six Qwen3-derived draft checkpoints. These checkpoints are draft
models, not replacements for their target models. p denotes pruning; 24D
and 31D denote retained transformer-layer counts.
| Target verifier | Draft family | Layers | D3D checkpoint |
|---|---|---|---|
| Qwen3-8B | 1.7B | 28 | D3D-1.7B-8B |
| Qwen3-8B | 8B | 36 | D3D-8B-8B |
| Qwen3-8B | Pruned 8B | 31 | D3D-31D-p8B-8B |
| Qwen3-32B | 1.7B | 28 | D3D-1.7B-32B |
| Qwen3-32B | 8B | 36 | D3D-8B-32B |
| Qwen3-32B | Pruned 1.7B | 24 | D3D-24D-p1.7B-32B |
git clone https://github.com/fudan-generative-vision/D3D.git
cd D3D
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
pip install flash-attn==2.8.3 --no-build-isolation
pip install -e . --no-depsexport CUDA_VISIBLE_DEVICES=0,1
# optional: override the defaults used below
export TARGET_MODEL=Qwen/Qwen3-8B
export DRAFT_MODEL=fudan-generative-ai/D3D-1.7B-8BThe target runs on cuda:0, and the draft runs on cuda:1.
Single-request — one MATH-500 sample, up to 128 generated tokens:
python inference/native/single_request/eval_async_packed.py \
--drafter "$DRAFT_MODEL" \
--verifier "$TARGET_MODEL" \
--dataset math500 \
--max-samples 1 \
--max-new-tokens 128 \
--verifier-device cuda:0 \
--draft-device cuda:1 \
--dtype bf16 \
--temperature 0 \
--block-size 16 \
--denoising-steps 1 \
--attention-mask-mode causal \
--anchor-num 3 \
--packed-accepted-counts all \
--output-json outputs/single_request/result.json \
--summary-json outputs/single_request/summary.jsonMulti-request — four samples at concurrency 4:
python inference/native/multi_request/bench.py \
--target "$TARGET_MODEL" \
--draft "$DRAFT_MODEL" \
--datasets math500 \
--max-samples 4 \
--max-new-tokens 128 \
--concurrency 4 \
--anchor-num 3 \
--pages 512 \
--output outputs/multi_request/result.jsonOverride TARGET_MODEL / DRAFT_MODEL for another model pair. Increase
--max-new-tokens for longer generations; for multi-request runs, increase
--concurrency and adjust --pages to fit the KV budget.
Run single-request evaluation over the standard datasets:
bash scripts/benchmark_single_request.shRun multi-request evaluation at request concurrency 1, 4, 8, and 16:
bash scripts/benchmark_multi_request.shDatasets are downloaded at runtime. See the benchmark protocol for sources and splits.
The repository supports native D3D inference and an optional two-GPU SGLang Hybrid backend. See the SGLang backend guide. Install SGLang separately:
bash scripts/sglang/install.shStart the two-GPU backend (GPU 0: Target + local Draft; GPU 1: optional Draft):
export TARGET_MODEL=Qwen/Qwen3-8B
export DRAFT_MODEL=fudan-generative-ai/D3D-1.7B-8B
export TARGET_GPU=0 DRAFT_GPU=1
export MAX_RUNNING_REQUESTS=1
bash scripts/sglang/launch.sh hybridThen run a MATH-500 evaluation from another terminal:
export MAX_RUNNING_REQUESTS=1
export TARGET_MODEL=Qwen/Qwen3-8B
.venv-sglang/bin/python scripts/sglang/benchmark.py \
--dataset math500 \
--tokenizer "$TARGET_MODEL" \
--samples 128 --concurrency "$MAX_RUNNING_REQUESTS" \
--repetitions 1 --max-new-tokens 512 \
--output outputs/sglang/example.jsonThe dataset is downloaded automatically and cached by Hugging Face Datasets.
Set MAX_RUNNING_REQUESTS to 1, 4, 8, or 16 in both terminals;
restart the backend after changing it.
SGLang Hybrid throughput on the complete seven-benchmark suite.
D3D-1.7B → Qwen3-8B, concurrency 1, anchor_num=1, block size 16,
max_new_tokens=512, greedy decoding, thinking disabled, and natural EOS.
| Dataset | tokens/s (speedup) |
|---|---|
| GSM8K | 658.10 (4.876×) |
| MATH-500 | 572.66 (4.243×) |
| AIME25 | 394.56 (2.924×) |
| HumanEval | 479.15 (3.550×) |
| MBPP | 428.39 (3.174×) |
| LCB-v6 | 267.42 (1.982×) |
| MT-Bench | 258.54 (1.916×) |
| Aggregate | 501.60 (3.717×) |
All corpora use Open-PerfectBlend prompts. See each dataset card for generation settings, statistics, and download instructions. Released models and datasets are grouped in the D3D collection.
| Corpus | Target Model | Samples | Dataset |
|---|---|---|---|
| PerfectBlend: Qwen3.8-27B Thinking | Qwen3.8-27B | 1,417,675 | PerfectBlend-Qwen3.8-27B-Thinking |
| PerfectBlend: Qwen3-4B Thinking (Greedy) | Qwen3-4B | 1,420,909 | PerfectBlend-Qwen3-4B-Thinking-Greedy |
| PerfectBlend: Qwen3-4B NoThink (Greedy) | Qwen3-4B | 1,420,908 | PerfectBlend-Qwen3-4B-NoThink-Greedy |
| PerfectBlend: Qwen3-4B NoThink (Sampling) | Qwen3-4B | 1,420,909 | PerfectBlend-Qwen3-4B-NoThink-Sampling |
| PerfectBlend: Qwen3-8B NoThink (Greedy) | Qwen3-8B | 1,420,908 | PerfectBlend-Qwen3-8B-NoThink-Greedy |
| PerfectBlend: Qwen3-8B NoThink (Sampling) | Qwen3-8B | 1,420,909 | PerfectBlend-Qwen3-8B-NoThink-Sampling |
| PerfectBlend: Qwen3-32B NoThink (Sampling) | Qwen3-32B | 1,420,500 | PerfectBlend-Qwen3-32B-NoThink-Sampling |
| PerfectBlend: Gemma-4-12B-IT (Greedy) | Gemma-4-12B-IT | 1,420,898 | PerfectBlend-Gemma-4-12B-IT-Greedy |
| PerfectBlend: Gemma-4-31B-IT (Greedy) | Gemma-4-31B-IT | 1,420,903 | PerfectBlend-Gemma-4-31B-IT-Greedy |
Each corpus targets the listed model; use the matching tokenizer for training.
Training code lives in training/train/ and training/pruning/, including
offline training on target-generated responses, layer selection, physical
pruning, recovery training, and recovery-checkpoint selection.
Use the same environment installed above with requirements.txt for
training and native inference.
export DRAFT_MODEL=checkpoints/Qwen3-1.7B
export TARGET_MODEL=checkpoints/Qwen3-8B
export DATA_PATH=data/train.jsonl
CUDA_VISIBLE_DEVICES=0,1 NUM_GPUS=2 bash scripts/train.sh \
--attn-implementation sdpa --disable-wandbDownload the initial checkpoints and prepare the jsonl dataset first. See the training and progressive-pruning guide for installation, data schemas, loss settings, and all four pruning stages.
If you use D3D, please cite:
@article{chen2026d3d,
title = {D3D: Decoupled Diffusion Drafting for Asynchronous Speculative Decoding},
author = {Baoyou Chen and Ruiqiao Mei and Guancong Lin and Tao Song and Weihao Yuan and Qun Liu and Siyu Zhu},
journal = {arXiv preprint arXiv:2611.06425},
year = {2026}
}