Diffusion drafts. The target verifies. D3D overlaps both.
A decoupled draft-and-verify architecture, powered by Qwen3-derived diffusion drafts.
D3D decouples a diffusion draft model from the target verifier on separate GPUs. Draft generation overlaps target verification, with verification feedback selecting reusable candidates for the next decoding step.
The contribution is the D3D execution architecture: independent drafting, overlapped verification, and feedback-driven candidate reuse.
| Asynchronous execution | Parallel diffusion drafts |
|---|---|
| Drafting and verification run on separate GPUs. | Block-wise candidate generation with packed outcomes. |
How it works: a target-selected anchor starts a parallel diffusion draft; the target verifies the pending block while the draft GPU prepares possible continuations; verification feedback selects the continuation to reuse. See the architecture guide for the execution flow.
Six Qwen3-derived draft checkpoints are grouped in fudan-generative-ai / D3D. These are
draft models, not replacements for the target models. p denotes pruning;
24D and 31D denote the retained transformer-layer counts.
| Target verifier | Draft family | Layers | D3D checkpoint |
|---|---|---|---|
| Qwen3-8B | 1.7B | 28 | D3D-1.7B-8B |
| Qwen3-8B | 8B | 36 | D3D-8B-8B |
| Qwen3-8B | Pruned 8B | 31 | D3D-31D-p8B-8B |
| Qwen3-32B | 1.7B | 28 | D3D-1.7B-32B |
| Qwen3-32B | 8B | 36 | D3D-8B-32B |
| Qwen3-32B | Pruned 1.7B | 24 | D3D-24D-p1.7B-32B |
Checkpoints retain their original FP32 tensors; inference uses BF16. Each Hub repository includes tokenizer assets, inference compatibility code, a model card, and release metadata. The family size in a name is not an exact post-pruning parameter count. Exact source steps, tensor counts, and release compatibility notes are in checkpoint details.
Training, layer pruning, and native single-request/concurrent inference share PyTorch 2.8.0, Transformers 4.57.3, FlashAttention 2.8.3, and CUDA 12.8. Python 3.10+ is supported; unified-environment GPU validation used Python 3.12 and two NVIDIA H800 80 GB GPUs.
git clone https://github.com/linguancong524/D3D.git
cd D3D
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
pip install flash-attn==2.8.3 --no-build-isolation
pip install -e . --no-depsInstall PyTorch before building FlashAttention. A matching prebuilt wheel can be used instead; source builds need a CUDA toolkit, a C++ compiler, and sufficient host memory. The concurrent backend requires FlashAttention. The optional SGLang backend still requires its separately locked environment; do not install its runtime lock into this environment.
export CUDA_VISIBLE_DEVICES=0,1
# optional: override the defaults for every script
export TARGET_MODEL=Qwen/Qwen3-8B
export DRAFT_MODEL=fudan-generative-ai/D3D-1.7B-8BThe target runs on cuda:0, and the draft runs on cuda:1. Model IDs also
accept local checkpoint directories. Public model downloads do not require
a Hugging Face token.
Every script forwards extra flags to its Python entry point, so prefer command
line options over environment variables. Single-request scripts take
--verifier / --drafter; concurrent scripts take --target / --draft.
Single request — one MATH-500 sample, up to 128 generated tokens:
bash scripts/quickstart_single.sh
# e.g. another pair with a longer generation cap
bash scripts/quickstart_single.sh \
--verifier Qwen/Qwen3-32B \
--drafter fudan-generative-ai/D3D-24D-p1.7B-32B \
--max-new-tokens 512Concurrent requests — four samples at concurrency 4:
bash scripts/quickstart_concurrent.sh
# e.g. concurrency 8 with 256 KV pages
CONCURRENCY=8 bash scripts/quickstart_concurrent.sh --pages 256Use the 32B target with the 24-layer pruned draft
bash scripts/quickstart_single.sh \
--verifier Qwen/Qwen3-32B \
--drafter fudan-generative-ai/D3D-24D-p1.7B-32B
bash scripts/quickstart_concurrent.sh \
--target Qwen/Qwen3-32B \
--draft fudan-generative-ai/D3D-24D-p1.7B-32B \
--concurrency 4 --pages 128Useful configuration options
Scripts pass their defaults as flags, and anything appended on the command line wins. Only a few environment variables remain:
| Variable | Purpose | Default |
|---|---|---|
TARGET_MODEL |
Target verifier used by every script | Qwen/Qwen3-8B |
DRAFT_MODEL |
Draft checkpoint used by every script | fudan-generative-ai/D3D-1.7B-8B |
DATASETS |
Dataset list for the two benchmark scripts | the seven benchmarks |
CONCURRENCY |
Quick-start request count, or benchmark sweep list | 4 / 1 4 8 16 |
D3D_DATA_CACHE |
Optional existing JSONL dataset cache | data_cache/ |
Everything else is a flag, for example --max-samples, --max-new-tokens,
--block-size, --anchor-num, --pages or --no-baseline.
Use python single/eval_async_packed.py --help or
python -m serving.bench --help for the full set of options.
Run single-request evaluation over the standard datasets:
bash scripts/benchmark.shRun concurrent evaluation at request concurrency 1, 4, 8, and 16:
bash scripts/benchmark_concurrent.shDatasets are downloaded at runtime, not bundled in this repository.
LiveCodeBench v6 uses test6.jsonl from
livecodebench/code_generation_lite; the other datasets use datasets.
See the benchmark protocol for sources, splits,
token caps, timing boundaries, and reporting rules.
The repository supports native D3D inference and an optional two-GPU SGLang Hybrid backend. Install SGLang separately:
bash scripts/sglang/install.shStart the two-GPU backend (GPU 0: Target + local Draft; GPU 1: optional Draft):
export TARGET_MODEL=Qwen/Qwen3-8B
export DRAFT_MODEL=fudan-generative-ai/D3D-1.7B-8B
export TARGET_GPU=0 DRAFT_GPU=1
export MAX_RUNNING_REQUESTS=1
bash scripts/sglang/launch.sh hybridThen run a MATH-500 evaluation from another terminal:
export MAX_RUNNING_REQUESTS=1
export TARGET_MODEL=Qwen/Qwen3-8B
.venv-sglang/bin/python scripts/sglang/benchmark.py \
--dataset math500 \
--tokenizer "$TARGET_MODEL" \
--samples 128 --concurrency "$MAX_RUNNING_REQUESTS" \
--repetitions 1 --max-new-tokens 512 \
--output outputs/sglang/example.jsonThe dataset is downloaded automatically and cached by Hugging Face Datasets.
Set MAX_RUNNING_REQUESTS to 1, 4, 8, or 16 in both terminals;
restart the backend after changing it.
SGLang Hybrid throughput on the complete seven-benchmark suite.
D3D-1.7B → Qwen3-8B, concurrency 1, anchor_num=1, block size 16,
max_new_tokens=512, greedy decoding, thinking disabled, and natural EOS.
| Dataset | TPS (speedup) |
|---|---|
| GSM8K | 658.10 (4.876×) |
| MATH-500 | 572.66 (4.243×) |
| AIME25 | 394.56 (2.924×) |
| HumanEval | 479.15 (3.550×) |
| MBPP | 428.39 (3.174×) |
| LCB-v6 | 267.42 (1.982×) |
| MT-Bench | 258.54 (1.916×) |
| Aggregate | 501.60 (3.717×) |
Hybrid uses two H800 GPUs; AR uses one. TPS includes prefill and generation; startup, tokenization, and warmup are excluded. These are throughput results, not answer-quality scores.
Training code lives in train/ and pruning/, including
offline training on target-generated responses, layer selection, physical
pruning, recovery training, and recovery-checkpoint selection.
Use the same environment installed above with requirements.txt for
training and native inference.
export DRAFT_MODEL=checkpoints/Qwen3-1.7B
export TARGET_MODEL=checkpoints/Qwen3-8B
export DATA_PATH=data/train.jsonl
CUDA_VISIBLE_DEVICES=0,1 NUM_GPUS=2 bash scripts/train.sh \
--attn-implementation sdpa --disable-wandbDownload the initial checkpoints and prepare the JSONL dataset first. See the training and progressive-pruning guide for installation, data schemas, loss settings, and all four pruning stages.
single/ Single-request asynchronous draft-and-verify decoding
serving/ Concurrent decoding, paged KV cache, and AR baselines
models/d3d/ Qwen3-derived D3D draft backbone and compatibility code
models/d3d_training/ Training-specific model port (shared dependencies)
train/ Offline training on target-generated response tokens
pruning/ Layer selection, physical pruning, recovery, checkpoint selection
common/ Shared helpers: devices, checkpoints, datasets, text, metrics
scripts/ Quick starts, smoke checks, and benchmark entry points
benchmarks/ Dataset sources and evaluation protocol
docs/ Architecture, model details, and validation notes
tests/ Model-loading, release, and metric-accounting checks
D3D-authored code is released under the MIT License. Included third-party code and Qwen-derived weights retain their own terms; see THIRD_PARTY_NOTICES.md and the individual model cards.
