Repository navigation
🗺️ Roadmap: Hugging Face Ecosystem Full Compatibility (transformers / accelerate / peft / trl) #204
Description
Goal
Make candle a drop-in PyTorch replacement for the entire HuggingFace ecosystem. Target:
USE_CANDLE=1 python -c "
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('gpt2')
model = AutoModelForCausalLM.from_pretrained('gpt2')
print(tokenizer.decode(model.generate(tokenizer('Hello', return_tensors='pt').input_ids, max_new_tokens=20)[0]))
"Priority follows the HF dependency chain: transformers (core) → accelerate → peft → trl.
Current State
| Capability | Status | Notes |
|---|---|---|
| nn.Module (state_dict, load_state_dict, hooks) | ✅ Ready | Full implementation |
| nn modules (Linear, Conv, Norm, RNN, Attention, etc.) | ✅ Ready | Comprehensive coverage |
| Optimizers & LR Schedulers | ✅ Ready | Adam, AdamW, SGD, etc. with state_dict |
| AMP (autocast + GradScaler) | ✅ Ready | Device-type dispatch |
| Serialization (save/load) | ✅ Ready | PyTorch zip checkpoint format |
| scaled_dot_product_attention | No Flash/fused attention | |
| DataLoader + Samplers | ✅ Ready | Multi-worker, pin_memory |
| Distributed (DDP, FSDP, collectives) | ✅ Ready | Gloo + HCCL backends |
| torch.utils.checkpoint | ✅ Exists | Needs HF integration validation |
| torch.library / custom_op | ✅ Ready | 2-arg Library ctor only |
| compat/transformers test framework | ✅ Ready | Tier 1-3 model test runner |
| Meta device / to_empty() | ❌ Missing | Blocks low_cpu_mem_usage loading |
| Native safetensors loading | ❌ In compat layer only | Not in src/candle/ |
| Text generation (GenerationMixin) | ❌ Missing | No generate(), KV-cache, sampling |
| torch.compile / _dynamo | Blocks compiled model paths | |
| Quantization (GPTQ/AWQ/bitsandbytes) | ❌ Stub only | quantize_per_tensor raises RuntimeError |
| PEFT (LoRA adapter injection) | ❌ Untested | Module internals may work |
| Accelerate integration | ❌ Untested | Low-level pieces exist |
Sub-Issues
Phase 1: Foundation (Target: v0.1.1 — June 2026)
- [Task][HF] 1.1: Meta device and Module.to_empty() implementation #223 — 1.1 Meta device and
Module.to_empty()⤴️ also vLLM [Task][vLLM] 1.3: Add meta/fake tensor support #237 - [Task][HF] 1.2: Native safetensors loading in candle source #224 — 1.2 Native safetensors loading in candle source
- [Task][HF] 1.3: Promote torch submodule stubs from conftest to candle source #225 — 1.3 Torch submodule stubs promoted to candle source
⤴️ partial overlap with vLLM [Task][vLLM] 1.5: Make torch.compile(..., backend="eager") a valid no-op #239 - [Task][HF] 1.4: Fix Library() 3-arg constructor #226 — 1.4 Fix
Library()3-arg constructor⤴️ also vLLM [Task][vLLM] 1.1: Fix Library() 3-arg constructor #235
Phase 2: HF-Compatible Inference (Target: v0.2.0 — September 2026)
- [Task][HF] 2.1: Text generation / GenerationMixin support #227 — 2.1 Text generation / GenerationMixin support
- [Task][HF] 2.2: SDPA optimization paths and attention mask handling #228 — 2.2 SDPA optimization paths and attention mask handling
- [Task][HF] 2.3: Gradient checkpointing integration with HF transformers #229 — 2.3 Gradient checkpointing integration
⤴️ also Megatron [Task][Megatron] 2.4: Validate activation checkpointing compatibility #258 - [Task][HF] 2.4: Run and baseline compat/transformers tier-1 report #230 — 2.4 Run and baseline compat/transformers tier-1 report
Phase 3: Training & Fine-Tuning (Target: v0.3.0 — December 2026)
- [Task][HF] 3.1: Accelerate Accelerator.prepare() compatibility #231 — 3.1 Accelerate
Accelerator.prepare()compatibility - [Task][HF] 3.2: Quantization foundation (bitsandbytes 4-bit/8-bit) #232 — 3.2 Quantization foundation (bitsandbytes 4-bit/8-bit)
- [Task][HF] 3.3: PEFT adapter injection compatibility (LoRA) #233 — 3.3 PEFT adapter injection compatibility
- [Task][HF] 3.4: TRL (Trainer/SFT/DPO) compatibility #234 — 3.4 TRL (Trainer/SFT/DPO) compatibility
Cross-Roadmap Duplicates
| Capability | HF | vLLM | Megatron | Notes |
|---|---|---|---|---|
| Meta device | #223 | #237 | — | Resolving one closes both |
| Library() 3-arg | #226 | #235 | — | Resolving one closes both |
| _dynamo / compiler stubs | #225 | #239 | — | Partial overlap; coordinate work |
| Activation checkpointing | #229 | — | #258 | HF = basic paths; Megatron adds selective recompute + RNG |
Success Criteria
- v0.1.1:
from transformers import AutoModel; model = AutoModel.from_pretrained("bert-base-uncased")works without compat conftest - v0.2.0:
model.generate()works for GPT-2/LLaMA-class models; tier-1 compat test pass rate ≥ 80% - v0.3.0: Fine-tuning with accelerate + PEFT LoRA works end-to-end
Activity