LFM3-1.7B-DSpark
A lossless DSpark draft model for LiquidAI/LFM3-1.7B, trained with speculators (training code).
- Up to 3.14× faster on math and 2.63× on code, 2.88× on average (default generation params)
- Mean acceptance length 4.10 on math and 3.24 on code, 2.83 overall (block size 8)
Usage
vLLM (recommended)
Requires vLLM from PR #66323 (Python-only changes, so precompiled kernels can be reused):
VLLM_USE_PRECOMPILED=1 uv pip install \
"vllm @ git+https://github.com/vllm-project/vllm.git@refs/pull/66323/head"
Then serve:
vllm serve LiquidAI/LFM3-1.7B --port 8000 \
--speculative-config '{"model": "rasyosef/LFM3-1.7B-DSpark", "num_speculative_tokens": 8}'
llama.cpp (GGUF)
A GGUF build of this drafter is available at rasyosef/LFM3-1.7B-DSpark-GGUF. Pair it with a LFM3-1.7B GGUF target:
llama-server -hf LiquidAI/LFM3-1.7B-GGUF:Q8_0 \
-hfd rasyosef/LFM3-1.7B-DSpark-GGUF:Q8_0 \
--spec-type draft-dspark --spec-draft-n-max 4 --spec-draft-n-min 0 \
-fa on -ngl 99
Details
- ~0.3B params, bfloat16, block size 8
- Draft vocab 32,000; aux hidden states from verifier layers 3/11/19/23
- 100,000 Open PerfectBlend prompts (filtered) with verifier-generated responses, sample length 2560, packed to 8192
- 4 epochs, lr 4e-4, cosine (4% warmup), 98/2 train/val split, loss
{"ce": 0.1, "tv": 0.9}
Evaluation
RedHatAI/speculator_benchmarks, default generation params (temperature 0.1, top_k 50, repetition_penalty 1.1 from the model's generation_config.json), single A100 40GB, concurrency 1.
Throughput (tokens/s)
Mean output throughput. Baseline is LFM3-1.7B without speculative decoding.
| subset | baseline (no drafter) | DSpark (this model) | speedup |
|---|---|---|---|
| math_reasoning | 220.8 | 692.9 | 3.14× |
| HumanEval | 221.0 | 580.2 | 2.63× |
| average | 220.9 | 636.6 | 2.88× |
Acceptance at default params
acceptance_length is mean tokens committed per verification step, including the bonus token (max 9.0 at block size 8).
| subset | acceptance_length | pos_0 | pos_1 | pos_2 | pos_3 | pos_4 | pos_5 | pos_6 | pos_7 |
|---|---|---|---|---|---|---|---|---|---|
| math_reasoning | 4.101 | 78.5% | 61.1% | 47.7% | 37.4% | 29.3% | 23.3% | 18.4% | 14.5% |
| HumanEval | 3.243 | 71.4% | 49.1% | 34.4% | 24.3% | 17.4% | 12.4% | 8.9% | 6.3% |
| rag | 2.971 | 68.7% | 46.4% | 31.1% | 20.7% | 13.4% | 8.4% | 5.2% | 3.1% |
| tool_call | 2.684 | 64.4% | 39.7% | 24.9% | 15.8% | 10.2% | 6.5% | 4.1% | 2.7% |
| translation | 2.630 | 62.6% | 39.4% | 24.4% | 15.1% | 9.4% | 6.0% | 3.7% | 2.3% |
| writing | 2.586 | 61.4% | 36.7% | 22.7% | 14.7% | 9.7% | 6.3% | 4.2% | 2.9% |
| question | 2.573 | 61.6% | 36.7% | 22.4% | 14.2% | 9.4% | 6.1% | 4.2% | 2.8% |
| summarization | 2.467 | 63.1% | 37.4% | 21.4% | 12.4% | 6.7% | 3.4% | 1.6% | 0.7% |
| qa | 2.446 | 61.6% | 36.2% | 21.0% | 12.0% | 6.8% | 3.8% | 2.1% | 1.2% |
Weighted across all subsets: 2.832 over 485,518 verification steps.
Limitations
Only works with LFM3-1.7B. Throughput was measured only on math_reasoning and HumanEval. Temperature 0, concurrent load, and contexts beyond the 2,560-token training samples were not evaluated.
License
Apache 2.0
- Downloads last month
- 112