LFM2.5-VL-3B-DSpark on an 8 GB Jetson: 3.2× decode with quantized drafters, no retraining
Hi Liquid team! For a side project I've been running LFM2.5-VL-3B with the DSpark drafter through llama.cpp on edge
devices, and wanted to share the results.
On an 8 GB Jetson Orin Nano Super, decode goes from 27.1 to 86.6 tokens/s: 3.2× over target-only and 1.72× over
the official F16 drafter, with the same greedy output up to near-ties. The drafters are exported from your F16
GGUF without retraining: quantized (P3: decoder Q4_K, Markov head Q8_0) and with a 32k-token draft vocabulary.
Three things that may be useful for the team:
- The Markov head is the edge bottleneck. Its 8 serial GEMVs over
markov_w2(33 MiB) take about half of the
draft time on the Jetson's 2 MB L2, while a desktop GPU's 48 MB L2 hides them. - Verification cost sets the gain per device: 1.07× a single-token decode on the Jetson, 1.6× on an RTX 5070,
1.8× on a Snapdragon 8 Elite NPU, for 3.2×, 1.85× and 1.29× overall. - Japanese gets about 70 % of the English speedup, with the loss concentrated on kanji, where the target itself
is least confident.
Files (GGUF, LFM Open License v1.0): https://huggingface.co/meit-ut/LFM2.5-VL-3B-DSpark-Edge-GGUF
Code, data and reproduction: https://github.com/meitmt/EdgeSpecQ
Technical note: https://github.com/meitmt/EdgeSpecQ/blob/main/docs/tech_note.md
Happy to share more data or run something on these devices if that helps!