FP8 checkpoint in Qwen3.8-27B-FP8's layout (vLLM, no flags)
#22
by processreap - opened
Thanks for releasing Arete. I made an FP8 checkpoint for anyone serving it with vLLM: https://huggingface.co/processreap/Arete-27B-FP8
It copies Qwen3.8-27B-FP8's layout tensor for tensor (e4m3, 128×128 blocks, same BF16 modules left unquantized), so vLLM reads the quantization from config.json with no --quantization flag. Static weight quantization, no calibration data. The converter script is in the repo.
Measured on 2× A6000:
- Throughput matches Qwen3.8-27B-FP8 (TTFT 2318.8 vs 2318.6 ms, TPOT 32.69 vs 32.75 ms)
- 11/11 on a narrow agentic tool-use benchmark, same as BF16 with load-time FP8
Not measured: Propaganda_Eval. Block FP8 at ~2.6% relative weight error should preserve it, but I haven't checked. If you run it, I'll add the numbers to the card. No MTP weights, since Arete doesn't ship them.
A link from your model card would help people find it.