Inquiry regarding official FP8 quantization and standard vLLM support for d3-ultra
Hi team, first of all, great work on the d3-ultra release! I'm currently working on deploying these decision models and have encountered a few practical bottlenecks. I'd appreciate some guidance:
Quantization for 48GB GPUs: The BF16 weights for d3-ultra are ~52.2GB. This makes it impossible to run on a single 48GB-class GPU (like the NVIDIA L20, 46GB). I noticed that d3-turbo has a dedicated official FP8 version (d3-turbo-FP8), but d3-ultra is currently only available in BF16. Are there plans to release an official FP8 (or other quantized) version for d3-ultra?
Standard vLLM compatibility vs. vllm-sr: During deployment, I realized that d3-turbo requires the prefix_bidir attention mode. As of now, standard upstream vLLM (even in newer 0.32.0/0.33.0 nightly versions) only accepts causal or noncausal_full_attention, meaning it fails to load these models. To support prefix_bidir, we currently have to use the vllm-sr native engine. Is there a roadmap to merge prefix_bidir support into the main vLLM repository so we don't need the heavy vllm-sr Docker runtime?
Hi, thanks for the kind words and the detailed write-up!
FP8 / quantized d3-ultra: This is on our radar. We’re seeing room for further improvement on d3-ultra, so the timing of a quantized release (FP8 or otherwise) will depend on when that updated version lands.
prefix_bidir support in upstream vLLM: We’re actively planning and working on this.
Please stay tuned, and we’ll update this thread as things progress. Thanks again for trying the models out!