nvidia

/

Qwen3.8-Flash-Next-NVFP4

Like 453

Follow

NVIDIA 75.2k

PLE embedding dtype mismatch (FP8 weights vs bf16 allocation)

Problem

sglang has "auto-switch" logic: if the incoming weights are in fp8 while VocabParallelEmbedding was created in bf16, sglang tries to recreate it in fp8 on the fly.

However, when ple_offload_embedding=True is enabled (passed as a server argument), the embedding is wrapped in Qwen4ExpPinnedHostEmbedding - the table has already been allocated in pinned host memory as bf16, and swapping to fp8 would cause a fault in the Triton kernel. Therefore, sglang explicitly raises a ValueError and asks to specify the dtype via the config.

Fix

Add the following field to text_config in config.json:

{
  "text_config": {
    "ple_embedding_dtype": "float8_e4m3fn"
  }
}

After that, Qwen4ExpNGramEmbedding.init allocates the table directly in fp8, and the error goes away.