RTX 5090 (32GB), llama.cpp: 5×64K and 2×128K with official NVFP4
I tested the official Flint NVFP4 GGUF on a single RTX 5090, mainly to see how much concurrent long-context serving fits in 32GB.
Five independent 64K slots fit with F16 KV, reaching 30.75 GiB of GPU memory. Two 128K slots also worked, at 28.80 GiB. The main limitation I found was prompt-processing latency: with five concurrent ~61K-token inputs, time to first token reached about 60 seconds at P95.
These measurements add a 32GB-card data point to the 3090 results and memory discussion.
Setup
- One RTX 5090; NVIDIA driver 595.99.02.
- Official RecursiveSI/flint-nvfp4-gguf weights.
- llama.cpp v0.8.0 / b11960, commit
8642541b2; no custom kernel patches. - Full GPU offload, F16 K/V, Flash Attention, default compact sliding-window cache.
- Independent context slots, without unified KV. K = 1,024 tokens; each slot's context includes input, reasoning and answer.
| Context per slot × slots | Sampled peak GPU memory | Result |
|---|---|---|
| 64K × 5 | 30.75 GiB | Concurrent long-input tests completed |
| 128K × 1 | 25.22 GiB | Focused long-context tests completed |
| 128K × 2 | 28.80 GiB | Both slots processed together; 6/6 controlled retrieval/code requests passed |
| 64K × 10 | — | OOM during allocation |
| 128K × 5 | — | OOM during allocation |
The 64K × 5 command:
llama-server -m flint-nvfp4.gguf --alias flint \
--device CUDA0 -ngl all -c 327680 -np 5 \
-fa on -ctk f16 -ctv f16 \
--no-kv-unified --fit off \
-b 2048 -ub 512 -cb \
--cache-ram 512 --ctx-checkpoints 8 \
--jinja --temp 1 --top-p 0.95 --top-k 20 --min-p 0 \
--host 127.0.0.1 --port 18080
For 2×128K, change the context/slot arguments to -c 262144 -np 2.
Five-request performance
Five requests were sent together per wave, using cold prompts and 512 generated tokens per request, including reasoning.
| Actual input tokens/request | Requests | Aggregate output tok/s | TTFT P95 | Completion P95 |
|---|---|---|---|---|
| 3,863 | 15 | 305.1 | 4.11 s | 8.60 s |
| 32,536 | 10 | 75.2 | 28.52 s | 34.29 s |
| 61,208 | 10 | 38.5 | 59.91 s | 66.73 s |
Aggregate throughput includes prompt processing, waiting and generation. EOS was ignored only for these fixed-output performance tests; quality tests used normal EOS. The P95 figures are small-sample observations, and GPU memory was sampled once per second.
A separate mixed-input run completed 85 requests in about 10.6 minutes with zero runtime errors. Cancellation and a follow-up request also worked.
Tool calling still needs attention in this setup: I observed incorrect string escaping and replies in ordinary content instead of an API tool call. A short-input control also failed, so I cannot attribute this to long context. These are controlled serving tests, not evidence of unchanged quality versus BF16. ARM64 Blackwell/Spark remains untested.