Arete-27B in CMF: a turbo file for long prompts on 16 GB cards
Hi Nous Research team, thanks for Arete-27B!
We added it in CMF, the single-file format of cortiq, a Rust inference engine with no Python and no ML framework: https://huggingface.co/infosave/Arete-27B-Turbo-cmf
It is a 4-bit (q4tp) file, 14.1 GB, with a turbo header for long prompts: past 8192 tokens the far field holds device memory in place, while the 248 rows that matter most (pass keys, codes, names) stay exact. Prompts under 8192 tokens run exact attention.
On an NVIDIA A40 (Vulkan), after a 64 000-token prompt:
| standard 4-bit | turbo | |
|---|---|---|
| peak VRAM | 21.5 GB | 14.8 GB |
| first token | 612 s | 443 s |
| decode | 16.2 tok/s | 27.2 tok/s |
With a 16 GB card's budget it stays at 14.3 GB. A fact planted in 100 000 characters came back 5 of 5 times.
cargo install cortiq-cli
hf download infosave/Arete-27B-Turbo-cmf arete-27b-turbo.cmf --local-dir .
cortiq run arete-27b-turbo.cmf --prompt-file report.txt --no-think
The file holds the language model only. The weights stay under Apache 2.0.
Feedback welcome!
Glad you called out the 16 GB budget separately. The A40 numbers are useful, but I'd read that capped run differently from a test on an actual 16 GB card.
Fair point, a capped A40 is not the same as a real 16 GB card: the driver keeps its own share, and the bus and clocks are different.
Arete itself hasn't been on one yet, but its twin has. OrcaCyber-Nano-1.0 is the same Qwen3.8-27B architecture in the same 4-bit q4tp format with the same turbo header and the same exact-row weights, and it was measured on an actual RTX 2000 Ada 16 GB (70 W) over Vulkan:
| prompt, tokens | peak VRAM, standard | peak VRAM, turbo |
|---|---|---|
| 1 000 | 14.1 GB | 14.1 GB |
| 4 000 | 14.5 GB | 14.5 GB |
| 16 000 | out of memory | 14.8 GB |
| 64 000 | out of memory | 14.8 GB |
So on the real card the plain file stops fitting at 16k, and the turbo one holds at 14.8 GB up to 64k. That's a bit above the 14.3 GB of the capped Arete run, and part of the gap is OrcaCyber's MTP head (its file is 0.26 GB larger). On a 16 GB card the header reads the first 4096 rows exactly instead of 8192 to leave headroom. The memory has to go somewhere though: host RAM grows while a long prompt is read (13.3 GB at 64k), or --o1-spill sends it to an SSD. Facts planted in 100 000 characters came back 6 of 6.
One difference to expect for Arete: it has no MTP head, so no draft tokens. Without drafts that card decodes these 27B files at about 13.5 tok/s; the 26 tok/s in the OrcaCyber table is with its MTP drafts on.