HyperArete: calibrated W4A16 checkpoint and HyperQwen recipe for a 24 GB GPU
We published an independently prepared quantization of Arete-27B, using HyperQwen's preparation and batch-serving optimizations:
- Weights + recipe: https://huggingface.co/surak/HyperArete-27B-W4A16
- Reproduction repository: https://github.com/surak/HyperArete
- Recipe offered upstream to HyperQwen: https://github.com/syv-ai/HyperQwen/pull/446
The language-model body is calibrated W4A16 (symmetric, group 128); embeddings and LM head are INT8. All 333 vision tensors were verified unchanged. The five weight shards total about 16.1 GB. This uses Arete's own weights, with no additional alignment training or merge with another Qwen checkpoint. Our export does not add an MTP head. Credit goes to Nous Research for Arete, Qwen for the base model, syv-ai/HyperQwen for the preparation/runtime optimizations, and Intel AutoRound for calibrated quantization.
We validated a single RTX 3090 24 GB, 150,000-token total-context batch profile with image support, FP8 KV cache, FP16 recurrent state, and HyperQwen's full-MLP INT8 activation path. Two independent workers passed text, image comprehension, JSON, tools, streaming, token-usage arithmetic, a real 149,744-token retrieval test, and Gumbel watermark detection. Source revisions and environment versions are pinned in the recipe.
Each worker measured about 47.6 output tokens/sec for one short request, 342 aggregate at eight concurrent requests, and 1,257–1,270 aggregate at 64. Across two simultaneous workers, a 128-request burst produced 16,384 output tokens in 6.301 seconds (~2,600 aggregate tokens/sec). These are short synthetic photosynthesis prompts with 128 output tokens per request, temperature 0.7, thinking disabled, ignore_eos=True, Gumbel enabled, and prefix caching enabled. Repeated prefixes can benefit from caching. These figures describe short bursts, not sustained production capacity or long-prompt throughput.
No broad quality evaluation against BF16 Arete has been completed. Functional checks do not establish preservation of Nous Research's accuracy or alignment results. Weight quantization, INT8 activations, FP8 KV and FP16 recurrent state may affect quality. The original source weights remain unchanged, and the release retains licenses and attribution.
Sharing this as a reproducible deployment option for people with 24 GB cards. Feedback and independent evaluations are welcome.