Invented-entity questions: Ripple-65B-Preview reasons until the budget runs out (M1 results)
Thanks for releasing the Preview and the base. We ran Ripple-65B-Preview through our M1 instrument and wanted to share what came out, since it may be useful while post-training is still in progress.
Setup:
Our own Q6_K, converted from your official BF16 (revision 30cb3b0e) with llama.cpp 67107ac (after #31692 and #31683).
Greedy, your ChatML template as shipped, 8192-token budget per answer, 4x Tesla P100.
M1 is 240 short questions built from Wikidata: 40 easy, 100 hard but real, and 100 about places, books and operas that do not exist. Every answer must end with "Exact Answer: ..." or "Exact Answer: UNKNOWN".
Results:
Easy: 35/40 correct.
Hard real: 60/100 correct. For reference on the same items: Qwen3.8-27B Q8_0 50, Nemotron 3.5 Super VL 86, Reflection Beam 1.0 (2-bit) 89.
Invented: 29/100 answered UNKNOWN, 27/100 gave a made-up answer, and 44/100 hit the 8192-token cap without writing an answer line.
The pattern we noticed: on real questions the model answers quickly (median about 175 tokens), but on invented ones it keeps trying to recall the nonexistent thing (median about 6,800 tokens). Over 2,000 tokens: 83 of the 100 invented items vs 7 of the 100 real ones. When it does finish an invented question, refusing and inventing are close to a coin flip (29 vs 27). That last part is our own after-the-fact analysis, not something we planned to test.
So a "this thing does not exist / I don't know" signal in post-training might reduce both the made-up answers and the long loops. Happy to rerun the same instrument on a later checkpoint and compare.
Full write-up, method and the raw per-item rows: https://github.com/apollo-mg/Project-Apollo/blob/main/data/receipts/ripple/RESULT_RIPPLE.md
Caveats: one quant (Q6_K), greedy decoding, one run per item, and M1 is our own instrument.
