kleos-decider-nano
A 150M decision model for agent memory. It is a fine-tune of tasksource/tasksource-decider-nano on memory-shaped pair judgements:
- does a new fact supersede an old one?
- is it a contradiction, a duplicate or compatible?
- do two mentions refer to the same entity?
- which relation holds between two statements?
It is the built-in decider of the kleos
database, where it backs decide(), the ::decider commands and the
supersession pipeline. The weights version is nano-tune-v1
(kleos_decider.json). kleos ships calibrated thresholds for this version.
Results
All numbers come from kleos's Rust inference (decide_eval). Thresholds and bias
were fitted on 60 adapt items only. The eval sets were never seen in training,
and 238 near-duplicates of them were removed from the training data. AUROC is
reported with 95% bootstrap CIs.
| base decider-nano | kleos-decider-nano | |
|---|---|---|
| Supersedes AUROC (66 eval items) | 0.938 [0.83, 1.0] | 0.991 [0.97, 1.0] |
| Supersedes auto-band coverage / false autos (α = 2%) | 0% / 0 | 36% / 0 |
| Contradicts AUROC | 0.944 | 0.991 |
| Duplicate AUROC | 0.997 | 0.994 |
| Same-entity AUROC | 0.993 | 1.000 |
| Relation, 3-way accuracy (bias-corrected) | 0.905 | 0.905 |
| VitaminC-300 (real Wikipedia revisions) | 0.679 [0.61, 0.74] | 0.805 [0.76, 0.85] |
| LongMemEval knowledge-update (held out) | 0.386 [0.27, 0.51] | 0.824 [0.71, 0.92] |
| MultiNLI dev, 3-way accuracy (forgetting check) | 0.805 | 0.817 |
Precision and memory (Apple M-series; kleos-infer, candle):
| b1 p50 | Footprint after a batch of 16 | |
|---|---|---|
| Metal q8 (kleos default) | 24.5 ms | 295 MB |
| Metal f32 | 28.0 ms | 701 MB |
| CPU q8 | 79–89 ms | 305 MB |
q8 vs f32: 1–2 of 922 decisions flip.
Usage
In kleos
kleos uses this model by default:
::decider enable nano {model: 'decider-nano'}
?[p] := p = decide_supersedes('nano', 'Alice lives in Berlin', 'Alice moved to Lisbon in May')
In Python
Use the bundled decider.py, which is unchanged from the base model:
from decider import Decider
m = Decider.from_pretrained("Mazino0/kleos-decider-nano")
Prompt format
The model was trained in the format kleos's templates render. Each option is
read as (state, "Question: <instructions>\nOption: <option>"). Options are
always descriptive choices, never bare yes/no.
Training
| Steps | 1 epoch, 809 steps of ≤ 2048 tokens |
| Optimiser | AdamW at 4e-5, 6% warm-up, linear decay, weight decay 0.01 |
| Loss | cross-entropy over the template's options |
| Frozen | token embeddings |
| Other | gradient checkpointing |
| Selection | learning rate chosen from {1e-5, 2e-5, 4e-5} by validation NLL on held-out training data, before evaluation |
| Weights | stored in bf16 (0 of 922 flips vs the f32 checkpoint) |
The training script and data builder are in kleos at
scripts/infer-bench/decide/train/. The run log is in kleos_tune.json.
About 19.5k template requests come from these sources:
| Source | Licence | Used for |
|---|---|---|
| VitaminC train (Schuster et al., 2021) | CC BY-SA 3.0 | claim/evidence and revision pairs |
| LongMemEval oracle (Wu et al., 2024) | MIT | knowledge-update pairs; 65 questions held out |
| PAWS labeled_final (Zhang et al., 2019) | attribution to Google | duplicate positives and hard negatives |
| MultiNLI (Williams et al., 2018) | OANC / CC BY-SA 3.0 / CC BY 3.0 | contradiction vs neutral |
Programmatic memory pairs (memgen.py) |
generated by kleos | 4-way update/contradiction/duplicate/compatible, entity typing, relation mapping |
Limitations
- The evaluation sets are small: 66 hand-written items and 90 LongMemEval pairs. The CIs are wide.
- With these weights the duplicate template ships with no auto band: the adapt-fitted threshold gave 3 false automatic duplicates on elaborations.
- The relation template did not improve over the base model.
- English only.
Licence
The fine-tuning changes are released under MIT (see LICENSE). The base model
and decider.py are Apache-2.0 (see LICENSE-APACHE-base). The training data
keeps its own licences, as listed above.
- Downloads last month
- 56
Model tree for Mazino0/kleos-decider-nano
Base model
jhu-clsp/ettin-encoder-150m