GameDecider v0.1

Status: v0.1, Apache-2.0.

GameDecider is a small decision model for games. You give it a scene (any text or JSON: who is here, what was said, what the player did), one or more questions, and a set of options for each question. It returns a calibrated probability for every option in a single forward pass, with no text generation. When it isn't confident enough it abstains, so your game can fall back to its own rules or ask an LLM.

Typical questions: "Is the player threatening this NPC?", "Which of these characters is the line addressed to?", "Did the player accept the quest?", "How does the shopkeeper feel now: pleased / wary / angry / afraid?".

• Base: Ettin encoder 400M (MIT), revision 7662476d.
• Weights: a uniform weight soup of two fine-tuned seeds (rel5_ettin400_mixI_B_safe3_soup_enc).
• Files: model/model.onnx (fp16 weights, 792.5 MB, the default), model/model_fp32.onnx (fp32, 1.58 GB), model/tokenizer.json, model/calibration.json, model/thresholds.json (default profile), model/thresholds.strict.json, model/run.json and model/model.json (SHA-256 of every file).
• Adapters: 39 compiled LoRA adapters in adapters/ (cookbook), plus 2 SkyrimNet-style adapters (skyrimnet/memory_match, skyrimnet/npc_mood) and 1 character-reaction adapter (motion/reaction): 42 in total.
• Runtime: the gamedecider 0.1.0 Python package in runtime/ (Apache-2.0; wheel and source).
• Language: English only.

Quick start

With the gamedecider runtime (Python, local server)

pip install https://huggingface.co/holdoutlabs/gamedecider/resolve/main/runtime/gamedecider-0.1.0-py3-none-any.whl
gamedecider check-model ./model --adapters ./adapters
gamedecider serve --model-dir ./model --adapters ./adapters        # http://127.0.0.1:8040, POST /judge, GET /health
gamedecider judge request.json --model-dir ./model --adapters ./adapters   # one request, no server

POST http://127.0.0.1:8040/judge          (or save it as request.json for gamedecider judge)
{
  "state": "Bolt the shopkeeper is closing up. The player walks in, draws a dagger and says: 'Open the till.'",
  "questions": [
    {"name": "player_intent", "type": "choice",
     "instructions": "What does the player want from Bolt?",
     "options": [{"name": "trade", "desc": "wants to buy or sell"},
                 {"name": "rob", "desc": "wants to take money by force"},
                 {"name": "chat", "desc": "small talk"}]},
    {"name": "is_threat", "type": "noul", "instructions": "Is the player threatening Bolt?"}
  ]
}

Each answer carries the option probabilities, the chosen option, and abstain: true when the confidence is below the threshold for that question type. The runtime also runs a minor-safety guard by default (see Safety). The Python API is Decider(model_dir, device, adapters_dir=None, thresholds="default").decide(rows); see runtime/ (the package README) for the answer fields and options. The runtime makes no network connections.

With ONNX Runtime directly

Inputs are built in the packed "layout C" format the runtime implements (the input layout is documented in the package README in runtime/, section "Raw ONNX input layout", and implemented in gamedecider/encoding.py): the state, then each question with its options, in one sequence with segment masks. Outputs are per-option logits. Divide the logits by the temperature in calibration.json for the question's type and option-count bucket, apply softmax, and answer only if the top probability is at least the threshold for that bucket.

Question types: choice (2-32 options), score (2-10 ordered options) and noul (yes/no). State limit 3,072 tokens.

Raw-ONNX users do not get the runtime's guard or option-order averaging. Add both yourself if you need them.

Threshold profiles

Both profiles use the same calibrated probabilities (temperatures refit on the dev split). They differ only in when the model abstains.

choice with 2 and with 11+ options never abstains. Use strict when a wrong answer costs more than no answer (moderation, anything visible to other players). Use default for flavour and NPC reactions.

Evaluation

All numbers are for this model, scored raw (guard off). The CIs in brackets are 95 % paired bootstrap intervals for the difference to the previous internal release candidate (a3, dm232_..._clean_add_soup_enc), in points; heldout_v3 uses a task-family cluster bootstrap and realgames a 12-game cluster bootstrap. Absolute CIs are given where we computed them.

Every eval set below is private and is not released. They were written for this project and are kept out of training by id and n-gram guards. None of them is a public benchmark.

heldout_v3 measures transfer to question families the model never saw. Expect in-distribution numbers (dev) to overstate real use. The ONNX files match PyTorch: 0 label flips on 200 bench rows and on 100 realgames rows (234 questions); max probability difference .0045.

What "not sure" buys you on heldout_v3

The 0.600 / 0.596 above are argmax over all 2,993 questions, with no abstention. This split is for single order, using the shipped thresholds. It shows how accuracy differs between the questions the model answers and the ones it declines. The CIs are 95 % task-family cluster bootstrap intervals (100 families, 2,000 draws).

For comparison, the default profile on our dev pool answers 97.4 % of questions at 0.943, and on cookbook it answers 97.3 % at 0.819.

How to read this:

• On new question families, "not sure" tracks error: the declined questions are about 22 points less accurate than the answered ones.
• The thresholds were fitted on dev, and on new families they are overconfident. Coverage drops to 66 %, but about a third of the answered questions are still wrong.
• Read "answered" as "more likely right", not as a calibrated error rate. If errors are costly in your game, use strict, or fit your own thresholds on a few dozen labelled questions from it.

Planning

On our procedural "what should I do next to reach the goal?" eval (DM24.1, 432 questions), the base model without an adapter scores next-step 0.939 (95 % CI .900-.972; 180 questions), needed .905, can-do-now .988, step count .631 (all types .882). The previous release candidate scored .256 on next-step.

This is not general planning ability. The training mix contains 9,000 planning rows from the same generator family (different vocabulary, own seeds, 0 world overlap with the eval), so this eval is near-distribution. Expect much lower accuracy on planning questions shaped differently from these crafting/gathering worlds, and use an LLM or a real planner for anything that matters.

Safety: minors and romance

Earlier internal versions rated underage NPCs "receptive" to player flirting. This release was retrained with minor-safety rows and gated on the raw model, with the guard off:

The two remaining raw errors are an "eighth-grader" flagged romanceable: true and a numeric age after a comma with the flag true.

The runtime guard (on by default in the runtime's judge, server and Decider.decide) blocks romance / flirt / date answers when the state has age or minor cues (for example "teenage"); with it on, both probe sets give 0 / 48 positive minors and adults are unchanged. If you run the ONNX file directly, you don't get the guard. Then you must make sure your game never asks romance questions about characters who are or may be minors.

Latency

Measured on a desktop (AMD Ryzen 5 3600X, 6 cores / 12 threads; NVIDIA RTX 3060 Ti, ONNX Runtime DirectML 1.24.4) with the runtime's Decider.decide(), after warm-up, on 200 held-out rows (median 325 tokens, p90 370). Model time, p50 / p90:

• Several questions about the same state cost about the same as one: put them in one row.
• CPU time grows with input length (it is a 400M-parameter encoder: about 1 s per ~300-token scene on a 6-core desktop CPU; shorter scenes are faster). The fp32 file is no faster than the fp16-weight file on CPU.
• Throughput: rows run one after another (no cross-row batching yet), so expect about 10 checks per second on a mid-range GPU and 1-2 per second on CPU. DirectML pads every input to 512 tokens, so GPU time does not depend on length.
• Use a GPU for anything interactive; on CPU, keep scenes short and ask several questions per row.

Adapters

The base model answers any question zero-shot. For questions you ask often, a compiled adapter (LoRA, rank 32, about 29 MB each, trained on a few hundred open-model-written examples per question) usually gives a large accuracy gain. Adapters never abstain. The runtime routes a question to its adapter by the question's exact wording.

Six cookbook adapters were left out because they lost 4 or more rows (about 7 points) against the previous base on their own eval: who_addressed, real_harm_request, encouragement, needs_clarification, objective_reported and reply_fit. The base model still answers these questions zero-shot.

Limits

• Option order matters. The model has a position bias. On realgames, 11.8 % of choice answers changed when the options were shuffled (measured on an earlier version); the runtime's two-order averaging (on by default) halves that, at about twice the latency for choice questions. Raw-ONNX users must average orders themselves.
• Abstention. With the default thresholds the model answers 97 % of dev and cookbook questions but only 66 % of heldout_v3 questions; on new question families, expect it to say "not sure" often. Strict answers 82 % of cookbook questions. On heldout_v3 the answered questions are still wrong about a third of the time (0.672 accuracy; see "What "not sure" buys you" under Evaluation).
• English only. Non-English lines work sometimes and are not evaluated.
• Simulation states. Earlier internal versions were trained on Open Sims simulator states; this release removes them, and accuracy on that dev pool dropped (0.785 for the previous candidate, 0.735 now; phone-call questions .705 → .638).
• Planning results come from in-family training rows (see Planning).
• Small eval sets outside heldout_v3, realgames and cookbook; the CIs are wide. continue-narration pairs (n = 28) fell from 5 to 2 correct.
• Calibration is fitted on our own dev pool. On your own questions, measure coverage and error before you trust a threshold.
• Long states: limit 3,072 tokens; very long or table-heavy states are a known weak spot.

Intended use

• Game logic and NPC behaviour: intent, addressee, mood, consent, quest progress, moderation of in-game text for your own game, choosing among authored reactions.
• As a fast, local filter in front of an LLM (answer the easy cases, abstain on the rest).

Out of scope

• Decisions about real people: hiring, credit, insurance, housing, medical, legal or immigration decisions, and moderation or enforcement against real users' accounts. The model was built and evaluated for fictional game situations only.
• Safety-critical use, surveillance or profiling.
• Sexual or romantic content involving minors, in any form. Don't rely on the raw model to refuse it.
• Anything that needs factual knowledge or general multi-step reasoning.
• Generating text (it doesn't).

Training data

333,760 rows (1,181,159 questions; a row can carry several questions). About half the rows are game scenes written by open-weight models (GLM-5.3-Flash, DeepSeek-V4.1-Flash) and labelled by two open-weight labellers that had to agree (DeepSeek-V4.1-Flash, gpt-oss-120b). The rest are recasts of public human-written datasets (CC BY 4.0, CC0, MIT, Apache-2.0, ODC-By), public-domain books and plays, and procedural rows (minor-safety and planning). Every source, its row count, licence, where the licence was checked and the attribution it needs are in DATA_SOURCES.md (machine-readable: data_sources.json); the credits are in NOTICE.md. The training data is not released. samples/ holds about 200 example rows that show the format.

Provenance note: some task templates, world seeds and question specs were drafted with an AI coding assistant (Claude) and were kept in the training data by the owner's decision.

Training recipe

• Two seeds (1 and 2) fine-tuned from Ettin-400M, then averaged (uniform float64 mean of all floating tensors; the output head copied from seed 1).
• Packed layout C (state + all questions in one sequence, segment masks), max length 3,072, token budget 4,096 per step, 2 epochs (120,402 steps per seed), AdamW lr 2e-5 (head 5e-4), weight decay 0.01, 5 % warm-up, grad clip 1.0, bf16.
• A 6-config lr × epochs sweep (DM24.0) kept 2e-5 / 2 epochs: no config beat it by the +1 pt dev bar, and 1 epoch was 1.9 pt worse on dev.
• Temperature scaling per question type and option bucket, fitted on the dev split, then the thresholds above.
• ONNX export (fp32, then fp16 weights); 0 label flips against PyTorch on 100 bench rows for both files.

Compute

Rented NVIDIA RTX 5090 GPUs: seed 1 trained in 13,665 s (about 3.8 h, 428 M tokens seen); two seeds are about 7.6 GPU-hours in total. Evaluation and calibration ran on a laptop RTX 3080 and a desktop RTX 3060 Ti.

Citation

@misc{gamedecider2026,
  title  = {GameDecider v0.1: a calibrated decision model for games},
  author = {Holdout Labs},
  year   = {2026},
  url    = {https://huggingface.co/holdoutlabs/gamedecider}
}