peekaboolean-450m (v11, release 0.4.0)

A LoRA adapter and yes/no head for LFM2.5-VL-450M (revision fc6221ca) that answers typed questions about an image by scoring caller-supplied options:

• choice: pick one of your named options → distribution over them
• score: place the image on your ordinal rubric (any length) → distribution and expected level
• noul: yes/no → P(yes)

It never generates text. Every question also gets a trained "can't tell" candidate (new in 0.4.0), and its probabilities are temperature-calibrated per question type, option count and image size (256/384/512 px). The request format is TypeSafe's Jev with an image, and the local server also answers POST /v1/systemone. A six-question request with 28 options takes 247 ms p50 on an M1 Max (MPS, fp16, torch 2.14) and 75 ms p95 on an RTX PRO 6000 (bf16).

Code, request format and training pipeline: https://github.com/bykof/peekaboolean. Research write-up: REPORT.md and newer-models-and-jev.md.

Usage

git clone https://github.com/bykof/peekaboolean && cd peekaboolean && uv sync
curl -L https://github.com/bykof/peekaboolean/releases/download/v0.4.0/peekaboolean-450m.tar.gz | tar xz
uv run python -m peekaboolean.serve --adapter peekaboolean-450m \
  --image photo.jpg --request requests/general.json --max-edge 512

--adapter bykof/peekaboolean-450m loads this repository from the Hub instead of the tarball.

Jev-style HTTP (the local UI server, uv run python -m peekaboolean.ui --adapter peekaboolean-450m):

curl -s http://127.0.0.1:8765/v1/systemone -H 'Content-Type: application/json' \
  -d '{"state": "...", "questions": {...}, "images": ["data:image/jpeg;base64,..."]}'

Can't tell

scorer_config.json has "unknown": true, so every answer carries two more fields:

• unknown_probability: the mass on "It cannot be determined from the image and the given information."
• abstained: true when that mass is larger than every option's.

A choice's or score's probabilities are the distribution given that the image answers the question, as imajev reports them. A noul's noul is P(yes) + unknown_probability / 2, Jev's convention. Treat abstained as "look at this one yourself".

Files

Training

1. Typed questions from The Cauldron training partitions (VQAv2, CLEVR, TextVQA, DocVQA, ChartQA, Screen2Words, AI2D), with same-form distractors.
2. 198,814 questions on 61,761 images from two local teachers: Qwen3.6-35B-A3B wrote and labelled them, Qwen3-VL-30B-A3B labelled them again, and the mean of both distributions is the target.
3. FairFace age, gender and child/adult questions.
4. New in 0.4.0: 72,735 questions on 24,063 of the same images from the same two teachers. Every labelling view offered a can't-tell option, and its mass became the target of the can't-tell candidate. Some states are records or rules the image has to be checked against. About one request in three got a question about a fact the image hides.

v0.3.0 (v10b) is v0.2.0's recipe on LFM2.5-VL-450M (1 to 3). v11 was fine-tuned from it on 1 to 4 (each new question twice): LoRA r = 32, lr 3e-5, half an epoch, --unknown. Early stopping on validation kept step 5,000.

Evaluation (held-out test split of 0.4.0's mixture, 512 px, 31,000 questions)

Both models are scored on the same rows. The test split is v0.2.0's plus the new teacher rows.

Public groups move by less than about one standard error. Teacher groups measure agreement with the teachers, not ground truth.

Can't tell, on the teacher's test questions:

ImajevBench v2.0-lite, dev and calibration splits (254 items with gold), run through its own harness against /v1/systemone: 118/254 (46.5%), against 110 for v0.3.0. By track: visual 65/103, joint (photo plus record or rule) 47/113, text only 6/38. Its 24 Unknown items stay at 0. They are mostly rules that the photo and the record cannot decide, while the teacher's can't-tell questions ask for facts the image hides. All 11 abstentions were text-only items, which the server answers against a blank grey image.

Intended use and limitations

For research and non-commercial prototypes that need fast, structured decisions about images: triage, tagging, routing, filtering.

• Not for decisions about individual people. Age, gender and child/adult estimates reflect FairFace's biases, are often wrong for individuals, and are also produced for drawings and cartoon characters.
• "Can't tell" fires for facts the image hides (text too small or cut off, a hidden side). It does not fire when a rule in the state cannot be decided from the photo and the record.
• A request without an image is answered against a blank grey image, and there v0.4.0 abstains where it should answer.
• Options are scored independently, so contrast questions ("the larger one") are out of scope.
• Rules and records in the state are read, but applying them to the photo is weak (ImajevBench joint track 42% on dev and calibration).
• Photo aesthetics were not learned.

Licence

The adapter is released under CC BY-NC 4.0, for the training-data reasons below. The base model is under the LFM Open License v1.0, which conditions commercial use on staying under its revenue threshold; using this adapter means using that base model under its own terms. Both teachers (Qwen3.6, Qwen3-VL) are Apache-2.0.

This summary is not legal advice. Check the upstream terms before any commercial use.