Introducing Umbra: the benchmark for cognitive simulation.

9 tasks · 5 of seven planes · 689,061 held-out records · public data only

Umbra¹ measures whether a simulated person behaves like a real one: what they choose, how long they take, how their mood moves, whether they stay the same person from one task to the next, and whether many of them add up to a real population. Survey benchmarks check the answers people give. Umbra checks what they do. Every task is scored in one unit, G², the share of the distance between guessing the population average and the best prediction people allow that a system closes.

We built it to be told when we are wrong, and that includes our own model. Myrias is scored here under the same rules as everything else, against hypotheses we wrote down before we saw a single score.

The bar is high. A free statistical model beats an off-the-shelf language model on all nine tasks, and given a person’s own history it matches or beats the language model given that history. Myrias clears that free model on five of the eight tasks it runs and falls behind on none. On the survey plane, Thesia beats Artificial Societies’ own figures on six of seven sources of their benchmark.

Seven questions a simulation has to answer

Umbra sorts its tests into seven planes. Each one asks a question a survey cannot answer on its own, and the last one contains the survey benchmarks unchanged, so anything scored on those can be scored here.

One number for every task

A task’s ceiling is how well people can be predicted at all: a person’s own answer on a retest, or the noise in an item’s outcomes when everyone sees the same item. G is 0 at the population average and 1 at that ceiling. Tasks with no defensible ceiling report their differences and no G, and say why.

On risky choice over problems it has never seen, a free model that reads only the problem already closes most of the gap, so that task leaves little room for anyone. On one person across surveys, predicting a person’s later answers from their earlier ones, the same model closes almost none of it. That is where a model of the individual has to earn its place.

Asked directly, a language model loses to a free model

An off-the-shelf language model gets each task’s information as text and returns probabilities, or a value with its uncertainty. Every arm is scored on the same 254 to 600 rows per task; the language-model arms run on that sample rather than on whole splits.

Lower is better for log loss and average error, higher for accuracy. On one person across surveys the language model reads each person’s full earlier survey and still predicts their later answers less accurately than the population average, 52.0% against 54.6%. On risky choice it picks the more likely option more often than the average does (60.2% against 55.0%), but its stated probabilities are miscalibrated, so its log loss is no better. It does not know each question’s base rate, and it guesses base rates badly.

History carries the gain

Give the language model a person’s last eight records and it improves sharply. Give the same eight records to the free model and it does as well or better. The difference below is the free model minus the language model, both given the history; below zero means the free model is ahead.

The free model sees only the earlier outcomes. The language model also sees each earlier record’s details, so the comparison favours it, and it still does not win. On mood at rest a handful of the language model’s answers state far too little uncertainty, which is why its log-loss interval is so wide; the average error tells the cleaner story there.

The survey plane, run unchanged

Umbra’s seventh plane is the Artificial Societies Benchmark, run with its own scorers on its sealed benchmark split. Thesia was scored there once, pre-registered, as a black box through our API. The measure is the distance between the simulated and the real answer distribution; lower is better.

Thesia is ahead of Artificial Societies’ own figures on six of seven sources and on the mean, and behind on the Albertson wording experiment. As everywhere else in Umbra, a model learned from real people carries the result: the prior alone scores 0.059. Artificial Societies’ numbers are their own and do not appear in the benchmark’s public scorecard. Method, ablations and the run kit are on the AS-Bench note.

Where Myrias stands

Myrias is scored against the strongest free arm on each task: the free model with the person’s history where a task has one, the free model alone where it does not. Myrias reads what each task allows a system to read: the person’s whole earlier record rather than the last eight entries, what they sampled in full, and Thymos’s modelled state. It runs on eight of the nine tasks; one person across surveys has no Myrias arm in this release. The Myrias scored here is the configuration we built for Umbra, fitted on Umbra’s training people only and held to the benchmark’s information rules; the model behind Thesia is configured separately.

Full sealed test split, the task’s own measure, Myrias minus the free arm; below zero is better for Myrias. Pre-registered before scoring.

Test S-4 asks whether modelled state predicts behaviour beyond what a person’s own history already predicts. We score it by running Myrias with and without Thymos’s state. The state adds measurably on momentary happiness, and adds no measurable gain on choice after sampling, dropping out of a course, response time, mood at rest and confidence. Most of Myrias’s lead comes from reading the person’s whole history well. The rest is the state.

A task clears a claim when Myrias beats the strongest free arm and every language-model arm on it, each interval below zero; a claim needs two tasks. Myrias clears three tasks in all. The rest of its wins over the free models are real, but on the shared sample of a few hundred rows they cannot yet be separated from the language model.

Where this release is narrow

One language model was tested, at one effort level, with one way of writing each task as text. A stronger model or better prompts may do better. Five tasks have no defensible ceiling, and the counterfactual plane is not scored yet. Myrias’s design predates Umbra, and the sealed test split, scored once, is the check on that. What Myrias is calibrated on is proprietary. For this benchmark it was fitted on Umbra’s training people only, and its configuration was fixed by a hash committed before scoring, which reviewers can audit under NDA. Rate limits cost 5.3% of the language-model calls on one person across surveys; those 32 rows are left out of every arm's score rather than counted as guesses, and were not re-sent, because re-sending would mean opening the sealed split a second time. Every number on this page comes from Umbra's sealed test split, opened once for this release: every arm, the same code as the validation run.

How these numbers stay honest

• Public datasets anyone can download. People never appear in both training and scoring, and every interval comes from a bootstrap over whole people.
• Every arm’s predictions are written to disk and hashed before a single outcome is opened.
• Our own model’s hypotheses were written down and hashed before it was scored. The ones it failed are reported with the ones it passed.
• No single Umbra score exists. Every task is reported, including the ones where we lose.