We ran Thesia on the Artificial Societies Benchmark. It scored 0.065.

In September, Artificial Societies published a benchmark for one question: when a model answers a survey as a particular person, does it answer the way real people did? The respondents come from public studies, the code is public, and nine language models are scored on it.

We ran Thesia, our flagship product, on the seven sources their chart reports, with the same respondents and questions as the benchmark’s published runs, scored with its code. We pre-registered the run and used the test respondents once.

Thesia came in closer to real answers than the figure Artificial Societies report for their own system, on six of the seven sources and on average.

Closer to real answers on six of the seven sources

The measure is total variation distance. Take the share of real people who picked each answer and the share of simulated people who picked it, then add up how far apart those shares are. Zero means identical. It is the measure on the Artificial Societies chart, computed with their code.

The Artificial Societies figures are self-reported and absent from the public scorecard, and their outputs are not published, so this compares against a claim we cannot check. Their page also counts 2,796 respondents; our run, on the benchmark’s published rosters, has 3,500 across the seven sources, so the two figures may not cover exactly the same people.

A demographic model beats every system on this benchmark

Fit a plain demographic model on the benchmark’s development data, or for GSS on the 2022 survey, with no language model and no simulation in it: for each question, how people with the same recorded age, sex, education, party and the like answered. It averages 0.059. That is closer to real answers than Thesia, than the figure Artificial Societies report, and than every language model on the scorecard. The benchmark’s own scorecard carries a baseline of the same kind, and on the two SAPA personality sources it scores 0.054 and 0.051, also ahead of Thesia.

That is the flaw. The headline score measures how answers spread across a population, and demographics settle that spread. A system that knows who the respondents are scores well whether or not it simulates anyone.

A test that a demographic model wins cannot tell you whether anyone simulated a person.

Where it falls short

The Albertson study is the clear miss: 0.098 against 0.025. The same study holds the measure Thesia does worst on. Jessee and Albertson asked one question two ways, once about spending on welfare and once about spending on assistance to the poor. Real respondents leaned 10.6 points more generous on the second wording. Thesia put the gap at 26.3 points, an error of 15.7 points, and the interval on that error does not reach zero (6.8 to 24.2). Artificial Societies report 1.4 points on this measure and the best language model 6.2 points. On the Jordan wording experiment the same error is 4.7 points.

Accuracy inside demographic subgroups is level with theirs, 0.061 and 0.058 against 0.06. On the spread of opinion Thesia is slightly further from the real figure, 0.010 against 0.006.

How the run was kept honest

• The configuration was frozen and pre-registered before a single answer was generated for the test respondents: the full system, the rosters, the metrics and the rule for what counts as on par. The file describes our configuration, so we keep it private and show it to reviewers under a confidentiality agreement. Its SHA-256 is 29d77fb924451214eb3dd3a85b1461ab354510d1487fe0c25fb9887227725de2, so anyone shown the file can confirm it is the one this page refers to. We recorded the hash before the run, in our own notes; it was not posted publicly at the time.
• The test respondents were used once. Everything earlier ran on data outside them.
• Respondents were chosen by the benchmark’s own roster code, the same people and counts as its published runs, and scored with its own metric code, unchanged.
• About 4–5% of calls returned no answer under 12 parallel workers; only those were re-sent with identical prompts. Unparseable answers stayed failures. Final coverage is 98.8% or higher on every source.

One correction before publication: our first GSS scorer counted real answers only from people Thesia answered for, where the benchmark counts everyone asked. Rescored the benchmark’s way with the same answers, the average moved from 0.064 to 0.065. The results file records both, with the score reports behind this page.

Answer-level outputs are tied to individual respondents in the source studies, and the benchmark’s authors leave them out of their release for that reason. We share ours with researchers on request.

A kit to rerun it yourself is coming

We are opening a run kit that reruns the benchmark split against Thesia through the Aetherya API. You download the source studies from their publishers, build the seven tasks and rosters with the benchmark’s own code, send each respondent to Thesia with an Aetherya API key, and score the answers with the benchmark’s own scorer. The kit prints your figures beside ours and beside the ones Artificial Societies report.

Thesia answers on our servers, in the configuration frozen for the sealed run. The language model’s replies vary from call to call, so a rerun lands near the published figures and seldom on them exactly. The kit will be posted on this page when the endpoint opens.

Research groups can check the findings before then, with us. We give your team API access and the kit, you run the kit on your own machines and score the answers with the benchmark’s code, and what you find is yours to report. Reviewers can also see the pre-registration file, under the same confidentiality agreement, and check it against the hash above.

What this benchmark leaves out

A population-spread score still catches a simulation that gets a population badly wrong. It says little about the decision one person makes with something they have never seen, how long they hesitate, or why they walk away, and that is the part a campaign team needs before the budget is committed.

We are building Umbra, our own benchmark, to measure that part.