Model evaluation · Reporting
StateBench: A long context can find an event and still get the state wrong
Ai2’s benchmark separates retrieval from tracking updates. Use a known-answer test to catch stale state before paying for a bigger context window.
A bigger window doesn't tell you which fact is current
Ai2 introduced StateBench on November 3 to test whether models can follow changes across long inputs. That matters if your API reads case histories, incident timelines or project logs. A response can quote a real event and still answer the wrong question: a case opened in Billing may now be closed in Support. Before you pay to send more history, test whether the model can distinguish those two facts.
The November 2 preprint varies the kinds of updates, from a single assignment through more dependent operations, and asks tested frontier models to generate the final state. In the authors’ November 3 summary, the four tested frontier models scored 13% or lower on the longest general finite problems. That is a result on the hardest category of a controlled benchmark, not an estimate of your support bot’s error rate. The framework also fixes the number of variables; it does not establish performance when new entities keep appearing.
Test the answer you actually need
Take the hypothetical case in the illustration. Retrieving Billing is correct if the question is where the case started. It is wrong if the question asks for the final queue. Make both questions part of the same fixture so a change that helps one and hurts the other is visible. Then add a dependent case: when two cases swap queues, their final queues depend on both previous assignments, not just the last sentence that mentions each case.
Write the expected state before the run and score fields against it. For a case lookup, that could mean an exact match on case ID, queue and status after normalization. Keep a separate count of responses that cite a superseded update. Otherwise a system that always returns a well-cited answer may look healthy while sending readers to the wrong team.
Three questions for the same case log
Scroll sideways to read all columns.
| Test | Question | Expected result |
|---|---|---|
| Retrieval | Which queue opened the case? | Billing |
| Current state | What are the final queue and status? | Support; closed |
| Dependent update | If cases A and B swap queues after the move, then A closes, where is A and what is its status? | If B began in Escalations and has not moved: Escalations; closed |
Vary length and placement independently
RULER treats tracing and retrieval as different tasks. Lost in the Middle shows why the position of relevant information also deserves a separate control. In your own evaluation, preserve the ordered events and change only the amount of irrelevant text, or move the intact history within the prompt. For the dependent-update test, keep a second fixture with the same length but a simpler update pattern. Don't mix a harder case, a longer prompt and a different model into one comparison.
Log the model ID, prompt format, input length, retries and the number of correct final states. Compare spend per correct result, not only the price of an input token. Use actual billed usage from an approved test; if no run produces a correct result, report that rather than dividing by zero.
Decide what to send on the next request
If the current queue and status already exist in an authoritative record, try passing those fields plus the relevant evidence alongside your full-log baseline. If the task is reconstructing an unknown state, use a verified event sequence and keep the final-state test. The useful decision is whether added history improves correct answers on the cases you care about. That is something you can measure without treating a context-window specification or a benchmark score as a purchase recommendation.
Sources & verification
Provider documentation supports the facts referenced below. Our explanations and recommendations are editorial interpretation.
- Ai2: StateBench release ↗Checked Nov 4, 2026
- Petty, Merrill and Sabharwal: A Fine-Grained Hierarchy and Benchmark for State Tracking in Language Models ↗Checked Nov 4, 2026
- Hsieh et al.: RULER ↗Checked Nov 4, 2026
- Liu et al.: Lost in the Middle ↗Checked Nov 4, 2026
Corrections: hello@aiapicost.com