What LoCoMo actually measures
LoCoMo is an academic benchmark that tests whether a system can answer questions about a long chat conversation it has already been shown. The public release is ten conversations between two fictional people, with 1,986 annotated question-answer pairs across them. A “LoCoMo score” is the share of those questions a system answered acceptably — where acceptably is decided by a scoring method the person running the evaluation chooses, and where the set of questions asked is also a choice. It is a real benchmark, built carefully, and it measures something narrower than the word “memory” suggests.
Every agent-memory vendor cites a number on it. This piece is about the instrument, not the scoreboard.
Where the conversations come from
The dataset is generated, not recorded. Two LLM agents — gpt-3.5-turbo in the paper’s pipeline —
are each given a persona expanded from the MSC dataset and a temporal event graph: up to 25
causally linked life events spread over 6 to 12 months, produced by conditioning text-davinci-003
on the persona. The agents then talk to each other across many dated sessions, sharing and reacting
to images, with each response conditioned on retrieved history.
Humans come in at the end, as editors rather than authors. The paper reports that annotators “edited nearly 15% of the dialog turns and removed or substituted approx. 19% images present in the LLM-generated dataset,” fixing long-range inconsistencies and checking that the dialogue matches the event graph.
The authors are direct about what that leaves them with, in their own limitations section: “Our dataset is sourced primarily from text generated by LLMs… we acknowledge that this dataset may not fully reflect the nuances of real-world online conversations.” So the substrate under every LoCoMo number is synthetic dialogue with a human editing pass, in English, between exactly two speakers, about personal life.
How long the conversations are, and which ten you get
The paper’s abstract says 300 turns and 9K tokens on average over up to 35 sessions. That describes the 50 conversations of the original 2024 release. What you can download today is a subset, and the repository says why: “We sampled a subset of the data to retain the longest conversations with high-quality annotations and for cost-effective evaluation of closed-source LLMs.”
Because the released ten are deliberately the longest, the headline averages understate them. I counted the file rather than trusting either number:
Measured over data/locomo10.json |
Value |
|---|---|
| Conversations | 10 |
| Sessions per conversation | 19 to 32 |
| Turns per conversation | 369 to 689 (mean 588) |
| Words of dialogue text per conversation | 8,019 to 16,165 (mean 13,377) |
| Annotated question-answer pairs | 1,986 |
The five question categories
The QA task is where nearly all published scores come from. The paper defines five reasoning categories, and the released file labels each question with a number from 1 to 5.
| Category | Questions in the release | What it asks for |
|---|---|---|
| Single-hop | 841 (42.3%) | A fact stated in one session |
| Adversarial | 446 (22.5%) | Nothing answerable — the system should decline |
| Temporal | 321 (16.2%) | When something happened, from time cues in the dialogue |
| Multi-hop | 282 (14.2%) | A fact assembled from several different sessions |
| Open-domain | 96 (4.8%) | An inference combining what was said with world knowledge |
The counts are mine, from the released file. The category names need a caveat: the file stores
integers, and the paper lists five categories without publishing the numbering. Adversarial is
certain — those entries carry an adversarial_answer field and no answer. The other four I
inferred from the questions themselves and from how many dialogue turns each cites as evidence: the
multi-hop group averages 3.13 evidence turns and 98% of it cites more than one, while the single-hop
group averages 1.07 and 5% cites more than one. That is strong, but it is an inference, and I have
labelled it as one.
Two structural facts follow from the table and matter more than any score. Forty-two per cent of the set is a single-session lookup, which is the easiest thing a retrieval system does. And more than a fifth of the set has no correct answer at all — the intended behaviour is refusal.
How answers are scored
The paper’s own metric is token-overlap F1 against a short gold answer. The repository’s evaluation code confirms the details: both the prediction and the gold answer are lowercased, stripped of punctuation and of the words a, an, the and and, split into words, Porter-stemmed, and compared as bags of tokens; the F1 of that overlap is the question’s score. The authors chose it knowingly, writing that they “ensure that answers in our QA annotations are directly taken from the conversations as much as possible” precisely so that partial string matching would work.
That metric gives partial credit for overlapping words and no credit for a correct answer phrased differently. Most vendors therefore do not use it. mem0’s paper argues the case plainly — that a system answering “July” where the truth is “March” still scores well on lexical overlap — and substitutes LLM-as-a-Judge: a second, stronger model reads the question, the gold answer and the response and rates it. Different instrument, different number, same benchmark name.
The same paper also drops a whole category: “The dataset originally included an adversarial question category… However, this category was excluded from our evaluation because ground truth answers were unavailable.” That is a defensible call, stated openly. It also removes 446 of 1,986 questions — the second-largest group in the set, and the one measuring whether a system knows when to say nothing.
The reference points in the paper
Two numbers from the paper are worth carrying, both from its Table 2, both measured with the F1
metric above on 2024-era models: a human benchmark of 87.9 overall, and a best model score of
32.1 for GPT-4-turbo at a 4,096-token context. Long-context models scored higher overall and
collapsed on the adversarial category — gpt-3.5-turbo-16k at 16K context reaches 37.8 overall and
2.1 on adversarial, against 70.2 for GPT-4-turbo at 4K. Given a long enough transcript, those
models answered the unanswerable almost every time.
What a score on it does not tell you
- Not your data. Every conversation is two friends discussing daily life. No documents, no tickets, no code, no tools, no third participant, no organisation.
- Not English-plus. The generation pipeline is English-only, which the paper states.
- Not real conversation. Synthetic dialogue, human-edited — good enough for comparison between systems, not a sample of how people actually talk to agents.
- Not fact updating. The questions ask what was said, not what is still true. A system that stores a superseded fact forever can score well.
- Not cost or latency. The paper’s QA metric is accuracy. Latency and token figures beside a LoCoMo score come from the vendor’s own harness, not from the benchmark.
- Not the same measurement twice. Two published scores may differ in subset, categories, metric, judge model and number of runs before any product difference is involved.
What to ask when a vendor shows you one
Five questions, all answerable from a vendor’s published method, and all cheap to ask: which release of the dataset, which categories, which metric, which judge model if any, and how many runs. If the answer is a single number with none of that attached, it is a claim rather than a measurement. The claim-versus-measurement test is not specific to memory benchmarks: CM-001, Do 95% of AI projects actually fail?, runs the same three questions — who was measured, how many of them, and with what instrument — over five of the most-repeated statistics in enterprise AI.
Scores on this benchmark are also actively disputed between vendors, with each side alleging the other misconfigured the thing being measured. That argument, and what it implies about published numbers, is in CM-008, What the agent-memory vendors actually differ on. For the underlying question of what memory is doing in an agent at all — CM-005, What agent memory actually is.
None of this makes LoCoMo a bad benchmark. It makes it a benchmark: a fixed instrument, aimed at one narrow task, whose numbers mean exactly as much as the setup that produced them and no more.