Open research · governed runtime · evidence laboratory
Memory that
can be questioned.
An auditable memory runtime for AI agents, built around governed recall, correction, currentness and forgetting. Its claims live alongside the evidence that can challenge them.
Delivery evidence / October 21, 2026
What the latest runs actually show
Accepted same-harness scores, experimental improvements and rejected regressions are different evidence classes. This checkpoint links to each source; no candidate run silently updates the accepted leaderboard.
Accepted AMB retrieval baseline (v6): Agent Memory 15/77 strict-success cases (mean precision 18.06%, mean recall 95.35%); Mem0 OSS 10/77 and BM25 8/77 on the same frozen retrieval-only protocol. The historical leaderboard below remains bound to its accepted snapshot. Accepted dashboard source ↗
October 20 selective-context experiment: 15/77 → 22/77. An opt-in lexical_content consumer package improved known-case AMB strict success by seven cases, with mean precision 18.06% → 32.43% and mean recall unchanged at 95.35%. This is a candidate diagnostic, not an accepted AMB lane: the full locked AMB CLI was not run and the 77 cases were already known. Raw paired evidence and limitations ↗
Why this mutation was not promoted: on the full external LongMemEval-S check, session complete recall@50 fell 419/419 → 418/419 and turn complete recall@50 fell 360/419 → 358/419. It lost required annotated evidence, and no-answer cases still received unrelated context. CHANGES REQUIRED, not a new default. Full 500-question-per-plane evaluation ↗
October 21 repaired v7 candidate, complete matrix: all 9/9 frozen benchmark rows are imported and re-verified against protected runtime tree f9b5ff9e and declaration 7de94f11 in draft PR #836. AMB is unchanged from v6: 15/77 Agent Memory, 10/77 Mem0 OSS, 8/77 BM25 (77/77 case equality in each row). LongMemEval-S includes six full 500-question session/turn rows for Agent Memory, lexical overlap and Mem0 OSS: all native metrics and required-evidence totals match v6; Mem0 turn differs in the ordering of 47/500 ranked lists, with no gold item moved and no metric change, so ranked-list equality is 453/500, not 500/500. This is source-qualified candidate parity, not an AMB improvement, a new external competitor, or an accepted v7 baseline. All nine-row results and comparator caveat ↗ · Independent input/import verification record ↗
Competitor coverage: the accepted same-harness external memory-product count is still one (Mem0 OSS). Hindsight and Letta have experimental runs that do not count as accepted products; Cognee, Graphiti, OpenViking, LangMem, Supermemory, TencentDB Agent Memory and the newly registered MemOS, EverOS, MIRIX and Acontext remain visible for qualification, never scored as zero. Browse accepted peer coverage, the expanded source registry and the live candidate execution feed.
Evidence checkpoint: October 21, 2026, draft #836 at 3df781c2. The accepted numeric projection remains pinned to October 7; candidate parity is displayed separately. Formal v7 release acceptance and the public API contract decision remain outstanding. Recheck linked run records for later evidence.
01 / Comparative evidence
Benchmark leaderboard
Real measured outcomes, including runs in which Agent Memory trails competitors. Every ranking is specific to its benchmark, population, native metric, and evidence status. Candidate results are visible, not silently accepted.
Where Agent Memory actually ranks
Current measured results first. Switch to accepted-only evidence, or inspect every scored benchmark profile. Unreviewed measurements remain explicitly labeled.
| Rank | Memory system | Role | Measured result | Scored cases | Gap to leading system | Evidence status | Source |
|---|---|---|---|---|---|---|---|
| Loading measured outcomes… | |||||||
Each row retains its own evidence status and source commit.
The AMB 15/77 strict-success result is a valid native local self-run. The separate published Hindsight 66/77 result uses structured filters and LLM extraction, so it is not ranked in that query-only comparison. The 50-question and 500-question LongMemEval populations are never combined. Lower results remain visible because they are precisely what the runtime must improve.
Accepted historical evidence
Older qualified, source-pinned profiles follow below. These retain their original acceptance decisions, but may omit newer candidate competitors. Use the measured leaderboard above for the current evidence picture.
Browse accepted historical profiles
Older, qualified benchmark comparisons. Current candidate results are ranked in the primary leaderboard above.
| System | Role | Measured result | Samples | Gap to leader | Evidence |
|---|---|---|---|---|---|
| Loading accepted evidence… | |||||
02 / Running the systems
Live benchmark executions.
Actual committed candidate results from the execution branches, checked at their latest public Git commits. The numbers below are not official placements or accepted gauntlet scores. Unlike the qualified leaderboard above, this execution tracker refreshes without a Pages deployment.
LongMemEval-S · retrieval-only candidate cohort
Preregistered 50-question subset. Forty-three questions are scored per completed plane, seven excluded. Scores reflect native recall-all metrics, not question answering.
Retrieving latest pinned competitor evidence…
An incomplete competitor run is reported by completed question count only, not by extrapolated scores.
Source-pinned execution registry
Automatically discovers new candidate run manifests, benchmark families and partial progress records committed by the Claude execution branches. Exact source identity, denominators and raw report digests are checked before displaying numeric results.
Discovering eligible candidate source manifests…
Checking source-pinned manifests…
Incomplete runs show progress only. Unknown benchmark families appear as evidence references without inferred scores.
Independent-harness outcomes
These are separate experimental protocols. Keep their scoring and evidence boundaries intact.
Checking external evaluation branch…
Loading upstream AMB and MemoryArena execution summaries…
Read-only refresh on page load, on demand, and approximately every 15 minutes while this tab is visible. Uses public immutable GitHub commit references, no personal GitHub credentials, no hosted benchmark CI, and no write access. If GitHub is unavailable, qualified historical scores remain available above. Execution ownership and qualification ↗
03 / The measurement gap
Where the evidence reaches.
A measurable peer is not the same as an untested competitor. This matrix is calculated from accepted source-pinned observations only, without borrowing vendor-published scores.
Benchmark families in the research and qualification queue
The accepted leaderboard is intentionally narrower than the benchmark programs under investigation. These links show separately owned methods; no run in one family earns a score in another.
AMB PrecisionMemBench: full 77-case retrieval-only same-harness comparison; accepted v6 and provisional v7 candidate evidence. Frozen lane qualification ↗
LongMemEval S and M: S provides separate session and turn retrieval planes and same-harness peers; M has accepted longitudinal Agent Memory evidence, but not an equivalent accepted multi-product cohort. Canonical profile interpretation ↗
AgentMemBench / MESA: source-retrieval, temporal currentness and mutation/semantic evidence; not interchangeable with answer-generation or AMB strict-success results. Formal MESA baseline ↗
SWE-ContextBench Lite: controlled software-agent context retrieval; preserve independent harness and model configurations before comparing products. Comparison protocol ↗
LoCoMo and next-generation task evaluations: external QA, lifecycle and cross-session task utility remain pending independently reproduced, profile-equivalent cohorts. LoCoMo comparator investigation ↗ · MemoryArena and external programs ↗
Benchmark-by-competitor coverage
Each cell indicates whether this exact profile contains an accepted measured result for that external memory system.
| Loading source-pinned competitor coverage… |
No result means there is no accepted comparable measurement in that profile. It does not mean zero performance, unsupported functionality or product failure. Baselines and Agent Memory are intentionally excluded from the competitor columns.
Loading source provenance coverage…
04 / External accountability
External evaluations & competitions.
We want Agent Memory tested by other evaluators using their own rules, not merely scored by a leaderboard we operate. These are links to independent evaluation programs, not claims of official placement.
Loading external benchmark references…
Loading independently maintained benchmark links…
Comparison rule: an AML score, an AMB result, and a MemoryArena task-success measurement are not interchangeable. No imported score changes our accepted leaderboard or creates a cross-program gap without an explicitly qualified shared protocol. See the execution program ↗.
05 / Coverage, not just winners
The competitive field
A system can be important to evaluate before its first accepted result. Missing is not failing.
The registry is a discovery surface. Inclusion is not an endorsement, compatibility claim or completed benchmark. Some systems may be unsupported for specific profiles.
06 / Research & architecture
Governance before convenience.
Persistent state is consequential. A retrieved memory does not grant itself authority.
PAMA & write authority ↗
Proposals and probabilistic interpretation cannot bypass admission and policy.
02Lifecycle & currentness ↗
Correction, supersession, temporal relevance and deletion have distinct contracts.
03Architecture catalog ↗
Canonical design intent and ADR states are maintained in the repository.
07 / Reproducibility
Show the work.
Then show the work again.
Each measured row links to its underlying evidence. Results remain bound to the benchmark input, profile, revision and execution history. A score without comparable execution identity is not a competitive delta.
Benchmark documentation ↗Evidence contract
Immutable input digest and system revision
Native metric and exact evaluation plane
Execution manifest: clock, hardware, dependencies, inference and costs
Raw outputs, evaluator, limitations and reproducibility links
Full execution-manifest enforcement is being tracked under #807. Historical runs may not satisfy every proposed field.
06 / External claims
Published references
Useful market context. Not scored against our same-harness results.
