Open research · governed runtime · evidence laboratory

Memory that
can be questioned.

An auditable memory runtime for AI agents, built around governed recall, correction, currentness and forgetting. Its claims live alongside the evidence that can challenge them.

Executable reference runtime. Qualified scope is bounded; production 1.0 is not claimed.
Every claim has a boundary.Every measurement has a method.Every omission stays visible.

Delivery evidence / October 21, 2026

What the latest runs actually show

Accepted same-harness scores, experimental improvements and rejected regressions are different evidence classes. This checkpoint links to each source; no candidate run silently updates the accepted leaderboard.

Accepted AMB retrieval baseline (v6): Agent Memory 15/77 strict-success cases (mean precision 18.06%, mean recall 95.35%); Mem0 OSS 10/77 and BM25 8/77 on the same frozen retrieval-only protocol. The historical leaderboard below remains bound to its accepted snapshot. Accepted dashboard source ↗

October 20 selective-context experiment: 15/77 → 22/77. An opt-in lexical_content consumer package improved known-case AMB strict success by seven cases, with mean precision 18.06% → 32.43% and mean recall unchanged at 95.35%. This is a candidate diagnostic, not an accepted AMB lane: the full locked AMB CLI was not run and the 77 cases were already known. Raw paired evidence and limitations ↗

Why this mutation was not promoted: on the full external LongMemEval-S check, session complete recall@50 fell 419/419 → 418/419 and turn complete recall@50 fell 360/419 → 358/419. It lost required annotated evidence, and no-answer cases still received unrelated context. CHANGES REQUIRED, not a new default. Full 500-question-per-plane evaluation ↗

October 21 repaired v7 candidate, complete matrix: all 9/9 frozen benchmark rows are imported and re-verified against protected runtime tree f9b5ff9e and declaration 7de94f11 in draft PR #836. AMB is unchanged from v6: 15/77 Agent Memory, 10/77 Mem0 OSS, 8/77 BM25 (77/77 case equality in each row). LongMemEval-S includes six full 500-question session/turn rows for Agent Memory, lexical overlap and Mem0 OSS: all native metrics and required-evidence totals match v6; Mem0 turn differs in the ordering of 47/500 ranked lists, with no gold item moved and no metric change, so ranked-list equality is 453/500, not 500/500. This is source-qualified candidate parity, not an AMB improvement, a new external competitor, or an accepted v7 baseline. All nine-row results and comparator caveat ↗ · Independent input/import verification record ↗

Competitor coverage: the accepted same-harness external memory-product count is still one (Mem0 OSS). Hindsight and Letta have experimental runs that do not count as accepted products; Cognee, Graphiti, OpenViking, LangMem, Supermemory, TencentDB Agent Memory and the newly registered MemOS, EverOS, MIRIX and Acontext remain visible for qualification, never scored as zero. Browse accepted peer coverage, the expanded source registry and the live candidate execution feed.

Evidence checkpoint: October 21, 2026, draft #836 at 3df781c2. The accepted numeric projection remains pinned to October 7; candidate parity is displayed separately. Formal v7 release acceptance and the public API contract decision remain outstanding. Recheck linked run records for later evidence.

01 / Comparative evidence

Benchmark leaderboard

Real measured outcomes, including runs in which Agent Memory trails competitors. Every ranking is specific to its benchmark, population, native metric, and evidence status. Candidate results are visible, not silently accepted.

Loading evidence snapshot…

Where Agent Memory actually ranks

Current measured results first. Switch to accepted-only evidence, or inspect every scored benchmark profile. Unreviewed measurements remain explicitly labeled.

Loading evidence…
RankMemory systemRoleMeasured resultScored casesGap to leading systemEvidence statusSource
Loading measured outcomes…

Each row retains its own evidence status and source commit.

The AMB 15/77 strict-success result is a valid native local self-run. The separate published Hindsight 66/77 result uses structured filters and LLM extraction, so it is not ranked in that query-only comparison. The 50-question and 500-question LongMemEval populations are never combined. Lower results remain visible because they are precisely what the runtime must improve.

Accepted historical evidence

Older qualified, source-pinned profiles follow below. These retain their original acceptance decisions, but may omit newer candidate competitors. Use the measured leaderboard above for the current evidence picture.

Browse accepted historical profiles

Older, qualified benchmark comparisons. Current candidate results are ranked in the primary leaderboard above.

Accepted evidence only · frozen
SystemRoleMeasured resultSamplesGap to leaderEvidence
Loading accepted evidence…

02 / Running the systems

Live benchmark executions.

Actual committed candidate results from the execution branches, checked at their latest public Git commits. The numbers below are not official placements or accepted gauntlet scores. Unlike the qualified leaderboard above, this execution tracker refreshes without a Pages deployment.

Two evidence levels, deliberately separate. The leaderboard above is frozen, independently accepted evidence. This section shows completed but unreviewed executions and honest progress for incomplete runs. Missing values never count as zero, and distinct benchmark families are never merged into a universal score.

LongMemEval-S · retrieval-only candidate cohort

Preregistered 50-question subset. Forty-three questions are scored per completed plane, seven excluded. Scores reflect native recall-all metrics, not question answering.

Candidate · not accepted
Checking latest GitHub benchmark commit…

Retrieving latest pinned competitor evidence…

An incomplete competitor run is reported by completed question count only, not by extrapolated scores.

Source-pinned execution registry

Automatically discovers new candidate run manifests, benchmark families and partial progress records committed by the Claude execution branches. Exact source identity, denominators and raw report digests are checked before displaying numeric results.

Discovering eligible candidate source manifests…

Checking source-pinned manifests…

Incomplete runs show progress only. Unknown benchmark families appear as evidence references without inferred scores.

Independent-harness outcomes

These are separate experimental protocols. Keep their scoring and evidence boundaries intact.

Checking external evaluation branch…

Loading upstream AMB and MemoryArena execution summaries…

Read-only refresh on page load, on demand, and approximately every 15 minutes while this tab is visible. Uses public immutable GitHub commit references, no personal GitHub credentials, no hosted benchmark CI, and no write access. If GitHub is unavailable, qualified historical scores remain available above. Execution ownership and qualification ↗

03 / The measurement gap

Where the evidence reaches.

A measurable peer is not the same as an untested competitor. This matrix is calculated from accepted source-pinned observations only, without borrowing vendor-published scores.

Benchmark families in the research and qualification queue

The accepted leaderboard is intentionally narrower than the benchmark programs under investigation. These links show separately owned methods; no run in one family earns a score in another.

AMB PrecisionMemBench: full 77-case retrieval-only same-harness comparison; accepted v6 and provisional v7 candidate evidence. Frozen lane qualification ↗

LongMemEval S and M: S provides separate session and turn retrieval planes and same-harness peers; M has accepted longitudinal Agent Memory evidence, but not an equivalent accepted multi-product cohort. Canonical profile interpretation ↗

AgentMemBench / MESA: source-retrieval, temporal currentness and mutation/semantic evidence; not interchangeable with answer-generation or AMB strict-success results. Formal MESA baseline ↗

SWE-ContextBench Lite: controlled software-agent context retrieval; preserve independent harness and model configurations before comparing products. Comparison protocol ↗

LoCoMo and next-generation task evaluations: external QA, lifecycle and cross-session task utility remain pending independently reproduced, profile-equivalent cohorts. LoCoMo comparator investigation ↗ · MemoryArena and external programs ↗

…Loading coverage evidence

Benchmark-by-competitor coverage

Each cell indicates whether this exact profile contains an accepted measured result for that external memory system.

Loading source-pinned competitor coverage…

No result means there is no accepted comparable measurement in that profile. It does not mean zero performance, unsupported functionality or product failure. Baselines and Agent Memory are intentionally excluded from the competitor columns.

Loading source provenance coverage…

Follow the independent competitor execution program ↗

04 / External accountability

External evaluations & competitions.

We want Agent Memory tested by other evaluators using their own rules, not merely scored by a leaderboard we operate. These are links to independent evaluation programs, not claims of official placement.

No official third-party Agent Memory ranking is verified here yet. Our accepted same-harness comparisons above remain separate. External results will be shown only after a complete run, public source evidence, and independent review.

Loading external benchmark references…

Loading independently maintained benchmark links…

Comparison rule: an AML score, an AMB result, and a MemoryArena task-success measurement are not interchangeable. No imported score changes our accepted leaderboard or creates a cross-program gap without an explicitly qualified shared protocol. See the execution program ↗.

05 / Coverage, not just winners

The competitive field

A system can be important to evaluate before its first accepted result. Missing is not failing.

Parallel benchmark execution tracker ↗

The registry is a discovery surface. Inclusion is not an endorsement, compatibility claim or completed benchmark. Some systems may be unsupported for specific profiles.

06 / Research & architecture

Governance before convenience.

Persistent state is consequential. A retrieved memory does not grant itself authority.

07 / Reproducibility

Show the work.
Then show the work again.

Each measured row links to its underlying evidence. Results remain bound to the benchmark input, profile, revision and execution history. A score without comparable execution identity is not a competitive delta.

Benchmark documentation ↗

Evidence contract

01

Immutable input digest and system revision

02

Native metric and exact evaluation plane

03

Execution manifest: clock, hardware, dependencies, inference and costs

04

Raw outputs, evaluator, limitations and reproducibility links

Full execution-manifest enforcement is being tracked under #807. Historical runs may not satisfy every proposed field.

06 / External claims

Published references

Useful market context. Not scored against our same-harness results.