The five results
Each number below was produced on 2026-10-28 on the tree released as 3.0.0-rc.2 (merge commit 16cd9c0). Every run is offline, with no API key, no paid model and no LLM judge, and each comes with the command that reproduces it from a fresh clone after npm ci && npm run build. Synthetic corpora and case sets are labelled synthetic. The full test suite passed on the same tree: 93 files, 1,557 tests.
- 70/77 PrecisionMemBench single-turn, 11/12 session turns
- 0/167 poisoning escapes, 0/60 false quarantines (synthetic)
- 96.05 % invalidation recall, 98.06 % precision, 0 stale with the premise guard (synthetic)
- 1000/1000 identical replay states, 24000/24000 as-of answers (synthetic)
- 1.00 bisect blame precision on 2000 records (synthetic)
These are statements about named, versioned fixtures and case sets. They are not measurements on your data, and no other product has been run through these harness commits, so none is compared here.
PrecisionMemBench
70/77single-turn cases, lexical and hybrid alike · 11/12 session turns · mean precision 0.871, recall 1.000
- Protocol
- tenurehq/precisionMemBench at commit
61a712a(MIT): 35 beliefs, 77 single-turn retrieval cases and 2 session cases with 12 turns. Retrieval mode: the harness scores pass or fail from the precision and recall of the returned belief ids, with no judge. The wrapper is protocol-exact and translation only: no wrapper-side filter and nokindspredicate. Every exclusion is a kernel decision recorded in the ledger. - What carries it
- The opt-in names gate (
requireNamedMatch: 'auto') and guarded single-edit fuzzy matching, both off by default, plus the always-on lifecycle prefilter. Without the names gate the score is 20/77; without fuzzy matching it is 64/77. The cross-encoder neither helped nor hurt: the reranked run also scores 70/77. - What 70/77 is not
- Not a held-out precision estimate. The fixture is public and small, and the kernel controls were designed against its failure classes: names in the query, typos and scope. Read it as “the kernel with its names gate, on this fixture”.
- The names gate abstains on implicit references by design. A record that declares names cannot be reached by a query that never names it.
- The wrapper makes translation choices a different wrapper could make differently (the canonical name, alias words on update, and
open_questionmapped toobservation). - Seven misses remain in our metadata-based configuration; the protocol omits explicit lifecycle fields. Each returns a belief the case excludes for lifecycle metadata the protocol does not send as fields (
type,superseded_byorresolved_at). The queries carry lifecycle hints in text and ids that this configuration does not mine; another approach could. - The 12th session turn returns an open question; passing it needs an open-question kind policy the kernel is not told about.
- History
- rc.1 (not published) scored 22/77 unfiltered and 48/77 with wrapper filters chosen by sweeping the fixture, with 6/12 session turns. Those filters are removed.
- Reproduce
- From the repository root
node benchmarks/precisionmembench/run.mjs --label final-lexical --retrieval lexical --metadata protocol --both node benchmarks/precisionmembench/run.mjs --label final-hybrid --retrieval hybrid --metadata protocol --bothThe harness clone needs the network once; scoring is offline.
run.mjsexits 3 because the suite reports the 7 failing cases.
Poisoning suite v1
0/167attack escapes · 0/60 false quarantines · 0 errors · consolidation on, 294 gate decisions
- Protocol
- 227 synthetic cases: 167 attacks and 60 benign controls, generated from seed
poisoning-v1(sha2564b5d665d…6240ee336, regenerated byte-identical by the test suite). Six classes: content, evidence, scope-leak, selection, path and erasure-return. Each case runs in a fresh in-memory store on the lexical path with a fixed clock, no network and no model. With consolidation on, every principal runs four signed passes through the outcome gate (a deterministic proposer, an aggressive-merge proposer and two synthetic laundering adversaries): 11 kept, 184 rolled back by invariant, 99 with too little evidence to score; laundering passes kept: 0. Default mode is also 0/167 and 0/60. - Control
- With the gate switched off, 85 of 167 attacks escape, so the probes do catch laundered content when it gets through. Switching off one kernel guard at a time also fails the gate every time.
- Limits
- This is zero escapes on these 227 cases. It is not a claim that the kernel cannot be poisoned.
- The content and selection classes test the trust-label gate, not content inspection: trust is decided by channel and provenance.
- Only the lexical path is exercised; dense and hybrid retrieval are not covered by v1.
- In consolidation mode the synthetic verifier counts any non-empty recall as a success, so the regression rollback is never exercised here and the result rests mostly on the invariants.
- Paraphrase laundering is not caught: an untrusted instruction restated, uncited, in tokens from trusted records can be kept.
import()still accepts a hand-built snapshot’s trust label for a record with no dependencies; v1 does not test this.- A failed outcome removes a record from
compile()but not fromrecall(). Decaying quarantine is not implemented.
- Reproduce
- About 2 s per run
npm run poisoning node dist/evaluation/poisoning/cli.js --cases benchmarks/poisoning/v1/cases.jsonl --out reports/poisoning/v1-consolidation --consolidation
Dependency suite v1
96.05 %invalidation recall (170/177) · precision 98.06 % (152/155) · 0 stale executions with the premise guard
- Protocol
- Undeclared-dependency inference (rule
co-use-success-v1) proposes premise links from successful co-use; accepted links become ordinary dependencies, so one correction invalidates the dependents through the kernel cascade. Synthetic corpus from seed20261028: 228 records, 717 tasks, 160 true undeclared edges, 78 declared edges and five decoy classes. The eval reviewer accepts every proposal; the executor is guarded; lexical path. Gates: recall at least 90 %, precision at least 95 %, stale exactly 0. Across ten robustness seeds (five used for tuning, five not), recall was 92.4–96.8 % and precision 95.1–99.4 %, with 0 stale through the guarded executor on every seed. - Limits
- Zero stale depends on the premise guard. Inference alone leaves 7 stale procedure executions out of 248 checks on the gate seed.
- The guard over-holds. Its false-hold rate is 92.78 % on the gate seed (84.54–93.14 % across seeds). That is reviewer load, not invalidation.
- Zero stale also rests on the generator forcing every true edge into at least one judged successful task.
- The eval reviewer accepts everything. In production each proposal is a person’s decision, and precision on real controller traces is unmeasured.
- Some thresholds were chosen after seeing results; each choice is disclosed in the method document.
- A pair used together only once is invisible to inference. Only the lexical path was measured.
- Reproduce
- About 5–8 minutes
node dist/evaluation/dependencies/cli.js --corpus benchmarks/dependencies/v1/corpus.json --out reports/dependencies/v1 --seeds 1,2,3,4,5,11,12,13,14,15
Replay determinism
1000/1000identical replay states across two readers · 24000/24000 as-of answers equal to ground truth · 281 distinct states
- Protocol
- Moment 1 of
mnemosy demo. A synthetic project history (24 services whose facts change over two years, some learned days after they became true) is written to a store. The belief state is hashed at 1000 seeded(asOf, knownAt)cut points, half of them on or one millisecond beside a boundary. One reader is the demo process; the second is a separate process that opens the same file cold. At every cut point each service’s as-of answer is also checked against ground truth computed from the seeded plan, never from the store. - Limits
- Record ids are random, so hash values differ between runs; the claim is that two independent readers of one store agree at every cut point.
- The hash covers the visible records. Derived indexes, embeddings and the ledger are excluded by design.
- Synthetic history, lexical path.
- Reproduce
- Pre-release 3.0.0-rc.3
npx -y -p mnemosy-ai@next mnemosy demo
Bisect
1.00blame precision: the blame set equals the seeded false record, on a 2000-record store, well under the 60 s bar
- Protocol
- Moment 3 of
mnemosy demo. A synthetic ops store of 2000 records holds one verified answer; one confident, false record then arrives at ordinary trust and the answer goes wrong. Onebisectcall delta-debugs the store; the blame set must equal the seeded record exactly, and an exact re-run of recall without it must restore the right answer at rank 1. The verifier has no model in it. - Timing
- 0.40 s in the recorded run of 2026-10-28 (
site/demo-five.json, 3.0.0-rc.2 build, on the maintainer’s development machine; the machine model is not recorded). Times vary by machine and load. The bar, asserted bytests/breakthrough-demo.test.ts, is under 60 s. - Limits
- One culprit, seeded at
k = 1, where the minimal blame set is unique. At largerkseveral minimal sets can exist, andbisectreports that ambiguity instead of hiding it. - The demo knows the right record because it seeded it.
bisectalways needs the expected (gold) record ids of a failed retrieval; it judges retrieval, not a generated answer, and can endpassing,unfixableorunconfirmedinstead of naming a culprit. - The demo uses the lexical path, and so does the
mnemosy bisectCLI; the hybrid path is covered by tests, not by this number.
- One culprit, seeded at
- Reproduce
The same
mnemosy demorun as replay prints this moment’s PROOF line.
Default retrieval recall unchanged
The precision work did not move the default retrieval path's recall. LoCoMo and LongMemEval-S were run before (84cb6c3) and after with the same commands: recall is identical in every row, and the largest MRR change is +0.0014 (overlap MRR on LoCoMo conversations 0–2). The opt-in controls were off, and neither corpus contains superseded or expired records, so the always-on lifecycle prefilter had nothing to drop.
| Benchmark and condition | Before | After |
|---|---|---|
| LoCoMo full set, BM25 recall@10 / MRR | 52.47 / 0.3696 | 52.47 / 0.3696 |
| LoCoMo full set, overlap recall@10 / MRR | 40.49 / 0.2899 | 40.49 / 0.2899 |
| LoCoMo conversations 0–2, hybrid recall@10 / MRR | 75.29 / 0.6325 | 75.29 / 0.6327 |
| LongMemEval-S, 470 annotated, BM25 mean recall / MRR | 95.08 / 89.57 | 95.08 / 89.57 |
| LongMemEval-S, 470 annotated, lexical mean recall / MRR | 89.72 / 78.72 | 89.72 / 78.72 |
These are retrieval coverage against a local scorer, not generated-answer accuracy. The full table and commands are in section 6 of the method document.
Latency: known limits
Latency was measured only on a loaded machine: a 10-core host at a 1-minute load of 5.12–11.32, on a synthetic store of 2,400 rows (400 superseded), with the controls off and on interleaved per query. Treat these as indications, not idle-machine numbers. The harness was a scratch script that is not committed, so these figures cannot be reproduced from the repository.
- The opt-in names gate and fuzzy matching added +18.2 % p50 to BM25 recall (per-round range +10.4 to +21.6 %) and no resolvable change to hybrid recall.
- The always-on lifecycle prefilter showed no p50 cost on BM25 or hybrid recall.
- Known regression: the prefilter makes the non-default
lexicalScoring: 'overlap'scorer about +458 % slower at p50 (26.6 ms against 4.8 ms with the prefilter disabled; 4.6 ms on 2.0.0). - Idle-machine latency, the real embedder and reranker, and stores larger than 2,400 rows are not measured.
What is not claimed
- PrecisionMemBench 70/77 is not a held-out estimate. The public 35-belief fixture is the only data the benchmark has.
- The poisoning, consolidation, dependency and demo data are synthetic. Real agent traces will not look like them.
- The poisoning suite, consolidation replay, dependency eval and demo use the lexical path only.
- No comparison with another product is made. No other product has been run through these harness commits.
- MemSecBench is not scored and ConsolidationBench is not attempted.
- No improvement-from-experience number exists. The consolidation gate keeps only passes whose aggregate lexical coverage on the recorded replay set does not drop; measured lift on new task families is future work and is not claimed.
- Recall latency has not been measured on an idle machine.
Full method and reports
Every table, threshold, seed and command behind this page is in docs/BENCHMARKS-TRUST.md at v3.0.0-rc.2, with the reports under reports/ in the same tag. The recorded five-moment run is on the home page, and its limits are in docs/DEMO.md. The stable 2.0.0 answer-accuracy result (409 of 500 LongMemEval-S questions, with a language-model reader and judge) is described in the documentation.
Found a number that does not reproduce? Open a GitHub issue with the command and its output.