Dakera Memory Engine Benchmark: 1M Vectors, Sub-2ms, No LLM

If you are choosing a memory layer for an AI agent, you are really making one decision under two constraints: can it recall the right thing, and can it do so fast enough and cheaply enough to sit in your request path at production volume. We publish the accuracy half already — 88.2% Recall@20 on LoCoMo. This post is the second half, measured with the same rigor: what the Dakera memory engine actually does in the read path when a namespace holds a million memories.

No sampled subset, no cherry-picked operating point, no "benchmark-only" build. Below: the full recall-vs-throughput frontier on a standard dataset, an explanation of why the curve has the shape it does, a head-to-head against both a purpose-built ANN library and the LLM-in-the-loop memory frameworks you're likely comparing us to, the production cost math, an honest account of what this measurement does and doesn't prove — and the complete harness to reproduce every number.

Dakera v0.11.108 · SIFT1M · 1,000,000 vectors · single 8-core box · one index, ef_search swept 10→400. Up and to the right is better.

One framing before the data, because it prevents a category error people make constantly: this is a retrieval-performance benchmark, not a conversational-accuracy benchmark. Different question, different dataset, different honest answer. The 88.2% accuracy number lives in the methodology post. This one is about latency, throughput, and recall in the read path at a million vectors.

The three-way choice every agent team faces

Before the numbers, the decision they serve. When you add memory to an agent, you pick one of three architectures — and each one trades retrieval speed against memory semantics:

The conventional wisdom is that you pick two of those three properties. The point of this benchmark is to show, with a reproducible number, that Dakera's read path is in the bare-vector-DB speed class while being a full memory engine. The rest of this post is the evidence.

Benchmark setup and methodology

We use SIFT1M, the standard one-million-vector ANN benchmark: 1,000,000 base vectors of 128-dimensional SIFT descriptors, Euclidean distance, with published ground-truth nearest neighbours for a held-out query set. The field standardised on it precisely because the ground truth is fixed and public — there is nowhere to hide, and results are comparable across implementations.

• DatasetSIFT1M — 1,000,000 base vectors, 128-dim
• IndexDakera HNSW, Euclidean metric
• Query setheld-out queries, recall@10 vs ground truth
• Hardwaresingle 8-core x86 box, 15 GiB RAM
• Acceleratorsnone — no GPU, no LLM in the hot path
• Index configHNSW M=16, ef_construction=200
• Build1M vectors in 869 s (~1,150 vec/s)
• EngineDakera v0.11.108 (single binary)

The measurement protocol is deliberately plain. We build one index over all one million base vectors, warm it, then issue the held-out queries serially — one at a time — recording end-to-end latency per query. Throughput is reported as the inverse of mean latency (QPS = 1 / mean latency), so every QPS figure here is single-query serial throughput on one core — not a concurrent peak. It is deliberately conservative: real deployments run queries in parallel and throughput scales roughly with cores, so treat these QPS numbers as a per-core floor, and read the latency columns as the primary result. (A concurrent-throughput sweep is on the list of things we're adding.) Recall@10 is the fraction of each query's true top-10 neighbours (per the dataset's ground truth) that appear in the ten Dakera returns, averaged over the query set.

The results: the whole curve

HNSW exposes one knob at query time — ef_search — that trades latency for recall. Rather than quote one flattering point, here is the full sweep behind the chart above. Every row is the same 1M-vector index on the same box; only ef_search changes.

Dakera v0.11.108 · SIFT1M · single 8-core box · latency p50/p95/p99/p99.9, QPS = 1/mean · recall vs published ground truth (harness self-check 0.998). recall@100 = 0.961 at ef 400.

Read the curve, not a single cell. At ef_search 100 you get 0.922 recall@1 / 0.930 recall@10 at 582 µs median latency — and, the number production teams actually watch, a 99th-percentile latency of just 879 µs, still under a millisecond, at 1,750 queries per second. Push to ef_search 400 and recall climbs to 0.976 while p50 stays under 2 ms (p99 ≈ 3 ms). Need raw throughput instead of top-end recall? Drop ef_search to 10 and the same index serves 9,785 QPS. One index, one build — you choose where on the frontier you operate.

Tail latency: p95 / p99 / p99.9

A median is a promise you keep half the time; an SLO is written against the tail. So this run publishes the full latency distribution per operating point, not just p50 — the p95/p99/p99.9 columns in the sweep table above are the numbers you'd actually put in a latency budget. The honest headline: at ef_search 100 — the 0.93-recall operating point most teams will run — the 99th percentile stays sub-millisecond at 879 µs, and even the 99.9th percentile is just 1.02 ms. Tail latency below a millisecond through ef 100 is the property that lets you drop this in a request path without a timeout budget blowing up on the unlucky reads.

Push for maximum recall and the tail widens predictably: at ef_search 400 (0.976 recall@10) p99 is 3.01 ms and p99.9 tops out at 3.36 ms — still low single-digit milliseconds, with the p99.9 barely 12% above p99, so there is no pathological long tail hiding past the 99th percentile. That flatness between p99 and p99.9 is the tell that the beam search is doing bounded work per query rather than occasionally falling off a cliff. Pick your operating point by the tail you can afford, not the median you'd like to quote.

Why the curve bends where it does

The shape of that frontier is not incidental; it's HNSW doing exactly what the algorithm guarantees, and understanding it is how you tune it. HNSW — Hierarchical Navigable Small World — stores vectors as a layered proximity graph. A query enters at the top layer, greedily walks toward the query vector, and drops down layers until it does a best-first beam search across the dense bottom layer.

ef_search is the width of that beam: the number of candidate nodes the search keeps "in play" at the bottom layer. Widen the beam and the search visits more neighbours, so it's more likely to find the true nearest ones — recall rises — but each extra candidate is another batch of distance computations, so latency rises with it, roughly linearly. That is the entire trade in one parameter, and it's why the left of the curve (small ef) is cheap-but-lossy and the right (large ef) is accurate-but-slower.

The diminishing returns at the top — 0.961 to 0.976 recall costs you nearly 2× the latency (ef 200 → 400) — are the second HNSW fact worth knowing. Recall saturates because, past a point, the true neighbours are already inside the beam; widening it further just re-confirms what you have. The height of that ceiling is fixed at build time by two other parameters — M (how many neighbours each node keeps, i.e. graph connectivity) and ef_construction (how hard the index works while being built). Higher M / ef_construction raise the achievable recall ceiling at the cost of a larger index and slower builds. Dakera ships conservative defaults for both; the curve above is what those shipped defaults deliver, and it's tunable if your workload wants a higher ceiling.

Does it hold at real embedding dimensions? SIFT vs GIST

SIFT1M is the standard ANN benchmark, but it is 128-dimensional — and agent memory runs on 768–1536-dimensional text embeddings. A number that only holds at 128 dimensions is not one you could trust in production. So we ran a second, harder dataset: GIST1M — one million 960-dimensional vectors, right at text-embedding scale. Same harness, same self-check (0.999 vs GIST's own ground truth), same box.

The first honest result: at the SIFT-tuned default (HNSW M=16), 960 dimensions is much harder — recall@10 tops out around 0.78. That is expected and not Dakera-specific: higher intrinsic dimensionality is harder for any ANN index, and GIST is one of the toughest standard datasets. But 0.78 is a configuration ceiling, not an engine ceiling — and the knob that sets it is M, the graph's connectivity. M=16 is right for low-dim SIFT; high dimensions need a denser graph. Rebuild GIST at M=32 and the ceiling lifts sharply:

Dakera v0.11.108 · GIST1M 960-dim · single 8-core box · self-check 0.999 vs ground truth. M=32 uses ef_construction=400. recall@100 = 0.925 at ef 800.

Two honest reads. First, the engine does scale to real-embedding dimensionality — tuned for high dim, it reaches 0.92–0.945 recall@10 at 960 dimensions. Second, high-dim high-recall genuinely costs more: latency climbs into single-digit milliseconds and throughput drops, because a 960-dim distance is 7.5× the arithmetic of a 128-dim one and a denser graph visits more nodes. That trade is intrinsic to the problem, not an artifact of the engine — which is why we publish the whole curve for both datasets rather than one flattering cell.

And on real cosine text embeddings?

SIFT and GIST are Euclidean image descriptors. To measure the metric agent memory actually uses — cosine — we ran the standard NYTimes-256 dataset: real news-article embeddings, 256-dim, angular/cosine, with the dataset's own published ground truth (no self-assembled labels). The harness self-check agrees with that ground truth at 0.993, so the cosine path is measured correctly.

Dakera v0.11.108 · NYTimes-256 (real text, cosine) · single 8-core box · dataset's own ground truth, self-check 0.993 · M=32, ef_construction=256.

On real cosine text embeddings, Dakera reaches 0.87 recall@10 at ~2ms p50, climbing to 0.91–0.925 at higher ef_search (and higher latency) — on a dataset the ANN literature treats as genuinely hard. Two honest notes: this is 256-dim, the highest dimensionality at which a standard, ground-truth-bearing cosine text benchmark exists publicly (nothing at 768–1536-dim ships a reproducible ground truth, so anything there would be self-labelled), and, as with GIST, recall scales with M — the tuned M=32 above lifts the whole curve versus a low-dim default. Across all three datasets the story is consistent: a competitive recall/latency envelope, with the operating point yours to tune.

Comparison 1 — against a purpose-built ANN library

The first fair question is whether a full memory engine can hold its own on raw ANN against a library that does nothing but nearest-neighbour search. The public reference is ANN-Benchmarks, the community leaderboard for this exact dataset. A representative hnswlib configuration (M=16, ef=64, single-threaded) lands around 0.942 recall@10 at roughly 1,940 QPS on SIFT-128-euclidean — the hollow marker on the chart.

hnswlib figures from ANN-Benchmarks (SIFT-128-euclidean); different hardware — treat as class-level positioning, not a controlled head-to-head

Here's the honest read: on v0.11.108 this is a dead heat. Dakera at ef_search 100 delivers 0.930 recall@10 at 1,750 QPS, sitting right on hnswlib's 0.942 at 1,940 QPS — within measurement noise, on different hardware. Turn ef_search up and Dakera clears 0.96–0.98 recall, past the reference operating point (at lower throughput, as the curve demands). Dakera is on the same recall-per-QPS line as a library that does nothing but nearest-neighbour search — while also carrying decay, importance weighting, hybrid vector+BM25 retrieval, an entity graph, lifecycle policies and a server around that index. For an agent builder the takeaway is the one that matters: you are not trading away ANN-class read speed to get real memory semantics.

A note on the wider field

The natural next question is how this compares to other engines. The honest answer: the only comparison we'd stake our name on is one run on identical hardware under one harness. Published QPS numbers from different machines and default configs are directional at best, and a favourable cross-hardware table is exactly the kind of thing that gets a vendor caught when someone reruns it matched. Neutral leaderboards like ANN-Benchmarks and peer-reviewed surveys (e.g. arXiv:2608.12812) put the field on a common footing; against the reference hnswlib point above, Dakera sits on the same recall-per-QPS line. A controlled, same-box A/B against other engines is the benchmark we're building next — and we'd rather publish that than a table that flatters us on a technicality.

Why the read path has no LLM in it

There's a second class of system agent teams evaluate — memory frameworks like Mem0, Zep and LangMem — and the difference from Dakera is architectural, not a leaderboard number. Those systems put a language model and/or a managed round-trip inside the retrieval path: a memory read triggers extraction, reranking, or a network hop, so published retrieval latencies for that class run from roughly 150 ms to several seconds, and the per-read cost tracks token prices. Dakera keeps the model out of the read path entirely — retrieval is Rust and on-device ANN, so a read is the sub-2ms figure measured above, and its cost is CPU you already own.

What sub-2ms with no LLM means in production math

Recall near 0.9 at a million vectors is a good result. The line that changes how you architect a system is the one under it: an 8-core commodity box, no GPU, no accelerator to provision, and no per-query token cost. Work the arithmetic the way a platform team would.

At ef_search 100 — 0.93 recall@10, 582 µs median, p99 under a millisecond — a single core turning queries over serially clears roughly 1,750 reads per second. That is north of 150 million reads per day from one core, before you fan out across the other seven or add concurrency. The marginal cost of each of those reads is CPU time on hardware you already run — an 8-core box rents for roughly $50–150/month — which puts the cost of a million memory reads in the small fractions of a cent, and, critically, flat: it does not rise when an LLM provider changes its pricing, and it does not add a network hop to every agent turn. Compare that to a read path with a model in it, where each recall is both slower and separately billed per token. For an agent doing many memory reads per turn, that is the difference between memory being a rounding error in your latency and cost budget and being the dominant term.

What this benchmark tells you — and what it doesn't

A benchmark is only as trustworthy as its stated limits. Here is exactly what you should and shouldn't conclude from these numbers:

Reproduce it — the complete harness

The recipe is deliberately boring, which is what makes it trustworthy. Read SIFT1M, validate the harness against ground truth before trusting it, build the index over all one million vectors, then sweep ef_search and record recall@1/@10, the full latency distribution (p50/p95/p99/p99.9), and QPS — the same SCALE_RECALL_JSON artifact our CI run publishes. The engine calls go through the documented public HTTP API; the scoring logic — the part that makes this a real benchmark and not a vendor claim — is engine-agnostic and fully shown, so you can point it at any Dakera instance and audit every number on the chart.

import time, requests, numpy as np

DAKERA = "http://localhost:3000"   # your Dakera binary
NS     = "sift1m-bench"

# --- 1. read SIFT1M .fvecs / .ivecs (standard formats) ---
def read_vecs(path, dtype):
    a = np.fromfile(path, dtype=np.int32)
    dim = a[0]
    return a.reshape(-1, dim + 1)[:, 1:].copy().view(dtype)

base  = read_vecs("sift/sift_base.fvecs",        np.float32)   # (1_000_000, 128)
query = read_vecs("sift/sift_query.fvecs",       np.float32)   # (10_000, 128)
gt    = read_vecs("sift/sift_groundtruth.ivecs", np.int32)     # (10_000, 100)

# --- 2. VALIDATE the harness before trusting it: brute-force L2 vs ground truth ---
def brute_topk(q, k=10):
    d = ((base - q) ** 2).sum(1)
    return set(np.argpartition(d, k)[:k].tolist())

agree = np.mean([len(brute_topk(query[i]) & set(gt[i, :10].tolist())) / 10
                 for i in range(100)])
assert agree >= 0.99, f"harness disagrees with ground truth ({agree:.3f}) — stop"
print(f"harness self-check vs ground truth: {agree:.3f}")

# --- 3. build the Dakera index over all 1M vectors (see /docs/api for fields) ---
requests.post(f"{DAKERA}/v1/namespaces",
              json={"name": NS, "dimension": 128, "distance": "euclidean"})
for i in range(0, len(base), 10_000):
    chunk = base[i:i + 10_000]
    requests.post(f"{DAKERA}/v1/namespaces/{NS}/vectors/bulk",
        json={"vectors": [{"id": str(i + j), "vector": v.tolist()}
                          for j, v in enumerate(chunk)]})

# --- 4. sweep ef_search, measure recall@1/@10 + full latency distribution ---
def recall_at_k(ids, truth, k):
    return len(set(int(x) for x in ids[:k]) & set(truth[:k].tolist())) / k

def pct(sorted_lat, p):
    return sorted_lat[min(len(sorted_lat) - 1, int(len(sorted_lat) * p))]

rows = []
for ef in [10, 25, 50, 100, 200, 400]:
    r1, r10, lat = [], [], []
    for i in range(len(query)):
        t = time.perf_counter()
        r = requests.post(f"{DAKERA}/v1/namespaces/{NS}/query",
            json={"vector": query[i].tolist(), "top_k": 10, "ef_search": ef}).json()
        lat.append(time.perf_counter() - t)
        ids = [m["id"] for m in r["results"]]
        r1.append(recall_at_k(ids, gt[i], 1))
        r10.append(recall_at_k(ids, gt[i], 10))
    lat.sort()
    row = {"ef_search": ef,
           "recall@1": round(float(np.mean(r1)), 3),
           "recall@10": round(float(np.mean(r10)), 3),
           "p50_us":   round(pct(lat, 0.50)   * 1e6, 1),
           "p95_us":   round(pct(lat, 0.95)   * 1e6, 1),
           "p99_us":   round(pct(lat, 0.99)   * 1e6, 1),
           "p99_9_us": round(pct(lat, 0.999)  * 1e6, 1),
           "qps":      round(1 / float(np.mean(lat)))}
    rows.append(row)
    print(f"ef={ef:>3}  r@1={row['recall@1']:.3f}  r@10={row['recall@10']:.3f}  "
          f"p50={row['p50_us']:.0f}us  p95={row['p95_us']:.0f}us  "
          f"p99={row['p99_us']:.0f}us  p99.9={row['p99_9_us']:.0f}us  qps={row['qps']}")

# emit the machine-readable artifact the CI run publishes (SCALE_RECALL_JSON)
import json, os
with open(os.environ.get("SCALE_RECALL_JSON", "scale_recall.json"), "w") as fh:
    json.dump({"dataset": "SIFT1M", "engine": "v0.11.108", "sweep": rows}, fh, indent=2)

Download SIFT1M from corpus-texmex (base, query, and ground-truth files), start the free Dakera binary, and run it. Same dataset, same metric, same knob we swept. Request and response field names track the documented API and may shift between engine versions; the scoring core — ground-truth self-check, recall@1/@10, the p50/p95/p99/p99.9 latency distribution, QPS — is what you're auditing, and it does not depend on us.

An honest note on the default cap

Building this harness surfaced a detail worth stating plainly, because it's exactly the kind of thing vendors bury. Dakera's HNSW index ships with a default max_elements bound of 50,000 per index — a deliberate memory-safety ceiling so an unbounded namespace can't silently exhaust RAM on a small box. This benchmark lifts that bound (via DAKERA_HNSW_MAX_ELEMENTS) so it measures the engine, not the shipped default. In production, a namespace that grows past the default needs that variable raised to keep every memory in the index. Recent builds emit a one-shot warning and a live-evictions metric the first time an index hits the cap, so the tradeoff is observable rather than silent. We treat that as a tunable to document, not a footnote to hide — which is the entire point of running a reproducible benchmark instead of quoting a number.

Accuracy and performance are different questions

To close the loop on the framing we opened with: this post measured retrieval performance at scale — throughput, latency, and recall on a standard ANN dataset. It is not our conversational-memory accuracy number. For how Dakera scores on remembering real conversations, and exactly what that measures and how it's judged, see the companion post on how we benchmark memory on LoCoMo. Two different questions, two honest numbers, each measured on its own terms and reproducible on its own dataset.

Frequently asked questions

Sources — ANN-Benchmarks (SIFT-128-euclidean HNSW reference); Mem0 (arXiv:2504.19413) and Zep retrieval-latency figures self-reported / third-party benchmarked; LangMem P95 from published latency benchmarks; full-context baseline from the Mem0 production-agents study. Dakera figures measured in-house on SIFT1M (128-dim, self-check 0.998) and GIST1M (960-dim, self-check 0.999), v0.11.108, single 8-core x64 runner — reproduce with the harness above.