Upgrading to v0.12 (from v0.11.108, v0.12.0 or v0.12.1)

This guide is for operators moving a v0.11.108, v0.12.0 or v0.12.1 deployment to v0.12.2, the current v0.12 release. It lists the steps to take before the upgrade, what happens on the first start, and every behavior change an existing deployment can notice, with what to do about each. A v0.12.1 deployment needs only the first section; a v0.12.0 deployment needs the first two.

An unchanged v0.11.108 deployment upgrades in place: stop v0.11.108, start v0.12.2 on the same data and the same environment. It starts, keeps its data and answers as before, except for the defects v0.12 fixes on purpose, which are listed below. One exception: a cluster node needs DAKERA_CLUSTER_SECRET from v0.12.1 (Cluster secret). Going back is supported for every deployment on a v0.11 embedding model, encrypted or not: stop v0.12, run dakera downgrade, start v0.11.108 (see Rolling back to v0.11). For a summary of the release, see What's new in v0.12.

Upgrading from v0.12.1 to v0.12.2

Nothing to migrate by hand. Back up first (POST /admin/backups), then replace the image or binary (ghcr.io/dakera-ai/dakera:0.12.2). Every change is in the Changelog under v0.12.2; these are the ones to check before or right after the upgrade:

1. Sessions now end automatically after 4 hours without activity (DAKERA_SESSION_IDLE_TIMEOUT_SECS, default 14400; 0 turns the server default off; at most 30 days). Sessions already open longer than that are closed, with their summaries, on the first passes after the upgrade. An agent that keeps a session open while idle sends POST /v1/sessions/{id}/touch, or starts it with idle_timeout_secs: 0. The timeout can also be changed at runtime with PUT /admin/config {"session_idle_timeout_secs": N}.
2. Keys: an entry ending in * on a key created before v0.12.2 stays a literal name (it granted nothing before and still grants nothing) until the key's namespaces are saved again with PATCH /admin/keys/{key_id}; the dashboard's API Keys page offers "Re-save to activate patterns". Keys no longer need _dakera_sessions for their agents' sessions. Invalid entries (a**b, blank, _dakera_*) are now refused when a key is created or edited. GET /v1/auth/whoami shows how the server reads a key, including its inert_namespaces.
3. Writes that are now refused with 400: a client-supplied dakera-curated tag or _dakera_* metadata key other than _dakera_content_date / _dakera_lang; caller ids of the form mem_s + 24 hex; memory-shaped records written through the raw vector routes of an agent namespace (use /v1/memory/*); agent ids longer than 241 bytes; content over DAKERA_MAX_MEMORY_CONTENT_BYTES (100000 bytes) on update as well as store (a memory already above it may still be updated to content no larger than it is now).
4. Listings: GET /v1/agents/{id}/memories and /wake-up leave sentence sub-memories out unless include_derived=true.
5. Storage: the RocksDB hot tier writes binary records (DAKERA_ROCKSDB_RECORD_FORMAT, default binary; json restores the previous writer). Rolling back to v0.12.0 or v0.12.1 needs nothing; before starting v0.11.108 on this data, run dakera downgrade, which rewrites them as JSON first.
6. On the first start the derivation reconciler heals existing sentence sub-memories in the background (stale and duplicate children deleted, missing ones derived, graph edges rebuilt where they were built from placeholder vectors); GET /admin/derivations/status shows its progress.

Clients: new request and response fields are additive and optional. What a client can notice is listed above: 400 refusals that name the field and the rule, 403 when a Read key ends a session, sessions ended by the server, and listings without sub-memories. The Dashboard 0.5.1 released with v0.12.2 uses its new routes and also works with v0.12.0 and v0.12.1.

Upgrading from v0.12.0 to v0.12.1

Stop v0.12.0 and start v0.12.1 on the same data and environment. Nothing is migrated by hand; the stored record, index and snapshot formats are those of v0.12.0. Upgrading from v0.12.0 straight to v0.12.2 is supported too: follow this section, then the one above. From v0.11.108, follow the rest of this page.

Before you start:

1. Take a backup: POST /admin/backups.
2. Cluster: every node needs the same DAKERA_CLUSTER_SECRET. If your v0.12.0 cluster runs without one, a v0.12.1 node refuses to start: set it and switch all nodes in one window (Cluster secret). A cluster that already has it upgrades node by node as usual. Single-node deployments are not affected.
3. Dashboards and reports: memory totals stop counting sentence sub-memories and drop; session totals count ended sessions and rise (Statistics totals). Tell the owners of dashboards, alerts and usage reports built on these fields.

What happens on the first start (nothing to do):

• Full-text indexes still on the v0.11 analyzer are re-analysed in the background while they keep serving (Full-text indexes). If you already ran the manual rebuild on v0.12.0, there is nothing left to do, and none is found.
• A changed DAKERA_MODEL is no longer refused: the store is re-embedded in the background, and recall stays correct throughout (Embedding model changes).
• Statistics build their counts from the store on first use: the first call per namespace reads it once, later calls answer from the maintained counts.

Check: GET /health/ready answers 200, GET /v1/agents lists every agent, and a keyword search returns hits.

Going back to v0.12.0 is not part of the release checks: restore the backup from step 1 if you need to. Going back to v0.11.108 works from v0.12.1 with the same dakera downgrade.

Upgrade checklist (from v0.11.108)

1. Take a backup: POST /admin/backups.
2. Check the configuration with the new image: run dakera --check-config against the deployment's environment and data volume. It lists every warning the first start will give (see Checking a configuration first).
3. Give gRPC clients an API key when authentication is on (gRPC authentication).
4. Check keys pinned to namespaces and backup tooling against the new permissions model.
5. Check namespace quotas. They are now enforced: a quota set on v0.11 (which ignored it) refuses writes over it (Quotas).
6. Decide about the backup schedule. A stored, enabled schedule starts running at its next slot; disable it first if you do not want that.
7. Point probes at /health/ready (readiness) and /health/live (liveness). The port now binds before the models load; the Helm chart and the compose file already do this.
8. Re-check any threshold on smart_score (or DAKERA_ABSTAIN_MIN_SMART_SCORE) (Recall and ranking).
9. Clusters need DAKERA_CLUSTER_SECRET (v0.12.1): generate one secret, set it on every node, and upgrade the nodes in one window. A cluster node without it refuses to start, and nodes with and without it do not see each other (Cluster secret). An encrypted cluster holds sealed changes for v0.11 nodes until they are upgraded.
10. Model volumes: do not run dakera models prune until you are staying on v0.12 (it removes the files v0.11.108 reads).
11. Memory and session counts change meaning (v0.12.1): memory totals stop counting sentence sub-memories and drop; session totals count ended sessions and rise. Warn the owners of dashboards, alerts and usage reports (Statistics totals).
12. Check the v0.12.2 changes: sessions end after 4 hours idle by default, stricter write validation, key patterns and listings without sub-memories (Upgrading from v0.12.1 to v0.12.2).
13. Stop v0.11.108 and start v0.12.2 on the same data and environment.
14. Full-text indexes need no step on v0.12.1: indexes built by v0.11 are re-analysed in the background at startup. Only if you upgrade to v0.12.0 instead, rebuild them once with POST /admin/fulltext/reindex and {"rebuild": true} (details and commands).
15. Watch /health: config_warnings, degraded, embed_migration, and embedding_model_change if you changed the model (and, with encryption, GET /admin/encryption/status).

Full-text indexes after the upgrade

Full-text indexes built by v0.11 may have no stemmer (a v0.11.108 deployment keeps the analyzer each index was created with). On them a keyword query such as release misses released, and keyword-routed recall can return nothing.

v0.12.1: automatic

At startup, every full-text index built with another analyzer than the configured one is re-analysed in the background, one namespace at a time, while it keeps serving: keyword search finds the stem variants meanwhile, writes made during it are kept, and a restart resumes. Each cluster node does its own (full-text indexes are node-local). An index holding documents that exist only in the index (POST /v1/namespaces/{ns}/fulltext/index), or agent memories that cannot be opened at that moment, keeps its analyzer instead of losing them; the server logs it. Watch the log for Full-text index re-analysed under the configured analyzer. There is no new setting.

v0.12.0: rebuild once by hand

On v0.12.0, rebuild the full-text indexes once after upgrading a deployment that holds data from v0.11.108. Until you do, keyword search and keyword-style recall can return nothing. Fresh installs do not need it. The rebuild re-analyses the indexes under v0.12's text analysis. It only rebuilds derived search indexes; your memories are not touched. On a production deployment with about 18,000 memories it took about 18 seconds. Run it once the server is ready (GET /health/ready answers 200), with a key of global admin scope:

curl -X POST https://<your-dakera>/admin/fulltext/reindex \
  -H "x-api-key: <global admin key>" -H "content-type: application/json" \
  -d '{"rebuild": true}'

Omit namespace to cover every agent memory namespace; add "namespace": "<ns>" to rebuild one. A vector-API namespace needs its own call with {"namespace": "<name>", "rebuild": true}; documents it holds only through POST /v1/namespaces/{ns}/fulltext/index are dropped by a rebuild, so re-index them. The response lists each namespace it processed (see Multilingual search for the fields).

Docker Compose (v0.12.0)

docker compose exec dakera curl -s -X POST http://localhost:3000/admin/fulltext/reindex \
  -H "x-api-key: $DAKERA_ROOT_API_KEY" -H 'content-type: application/json' -d '{"rebuild":true}'

This needs curl inside the container; the v0.12 image's healthcheck uses it. If your image does not have it, run the host-side curl above against the published port (for example http://localhost:3000).

Kubernetes (v0.12.0)

Forward the service port and run the same curl from your workstation:

kubectl port-forward svc/dakera 3000:3000 &
curl -X POST http://localhost:3000/admin/fulltext/reindex \
  -H "x-api-key: $ADMIN_KEY" -H "content-type: application/json" \
  -d '{"rebuild": true}'

Or run it as a one-off Job from an image that has curl, pointing at the in-cluster service URL. Adjust the service name and namespace to your release.

Check that it worked

On v0.12.1 after the background re-analysis, or on v0.12.0 after the rebuild:

• A keyword search returns hits: POST /v1/namespaces/<ns>/fulltext/search with a common word from your data.
• A short keyword recall returns memories.

If either still returns nothing, see Troubleshooting.

What happens on the first start

Nothing below needs an operator step. Recall and ingest are served throughout.

Stored records are written byte-for-byte as in v0.11 (the binary record format is read-only in v0.12). Index snapshots and the BM25 base format are unchanged; HNSW keeps its in-RAM vectors in f16 with no on-disk change.

Embedding model changes

v0.12.1 (no step): set DAKERA_MODEL to another model of the registry and restart. The server starts and re-embeds the store in the background, one agent namespace at a time (on the leader in a cluster). Until its turn, each namespace is answered, queries and writes alike, with the model its memories are in, so recall stays correct. At the same dimension a namespace is re-embedded in place while it serves; at a new dimension it is rewritten through a staging copy, and its writes answer 503 with Retry-After for the few seconds that takes. Progress: /health embedding_model_change (from, to, namespaces_switched, namespaces_left) and GET /v1/capabilities "reembed_pending": true. A restart resumes where the change stopped.

Startup is still refused where the change cannot run in the background: under DAKERA_TIERED=1, with late-interaction scoring, with the visual lane, or when the recorded model is not in this build's registry. There, and for every model change on v0.12.0, start once with DAKERA_ALLOW_MODEL_CHANGE=1 (it only acknowledges the change), then re-embed with POST /admin/namespaces/migrate-dimensions and {"target_dimension": <dim>, "reembed_same_dimension": true} (see Switching an existing store).

Startup and configuration changes

• Explicitly configured features refuse a fresh install instead of running without themselves. DAKERA_REDIS_URL set but Redis unreachable, a DAKERA_DISK_CACHE_DIR that cannot be opened, an explicitly enabled ML classifier that fails, a graph file that cannot be opened, an S3 backup store that cannot be opened, gossip that cannot start, a cluster outbox directory (DAKERA_CLUSTER_OUTBOX_DIR) that cannot be opened. An upgraded deployment starts as v0.11 did, without the component, and lists it as degraded in /health (redis_cache, disk_cache, ml_classifier, knowledge_graph, backups, cluster_gossip, cluster_outbox). Redis is checked every 15 s and leaves the list as soon as it answers.
• Fresh install or upgrade is decided by the data root. A data root that already holds data (a write-ahead log or snapshot, filesystem or warm-tier namespaces, a RocksDB hot tier with data) is an upgrade. With S3 storage and nothing on local disk, the bucket is listed at startup (a namespace there means an upgrade); if it cannot be listed after three attempts the node starts as an upgraded deployment, with a deployment entry in config_warnings, never as a fresh install.
• One server per data root. A server locks its data root (.dakera.lock) while it runs; a second server started on it refuses to start ("REFUSING TO START ... locked by a running Dakera process"). Not locked: plain S3 (nothing on disk) and an in-memory store with DAKERA_WAL=false. The lock is flock: on NFS it holds across hosts only where the mount supports locks.
• The Helm chart sets a pod security context: dakera.podSecurityContext defaults to uid/gid 1000 (the image's user) with fsGroup: 1000 and fsGroupChangePolicy: OnRootMismatch, so a volume whose root is root:root 0755 is writable. On the first rollout Kubernetes chowns a volume whose root does not match, once. Set dakera.podSecurityContext: {} where the platform assigns uids.
• POST /ops/shutdown shuts the server down with the same graceful shutdown as SIGTERM: both ports stop accepting, in-flight requests finish, then the shutdown flushes run and the process exits. It needs a key of global admin scope. A container or service manager restarts it according to its restart policy; with none, the server stays down. Any tooling that called it as a health or smoke test must stop doing so. The graceful shutdown now drains gRPC as well as REST.

Health endpoints

The REST port now answers while the server starts. v0.11 bound the port only after the embedding model had loaded, so a model that was not in the image (bge-m3, colbert-small) downloaded behind a closed port, and a Docker HEALTHCHECK or Kubernetes liveness probe restarted the process.

Every other request answers 503 with Retry-After: 5 until the models are loaded. A client that waited for the connection to be accepted should wait for /health/ready (or retry the 503). A download that was cut short continues where it stopped when the server serves byte ranges. The images' HEALTHCHECK start period is now 10 minutes, and the Helm chart's liveness probe reads /health/live behind a 10-minute startup probe.

Checking a configuration first

dakera --check-config runs the startup configuration check alone (same environment, same data root, same decision), prints what it found and exits 0 (the server would start, warnings included) or 78 (it would refuse). It starts nothing and writes nothing.

docker run --rm -e DAKERA_ROOT_API_KEY=... -v dakera-data:/data \
  ghcr.io/dakera-ai/dakera:0.12.2 --check-config

From v0.12.1 it also reports a cluster node without DAKERA_CLUSTER_SECRET, or with one shorter than 16 characters. Once running, the gauge dakera_config_warnings counts the config_warnings entries; alert on > 0.

Security and access

gRPC authentication and validation

v0.11.108's gRPC port (DAKERA_GRPC_PORT, default 50051) had no authentication. With DAKERA_AUTH_ENABLED on (the default), every RPC except Health now needs a key the REST API accepts, sent in the call metadata:

x-api-key: <key>
authorization: Bearer <key>

• Without a key the call fails UNAUTHENTICATED; a key without the scope or namespace the RPC needs fails PERMISSION_DENIED, as the REST twin answers 401 / 403.
• Every authenticated call takes a token from the server-wide rate limit (DAKERA_RATE_LIMIT_RPS / DAKERA_RATE_LIMIT_BURST); when spent the call fails RESOURCE_EXHAUSTED.
• Every RPC is bounded by DAKERA_REQUEST_TIMEOUT, and Query by query_timeout_ms of PUT /admin/config (DEADLINE_EXCEEDED).
• While maintenance mode rejects requests, every RPC but Health fails UNAVAILABLE. Calls are written to the audit log.
• gRPC validates like REST: Query with top_k unset (proto3 sends 0) is INVALID_ARGUMENT (set top_k between 1 and 10,000); an unknown distance_metric is INVALID_ARGUMENT; an Upsert whose metadata_json is not valid JSON is INVALID_ARGUMENT.

What to do: give every gRPC client a key in its metadata before upgrading, or keep DAKERA_AUTH_ENABLED=false if the deployment ran without authentication.

Permissions

A key pinned to namespaces (for example an admin key minted by POST /v1/namespaces/{ns}/keys) used to pass every check that asked for a scope only. These routes now answer 403 where v0.11.108 did not:

Deployments with authentication off (every request is super_admin) see none of these. An unrestricted admin key can no longer download, upload or restore a backup bundle: use a global super_admin key. POST /ops/shutdown needs a key of global admin scope.

Other security changes

• Sealed values from clients are refused. A vector write whose _text or content metadata starts with $enc$v1$ or $enc$v2$ is refused (400; gRPC INVALID_ARGUMENT), because storing a client-supplied ciphertext made the server a decryption oracle.
• Importing a v0.11 export of an encrypted deployment needs a key of global admin or super_admin scope (or authentication off) and the same DAKERA_ENCRYPTION_KEY that sealed it. Exports taken from v0.12 carry plaintext and import with any key allowed to write the agent.
• With authentication off, CORS allows loopback origins only by default, and a state-changing request from a web page of another origin is refused with 403 CROSS_ORIGIN_REQUEST_REFUSED. SDKs and curl are unaffected.
• Knowledge graph edges are per agent; the existing graph is migrated on the first start. POST /v1/memories/{id}/links answers 404 unless both ids are live memories of the agent, and path searches (GET /v1/memories/{id}/path, GET /v1/knowledge/path) stop at 10 hops or 4096 memories and answer 400 saying so.
• Audit body capture (DAKERA_AUDIT_REQUEST_BODY, DAKERA_AUDIT_RESPONSE_BODY) and DAKERA_AUDIT_FORMAT=text now work; with the defaults the audit output is unchanged.
• Raw vector APIs return memory content as stored (unchanged from v0.11): /v1/namespaces/{ns}/query, batch and multi-vector query, /v1/namespaces/{ns}/export, /v1/namespaces/{ns}/records/{id} and gRPC Query return an agent memory's content metadata in its at-rest form (z64: for content over 256 bytes, $enc$v2$ with encryption). Read memories through the memory API, which returns plaintext.
• A caller-chosen extractor base_url is not fetched through the system proxy, and provider responses are capped at 4 MiB. An uploaded backup bundle may inflate to at most 16 times DAKERA_MAX_BODY_SIZE (413 beyond).
• Retired operator key masters can be revoked with POST /admin/encryption/keys/{key_id}/revoke-master (global admin). Deleting a namespace removes its own encryption key assignment.

Storage, data and backups

Quotas are enforced

v0.11 stored, returned and replicated namespace quotas (/admin/quotas..., and max_vectors_per_namespace of PUT /admin/config) but no write consulted them. Every write path (REST, gRPC, memory, import, re-embed, background and admin writes) now goes through one quota check. With the default hard enforcement, a write that would take a namespace over max_vectors, max_storage_bytes, max_dimensions or max_metadata_bytes is refused with 413 QUOTA_EXCEEDED before anything is written. soft logs a warning and writes; none only tracks. A deployment that set quotas on v0.11 and relies on them not biting should raise them, or set enforcement to soft or none, before upgrading.

Backups

• A stored backup schedule now runs. A backup is taken at every slot of its cron (UTC; on the leader in a cluster) and scheduled backups beyond retention_days / max_backups are deleted (the newest completed one is always kept; manual and uploaded backups are never deleted by it). An upgraded deployment whose stored schedule is enabled gets a backup_schedule entry in config_warnings; the first backup is taken at the first slot after the start. To keep v0.11's behavior, send POST /admin/backups/schedule with {"enabled": false} before that slot.
• Restore without overwrite behaves as in v0.11; with "overwrite": true it gives the exact point in time of the backup. A restore no longer restores the backup schedule: the live schedule is kept.
• A backup interrupted by a stop is marked failed at the next start instead of staying inprogress.
• Restore does not pause serving: restore in a maintenance window.

Other storage changes

• RocksDB hot-tier writes are fsynced (DAKERA_ROCKSDB_SYNC, default true).
• Concurrent writes into one namespace share the write-ahead log's fsync.
• Memories report the TTL they were stored with on every backend; spaced-repetition TTL extensions now reach the TTL sweep.
• Forget removes a memory's sentence sub-memories, and TTL expiry removes expired memories from full-text search, the graph and the counters.
• A write whose client gave up still applies, and replicates. In a cluster it is now replicated to the peers like any write, with its tombstone for a delete, instead of waiting for the next reconciliation (up to 5 minutes). A timed-out write may have taken effect: read it back before retrying a non-idempotent change (memory stores with an id and deletes are safe to retry). A storage backend that stops answering mid-write is no longer cut loose by the request timeout: the write waits for it, is logged at error every 30 s and counted in dakera_storage_write_stalls_total; alert on any increase.
• Backup schedule details. The schedule runs with its namespaces, type, encrypt and compression. A schedule with an invalid expression, or an encrypt / compression a backup cannot honor (lz4, encrypt without DAKERA_ENCRYPTION_KEY), is refused with 400 when set. A stored one with an invalid expression logs an error and takes no backup. DELETE /admin/backups/{id} accepts an inprogress backup that no node is taking.
• A restore keeps the live schedule. When the backup's schedule differs, the restore logs it, and the operator adopts it with POST /admin/backups/schedule. Revoked API keys stay revoked, and keyring records are restored only where missing.
• Session ids belong to their agent. POST /v1/sessions/start with the id of another agent's session answers 409 (v0.11 replaced that session, owner included).
• Deep agent listings are cheaper. GET /v1/agents/{id}/memories pages cost in proportion to offset + limit (they grew with the square of offset), and a page read while the agent is written lists each memory once.
• Import and export are complete. POST /v1/import obeys the 100,000-byte content limit: an item over it is refused and named in the job's errors. GET /v1/export carries plaintext content and each memory's TTL, and no longer exports sentence sub-memories.

Recall and ranking

Recall no longer ranks what earlier recalls returned higher. v0.11 raised the importance of every memory a recall returned and scored the access count; both are query-independent, so the results of one query crowded out more relevant memories in every later one. In v0.12 a recall records access_count and last_accessed_at but changes neither importance nor the score. The score's recency term counts from the memory's created_at. POST /v1/memory/feedback (upvote) still raises importance.

smart_score values move with this. The score is w_vec * relevance + w_imp * importance + w_rec * recency; the weights sum to:

v0.11 gave the rest to the frequency term. A memory that no recall, feedback or session promotion had touched scores exactly as in v0.11; one that v0.11 recalls had returned scores up to 0.27 lower, and one last used long after it was stored up to 0.13 lower. An absolute cut on smart_score calibrated on v0.11 (DAKERA_ABSTAIN_MIN_SMART_SCORE, or a client filtering on the score) can drop such memories: re-check it against v0.12's scores. Relative order within a query is what changed on purpose.

Other behavior changes:

• Importance decay no longer compounds. Decayed importances are higher than v0.11 would have left them.
• Auto-pilot dedup merges only true duplicates, and POST /v1/knowledge/deduplicate uses the same safeguards (so duplicates_merged can be lower than duplicates_found).
• Recall admission: at most two recalls per core (at least four) run at once; the rest queue and answer 503 with Retry-After only after 20 s without progress (at most half of DAKERA_REQUEST_TIMEOUT when that is shorter).
• An omitted distance_metric resolves to the namespace's metric instead of always cosine.
• The rerank report is rerank_report (applied, candidates, skipped); rerank in a request stays the on/off switch.
• Page caps are stated in x-dakera-limit; POST /v1/memories/recall/batch returns at most 10,000 memories.

More recall and write behavior

• Feedback, importance and update writes apply to the memory as stored. POST /v1/memory/feedback, POST /v1/memories/{id}/feedback, POST /v1/memory/importance, PATCH /v1/memories/{id}/importance and PUT /v1/memory/update/{id} used to write back their read copy, losing an access count or content update that landed meanwhile. They now write only over an unchanged memory and retry; a memory that keeps changing through five attempts answers 503 with Retry-After: 1. On an expired memory they answer 404. One that would take the namespace over its storage quota is refused.
• Importance set by a client decays from the moment it was set. An importance a client sets, or feedback changes, decays from that moment. Importances boosted by past v0.11 recalls are kept and decay from there. GET /v1/agents/{id}/wake-up still orders by importance and recency of use.
• Consolidation refuses memories it would lose data from. POST /v1/memory/consolidate with explicit memory_ids answers 400 for a memory with an attachment, a TTL, user metadata or that has expired.
• Admission queueing ends by the 20 s mark. The 20 s counts the whole request: a recall that waited for its slot, or a store that waited for memory or a model load, waits only what is left of it. 20 s, not 30 s, because every official SDK times a request out at 30 s. POST /v1/memories/recall/batch now waits for an admission slot like every recall.
• Stored memories are embedded on the document side, and gte-modernbert-base and modernbert-embed-base are corrected against upstream, so their vectors are re-embedded by the migration.

Errors and API responses

Statistics totals (v0.12.1)

The count endpoints disagreed with each other: one agent that stored one 3-sentence memory was reported as 5, 4 or 5 "memories" depending on the endpoint, because each counted different records: the memory, the sentence sub-memories Dakera derives from it for keyword search (one per sentence, created in the background for a memory of two sentences or more), and the namespace's bookkeeping records. v0.12.1 uses one definition everywhere:

• a memory is a top-level memory an agent stored;
• a sub-memory is a sentence record derived from one;
• a vector is any stored record.

Field names and types do not change; the values of these fields do:

New fields only: GET /v1/ops/stats total_memories, internal_vectors, internal_namespaces, agent_namespaces; GET /v1/analytics/overview total_memories, internal_vectors. Summed over agents, GET /v1/agents memory_count equals total_memories. GET /v1/agents also lists agents that have memories but no session (they were missing).

What to expect after the upgrade. Memory counts drop, by roughly the average number of sentences per memory for agents that store long, multi-sentence memories, and session counts rise, because ended sessions are counted. No data changes: storage, recall and namespace quotas (which count stored records, max_vectors) are unaffected.

What to do. Dashboards, alerts, usage or billing reports and any text that shows "N memories" from these fields will step on upgrade day: annotate the step, and re-base thresholds on the new figures. A consumer that needs the previous GET /v1/agents figure reads vector_count. Clients parse the responses unchanged (only fields were added).

Other error and response changes

• Every error body is JSON ({"error", "code", "status"}), including framework refusals: invalid body (400), wrong shape (422), wrong content type (415), unknown path (404 ROUTE_NOT_FOUND), wrong method (405), timeout (408) and the server-wide rate limit (429).

• Every 503 carries Retry-After. Two refusals that no retry can change are now 501 NOT_IMPLEMENTED with the settings to change in details: DAKERA_SCORING_STRATEGY=late-interaction on a model without a late-interaction recipe or under DAKERA_TIERED=1, and an invalid DAKERA_EXTRACTOR_* configuration.

• A body over the size limit is 413 (attachments over DAKERA_ATTACHMENT_MAX_BYTES, imports over DAKERA_MAX_BODY_SIZE).

• Namespace config: PATCH /v1/namespaces/{ns}/config merges what it is sent and answers 400 naming an unknown field; PUT is a full replacement (omitted entity_types clears the list). Persisted memory policies are re-applied at startup (v0.11 lost them at every restart).

• PUT /admin/config settings take effect (max_vectors_per_namespace, cache settings, rate limit, query_timeout_ms), are re-applied at startup over their environment variable, are listed in runtime_overrides and replicate to every cluster node. query_timeout_ms is a deadline on query, recall and search routes and gRPC Query (504 QUERY_TIMEOUT; default 0 = none). default_index_type accepts only hnsw; GET reports the effective values.

• 404 bodies name their resource, and error text is redacted everywhere. resource is one of namespace, vector, memory, session, job, attachment, path, encryption_key, api_key. Memories, sessions, jobs, attachments and graph paths keep the v0.11 code VECTOR_NOT_FOUND, and their error loses the "Vector not found:" prefix. An unknown API key is API_KEY_NOT_FOUND; an unknown /ops/jobs/{id} is JOB_NOT_FOUND (v0.11 answered a bare null). Every 403's details names scopes as keys are created with them (required: admin, actual: read).

• New error codes: ROUTE_NOT_FOUND, METHOD_NOT_ALLOWED, UNSUPPORTED_MEDIA_TYPE, REQUEST_TIMEOUT. Their statuses are unchanged (the timeout stays 408), and the headers (Allow, Retry-After, X-RateLimit-*) are kept. The authentication middleware's 401 keeps its v0.11 fields and gains code and status.

• The re-embed drain answers inside the request timeout. POST /admin/reembed/drain answers after at most 4/5 of DAKERA_REQUEST_TIMEOUT with timed_out: true; a second drain is 409 CONFLICT (was 503).

• Validation names its rule; ../ is accepted in text. Metadata values, tags and filter values containing ../ or ..\ are no longer refused as path traversal (ids and keys still are).

• Maintenance mode is held by the node that receives POST /admin/cluster/maintenance/enable: not replicated, not persisted, and it refuses requests only on that node, only when node_ids is empty or lists it. To take several nodes into maintenance, send the request to each. Its 503 carries Retry-After; reject_requests is applied; POST /admin/cluster/maintenance/disable without force is 409 while jobs run.

• Smaller API corrections: the importance routes answer 409 for a memory whose content does not open on this node; namespace event streams are capped at 128 per key (1,024 per process); POST /v1/namespaces/{ns}/explain with execute: true returns real actual_stats; an admin import into another agent's namespace rewrites the memories' agent_id to that agent.

Logs, metrics and alerts

• KPI and analytics figures count memory operations (v0.12.1). Batch recall, batch store and memory search were not counted, and every HTTP request (health probes and admin scans included) fed the query latency of /v1/analytics/*. Recalls (single and batch) and searches are now queries with their own latency, stores (single and batch) feed the store latency, and the analytics latency covers queries only. GET /v1/kpis gains a latency object (window_seconds, recall_samples, store_samples, recalls_total, stores_total). GET /admin/cache/stats and the cache figures of GET /v1/analytics/overview include the in-process cache, not only Redis.
• Quieter logs. Per-request INFO lines of the data path are DEBUG; the audit log still records every request. A request without a valid key is a DEBUG line with at most one WARN per minute (dakera_errors_total{type="auth"} counts every one). Log messages no longer carry internal ticket names: alerts matching those words need updating (for example SEC-3 keyring is now encryption keyring). The compose dev profile logs at info.
• Refusals are counted. dakera_http_requests_total and dakera_http_request_duration_seconds now include 429, 401 / 403, 408, 413 and unmatched paths (path="unmatched"), so expect those statuses in dashboards; the traffic itself is unchanged. dakera_active_requests no longer drifts on disconnects.
• New metrics: dakera_model_loads_total{model,outcome}, dakera_model_load_duration_seconds{model}, dakera_backups_total{outcome}, dakera_wal_write_failures_total, dakera_memory_read_failures_total, dakera_cache_redis_failed_writes_total, dakera_cache_redis_failed_publishes_total, dakera_cache_redis_failed_evictions_total, dakera_cluster_sync_held_entries, dakera_memory_budget_reserved_bytes, dakera_component_degraded{component} and dakera_config_warnings. wal_replay in /health degraded means the write-ahead-log replay dropped entries the store refused.
• Alert rules and dashboard: monitoring/dakera.rules.yml in dakera-deploy monitoring/ (v0.12.0 versions arrive with the dakera-deploy v0.12.0 release) covers these signals plus availability rules (target down, 5xx and 503 ratios, memory-budget refusals, cold-tier circuit and backlog, RocksDB write stop, dropped cluster changes, clock skew, corrupt records). The Grafana dashboard reads only metrics the server emits; panels that read names nothing emitted are gone.

Cluster upgrade

Cluster secret (required from v0.12.1)

What changes. Every node in cluster mode (DAKERA_CLUSTER_MODE=true) needs DAKERA_CLUSTER_SECRET: at least 16 characters, the same value on every node. It authenticates every node-to-node request (/internal/*, sent in a header) and every gossip packet (HMAC-SHA256 with a send time and a nonce, so replays are dropped). It authenticates; it does not encrypt, so keep node-to-node traffic on a private network or behind TLS. Single-node deployments (cluster mode off) are not affected.

Why it is required now. v0.11 had no secret: anyone who reached a node's API port could call the node-to-node routes (POST /internal/sync/key/add minted a SuperAdmin API key), and anyone who reached the gossip port could join the cluster. v0.12.0 required the secret on fresh installs only and let an upgraded node start without one, still open. From v0.12.1 a cluster node without the secret refuses to start, fresh install or upgraded.

What a node without it does. It exits with code 78 before it serves anything, logging:

cluster mode needs DAKERA_CLUSTER_SECRET (at least 16 characters, the same on every node): it authenticates every node-to-node request (/internal/*, gossip), …

dakera --check-config reports the same (DAKERA_CLUSTER_SECRET missing, or shorter than 16 characters), so run it first.

Generate and set it.

openssl rand -hex 32        # 64 characters; any value of 16 or more works

Store it like any other credential (a Kubernetes Secret, your secret manager, an .env file outside version control) and pass the same value to every node as DAKERA_CLUSTER_SECRET. The repository's docker-compose.yml has a development default (dev-cluster-secret-change-me): set your own.

Upgrade procedure. Nodes with and without the secret cannot talk: a node with it answers a node without it with 401 and drops its gossip, and a node without it cannot read authenticated gossip. So the cluster must not run half-and-half for long.

Check it worked. On every node, GET /admin/cluster/status lists every other node as a healthy member and the same leader, and the log has no refused a node-to-node request without the cluster secret warnings (a node logs that when a peer sends the wrong secret or none). For replication, write on one node and read on another.

Rotating the secret needs the same window: a node accepts one secret, so nodes on the old and new values are split until all have the new one.

Rolling back to v0.11.108: v0.11 ignores the variable; roll the whole cluster back together (Rolling back: clusters).

Node id and mixed v0.11 / v0.12 periods

• DAKERA_CLUSTER_NODE_ID stays required on a fresh install. An upgraded node without one (unset, or set but empty) no longer takes node-<pid> (a new member on every restart): it generates one once and keeps it in {data root}/cluster-node-id, so it stays the same member across restarts, and is degraded in /health (cluster_auth) until it is set. It falls back to node-<pid> only when that file cannot be written.
• Mixed v0.11 / v0.12 nodes. On v0.12.1 the v0.11 nodes are in the other group until they are upgraded (above). The next three items describe a mixed period on v0.12.0, where an upgraded node could still run without the secret.
• An encrypted cluster (DAKERA_ENCRYPTION_KEY) upgrades node by node, but v0.12 nodes hold changes carrying $enc$v2$ text for peers that still run v0.11, in a durable outbox, and deliver them once the peer runs v0.12 (/health lists cluster_sync_held; the metric is dakera_cluster_sync_held_entries). Clients reading from a v0.11 node do not see what was written on v0.12 nodes until it is upgraded. Deletes and key revocations are not held.
• During a mixed period, a v0.12 node applies what a v0.11 node sends as made when it arrives, as v0.11 does. A change a v0.11 node has no route for (namespace and runtime configuration, attachments, record writes) is kept in the sending node's durable outbox and delivered once that node runs v0.12; it holds none of the node's other changes back.
• Encrypted clusters: v0.12 nodes tell v0.11 peers apart by the X-Dakera-Peer-Protocol header. Changes without encrypted text are not held: API key deletes and deactivations, quotas, memory, namespace and attachment deletes reach the v0.11 nodes, so a key revoked on a v0.12 node stops authenticating there too. With a stable DAKERA_CLUSTER_NODE_ID the held changes are delivered when the upgraded node answers; without one, the upgraded node gets the data from its startup reconciliation instead. The outbox holds at most 64 MiB; what it evicts is repaired by reconciliation.

Storage, backups and timings

• Cluster nodes must not share one S3 bucket as their store. Every cluster node on S3 keeps a heartbeat marker in its bucket. A fresh bucket that finds another node's live marker refuses to boot (exit 78); an existing one starts with a warning and shared_store degraded. The repository's compose ha profile now gives each node its own bucket.
• Backups with one bucket per node: a backup's files live in the backup store of the node that took it (node_id in GET /admin/backups). Restore it through that node, or give the nodes one shared backup bucket.
• The compose ha profile now gives each node its own bucket: dakera-1 keeps dakera, dakera-2 and dakera-3 use the new buckets dakera-2 and dakera-3. To move an existing compose deployment, stop it (docker compose --profile ha down, volumes kept), pull the new file and start it again; dakera-2 and dakera-3 start on empty buckets and are filled from their peers by the startup reconciliation (watch GET /admin/cluster/status, replication.startup_reconciled, and compare vectors/count across the nodes). Set your own DAKERA_CLUSTER_SECRET: the compose files carry a development default.
• Shared-bucket check: each node's marker is _dakera_cluster/nodes/<node id>.json, refreshed every 30 s; another node's marker younger than 120 s means the bucket is shared. Standalone replicas (cluster mode off) sharing a bucket are not checked.
• Deleting a backup that one node took also deletes its files: the node that took it removes them from its bucket when the delete reaches it. Backups taken before the upgrade have no node_id.
• Cluster timings now all follow DAKERA_ELECTION_LEASE_MS (default 15000, accepted 1000 to 600000).

Environment variables

Names that now only warn

Set by v0.11 deployments but never read, or no longer read. On an upgraded deployment each is a warning; on a fresh install it refuses the boot. Remove them, or set what the right column says.

Removed variables

Read by a v0.11 release, not by v0.12 (a warning on an upgraded deployment):

Variables added in v0.12

All opt-in or deploy inputs; a deployment that sets none behaves as before apart from the changes above. DAKERA_HOT_TIER is not new, but its default changed to rocksdb (memory keeps the v0.11 tier).

The cluster timings all follow DAKERA_ELECTION_LEASE_MS, accepted between 1000 and 600000 (a value outside is invalid like any other); scheduled reconciliation runs every 5 minutes. RATE_LIMIT_RPS and RATE_LIMIT_BURST still work as deprecated aliases, with a warning. Several tuning variables that appeared during v0.12 development (cluster timings other than the lease, DAKERA_RECALL_TRACK_ACCESS, DAKERA_EMBED_* threading knobs and others) were removed before release. A deployment that ran a v0.12 pre-release and still sets one starts with a warning if it holds v0.11 data; a deployment that a pre-release installed refuses to start naming the variable (remove it, or start once with DAKERA_CONFIG_LENIENT=1).

Names that do nothing where they are set

A variable set where it has no effect is logged and listed in config_warnings as env_ignored, never refused: DAKERA_MODEL other than bge-large together with DAKERA_TIERED=1 (the tiered embedding engine always embeds with bge-large; tiered storage is DAKERA_TIERED_STORAGE), tiered-storage settings without DAKERA_TIERED_STORAGE=true, DAKERA_NODE_ID different from DAKERA_CLUSTER_NODE_ID, and DAKERA_GRPC_PORT with gRPC off.

Booleans

A fresh v0.12 install reads 1/true/yes/on and 0/false/no/off (trimmed, any case) as written on every boolean variable. v0.11.108 read some spellings the other way (DAKERA_GRPC_ENABLED=yes disabled gRPC, DAKERA_WAL=off left the log on, DAKERA_AUTH_ENABLED=no left authentication on). An upgraded deployment keeps v0.11.108's reading for this release; each such value is listed in config_warnings (env_reading) with the true or false spelling that both versions read alike.

One value refuses on every deployment: DAKERA_WAL_SYNC=periodic:0, a write-ahead log that never fsyncs. Set everywrite, periodic:N with N greater than 0, or manual.

Docker model store and air-gapped installs

The default image ships bge-large and the reranker, already converted, and needs no network (--network none works), writes nothing at start, and works with a read-only root filesystem. Anything else is downloaded on first use into the model cache (HF_HOME, /app/models in the images).

• A model volume kept from v0.11 holds copies of the models the image now ships (about 0.9 GB): dakera models prune frees them, but they are what v0.11.108 reads, so prune only once you are staying on v0.12.
• The reranker and GLiNER are now pinned by SHA-256. A v0.11 cache may hold another upstream revision: v0.12 replaces it at first load. With HF_HUB_OFFLINE=1 nothing can be fetched, so fetch the pinned files where the Hub is reachable (dakera models pull --dir <model cache> reranker gliner) and copy them in.
• Downloads honor HTTPS_PROXY, HTTP_PROXY, ALL_PROXY, NO_PROXY, HF_ENDPOINT (a mirror) and HF_HUB_OFFLINE. v0.11 connected to huggingface.co directly whatever the environment said.
• To seed an air-gapped host, run docker run --rm -v "$PWD/models:/seed" ghcr.io/dakera-ai/dakera:0.12.2 models pull --dir /seed bge-m3 whisper vision on a connected machine, copy models/ over, mount it at /app/models and set HF_HUB_OFFLINE=1.

The full reference is in Model store and air-gapped installs.

Models, media and inference

• A store into a GLiNER namespace no longer waits for a first download: the model loads in the background, and a store waits up to 10 seconds, then answers 503 with Retry-After. Run dakera models pull gliner ahead of time.

• Speech to text defaults to whisper-base. Speech is new in v0.12.0 and opt-in. The default model is whisper-base (multilingual, language auto-detected), not English-only whisper-tiny.en: a deployment that turns on attachments and sets no DAKERA_WHISPER_MODEL transcribes with whisper-base. Set DAKERA_WHISPER_MODEL=whisper-tiny.en to keep the lightest English model.

• Idle models (the visual model, whisper, GLiNER) are unloaded after DAKERA_HNSW_CACHE_TTI_SECS (default 1 hour).

• Media jobs and entity extraction reserve their estimated peak memory against the memory limit before they start, and answer 503 with Retry-After when it does not fit. DAKERA_MEM_BACKPRESSURE=0 turns these refusals off.

• The reranker and GLiNER are pinned by SHA-256; the first load replaces another upstream revision with the pinned file (a download of about 570 MB and 780 MB). The one-time ORT-format conversion of a model's first load reserves its peak from the memory budget: where it does not fit, the request or job is answered 503 naming the conversion's size, the budget and the remedy, and /health shows the model as degraded. A server with about 2 GiB and entity types should run dakera models pull gliner once.

• Proxy variables: the proxy is chosen as curl chooses it (https_proxy first for the Hub's URLs, all_proxy only when neither is set); HTTP and SOCKS (4, 4a, 5, 5h) proxies work. A proxy value the downloader cannot use (an https:// proxy URL, an unknown scheme, an IPv6 proxy address) fails the download naming the variable; use the proxy's http:// address. Add huggingface.co to NO_PROXY if the proxy cannot reach it.

• Media jobs (new in v0.12): the first job of a model loads it in the job, so the 202 no longer waits for a download; progress reflects the work; a failed job carries error: {status, code}; ids are job_<run>_<n>, and a poll of a job lost to a restart answers 404 JOB_NOT_FOUND. In a cluster any node answers a job poll. A job with no id stores under a derived id, so a resubmission replaces instead of duplicating. Media that cannot be decoded (a non-WAV transcription, an image over the pixel limit) is refused with 400 before a job exists.

• Vision lane: the visual model's prompt cap is its own (2,048 tokens for colmodernvbert); DAKERA_MAX_SEQ_LENGTH applies to the text models only.

Notes for upgrading

• v0.11 SDKs and the startup window: wait on GET /health/ready before sending traffic to a starting server.
• v0.11 Go SDK and namespace config: clear entity_types with PUT /v1/namespaces/{ns}/config.
• Persistent storage: keep the data root on a persistent volume, also with S3 storage.
• Metrics: about 25 Prometheus series per active agent namespace; size the scrape for your agent count.
• Release gate: every release is published only after the upgrade, topology, cluster and rerank checks and the benchmark gates pass on the tagged commit.

See also What's new in v0.12, Configuration, Deployment and Security.