Rolling back to v0.11
This page is for operators who need to take a v0.12.0 deployment back to v0.11.108. It covers the procedure, the exit codes and report of dakera downgrade, what the command converts, and the cases it refuses.
No new v0.11 release is needed: the v0.12.0 binary carries the rollback. dakera downgrade runs on the stopped deployment, with the same environment and working directory as the server, and rewrites the few things v0.11.108 cannot read into the form v0.11.108 reads. It adds no variable, no flag and no storage name, and nothing has to be prepared before a rollback is needed. The target is v0.11.108; the supported starting point is any deployment that embeds with a v0.11 model, encrypted or not.
stop v0.12 -> dakera downgrade -> start v0.11.108
Procedure
Stop v0.12. Graceful or not; a crash is covered. In a cluster, see Clusters.
Run
dakera downgradeonce, with the same environment and working directory as the server (sameDAKERA_*variables, same volumes), as a one-off job, after the server has stopped. Never run it as the server's owncommandin a rolling update: it refuses to run beside a live server. The image's entrypoint isdakera, so:# Docker docker run --rm <same env and volumes> ghcr.io/dakera-ai/dakera:0.12.0 downgrade # Docker Compose docker compose run --rm <service> downgrade# Kubernetes Job (same env, same volumes as the server's pod) containers: - name: downgrade image: ghcr.io/dakera-ai/dakera:0.12.0 args: ["downgrade"]Read its JSON report (alone on stdout; its log and closing line go to stderr, so
dakera downgrade > report.jsonkeeps just the report). It contains counts,notes,errors,completed,environment_for_v011andmemory_policies_to_reapply.Apply what the report lists (below) and start v0.11.108 on the same data and the same environment.
Any other argument is ignored with a warning, as v0.11.108 ignored every argument: a typo starts the server, and its log says so.
Exit codes
| Exit | Meaning | What to do |
|---|---|---|
0 |
The data is in v0.11.108's form. | Start v0.11.108. |
1 |
Not yet: some values could not be converted (for example under a key this DAKERA_ cannot unwrap), or a step failed (completed: false; what ran before is converted, the rest is not). |
Do not start v0.11.108. Fix the cause and run it again. |
78 |
Refused, and nothing was changed. | Read the message; see below. |
The command is idempotent. Starting v0.12 again on converted data is safe: it re-applies its own formats in the background, as on the first upgrade.
It refuses (exit 78, nothing changed) when:
- it finds no data where the configuration looks (a wrong working directory, an unset
DAKERA_STORAGE_PATH, a volume that is not mounted); - a server runs on the data: one answering on this configuration's port, one holding the data root (every server locks it while it runs), or a live server marker in the S3 bucket (a killed server's marker counts for 2 minutes). While it runs on an S3 bucket it keeps a marker there too, and a server starting on that bucket meanwhile refuses until it is done;
- the deployment embeds with a model or lane v0.11.108 does not have (see What blocks a rollback);
- the store's recorded embedding model is not the one the environment names, a model change is still re-embedding, or agent namespaces hold vectors of another dimension (another model's vectors, or the visual lane's);
- a re-embed that a stop interrupted mid-swap is still parked in a staging namespace (the live namespace is then empty or partial). Start v0.12 once: it finishes such a run at its next start. Then stop it and run the downgrade again.
The report
dakera downgrade always prints one JSON report on stdout. Its fields:
| Field | Meaning |
|---|---|
namespaces_, records_scanned, records_ |
How much it read and rewrote |
contents_, texts_opened |
Memory contents re-sealed as $enc$v1$ under DAKERA_; _text values opened to plaintext |
values_ |
Sealed values it could not open (key not held, moved or damaged); left as they are |
memories_, texts_ |
Memories and text vectors re-embedded in v0.11.108's recipe |
fulltext_, fulltext_, fulltext_, fulltext_ |
Full-text indexes rebuilt as v0.11.108's plaintext base, with the delta log folded in; indexes that could not be rebuilt are listed |
keyring_ |
Bookkeeping of the background re-seal that would mislead the next v0.12 start |
cold_ |
Tiered storage's owed cold writes and deletes flushed |
write_ahead_log, caches_cleared |
What it did with the write-ahead log, and which caches (L2 disk cache, Redis) it emptied |
environment_ |
Boolean variables v0.11.108 reads the other way, with the spelling to give it (see below) |
memory_ |
Namespace to memory policy body, to put again once v0.11.108 serves (see below) |
notes, errors |
Facts to know (for example the DAKERA_DATA_DIR for the knowledge graph); what could not be done |
completed |
Every step ran to its end. false means the run stopped at the last error; what ran before is kept |
Exit 0 requires completed: true, no unreadable values, no unreadable full-text indexes and no errors.
What it converts
| Area | v0.11.108 without the downgrade | After dakera downgrade |
|---|---|---|
Encryption at rest (DAKERA_) |
Serves $enc$v2$ ciphertext as content; full-text search empty |
Every value opened with the keyring (rotations since the upgrade included) and re-sealed as v0.11.108 reads it: $enc$v1$ under DAKERA_; _text and BM25 indexes plaintext |
| Full-text indexes | Misses what was still in v0.12's BM25 delta log (after a crash, or a stop that followed one) | Each index rebuilt complete, delta log folded in |
| Write-ahead log (memory store, filesystem) | Skips entries it cannot parse (records, attachments) in silence; boots empty where v0.12 kept the log under the data root | Snapshotted, truncated, and placed where v0.11.108 reads it (./data/wal, or its DAKERA_WAL_DIR) |
| S3 | Restores its pre-upgrade ./data/wal snapshot into the bucket, reverting changes and bringing deletes back |
That stale log is moved aside |
| Tiered storage | Serves deleted records from cold and misses writes not flushed at a crash | Every owed cold write and delete flushed (the command fails, losing nothing, if S3 does not answer) |
| L2 disk cache, Redis | Serves the copies v0.12 cached ($enc$v2$ content, v0.12-recipe vectors) before the store |
Both emptied (the command fails, nothing lost, if Redis cannot be reached) |
gte-, modernbert- |
Queries in v0.11.108's recipe against vectors in v0.12's | Memories and text vectors re-embedded in v0.11.108's recipe |
Left as they are, because v0.11.108 ignores them and a later upgrade uses them again: multi-vector sidecars and attachments (v0.11.108 serves a record as its dense vector), the keyring, index files and bucket markers.
Environment for v0.11
A deployment first installed with v0.12 reads booleans such as DAKERA_TIERED_STORAGE=yes or DAKERA_GRPC_ENABLED=on as written; v0.11.108 reads those spellings as off. The report lists every such variable in environment_for_v011, with the true or false spelling to set before starting v0.11.108. An upgraded deployment already reads them v0.11.108's way, so nothing is listed.
Memory policies
v0.11.108 keeps memory policies (PUT /v1/namespaces/{namespace}/memory_policy) in memory only and never loads the persisted ones when it starts: after any restart of v0.11.108 a namespace's policy reads as the defaults. The report lists every namespace's policy under memory_policies_to_reapply. Once v0.11.108 serves, put each one again; the listed body is the request body.
Knowledge graph
v0.11.108 keeps its graph on disk only when DAKERA_DATA_DIR is set. The report's notes name the location v0.12 used and the DAKERA_DATA_DIR that keeps its edges under v0.11.108.
What blocks a rollback
The downgrade refuses (exit 78, nothing changed) a deployment embedding with a model or lane v0.11.108 does not have. The models v0.11.108 knows are bge-large, minilm, bge-small, e5-small, modernbert-embed-base and gte-modernbert-base. Any other DAKERA_MODEL made v0.11.108 fall back to bge-large in silence, which would embed queries into a store of other vectors with no error and wrong results. Refused:
bge-m3(multilingual embeddings);colbert-smallorDAKERA_SCORING_STRATEGY=late-interaction;- the visual lane (
DAKERA_VISION).
Move such a deployment to a v0.11 model under v0.12 first: set DAKERA_MODEL with DAKERA_ALLOW_MODEL_CHANGE=1, then run POST /admin/namespaces/migrate-dimensions. Then run the downgrade.
Backups
Backups taken under v0.12 are restored by v0.12: restore, then run dakera downgrade. The keyring is kept, so old and new backups stay restorable.
Clusters
Roll back the whole cluster, not one node at a time:
- Wait until every node's sync outbox is empty (
GET /admin/cluster/status). - Stop every node.
- Run
dakera downgradeon each. - Start v0.11.108 everywhere.
A mixed cluster does not work in this direction: with DAKERA_CLUSTER_SECRET the two versions do not see each other, and a v0.12 node can send a v0.11.108 node encrypted values before it learns its release.
Model volume and containers
- Model volume:
dakera models prune(v0.12) removes the volume's copies of the models the v0.12 image ships, exactly the files v0.11.108 reads there (bge-large and the reranker). v0.11.108 on a pruned volume downloads them again at its first start (about 0.9 GB). Air-gapped, re-seed the volume first, where the Hub is reachable:docker run --rm -v <volume>:/app/models <v0.12 image> models pull --dir /app/models default. - Containers: run
dakera downgradein the v0.12 image with the volumes the server has. v0.11.108's image reads its log at/app/data/wal; an upgraded deployment kept it there. A deployment first installed with v0.12 keeps it under/data: the downgrade does not move it off the volume, and exits1naming theDAKERA_WAL_DIRandDAKERA_WAL_SNAPSHOT_DIRto start v0.11.108 with (both are v0.11.108 variables).
After the rollback
- v0.11 clients and SDKs work as before. Features that exist only in v0.12 (attachments, records,
GET /v1/capabilities) are not available on v0.11.108. On v0.11.108 the REST port binds only after the embedding model has loaded, so keep probe start periods long enough for a model download. - The v0.12 permissions, quota enforcement and gRPC authentication no longer apply.
See also Upgrading from v0.11.108, What's new in v0.12.0 and Deployment.