Multilingual search

Dakera v0.12.0 can store and recall memories in many languages. This page is for deployments whose agents store or query text that is not English: it covers the multilingual embedding model, full-text analysis per language, dates and query routing in seven languages, and the per-request lang field. Four independent settings make that work, and each one is opt-in, so a server that sets none of them behaves as v0.11.108 did (English):

Use them together for a multilingual deployment, or separately: a German-only deployment needs only DAKERA_FULLTEXT_LANGUAGE=de (its query language follows it), while a deployment that mixes languages benefits most from bge-m3 plus DAKERA_FULLTEXT_LANGUAGE=multilingual.

Multilingual embeddings: bge-m3

DAKERA_MODEL=bge-m3 selects BAAI bge-m3, a multilingual model covering 100+ languages.

Why 2048 and not 8192: a full-attention encoder needs heads x tokens^2 x 4 bytes of attention scores per layer for one row. For bge-m3 (16 heads) that is about 4 GiB at 8192 tokens and about 256 MiB at 2048. Raise DAKERA_MAX_SEQ_LENGTH only if you have the memory and long texts.

Constraints

• CPU ONNX backend only. GPU mode (DAKERA_USE_GPU=1) is refused for this model (its FP32 export is an external-data graph the loader cannot fetch), the Candle backend refuses it ("unsupported by the Candle backend"), and the static backend cannot use it (a different vocabulary). Dakera refuses at startup instead of producing wrong vectors.
• Not in the image. The images ship bge-large and the reranker. bge-m3 downloads on first start into the model cache, or ahead of time with dakera models pull bge-m3 (see Models and Docker). The first load converts it once to the memory-mapped ORT form, which needs disk about the size of the model again.
• Not with DAKERA_TIERED=1. The tiered engine always embeds with the default model and ignores DAKERA_MODEL.
• No rollback to v0.11 while it is in use. v0.11.108 does not know bge-m3; dakera downgrade refuses (exit 78, nothing changed) a store embedded with it. Move to a v0.11 model under v0.12 first (see Rolling back to v0.11).

Choosing it on a new store

docker run -d -p 3000:3000 -v dakera-data:/data -v dakera-models:/app/models \
  -e DAKERA_ROOT_API_KEY=dk-mykey \
  -e DAKERA_MODEL=bge-m3 \
  -e DAKERA_FULLTEXT_LANGUAGE=multilingual \
  ghcr.io/dakera-ai/dakera:latest

Switching an existing store

The store records which model embedded it. A request whose model differs from the server's model is 400, because bge-large and bge-m3 are both 1024-d and mixing them would trip no dimension check.

From v0.12.1, take a backup (POST /admin/backups), set DAKERA_MODEL=bge-m3 and restart. The server starts and re-embeds the store in the background, one agent namespace at a time (on the leader in a cluster). No flag and no admin call are needed. Until its turn, each namespace is answered, queries and writes alike, with the model its memories are in, so recall stays correct throughout; the previous model stays loaded beside the new one while the change runs. bge-large to bge-m3 keeps the dimension (1024), so each namespace is re-embedded in place while it serves. A change to a model of another dimension rewrites each namespace through a staging copy, and that namespace's writes answer 503 with Retry-After for the few seconds it takes. Progress is in /health embedding_model_change (from, to, namespaces_switched, namespaces_left), and GET /v1/capabilities reports "reembed_pending": true while it runs. A restart resumes where the change stopped.

Startup is still refused (both models named) where the change cannot run in the background: under DAKERA_TIERED=1, with late-interaction scoring, with the visual lane, or when the recorded model is not in this build's registry. On v0.12.0 every model change is refused this way. There, switch by hand:

1. Take a backup (POST /admin/backups).
2. Start once with DAKERA_MODEL=bge-m3 and DAKERA_ALLOW_MODEL_CHANGE=1. This only acknowledges the change; it re-embeds nothing.
3. Re-embed with an admin key:

curl -s -X POST localhost:3000/admin/namespaces/migrate-dimensions \
  -H "Authorization: Bearer $ADMIN_KEY" -H "Content-Type: application/json" \
  -d '{"target_dimension": 1024, "reembed_same_dimension": true}'

The request takes namespaces (default: every agent memory namespace), target_dimension (default 1024) and reembed_same_dimension (re-embed namespaces already at that dimension, needed here because both models are 1024-d). The response lists migrated, failed, already_current, per-namespace results (status: success, failed or skipped_already_current) and a job_id readable at GET /ops/jobs/{id}. The work runs on its own task, so a request that times out does not stop it. Vectors that clients upserted directly are not re-embedded, since Dakera never held their text.

On this manual path, until every agent namespace has been re-embedded, recall mixes the two embedding spaces and GET /v1/capabilities reports "reembed_pending": true.

Re-embedding does not change the full-text analyzer. If the data is not English, also set DAKERA_FULLTEXT_LANGUAGE: from v0.12.1 existing indexes are re-analysed at the next start; on v0.12.0, rebuild each namespace's index (see below).

Full-text languages

Full-text (BM25) search runs on every memory. DAKERA_FULLTEXT_LANGUAGE chooses its analyzer: from v0.12.1 for every index (an existing one built with another analyzer is re-analysed at the next start), on v0.12.0 for newly created indexes only. Unset, empty or unrecognized means English with stemming, exactly as in v0.11.108. An unrecognized value refuses a fresh install and, on a deployment that already holds data, becomes a config_warnings entry on /health while English stays in effect.

Accepted values are an ISO 639-1 code or the English name, case-insensitive, with an optional region suffix (de, German, pt-BR, de_DE):

These 18 languages use a Snowball stemmer and their own stop-word list (English keeps its existing list). English stop words are content words in other languages (die, was, an), which is why a German deployment must not keep the English list.

No stemming: zh, ja, ko, th (and chinese, japanese, korean, thai), none, off, multi and multilingual. These select an analyzer with no stemmer and no stop words. Use multilingual for a corpus that mixes languages, where any single stemmer would damage the others.

What the analyzer does

• Text is Unicode NFKC-normalized first, so width and composition variants meet: half-width katakana, full-width digits and letters, decomposed kana.
• Combining marks stay inside the word, for every script. Thai tone marks, the Tamil pulli, the Devanagari virama and Arabic vowel marks no longer split a word. Optional Arabic vocalization and the tatweel stretch are dropped, so a vocalized and a plain spelling of a word are one term.
• Token length limits count characters, not bytes.

CJK and other unsegmented scripts: DAKERA_FULLTEXT_CJK_BIGRAMS

Chinese, Japanese, Korean and Thai are written without spaces, so whitespace tokenization turns a sentence into one very long token. In v0.11.108 that token exceeded the token length limit and was dropped: such text was not indexed at all.

With bigrams on, runs of unsegmented script are indexed as overlapping character bigrams (and as single characters). A query of two or more characters searches by bigrams, and a one-character query searches that character, so single-character words are findable. Scripts covered: Han, Hiragana, Katakana, Hangul, Thai, Lao, Khmer, Myanmar and Tibetan (Thai, Khmer, Myanmar and Tibetan clusters keep their combining marks together).

A value that is not a boolean is ignored with a warning and the language default applies.

The analyzer is stored with the index

Each full-text index persists the analyzer it was built with, so an index and its queries always agree.

From v0.12.1, changing DAKERA_FULLTEXT_LANGUAGE or DAKERA_FULLTEXT_CJK_BIGRAMS applies to existing indexes too: at the next start, every full-text index built with another analyzer than the configured one is re-analysed in the background, one namespace at a time, while it keeps serving (it answers as built until its turn). Writes made meanwhile are kept, a restart resumes, and each cluster node does its own (full-text indexes are node-local). An index holding documents that exist only in the index (added with POST /v1/namespaces/{ns}/fulltext/index), or agent memories that cannot be opened at that moment, keeps its analyzer instead of losing them; the server logs it. A switched index is logged as Full-text index re-analysed under the configured analyzer. No setting and no call is needed.

On v0.12.0 a change of these settings affects new indexes only. An existing namespace keeps its analyzer (the server logs that the restored index was built with a different one) until you rebuild it with the call below.

Rebuilding a namespace: POST /admin/fulltext/reindex

Requires a key with the global admin scope (not restricted to namespaces).

curl -s -X POST localhost:3000/admin/fulltext/reindex \
  -H "Authorization: Bearer $ADMIN_KEY" -H "Content-Type: application/json" \
  -d '{"namespace": "_dakera_agent_support-de", "rebuild": true}'

{
  "namespaces_processed": 1,
  "total_indexed": 1520,
  "total_skipped": 0,
  "details": [
    {
      "namespace": "_dakera_agent_support-de",
      "vectors_scanned": 1520,
      "newly_indexed": 1520,
      "already_indexed": 0,
      "parse_failures": 0,
      "rebuilt": true,
      "analyzer_language": "de",
      "text_vectors_resealed": 0
    }
  ]
}

• Without rebuild, and when the analyzer is unchanged, only memories missing from the index are added (a backfill).
• parse_failures counts vectors that are not memories or whose content could not be read; they are never indexed as ciphertext.
• text_vectors_resealed is non-zero only with rebuild and encryption at rest: text documents are rewritten sealed under the current key.
• The call runs inside the request and reads every vector of each namespace it processes. On a large store run it one namespace at a time, in a quiet period.
• Memories are re-analyzed in the language they were written in (_dakera_lang, below).

Query languages and dates

DAKERA_QUERY_LANG sets the language of the rule-based query handling:

• the patterns that classify a question ("when", "how long ago", "after she ...") and so choose how recall weighs dense and keyword search;
• temporal expressions in queries ("last week", "in March 2023");
• dates found in stored content and rule-based date entities.

Supported: en, de, fr, es, it, pt, nl. All seven use one calendar registry for month names, seasons and relative expressions. Values: one of those codes (or its English or native name), or auto.

Examples the date parser resolves (verified by the engine's tests): German "Worüber haben wir gestern gesprochen?" (yesterday), "vor zwei Wochen" (two weeks ago), "im Sommer 2022"; French "il y a deux semaines", "le mois dernier", "au mois d'août"; Spanish, Italian, Portuguese and Dutch have equivalent sets. Names that look like months or seasons are not treated as dates ("Frau Sommer", "Mai hat mir eine Nachricht geschickt").

Auto-detection is opt-in because a statistical detector can misjudge a short English query. Prefer a fixed DAKERA_QUERY_LANG or the per-request lang when you know the language.

A note on classifiers: Dakera's learned query classifier uses English prototypes. With an English-only embedding model, a query that is not English falls back to the rule-based classifier for its language. With bge-m3 the learned classifier applies to every language.

Per-request lang

lang accepts the same values (de, German, deutsch, pt-BR). Omitted or blank means the server-wide choice. Anything else is 400 naming the supported languages. It never translates or filters.

POST /v1/memories/recall/batch uses the server-wide language.

On a write, the canonical code is recorded in the memory's metadata as _dakera_lang. Sentence sub-memories, updates, imports, peer sync, consolidation and POST /admin/fulltext/reindex all resolve a memory's language through that record, so a memory is always parsed the way its write parsed it.

# Store a German memory; the date "vor drei Tagen" is parsed with German rules.
curl -s -X POST localhost:3000/v1/memory/store \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"agent_id": "support-de", "lang": "de",
       "content": "Vor drei Tagen hat der Kunde das Konto gekündigt.", "importance": 0.7}'
# 200 {"memory": {"id": "...", "metadata": {"_dakera_lang": "de", ...}, ...}, "embedding_time_ms": 41}

# Recall in German
curl -s -X POST localhost:3000/v1/memory/recall \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"agent_id": "support-de", "lang": "de", "query": "Wann hat der Kunde gekündigt?", "top_k": 5}'

Recall of Chinese, Japanese and Thai queries

A query in a space-free script has no spaces to count, so v0.11.108 counted it as one word and routed it as a keyword lookup, which weighs the dense lane at 0.25 instead of 0.50 or 0.75. v0.12 estimates the length of such queries from their characters (about two characters per word), recognizes the full-width question mark and routes them like their English equivalents.

Defects fixed in v0.12.0

These affected non-English text in v0.11.108 and need a rebuild of an existing index to take effect (POST /admin/fulltext/reindex with rebuild):

• Eleven accepted stemmer languages had an empty stop-word list.
• Token length limits counted bytes, so long Cyrillic, Greek and Arabic words (over 25 letters) and Tamil and Devanagari words (over 16) were dropped.
• Combining marks split Tamil and Devanagari words (and Thai, Arabic).
• Lao, Khmer, Myanmar and Tibetan text was dropped whole.
• Pseudo-relevance feedback injected non-English stop words into the query.
• CJK queries were routed as keyword queries.

Verify it works

# What the server is configured for
curl -s localhost:3000/v1/capabilities -H "Authorization: Bearer $KEY" \
  | jq '{default_model, fulltext_language, query_languages, reembed_pending,
         bge_m3: (.models[] | select(.name=="bge-m3") | {active, dimension, max_seq_length, effective_max_seq_length})}'

Expect default_model: "bge-m3", bge_m3.active: true, effective_max_seq_length: 2048 (or your DAKERA_MAX_SEQ_LENGTH), fulltext_language as configured ("none" for multilingual and the CJK choices), query_languages: ["en","de","fr","es","it","pt","nl"] and reembed_pending: false once any model switch has finished.

Then:

1. Store two or three memories in the target language and recall with a paraphrase. With bge-m3 a query in another language than the stored text should still find it.
2. For an existing namespace after a language change: on v0.12.1, look for Full-text index re-analysed under the configured analyzer in the log after the restart; on v0.12.0, run the reindex and check analyzer_language and rebuilt: true in the response.
3. GET /health lists a config_warnings entry for an unrecognized DAKERA_FULLTEXT_LANGUAGE or DAKERA_QUERY_LANG on an upgraded deployment; an empty list means every value was honored.
4. After a CJK reindex, a full-text search for a two-character word returns its documents.

Resource implications

• bge-m3: about 570 MB download plus an ORT-format copy on disk; slower per token than bge-large (it has about 568M parameters). Its attention memory grows with the square of the truncation length.
• Stemming, stop words and bigrams add no model and no measurable memory: they change what is indexed. Bigrams index each character as well as each pair, so a CJK namespace's full-text index is larger than the same text would be unsegmented.
• Re-embedding a store embeds every memory again, so budget CPU time in proportion to the number of memories. The background change of v0.12.1 reports its progress in /health embedding_model_change; the manual re-embed runs as a job (GET /ops/jobs/{id}).

See also: Configuration, Models and Docker, Multimodal memory, Search and ranking.