High Availability

Dakera supports production-grade multi-node clustering with gossip-based membership (SWIM protocol), lease-based leader election, consistent-hash sharding, configurable replication, and automatic rebalancing. Nodes discover each other via seed addresses and coordinate through fencing-token-protected leases. Cold-tier data is stored on shared S3-compatible object storage.

Cluster environment variables

Failure modes & recovery

Understanding how the cluster handles failures is critical for operations. Here's what happens in each scenario:

Split-brain prevention

Dakera prevents split-brain using fencing tokens — monotonically increasing integers assigned to each leader lease. When a leader's lease expires and a new leader is elected, the new leader receives a higher fencing token. Any operations from a stale leader with a lower token are rejected by other nodes.

Replication

Data is replicated across nodes based on the replication factor. Each shard's data exists on multiple nodes for durability. The consistency model is eventual consistency with gossip-driven convergence — writes are acknowledged after the primary shard confirms, and replicas converge within milliseconds under normal network conditions.

Scaling procedures

Adding a node

# 1. Start a new node with seed addresses pointing to existing cluster
docker run -d \
  --name dakera-4 \
  -e DAKERA_CLUSTER_MODE=true \
  -e DAKERA_NODE_ID=node-4 \
  -e DAKERA_CLUSTER_SEEDS=dakera-1:7946,dakera-2:7946 \
  -e DAKERA_ROOT_API_KEY=$DAKERA_ROOT_API_KEY \
  -p 3303:3300 -p 7949:7946 \
  ghcr.io/dakera-ai/dakera:latest

# 2. Verify node joined the cluster
curl -H "Authorization: Bearer $DAKERA_ROOT_API_KEY" \
  http://localhost:3300/admin/cluster/nodes
# Node 4 appears with state: "Alive"

# 3. Automatic shard rebalancing begins — monitor progress
curl -H "Authorization: Bearer $DAKERA_ROOT_API_KEY" \
  http://localhost:3300/admin/cluster/status

Removing a node

# 1. Enable maintenance mode (drains traffic and migrates shards)
curl -X POST http://dakera-4:3300/admin/maintenance/enable \
  -H "Authorization: Bearer $DAKERA_ROOT_API_KEY"

# 2. Wait for drain to complete
curl http://dakera-4:3300/admin/maintenance/status \
  -H "Authorization: Bearer $DAKERA_ROOT_API_KEY"
# {"status":"drained","shards_remaining":0}

# 3. Stop the node
docker stop dakera-4 && docker rm dakera-4

# 4. Remove from seed list in remaining nodes (optional — gossip handles it)

Rolling upgrade procedure

Upgrade cluster nodes one at a time with zero downtime:

Docker Compose cluster

The dakera-deploy repo includes a multi-node Docker Compose configuration. A 3-node cluster with Traefik load balancer, Redis cache, and shared MinIO storage:

# Clone and start a 3-node cluster
git clone https://github.com/dakera-ai/dakera-deploy
cd dakera-deploy/ha

# Generate secrets and start
cp .env.example .env  # edit with your keys
docker compose up -d

# Verify cluster health
curl -H "Authorization: Bearer $DAKERA_ROOT_API_KEY" \
  http://localhost:3100/admin/cluster/status

Kubernetes HA

Use the Helm chart with dakera.replicaCount set to 3 or more. The chart automatically configures gossip seeds, Redis, and shared storage:

helm install dakera oci://ghcr.io/dakera-ai/dakera-helm/dakera \
  --namespace dakera --create-namespace \
  --set dakera.replicaCount=3 \
  --set dakera.rootApiKey="$(openssl rand -hex 32)" \
  --set minio.rootPassword="$(openssl rand -hex 16)"

Monitoring HA health

Key metrics to watch in a clustered deployment:

Performance characteristics

Gossip tuning

For most deployments, the default gossip settings work well. Tune these for high-latency or large clusters:

Disaster recovery

For cross-region disaster recovery:

• Backup to remote S3 — schedule daily backups to an S3 bucket in a different region via the admin API (see Deployment → Backup & Restore).
• Full cluster restore — provision new nodes, start with DAKERA_STORAGE=s3 pointing to the backup bucket, and upload the backup bundle via /admin/backups/restore.
• RTO expectation — cluster bootstrap + restore: 5-15 minutes depending on data size.
• RPO expectation — equal to backup frequency (daily = up to 24h data loss). For lower RPO, increase backup frequency or use cross-region S3 replication on the primary bucket.

Admin endpoints

Production checklist

• Odd replica counts — run 3 or 5 nodes for clean quorum math
• Dedicated S3/MinIO — do not co-locate cold storage on Dakera nodes
• External load balancer — place outside the Dakera node pool
• Monitor all nodes — scrape /metrics on every node; alert on gossip membership count and replication lag
• Firewall gossip port 7946 — restrict to cluster nodes only (TCP + UDP)
• Rolling upgrades — use maintenance mode to drain nodes one at a time
• Cross-region backups — replicate to a separate S3 bucket for disaster recovery
• Test failover — periodically stop a node and verify automatic recovery