> For the complete documentation index, see [llms.txt](https://docs.envector.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.envector.io/1.4.x/operations-and-management/high-availability.md).

# High Availability & Failover

This document describes the **Kubernetes-local HA model** for enVector. Cross-region or cross-cluster active/standby designs are out of scope.

***

## 1. HA Topology Overview

The table below describes the **HA-enabled** topology (`election.backend: k8s-lease`). Without HA, `compute` and `shaper` deploy as `Deployment` instead of `StatefulSet`.

| Service        | HA Pattern            | Workload      | Election           |
| -------------- | --------------------- | ------------- | ------------------ |
| `endpoint`     | Active-Active         | `Deployment`  | None               |
| `orchestrator` | Leader-only           | `StatefulSet` | Kubernetes `Lease` |
| `backend`      | Leader-only           | `Deployment`  | Kubernetes `Lease` |
| `compute`      | Active-Standby (cold) | `StatefulSet` | Kubernetes `Lease` |
| `shaper`       | Active-Standby (cold) | `StatefulSet` | Kubernetes `Lease` |

Common building blocks:

* **`Lease`-based leader election** for components owning singleton or stateful work
* **Leader label + Service selector** so traffic automatically follows the active pod
* **Process exit on leadership loss** for predictable failover semantics
* **Cold standby recovery** instead of in-memory state replication

***

## 2. Per-Component HA

### 2.1 Endpoint — Active-Active

`endpoint` is a stateless gateway. All ready replicas serve traffic concurrently behind the Kubernetes Service. No leader election.

* Hard readiness gate: `backend` only
* `orchestrator`, `compute`, `shaper` are observed via `/health/deps` for diagnostics but do **not** gate request admission
* This prevents normal HA states (e.g. an `orchestrator` follower being `NotReady`) from ejecting `endpoint` pods

### 2.2 Orchestrator — Leader-only

Two replicas participate in one Lease. The leader runs `recoverState()`, node supervision, and reconciliation. The follower stays alive but `NotReady` for business traffic.

* Election backends: `postgres` (non-Kubernetes), `k8s-lease` (Kubernetes)
* On leadership loss: process exits

### 2.3 Backend — Leader-only

Two replicas with a Service selector matching only the leader label. **No follower-side request forwarding.**

The leader owns `RecoverTasks()`, `RecoverVCTCaches()`, the append queue worker, the cleanup worker, and index delete scheduling. The follower starts none of these.

* Election backends: `single-node` (non-Kubernetes), `k8s-lease`
* On leadership loss: process exits

### 2.4 Compute — Active-Standby (cold)

Two replicas with stable per-pod identity (headless Service). The Service routes only to the leader (`envector.ai/compute-leader=true`).

* Only the active pod calls `AddCompute` to register with `orchestrator`
* The standby is alive but unregistered
* On failover, `orchestrator` reassigns and reloads shards from object storage onto the promoted pod
* **Target failover window: 30–90 seconds** (cold-standby shard reload)

Setting `election.backend: ""` disables HA on `compute` and runs all replicas active-active for scale-out.

### 2.5 Shaper — Active-Standby (cold)

Two replicas with stable identity. The Service routes only to the leader (`envector.ai/shaper-leader=true`).

The standby does **not** preload keys or mirror in-flight jobs. Recovery is owned by `backend`:

* on shaper reconnect, only the **backend leader** runs the reconnect flow
* `backend` reloads keys for all loaded indexes into the new shaper leader
* in-flight jobs lost on failover are replayed from `backend`'s persistent `task_queue`

A `preStop` drain hook releases the Lease before shutdown. Default `terminationGracePeriodSeconds` is `120s`.

***

## 3. Health Endpoints

All HA-aware services expose two distinct readiness views:

| Endpoint            | Used by                              | Leader | Follower / Standby       |
| ------------------- | ------------------------------------ | ------ | ------------------------ |
| `/health`           | liveness                             | `200`  | `200`                    |
| `/health/ready`     | gRPC interceptor / Service admission | `200`  | `503`                    |
| `/health/pod-ready` | k8s readiness probe (HA mode)        | `200`  | `200` (when initialized) |

`/health/pod-ready` keeps standby pods running and observable; `/health/ready` is what gates business traffic. `endpoint` adds `/health/deps` and `/health/ha` for HA diagnostics.

***

## 4. Helm Configuration

Recommended Lease defaults differ by component to balance leadership stability against failover latency:

| Component                 | `leaseDurationSeconds` | `renewDeadlineSeconds` | `retryPeriodSeconds` |
| ------------------------- | ---------------------- | ---------------------- | -------------------- |
| `backend`, `shaper`       | `15`                   | `10`                   | `2`                  |
| `orchestrator`, `compute` | `6`                    | `4`                    | `1`                  |

`backend` and `shaper` use longer leases that favor stable leadership over the fastest possible handoff. `orchestrator` and `compute` use shorter leases so cold-standby relocation completes quickly when a leader is lost.

Example `values.yaml`:

```yaml
endpoint:
  replicas: 2

orchestrator:
  replicas: 2
  election:
    backend: k8s-lease        # "postgres" for non-K8s

backend:
  replicas: 2
  election:
    backend: k8s-lease        # "single-node" for non-K8s

compute:
  replicas: 2
  election:
    backend: k8s-lease        # "" disables HA (active-active scale-out)

shaper:
  replicas: 2
  election:
    backend: k8s-lease        # default "" — HA is opt-in for shaper
  terminationGracePeriodSeconds: 120
```

Setting `election.backend: k8s-lease` activates the Kubernetes HA path; there is no separate `enabled` flag.

**Auto-generated when `k8s-lease` is enabled:**

* `Role` + `RoleBinding` for `coordination.k8s.io/leases` and `core/pods` (for leader label patching)
* `PodDisruptionBudget` with `minAvailable: 1` (`backend`, `orchestrator`, `compute` only — `shaper` does not auto-generate a PDB)
* `preferred` pod anti-affinity

***

## 5. Failover Behavior

### Leader pod failure

1. Lease holder stops renewing
2. Lease expires after `leaseDurationSeconds`
3. follower acquires the Lease and runs leader-only initialization
4. Service endpoints flip to the new leader
5. Kubernetes restarts the failed pod, which rejoins as a follower

**Expected windows** (track the lease groups in §4):

* `orchestrator`, `compute`: 6–15 seconds (lease 6/4/1; for `compute` add cold-standby shard reload, see below)
* `backend`, `shaper`: 15–30 seconds (lease 15/10/2; for `shaper` add cold-standby state reload, see below)
* Cold-standby reload (`compute`, `shaper`) extends the window to **30–90 seconds** total

### Direct call to follower / standby

Followers reject business traffic. `/health/ready` returns `503`, gRPC RPCs return `Unavailable`. There is no follower-side request forwarding.

### Rolling restart

`kubectl rollout restart` recreates pods one at a time. The cluster converges back to one leader and one follower per HA service.

***

## 6. Validation

Recommended minikube test settings:

```yaml
embeddedPostgres: { enabled: true }
embeddedMinio:    { enabled: true }
endpoint:        { replicas: 2 }
orchestrator:    { replicas: 2, election: { backend: k8s-lease } }
backend:         { replicas: 2, election: { backend: k8s-lease } }
compute:         { replicas: 2, election: { backend: k8s-lease } }
shaper:          { replicas: 2, election: { backend: k8s-lease } }
```

**Expected post-deployment state:**

* exactly one pod per HA service holds the leader label
* the Kubernetes `Lease` holder matches the leader-labelled pod
* both `endpoint` replicas are `Ready`

**Useful commands:**

```bash
# Pods and leader labels
kubectl get pods -L envector.ai/orchestrator-leader \
                 -L envector.ai/backend-leader \
                 -L envector.ai/compute-leader \
                 -L envector.ai/shaper-leader

# Leases and Service endpoints
kubectl get lease
kubectl get endpoints,endpointslices

# Failover trigger
kubectl delete pod <leader-pod>

# Health checks
kubectl port-forward <pod> 8080:8080
curl -s localhost:8080/health/ready
curl -s localhost:8080/health/whoami
```


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.envector.io/1.4.x/operations-and-management/high-availability.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.