> ## Documentation Index
> Fetch the complete documentation index at: https://www.datum.net/docs/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Agent home for Datum, with what it is, pricing, and how to act: https://agents.datum.net
> If you can only fetch URLs: read https://www.datum.net/docs/llms.txt in full and open the most specific page for the task. Commands, limits, and field paths are on task, operations, and limits pages, not on section overviews. Pages described as "Documentation is coming soon" are placeholders.
> If you have a shell or can save files: download https://www.datum.net/docs/llms-full.txt (every page, about 740 KB) once and search it with grep, instead of fetching pages one at a time.
> datumctl describes itself: `datumctl api-resources` lists resource types and `datumctl explain <resource>` shows a schema. To act on live resources, use Datum MCP or datumctl. To search the docs, use the Docs MCP server at https://www.datum.net/docs/mcp.

# Failover between origins

> Configure automatic failover between a primary and fallback backend for applications behind a Datum gateway using datumctl and Envoy Gateway Backend, BackendTLSPolicy, and BackendTrafficPolicy resources.

***

## Overview

Failover routes traffic to a fallback origin only after the primary origin starts failing, and returns traffic to the primary once it recovers.

At a high level, this setup:

1. Defines two **`Backend`** resources — a primary and a fallback origin
2. Pairs each `Backend` with a **`BackendTLSPolicy`** so [Envoy](https://www.datum.net/docs/platform/envoy.md) presents the correct SNI to the origin
3. References both backends from an **`HTTPRoute`**, with the fallback `Backend` marked `fallback: true`
4. Applies a **`BackendTrafficPolicy`** that configures passive (outlier-detection) health checking
5. Verifies failover by forcing the primary to fail, then verifies automatic recovery

***

## Prerequisites

* `datumctl` installed and authenticated
* A valid **Project**
* Existing:
  * `Gateway`
  * `HTTPRoute`
* Two backend origins reachable over TLS on port 443 — a primary and a fallback — each with its own publicly resolvable hostname
* Permission to create:
  * `Backend`
  * `BackendTLSPolicy`
  * `BackendTrafficPolicy`

Verify access:

```bash theme={null}
datumctl get gateway
datumctl get httproute
```

***

## Critical requirement: BackendTLSPolicy for FQDN backends

A `Backend` that targets an external FQDN **must** be paired with a `BackendTLSPolicy` that sets `spec.validation.hostname`.

### Why this matters

* An FQDN `Backend`'s inline `spec.tls` block alone does **not** set outbound SNI
* Origins that route by SNI (most shared-IP hosting platforms) reject connections with no or incorrect SNI
* This fails as `upstream_reset_before_response_started{remote_connection_failure}` — it is **not** a certificate-trust error, and setting `insecureSkipVerify: true` will not fix it

Origins that do not inspect SNI (single-tenant IPs, some test endpoints) will work without this, which can hide the problem until you move to a shared-IP host.

***

## Configuration steps

### Step 1: Set variables

> [tab] macOS / Linux
>
> ```bash theme={null}
> project="your-project-id"
> namespace="default"
> route="your-route-name"
> gateway="your-gateway-name"
> primaryHostname="primary.your-origin.example.com"
> fallbackHostname="fallback.your-origin.example.com"
> primaryBackend="primary-backend"
> fallbackBackend="fallback-backend"
> ```

> [tab] Windows (PowerShell)
>
> ```powershell theme={null}
> $project          = "your-project-id"
> $namespace        = "default"
> $route            = "your-route-name"
> $gateway          = "your-gateway-name"
> $primaryHostname  = "primary.your-origin.example.com"
> $fallbackHostname = "fallback.your-origin.example.com"
> $primaryBackend   = "primary-backend"
> $fallbackBackend  = "fallback-backend"
> ```

***

### Step 2: Create the primary and fallback backends

Set `fallback: true` on the fallback `Backend` only. Envoy sends traffic to a `fallback: true` backend only once every non-fallback backend referenced by the same route is ejected.

> [tab] macOS / Linux
>
> ```bash theme={null}
> cat <<EOF | datumctl apply --project $project --namespace $namespace -f -
> apiVersion: gateway.envoyproxy.io/v1alpha1
> kind: Backend
> metadata:
>   name: $primaryBackend
> spec:
>   endpoints:
>   - fqdn:
>       hostname: $primaryHostname
>       port: 443
>   fallback: false
>   tls:
>     wellKnownCACertificates: System
> ---
> apiVersion: gateway.envoyproxy.io/v1alpha1
> kind: Backend
> metadata:
>   name: $fallbackBackend
> spec:
>   endpoints:
>   - fqdn:
>       hostname: $fallbackHostname
>       port: 443
>   fallback: true
>   tls:
>     wellKnownCACertificates: System
> EOF
> ```

> [tab] Windows (PowerShell)
>
> ```powershell theme={null}
> @"
> apiVersion: gateway.envoyproxy.io/v1alpha1
> kind: Backend
> metadata:
>   name: $primaryBackend
> spec:
>   endpoints:
>   - fqdn:
>       hostname: $primaryHostname
>       port: 443
>   fallback: false
>   tls:
>     wellKnownCACertificates: System
> ---
> apiVersion: gateway.envoyproxy.io/v1alpha1
> kind: Backend
> metadata:
>   name: $fallbackBackend
> spec:
>   endpoints:
>   - fqdn:
>       hostname: $fallbackHostname
>       port: 443
>   fallback: true
>   tls:
>     wellKnownCACertificates: System
> "@ | datumctl apply --project $project --namespace $namespace -f -
> ```

***

### Step 3: Attach a BackendTLSPolicy to each backend

> [tab] macOS / Linux
>
> ```bash theme={null}
> cat <<EOF | datumctl apply --project $project --namespace $namespace -f -
> apiVersion: gateway.networking.k8s.io/v1
> kind: BackendTLSPolicy
> metadata:
>   name: ${primaryBackend}-tls
> spec:
>   targetRefs:
>   - group: gateway.envoyproxy.io
>     kind: Backend
>     name: $primaryBackend
>   validation:
>     hostname: $primaryHostname
>     wellKnownCACertificates: System
> ---
> apiVersion: gateway.networking.k8s.io/v1
> kind: BackendTLSPolicy
> metadata:
>   name: ${fallbackBackend}-tls
> spec:
>   targetRefs:
>   - group: gateway.envoyproxy.io
>     kind: Backend
>     name: $fallbackBackend
>   validation:
>     hostname: $fallbackHostname
>     wellKnownCACertificates: System
> EOF
> ```

> [tab] Windows (PowerShell)
>
> ```powershell theme={null}
> @"
> apiVersion: gateway.networking.k8s.io/v1
> kind: BackendTLSPolicy
> metadata:
>   name: ${primaryBackend}-tls
> spec:
>   targetRefs:
>   - group: gateway.envoyproxy.io
>     kind: Backend
>     name: $primaryBackend
>   validation:
>     hostname: $primaryHostname
>     wellKnownCACertificates: System
> ---
> apiVersion: gateway.networking.k8s.io/v1
> kind: BackendTLSPolicy
> metadata:
>   name: ${fallbackBackend}-tls
> spec:
>   targetRefs:
>   - group: gateway.envoyproxy.io
>     kind: Backend
>     name: $fallbackBackend
>   validation:
>     hostname: $fallbackHostname
>     wellKnownCACertificates: System
> "@ | datumctl apply --project $project --namespace $namespace -f -
> ```

***

### Step 4: Update the HTTPRoute to reference both backends

Add both backends to the route's `backendRefs`. Give them equal `weight` — the fallback's `fallback: true` flag, not the weight, is what keeps it out of rotation while the primary is healthy.

> **Warning:** This manifest replaces the `HTTPRoute`'s entire `spec`. If your existing `HTTPRoute` has additional rules, matches, or filters, merge them into the manifest below instead of applying this as-is. Adjust `sectionName` to match the listener your route currently attaches to.

> **Warning:** Do not add a `hostnames` field to the `HTTPRoute`. Hostname routing is controlled solely by the parent `Gateway`'s listener, and `spec.hostnames` on the route will be rejected by admission.

> [tab] macOS / Linux
>
> ```bash theme={null}
> cat <<EOF | datumctl apply --project $project --namespace $namespace -f -
> apiVersion: gateway.networking.k8s.io/v1
> kind: HTTPRoute
> metadata:
>   name: $route
> spec:
>   parentRefs:
>   - name: $gateway
>     sectionName: default-https
>   rules:
>   - matches:
>     - path:
>         type: PathPrefix
>         value: /
>     backendRefs:
>     - group: gateway.envoyproxy.io
>       kind: Backend
>       name: $primaryBackend
>       port: 443
>       weight: 1
>     - group: gateway.envoyproxy.io
>       kind: Backend
>       name: $fallbackBackend
>       port: 443
>       weight: 1
> EOF
> ```

> [tab] Windows (PowerShell)
>
> ```powershell theme={null}
> @"
> apiVersion: gateway.networking.k8s.io/v1
> kind: HTTPRoute
> metadata:
>   name: $route
> spec:
>   parentRefs:
>   - name: $gateway
>     sectionName: default-https
>   rules:
>   - matches:
>     - path:
>         type: PathPrefix
>         value: /
>     backendRefs:
>     - group: gateway.envoyproxy.io
>       kind: Backend
>       name: $primaryBackend
>       port: 443
>       weight: 1
>     - group: gateway.envoyproxy.io
>       kind: Backend
>       name: $fallbackBackend
>       port: 443
>       weight: 1
> "@ | datumctl apply --project $project --namespace $namespace -f -
> ```

***

### Step 5: Apply passive health checking with a BackendTrafficPolicy

> **Note:** Only passive (outlier-detection) health checking is supported. A `healthCheck.active` block is rejected by admission — active `/healthz` polling is not available on this platform.

The thresholds below eject a backend after a single error and hold it out of rotation for 30 seconds. Tune `consecutive5XxErrors`, `consecutiveGatewayErrors`, and `interval` for production traffic (see [Best practices](#best-practices)), but leave `maxEjectionPercent` at `100`. With a single primary backend, a lower value can prevent ejection from happening at all.

> [tab] macOS / Linux
>
> ```bash theme={null}
> cat <<EOF | datumctl apply --project $project --namespace $namespace -f -
> apiVersion: gateway.envoyproxy.io/v1alpha1
> kind: BackendTrafficPolicy
> metadata:
>   name: ${route}-failover-policy
> spec:
>   targetRefs:
>   - group: gateway.networking.k8s.io
>     kind: HTTPRoute
>     name: $route
>   healthCheck:
>     passive:
>       baseEjectionTime: 30s
>       consecutive5XxErrors: 1
>       consecutiveGatewayErrors: 1
>       consecutiveLocalOriginFailures: 1
>       interval: 30s
>       maxEjectionPercent: 100
> EOF
> ```

> [tab] Windows (PowerShell)
>
> ```powershell theme={null}
> @"
> apiVersion: gateway.envoyproxy.io/v1alpha1
> kind: BackendTrafficPolicy
> metadata:
>   name: ${route}-failover-policy
> spec:
>   targetRefs:
>   - group: gateway.networking.k8s.io
>     kind: HTTPRoute
>     name: $route
>   healthCheck:
>     passive:
>       baseEjectionTime: 30s
>       consecutive5XxErrors: 1
>       consecutiveGatewayErrors: 1
>       consecutiveLocalOriginFailures: 1
>       interval: 30s
>       maxEjectionPercent: 100
> "@ | datumctl apply --project $project --namespace $namespace -f -
> ```

***

## Verification

### Primary healthy

```bash theme={null}
curl -I https://your-app.example.com
```

Expected response:

```
HTTP/1.1 200 OK
```

Traffic is served by the primary backend.

***

### Primary failing

> **Warning:** This test interrupts live traffic to the primary. Run it against a non-production hostname or during a maintenance window.

Make the primary origin return a 5xx status (or take it offline). Passive health checking is re-evaluated on each `interval` (30 seconds in this example), not per request, so poll for at least that long:

```bash theme={null}
for i in $(seq 1 10); do
  curl -s -o /dev/null -w "%{http_code}\n" https://your-app.example.com
  sleep 5
done
```

Expected behavior:

* Requests may still reach the failing primary for up to one `interval` after it starts failing
* Once `consecutive5XxErrors`/`consecutiveGatewayErrors` is reached at the next interval, the primary is ejected for `baseEjectionTime`
* Remaining requests return `200 OK`, served by the fallback backend

***

### Recovery

Restore the primary origin to a healthy state and wait out `baseEjectionTime`:

```bash theme={null}
sleep 30
curl -I https://your-app.example.com
```

Expected response:

```
HTTP/1.1 200 OK
```

Traffic returns to the primary automatically — no manual re-enable step is required.

***

## Cleanup / disable failover

> [tab] macOS / Linux
>
> ```bash theme={null}
> datumctl delete backendtrafficpolicy ${route}-failover-policy \
>   --project $project --namespace $namespace --ignore-not-found
>
> datumctl delete backendtlspolicy ${primaryBackend}-tls ${fallbackBackend}-tls \
>   --project $project --namespace $namespace --ignore-not-found
>
> datumctl delete backend $primaryBackend $fallbackBackend \
>   --project $project --namespace $namespace --ignore-not-found
> ```
>
> > **Note:** Remove the fallback `backendRef` from the `HTTPRoute` as well if you no longer want it defined, or the route will be left pointing at a deleted `Backend`.

> [tab] Windows (PowerShell)
>
> ```powershell theme={null}
> datumctl delete backendtrafficpolicy ${route}-failover-policy `
>   --project $project --namespace $namespace --ignore-not-found
>
> datumctl delete backendtlspolicy ${primaryBackend}-tls ${fallbackBackend}-tls `
>   --project $project --namespace $namespace --ignore-not-found
>
> datumctl delete backend $primaryBackend $fallbackBackend `
>   --project $project --namespace $namespace --ignore-not-found
> ```

***

## Troubleshooting

| Symptom | Root Cause | Resolution |
| - | - | - |
| `upstream_reset_before_response_started{remote_connection_failure}` even though the origin is healthy | Missing `BackendTLSPolicy` — Envoy sent no or incorrect SNI | Add a `BackendTLSPolicy` targeting the `Backend` with `validation.hostname` set to the backend's FQDN |
| `HTTPRoute` rejected by admission | `spec.hostnames` set on the `HTTPRoute` | Remove `hostnames` — hostname routing belongs to the `Gateway` listener |
| `BackendTrafficPolicy` rejected by admission | `healthCheck.active` block included | Remove `active` — only `passive` (outlier detection) health checking is supported |
| Traffic never shifts to the fallback backend | Fallback `Backend` missing `fallback: true`, or `BackendTrafficPolicy.spec.targetRefs` doesn't match the `HTTPRoute` | Set `fallback: true` on the fallback `Backend`; confirm the `targetRefs` name matches |
| Fallback keeps serving traffic after the primary recovers | Still inside the `baseEjectionTime` window | Wait out the ejection window — Envoy re-admits the primary automatically once it passes health checks again |

### Useful debug commands

```bash theme={null}
datumctl get backend --project $project --namespace $namespace
datumctl get backendtlspolicy --project $project --namespace $namespace
datumctl get backendtrafficpolicy --project $project --namespace $namespace
datumctl get httproute $route --project $project --namespace $namespace -o yaml
```

***

## Best practices

* Set `consecutive5XxErrors` / `consecutiveGatewayErrors` above `1` in production to avoid ejecting on a single transient error
* Keep `maxEjectionPercent` at `100` when there is only one primary backend — a lower value can block ejection entirely since there is only one host to eject from
* Keep the fallback origin provisioned at production-equivalent capacity — it may receive 100% of traffic during an outage
* Test failover on a recurring schedule, not only at initial setup
* Monitor ejection and recovery events directly rather than relying on `Gateway` status conditions as the sole health signal

***

## Summary

* Failover is expressed with two `Backend` resources — the fallback one set to `fallback: true`
* FQDN backends require a paired `BackendTLSPolicy` setting `validation.hostname`, or SNI-sensitive origins will reset the connection
* Only passive (outlier-detection) health checking is supported — `active` health checks are rejected by admission
* The `HTTPRoute` cannot set `spec.hostnames` — hostname routing belongs to the `Gateway` listener
* Recovery to the primary is automatic once `baseEjectionTime` elapses and it passes health checks again


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.