Production deployment (HA)

A literal, ordered, copy-paste sequence to stand up the full Opsta AI Gateway appliance in a high-availability configuration on a conformant Kubernetes cluster. The whole bundled stack — Higress, PostgreSQL, Redis, Keycloak, the LGTM observability stack, cert-manager, and our control-plane/console — installs as one versioned unit.

Who this is for

The platform engineer doing the install. You bring a conformant HA Kubernetes cluster, storage, and DNS; optionally an external IdP and object store. The appliance installs the same way on any supported distribution — only how you provision the cluster differs (see Phase 0). For a single-node pilot, see the Standalone quick-start.

Verify before you trust

Every command here is run-once-for-real on a live cluster before publication, and every "expected output" is one that was actually observed. Each step pairs an exact command with what you should see and how to verify it. Do not skip the verification steps.

Before you start, read the Reference architecture (what you're deploying and why) and clear the Production-readiness checklist. This runbook executes that architecture; it doesn't restate it.

Supported platforms

The gateway runs on native Kubernetes of your choice — install (Phases 1–5) is identical across all of them; only Phase 0 (provisioning the cluster) is platform-specific.

Environment	Distribution	Provision the cluster with
On-prem (default)	Rancher / RKE2	Your platform team (Opsta SE can lead this) — see Phase 0
On-prem	OpenShift	Red Hat's installer (IPI/UPI)
On-prem	VMware Tanzu (TKG)	Tanzu's cluster tooling
Cloud	GKE / AKS / EKS / CCE / OKE	The provider's managed-Kubernetes service

---

Pre-flight — what you must have before you start

Fill in this worksheet first. Every later phase references it. Nothing below is run yet.

1. A conformant HA cluster

Provisioned in Phase 0. Sizing from the Requirements page: ≥3 worker nodes (so HA replicas + anti-affinity spread), each 4 vCPU / 8 GiB minimum → 8 / 16 GiB comfortable, 100 GiB SSD. The cluster's CNI must enforce NetworkPolicy and a default StorageClass (RWO) must exist.

• Distribution chosen: __________ · nodes (count + size): __________

2. DNS

• *.<baseDomain> → the ingress IP (from step 5). baseDomain: __________
• Subdomains used (all under the wildcard): api, console, auth, grafana, mcp.

3. TLS

• letsencrypt (cert-manager issues + renews via DNS-01/HTTP-01) or provided (you supply a wildcard cert + key for *.<baseDomain>, placed in secrets, never in git).
• Chosen: __________ · (if provided) cert + key ready: ☐

4. Object store (HA)

S3-compatible bucket(s) + credentials for the LGTM stack (logs/metrics/traces) and PostgreSQL backups (a readiness gate; self-hosted → SeaweedFS).

• Endpoint: __________ · bucket(s): __________ · creds in your vault: ☐

5. Load balancer (the ingress IP)

The gateway is reached through one LoadBalancer Service; *.<baseDomain> points at its external IP.

• Cloud LoadBalancer — provisioned automatically for the type: LoadBalancer Service.
• Customer-provided / existing load balancer — point your L4 LB (F5, HAProxy, NSX, etc.) at the node IPs/NodePorts, or hand the Service an external IP your LB fronts.
• MetalLB — bare-metal with no external LB; hands out an IP from a pool you own.
• Chosen: __________ · ingress IP (once known): __________

6. External identity provider (optional)

• Type: __________ (OIDC / SAML) · client id + secret in vault: ☐ · allowed email domain(s): __________

7. LLM provider keys

• At least one provider key, kept in secrets (never in git). Provider(s): __________ · in vault: ☐

8. Tooling on your workstation

kubectl, helm (pinned — see Upgrades), task, yq, jq, git, ssh-keygen, plus your TLS/secret tooling (helmfile only for the fallback path). The released chart is pulled from oci://ghcr.io/opsta/opsta-ai-gateway/charts at a pinned vX.Y.Z — no repo clone needed for the install.

---

Validation status

What's been run for real

The appliance install + data path (Phases 1–3) are validated from scratch on a clean cluster at v1.12.0 (clean helmfile sync → all core pods Running → 120/120 governed chat completions at 200, guardrails blocking). The released-OCI-chart install path is exercised by every production deploy.

Validated on a real multi-node managed cloud cluster (4-node GKE, 2026-06-18): the full install on a generic cluster off the released chart + private images — every component (Higress, control-plane, console, Postgres/CNPG, Redis, Keycloak, the LGTM stack) reached Running on a clean install, with a real cloud LoadBalancer as the ingress, governed routing live (no-key API request → 401 from key-auth, console → 302 SSO redirect, Keycloak realm reachable), and the bundled-Keycloak local admin seeded. Two multi-node-only install issues surfaced and were fixed: the Gateway API CRDs must be applied before the chart (Phase 2, step 1 — managed clusters don't pre-ship them), and the embedding model-seed job must not share the model PVC across nodes.

HA failover validated on a real multi-node cluster (4-node GKE, 2026-07-04): a full HA install off the released OCI chart (v1.27.0) via the GitOps path — 18/18 apps Healthy, CNPG Postgres ×3, Redis 3 + Sentinel 3, control-plane ×2 spread across distinct nodes, real cloud LoadBalancer + Let's Encrypt wildcard (DNS-01). Failover proven: a live https://api.<domain>/ request stayed 200 before, during, and after deleting a control-plane pod and a CNPG replica (the killed pods rescheduled). Six install issues surfaced and were fixed in the chart/tooling — called out inline below (storage class, domain-at-bootstrap, the LoadBalancer edge).

Multi-zone (regional) HA confirmed (regional GKE, 1 node/zone, 2026-07-04): on a regional cluster with one node in each of three zones, CNPG Postgres ×3 and Redis ×3 spread across all three zones, the gateway data plane ran 2 replicas across two zones, a real governed chat completion returned 200 end-to-end, and killing one of the two gateway replicas left the surviving zone serving 6/6 requests with zero drops. A full zone loss is survivable.

Still pending: the backup restore drill on a long-lived HA cluster.

The deployment flow

Each phase follows the same shape: intent → exact command → expected output → verification. The exact commands + observed outputs are captured from the validated run.

Phase 0 — Provision a conformant cluster

Stand up an HA Kubernetes cluster on your chosen platform (table above) with: ≥3 worker nodes; a NetworkPolicy-enforcing CNI; a default RWO StorageClass; a reachable object store (HA); and an ingress IP via your chosen LoadBalancer option. On-prem Rancher/RKE2 is Opsta's default — your Opsta SE provisions it from Opsta's internal cluster runbooks (compute → RKE2 → Rancher → downstream). For OpenShift/Tanzu or a managed cloud (GKE/AKS/EKS/CCE/OKE), provision per that platform's docs. The rest of this runbook is identical regardless of how the cluster was created.

Phase 1 — Configure the appliance

Intent: decide the config customer:init (Phase 2) bakes into your GitOps repo, and fill the git-ignored secrets.yaml it generates. Nothing is applied yet.

1. Config decisions — these become customer:init flags (Phase 2):

   Decision	customer:init flag	Notes
   Domain	DOMAIN=<your-domain>	→ global.baseDomain (see warning below)
   HA vs standalone	HA=true | HA=false	HA = 3× DB/Redis, 2× control/data plane; standalone = 1 each
   Storage class	STORAGE_CLASS=<your-default-sc>	your cluster's real default RWO class (see warning)
   Edge	ACCESS_MODE=lb | tunnel	lb = LoadBalancer + DNS A-records; tunnel = Cloudflare Tunnel
   TLS	TLS_MODE=letsencrypt | provided	letsencrypt needs DNS_ZONE=<zone> for DNS-01

   Two settings that must be right before you install

   • STORAGE_CLASS must be your cluster's real default RWO class (standard-rwo on GKE, gp3/ebs-sc on EKS, your CSI class on RKE2). A wrong/empty value leaves pods Pending on unbound PersistentVolumeClaims at bootstrap.
   • DOMAIN must be the final domain from the first install. Changing it after the platform is up rebuilds the routes but leaves the gateway's governance plugins scoped to the old domain, so /v1 requests return 404 until a clean re-reconcile. Set it once, correctly, up front.

2. Secrets — customer:init writes a secrets.example.yaml; copy it to a git-ignoredsecrets.yaml and fill it (or generate a self-consistent set, then override the reals): provider API key(s), the initial admin password, the DNS-01 API token, the IdP client secret, and (if TLS_MODE=provided) the wildcard cert + key. These are seeded into Vault → External Secrets at bootstrap; they never enter git.

3. Image pull — the released chart + images are pulled from GHCR with a read:packages token, passed to the bootstrap as GHCR_USER/GHCR_TOKEN (Phase 2). The chart templates the pull secret into every namespace it owns — no manual per-namespace kubectl create secret.

Phase 2 — Install (GitOps — the validated path)

Intent: scaffold a per-environment GitOps repo, run one imperative bootstrap, and let ArgoCD reconcile the whole appliance from the released OCI chart (pinned automatically to the current release — no version to hardcode). This is the path the multi-zone validation above exercised end-to-end. Full walkthrough: GitOps install. (The older helmfile sync flow remains as a fallback — see Install.)

Standalone vs HA is one flag: HA=true below for HA; HA=false for the single-node quick-start (see Appendix A). Everything else is identical.

# 1. Scaffold your environment repo. customer:init bakes your domain, HA, LB/tunnel, and the cluster's
#    storage class into a ready-to-commit GitOps repo pinned to the released OCI chart.
CUSTOMER=acme DOMAIN=gw.acme.com REPO_URL=git@github.com:acme/aigw-gitops.git \
  HA=true ACCESS_MODE=lb STORAGE_CLASS=<your-default-sc> DNS_ZONE=<your-dns-zone> \
  TLS_MODE=letsencrypt task customer:init

# 2. Fill the git-ignored secrets.yaml (provider keys, admin password, DNS/registry tokens — never
#    committed), commit + push the generated ./gitops-acme/, and add a READ-ONLY deploy key to it.

# 3. Bootstrap — one command stands up ArgoCD + Vault + operators, seeds Vault, and points ArgoCD at
#    your repo. IMPORTANT: pass the routing vars as CLI ARGS *after* the task name (not leading env vars).
GHCR_USER=<user> GHCR_TOKEN=<token-with-read:packages> task gitops:bootstrap:prod \
  GKCTX_PROD=<your-kube-context> GCLUSTER_PROD=<cluster-name> \
  PROD_REPO_URL=git@github.com:acme/aigw-gitops.git PROD_DEPLOY_KEY=~/.ssh/acme-aigw \
  GITOPS_SECRETS=./gitops-acme/secrets.yaml \
  NS_PREFIX=acme- REGISTRY=ghcr.io STORAGE_CLASS=<your-default-sc>

# The bootstrap prints a TARGET banner (context / repo / secrets) before it acts — read it. It installs
# the Gateway API CRDs as an ArgoCD app (no manual apply). Watch ArgoCD converge:
kubectl --context <your-kube-context> -n argocd get applications -w

Expected: every ArgoCD Application reaches Synced / Healthy (first HA reconcile ~10–20 min) — Higress gateway + controller, control-plane, console, Postgres (CNPG), Redis, Keycloak, and the LGTM stack, with the WasmPlugin chain programmed (kubectl get wasmplugin -A shows key-auth, ai-statistics, model-router, the guardrail/cache/MCP plugins). If a bootstrap PVC (argocd-redis, vault-raft) sticks Pending, your STORAGE_CLASS is wrong — set it to the cluster's real default RWO class.

Phase 3 — Verify the install

Intent: prove the data path works, not just that pods are up.

# Real governed chat completion through the gateway with a project key → HTTP 200
curl -s https://api-<baseDomain>/v1/chat/completions \
  -H "Authorization: Bearer <project-key>" -H 'Content-Type: application/json' \
  -d '{"model":"<your-model>","messages":[{"role":"user","content":"say hello"}]}'

Expected: a 200 with a normal completion body. Then confirm the surfaces:

• API answers governed traffic — validated end-to-end: 120/120 seeded chat completions returned 200, and 3/3 prompt-injection attempts were blocked by guardrails.
• Console (https://console-<baseDomain>/) → SSO login → admin home renders.
• Grafana (https://grafana-<baseDomain>/) → overview dashboard shows token/spend panels.
• Keycloak (https://auth-<baseDomain>/) reachable; control-plane reconcile healthy.
• (HA) the first PostgreSQL backup job succeeded (see Backup & DR).

Verified

The GitOps path in Phase 2 was run from scratch on real GKE clusters (zonal and regional) against the current released OCI chart: clean bootstrap, all ArgoCD apps Synced / Healthy, and real governed chat completions returned 200 over publicly-trusted TLS. The released-OCI-chart install is additionally exercised by every production deploy.

Phase 4 — Day-1 configuration

Intent: make the gateway usable — an admin signs in, identity is wired, and a first project + key serves governed traffic. Each step links the detail page; do them in this order.

1. Sign in as the bootstrap admin. Open https://console-<baseDomain>/ and sign in. The break-glass/bootstrap admin is set at install (controlPlane.bootstrapAdmin); store its credential in your vault. See the Console tour.
2. Wire the external IdP + restrict the email domain so staff log in via your SSO and only your domain(s) are allowed. See SSO & IdP brokering.
3. Create an organization → project → members. See Organizations & members.
4. Issue a project API key (one key for chat and tools). See Manage API keys.
5. Set a budget and enable guardrails for the project. See Budgets & limits and Guardrails.
6. Prove it end-to-end: hand the key to an app (Connect a client), send a request, and confirm it flows and appears in usage. (This is the same governed 200 path verified in Phase 3.)

Phase 5 — Production hardening sign-off

Intent: the go-live gate. Walk the Production-readiness checklist top to bottom and record the result:

• Confirm every checklist item, or log it as an accepted risk with an owner (see the Shared-responsibility & maturity matrix for what you vs. Opsta own).
• Record RTO/RPO from the backup restore drill.
• Confirm alerts are wired to your paging (Platform observability).
• Assign secret-rotation owners for provider keys, IdP secrets, and TLS material.

Sign the checklist's sign-off table; you're in production.

---

Appendices

Appendix A — Standalone / PoC quick-start

The same runbook — Phases 0–3 are identical — with a single-node footprint. It's the shortest path to a demo, and the configuration validated end-to-end above (32 pods Running, 120/120 governed completions at 200). Only the Phase 1 values differ:

global:
  baseDomain: <your-domain>       # still set correctly up front (see the Phase 1 warning)
  highAvailability: false         # 1 replica per component, no PDB/anti-affinity
  storageClass: <your-default-sc> # still the cluster's real default RWO class
tls:
  mode: letsencrypt               # or "provided"
postgres:
  backup: { enabled: false }      # backups optional for a PoC (turn on for anything kept)
observability:
  storage: local                  # filesystem PVC — single-binary Loki/Tempo + all-in-one Mimir
                                   # (HA uses object storage instead)

What changes vs. HA: 1 node is enough; Postgres/Redis/control-plane run a single replica each (no failover); observability uses a filesystem PVC instead of an object store; DNS can point at a single node IP / hosts entry rather than a LoadBalancer. Everything else — the install commands, the WasmPlugin chain, the day-1 provider/model/key setup — is the same. To grow a PoC into HA later, set highAvailability: true (+ ≥3 nodes and an object store) and re-sync.

Appendix B — Air-gap

No outbound internet: mirror the chart + every image into your internal registry (e.g. Harbor) and point the install at it, plus the LLM-provider egress allowlist. Full procedure: Air-gapped install.

Appendix C — BYO operators (advanced)

Reuse an existing CloudNativePG / Redis / cert-manager operator instead of the bundled ones (toggles in values). The appliance is still one support boundary, but the operator lifecycle becomes yours. See Reuse existing operators.

Appendix D — Upgrade & rollback

Upgrade = bump the OCI chart pin in your GitOps repo (the released vX.Y.Z) and let ArgoCD reconcile — no imperative sync. Always back up the database first — control-plane migrations are forward-only at runtime, so a rollback across a schema change is not automatic (see Shared-responsibility). Full upgrade/rollback procedure: Upgrades.

Appendix E — Uninstall / teardown

# Remove the root app-of-apps (ArgoCD prunes the appliance; keeps the cluster). PVCs are retained.
kubectl -n argocd delete application platform
# PVCs are retained by default — delete them to remove data (irreversible):
kubectl get pvc -A | grep -E 'opsta|observability|cnpg|redis'
# kubectl delete pvc <name> -n <ns>   # only when you mean it

Then delete the cluster per your platform (e.g. via Rancher, or your cloud console).

Appendix F — Troubleshooting

Common failures per phase, and the standard diagnostics to gather (kubectl get pods -A, kubectl describe, kubectl logs, kubectl get events -A --sort-by=.lastTimestamp, helm -n <ns> history <release>): Troubleshooting.