# Deployment

Target: a patched Ubuntu 24.04 LTS host with Docker Engine/Compose, encrypted storage, restricted
administration, outbound policy enforcement, DNS, and valid TLS certificates. The repository
defines the deployment but this workstation cannot execute it because Docker is unavailable.

## 1. Prepare protected configuration

1. Copy `.env.example` to `.env`. Set the external HTTPS origin, independent database/Redis
   credentials, an owner-approved OpenAI model, and the real dedicated-VM gateway. The production
   overlay forces `APP_ENV=production`; changing the base file cannot weaken that setting.
2. Create the Redis ACL, OpenAI provider key, and compute-gateway token from protected operator
   sources. The compute token must be independently scoped to the restricted gateway, not the
   hypervisor administration API. Files under `secrets/` must be owned by the deployment operator,
   readable only by the required service, excluded from backups where appropriate, and never
   committed.
3. Run `python scripts/generate_security_keys.py --secrets-root secrets` once from a protected shell.
   It creates the application privacy secret, AES data key, Ed25519 Computer capability keypair,
   independent AI/Computer service tokens, and one-time owner bootstrap token with exclusive
   creation; it refuses to overwrite existing material. Back up encryption/signing keys separately
   from the database.
4. Put `fullchain.pem` and `privkey.pem` in the protected directory named by `HAYVA_TLS_DIR`.
5. Keep the migration URL bound to `POSTGRES_USER`; keep `DATABASE_URL` bound to
   `POSTGRES_RUNTIME_USER`. They must use different passwords.

Development and staging report missing provider credentials as `unconfigured`. Production is
stricter: Core, AI Agent, and Computer Agent refuse startup when required file-mounted secrets,
the public HTTPS origin, the AI model, safe storage roots, or the real VM gateway are absent or
unsafe. Do not place provider keys, TOTP data keys, signing private keys, or service tokens directly
in source or images.

## 2. Validate and start

Run the fail-closed preflight on a clean, remotely traceable release checkout. It verifies protected
material without printing it, TLS presence/expiry, independent database identities, authenticated
Redis, exact release revision, Docker availability, and the effective Compose render:

```sh
python scripts/production_preflight.py --env-file .env \
  --evidence /protected/hayva-evidence/preflight.json
```

The evidence path uses exclusive creation. Preserve it outside the repository. A pass is necessary,
but does not replace the live checks below. Then build and start on the target host:

```sh
docker compose --env-file .env -f docker-compose.yml -f docker-compose.production.yml config
docker compose --env-file .env -f docker-compose.yml -f docker-compose.production.yml build --pull
docker compose --env-file .env -f docker-compose.yml -f docker-compose.production.yml up -d
```

PostgreSQL initialization creates the runtime identity. The `migrate` one-shot uses only the
migration URL; `db-permissions` then grants normal DML while revoking runtime DDL, Alembic writes,
and update/delete/truncate access to audit/login evidence. Core starts only after both steps succeed.
Aggregate readiness also fails until the AI and Computer workers report configured production
state. Public API requests with a Host different from `APP_BASE_URL` are rejected.

The production overlay publishes 80/443 and replaces the local gateway with TLS 1.2/1.3, HSTS,
CSP, cross-site request rejection, and edge limits. The base file alone binds only
`127.0.0.1:8080` and is for local evaluation.

## 3. Required verification before traffic

- All containers are healthy and use expected non-root identities, read-only roots, dropped
  capabilities, resource limits, bounded logs, protected mounts, and expected networks.
- Only 80/443 and restricted SSH are host-reachable. PostgreSQL, Redis, Core, Web, AI Agent,
  Computer Agent, Docker API, CDP, and VNC are not public.
- `/health` is minimal; detailed database/Redis/AI/browser/readiness probes are reachable only from
  an authorized internal/admin surface.
- A runtime-role SQL probe can perform expected application DML but cannot create/alter/drop tables,
  modify `alembic_version`, or update/delete/truncate `audit_events` and `login_events`.
- TLS certificate chain, hostname, renewal, protocols, HSTS, CSP, framing denial, request-size/rate
  limits, proxy IP normalization, and cookies are verified externally.
- Browser egress is enforced by host/network policy and cannot reach private, metadata, Docker, or
  service networks even if application checks fail.
- Owner setup succeeds once. Immediately replace the bootstrap secret with a disabled marker and
  restart Core; the database singleton must continue to reject a second setup.
- MFA enrollment/login/recovery, password change/reset, idle expiry, session rotation/revocation,
  emergency stop, takeover, unknown-write reconciliation, and artifact retention pass staging E2E.
- Security workflows have passed for the exact commit and release images; image digests and SBOM are
  retained with the release.

## 4. Backup and reset operations

Install `age` on the operator host and set an off-host staging directory and separately controlled
recipient:

```sh
HAYVA_BACKUP_DIR=/protected/offhost-staging AGE_RECIPIENT=age1... \
  scripts/backup_production.sh
```

Transfer both output files to immutable off-host storage. Prove restoration with
`scripts/restore_drill.sh` and record the evidence described in `RECOVERY.md`.

Password reset is deliberately operator-assisted until an authenticated delivery integration is
implemented:

```sh
python scripts/create_password_reset_token.py --email '<account-email>' --workspace-id '<workspace-uuid>'
```

Run it only in a trusted Core environment. Transmit the single printed capability over an
authenticated out-of-band channel; it expires quickly and is invalidated on use or replacement.

## 5. Remaining external release gates

Volume encryption/key rotation, firewall/SSH policy, browser egress enforcement, centralized
metrics/logging/alerts, certificate renewal alerts, scheduled off-host backups, restore evidence,
load/failure drills, CI results, and independent Security/QA approval cannot be proven by repository
configuration. Production stays blocked until those gates have dated evidence.
