who_need_help/docs/operations.md

# Operations runbook

This runbook describes the commands that are implemented and verified in this
repository. It does not claim a production recovery point objective, recovery
time objective, retention period, storage capacity, or high-availability model;
those values require product policy and measurements from the eventual
production environment.

## Private Gitea continuous integration

The observed private Git remote is hosted by Gitea 1.22.0. Gitea 1.22 reads
repository workflows from `.gitea/workflows/`, so the repository keeps a
Gitea-specific quality workflow in addition to the GitHub-compatible workflow.
Both files are checked by the pinned local actionlint gate.

The Gitea workflow deliberately targets only the custom
`who-need-help-ci` runner label. Its jobs build and run many Docker images and
must not be scheduled onto an unrelated shared runner or onto the production
application host. Its gates run sequentially in one job so a single push cannot
start four heavy Docker suites concurrently. Before enabling it:

1. enable Actions in the private repository settings;
2. register a trusted runner controlled by this project;
3. assign `who-need-help-ci:host` only to a Linux host where the exact Docker
   Engine and Compose commands used by the repository have been verified;
4. install the runtime required by the pinned checkout action on that host;
5. keep the runner registration state and any Docker socket access outside the
   repository and away from untrusted projects.

The runner's availability, labels, software versions, free resources and
repository Actions setting cannot be established from the unauthenticated
repository clone. Do not treat the presence of the workflow file as evidence
that remote CI has run. After provisioning the runner, trigger the workflow
manually, require the full job and all four gate steps to pass, and record the
run URL and commit before protecting `main`.

## First production Compose environment

The deployment environment selects topology and database ownership:

| Setting | Result |
| --- | --- |
| `APP_TOPOLOGY=compact` | One Phoenix+Oban container, directly published to the host; default for the first server. |
| `APP_TOPOLOGY=split` | Traefik, `WEB_REPLICAS` web containers, and `WORKER_REPLICAS` worker containers. |
| `DATABASE_MODE=container` | Start the project-owned PostgreSQL/PostGIS container and volume. |
| `DATABASE_MODE=external` | Do not create/start a Compose database; use only `DATABASE_URL`. |

There is no Redis dependency. Queues, rate-limit counters, Oban leadership,
and durable application state use PostgreSQL.

### Public HTTPS edge and isolated staging

The first single-server deployment uses one separately managed Caddy container
for ports 80/443 and two independent application projects. Caddy 2.11.4 is
built as a static non-root binary with pinned Go 1.26.5 and a scratch runtime.
Its persistent volume holds ACME account and
certificate state; do not remove that volume during an ordinary application
deploy. Production and staging each have their own application secret file,
PostGIS volume, migrations, and Docker DNS alias. They share only the named
`who_need_help_public_edge` network, so load and browser records created in
staging cannot enter the production database.

Generate and start the edge before either application project:

```bash
./scripts/init-edge-env.sh \
  whoneedhelp.com staging.whoneedhelp.com .env.edge
./scripts/edge-up.sh .env.edge
```

The authoritative A records for `whoneedhelp.com`, `www.whoneedhelp.com`, and
`staging.whoneedhelp.com` must point to the observed server address, and inbound
TCP 80/443 plus UDP 443 must be permitted before public certificate issuance.
Caddy obtains and renews public certificates and redirects HTTP to HTTPS. The
Phoenix release remains plain HTTP on the internal shared Docker network;
Caddy replaces incoming forwarding headers and WebSocket proxying is automatic.
The compact app's optional host-published HTTP port stays bound to loopback for
operator health checks and is not a public entry point.

Create staging with independent names and secrets:

```bash
PRODUCTION_COMPOSE_PROJECT_NAME=who_need_help_staging \
PRODUCTION_PUBLIC_UPSTREAM_NAME=who-need-help-staging \
PRODUCTION_HTTP_PORT=4011 \
PRODUCTION_CODEX_SESSION_ID=YOUR_MAIN_CODEX_SESSION_ID \
./scripts/init-production-env.sh \
  staging.whoneedhelp.com .env.staging
```

Create production separately:

```bash
PRODUCTION_COMPOSE_PROJECT_NAME=who_need_help_production \
PRODUCTION_PUBLIC_UPSTREAM_NAME=who-need-help-production \
PRODUCTION_HTTP_PORT=4010 \
PRODUCTION_CODEX_SESSION_ID=YOUR_MAIN_CODEX_SESSION_ID \
./scripts/init-production-env.sh \
  whoneedhelp.com .env.production
```

Configure and validate email delivery/OAuth independently in each ignored file. Start
staging first, run database/load/browser/Android verification there, and then
start the clean production project. Stopping staging does not stop Caddy or
production:

```bash
./scripts/compose.sh .env.staging down
```

Do not add `--volumes` unless the exact staging database has been inspected and
its deletion is the intended operation.

Run the initializer on the target Docker host after its final public hostname
is known:

```bash
./scripts/init-production-env.sh whoneedhelp.com .env.production
```

The initializer reads `.env.example`, derives the numeric group of that host's
Docker socket, generates independent random values for PostgreSQL,
`SECRET_KEY_BASE`, handover codes, the BEAM release cookie, and metrics access,
and writes an ignored mode-`0600` file. It does not print those values and
refuses to overwrite an existing destination.

By default, the generated public proxy port binds to `127.0.0.1`, which is
appropriate only when the verified reverse proxy reaches the application on
the same host. Set `PRODUCTION_HTTP_BIND_ADDRESS` when generating the file, or
edit `HTTP_BIND_ADDRESS` afterward, to match the observed target topology.
Replace `TRAEFIK_TRUSTED_IPS` with the exact source IP/CIDR observed at Traefik;
do not copy the temporary VPN value into an unrelated server.

Configure the transactional email provider and a sender accepted by it. SMTP
uses `EMAIL_DELIVERY_PROVIDER=smtp` plus the `SMTP_*` settings. UniSender Go's
HTTPS Web API uses `EMAIL_DELIVERY_PROVIDER=unisender_go`,
`UNISENDER_GO_API_KEY`, and the default documented API base URL. The initializer
accepts the corresponding `PRODUCTION_EMAIL_DELIVERY_PROVIDER`,
`PRODUCTION_SMTP_*`, and `PRODUCTION_UNISENDER_GO_*` inputs. Optional
`EMAIL_HTTP_CONNECT_TIMEOUT_MS` and `EMAIL_HTTP_RECEIVE_TIMEOUT_MS` override the
Req/Finch defaults only when deployment measurements justify explicit values.
Set optional `SUPPORT_INBOX_ADDRESS` to
a monitored address for support/removal queue alerts and email `Reply-To`;
leaving it empty disables operator email alerts, not the protected queues. If
Google registration/sign-in is
enabled, also set both Google Web client credentials and register
`https://YOUR_PHX_HOST/auth/google/callback` as the exact authorized redirect
URI. Leave both credentials empty to keep the feature disabled. Then run:

```bash
./scripts/validate-production-env.sh .env.production whoneedhelp.com
./scripts/deploy-up.sh .env.production
```

The production override keeps Mailpit behind its inactive `local-mail` profile,
so public registration cannot appear to succeed while mail is only retained
locally. The validator checks file ownership/mode, origin consistency,
template markers, independent generated secrets, settings required by the
selected email provider, a
complete-or-empty Google credential pair, the generated database URL, and the
final Compose render without printing secrets.
It does not contact DNS, TLS, the email provider, the reverse proxy, or the application.
After deployment, verify `/healthz/ready`, inspect all replica health and logs,
register a unique address through the public browser, receive its message at
the real mailbox, follow the HTTPS confirmation link, and remove only that
run-scoped account.

### External PostgreSQL/PostGIS

#### PostgreSQL on the same Linux host

The server-local mode uses the PostgreSQL Unix socket instead of exposing the
database on a Docker-reachable TCP address. Ecto/Postgrex receive
`DATABASE_SOCKET_DIR`; `scripts/compose.sh` then adds
`compose.external-db-socket.yaml` and mounts that exact directory read-only
into the active application and migration services. An empty setting preserves
the normal remote-provider TCP behavior.

After read-only inspection confirms the intended PostgreSQL 18 cluster,
installed PostGIS package, socket directory, and absence of the project-scoped
roles/databases, run the root-only provisioner:

```bash
sudo ./scripts/provision-host-postgres.sh "$USER"
```

Its exact mutation scope is:

- prepend two database-and-role-specific `local ... scram-sha-256` rules to the
  active `pg_hba.conf`, retaining a mode-`0600` copy under
  `/var/backups/who_need_help/`, and reload that cluster;
- create login roles `wnh_production` and `wnh_staging` without superuser,
  database-creation, role-creation, replication, or row-security bypass rights;
- create empty owner databases `who_need_help_production` and
  `who_need_help_staging`, revoke public connect, and preload `citext` and
  PostGIS so the non-superuser Ecto migrations can run;
- verify both credentials through the observed Unix socket and write separate
  mode-`0600` initializer fragments under
  `~/.config/who_need_help/` without printing passwords.

The command refuses to overwrite credential fragments, refuses any matching
pre-existing role/database or managed HBA marker, validates the candidate HBA
rules before reload, and restores its HBA backup plus removes only objects it
created if provisioning fails. It does not inspect or migrate application
tables; the release migration runner remains authoritative for schema.

Generate staging or production after loading only the matching trusted
fragment:

```bash
set -a
. "$HOME/.config/who_need_help/database-production.env"
set +a
./scripts/init-production-env.sh whoneedhelp.com .env.production
unset PRODUCTION_DATABASE_MODE PRODUCTION_DATABASE_URL \
  PRODUCTION_DATABASE_SOCKET_DIR
```

For staging, load `database-staging.env` and pass the staging domain, project,
upstream alias, and HTTP port described above. Do not load both fragments into
one shell.

Provision the database and role first, then generate the environment without
placing its credentials on a command line that is retained in shell history:

```bash
export PRODUCTION_DATABASE_MODE=external
read -rsp 'External DATABASE_URL: ' PRODUCTION_DATABASE_URL && echo
export PRODUCTION_DATABASE_URL
./scripts/init-production-env.sh whoneedhelp.com .env.production
unset PRODUCTION_DATABASE_URL
```

Set `PRODUCTION_APP_TOPOLOGY=split` during generation when independent web and
worker scaling is required. In external mode `scripts/compose-up.sh` builds the
migration image, runs `scripts/check-database.sh`, and fails before migration
or application startup unless PostgreSQL and `PostGIS_Version()` are reachable.
The check also reports the observed TLS flag. The connection URL must follow
the database provider's verified TLS/CA requirements.

`scripts/compose.sh .env.production config --services` is the canonical
read-only render check. In external mode its output must not contain `db`.
Use `scripts/compose.sh .env.production ps -a` and
`scripts/compose.sh .env.production logs web worker` for the selected mode.

The repository's Compose backup, restore, rotation, and rollback-based staging
scripts deliberately refuse `DATABASE_MODE=external`; they are scoped to the
project-owned database container. For an external database, use the provider's
verified backup/restore procedure and test recovery before public launch.

## Compose database backup

Create a PostgreSQL 18 custom-format archive, validate its table of contents,
and write a SHA-256 manifest:

```bash
./scripts/backup-compose.sh
```

The default destination is the ignored `output/backups/` directory. An explicit
new destination may be supplied as the only argument. The command refuses to
overwrite either an archive or its checksum manifest and writes through
temporary files before publishing the final pair. Files and newly created
directories are restricted by `umask 077`.

The archive covers the configured application database. PostgreSQL cluster
globals such as roles and tablespaces are not part of `pg_dump`; deployment
credentials and database roles must be provisioned separately from secrets.
Local backup files on the same workstation are not an off-site backup.

## Encrypted local S3-compatible backup drill

The isolated load project can run a complete encrypted Restic/MinIO drill:

```bash
./scripts/load-stack-up.sh
./scripts/backup-s3-drill.sh local-encrypted-backup
```

The script refuses the staging Compose project and validates the project and
service labels of every pre-existing container in its scope. On first use,
`scripts/ensure-local-load-env.sh` generates independent random MinIO and
Restic credentials in ignored `.env.load` and restricts that file to mode
`0600`. MinIO publishes Docker-assigned ports only on `127.0.0.1`; the observed
API and console URLs are printed after a successful run.

The backup tool combines the matching PostgreSQL 18 client with pinned Restic
rebuilt on Go 1.26.5. MinIO server and client are also rebuilt as non-root
Alpine images from checksum-pinned upstream source commits with the exact
dependency updates recorded in `Dockerfile.minio`. The quality gate verifies
their reported release, commit, Go runtime, configured user, and current
HIGH/CRITICAL vulnerability scan. `restic backup --stdin-from-command` runs a
custom-format `pg_dump`, checks the producer exit status, encrypts the data,
and uploads it directly to MinIO. No plaintext database dump is written to the
host. The drill then:

1. runs `restic check --read-data`;
2. streams `restic dump` into `pg_restore --list`;
3. restores into a uniquely named database created from `template0`;
4. checks tables, current Ecto migrations, PostGIS, and release migration
   readiness before removing that exact database;
5. clones and corrupts an isolated repository and requires both check and dump
   to fail;
6. stops an exact scoped in-progress backup container only after encrypted
   objects reach MinIO, requires zero published snapshots, prunes unreferenced
   packs, and rechecks the repository;
7. removes and verifies removal of the corruption/interruption buckets and
   requires source table counts to remain unchanged.

The successful encrypted bucket is deliberately retained in the named local
MinIO volume. Non-secret evidence is written under ignored
`output/backups-s3/<run-label>/`; the runtime scratch directory is under
ignored `tmp/backup-s3/`. Use a unique lowercase run label of at most 32
characters. The command refuses to replace an existing retained bucket.

MinIO server and client are AGPLv3. `Dockerfile.minio` identifies the exact
upstream source commits and contains the dependency changes and complete build
commands used here. Before distributing or publicly operating modified images,
review the license and make the corresponding source available as required;
this runbook does not provide legal advice.

This verifies encryption, local S3 protocol use, restore mechanics, and two
failure paths on the observed workstation. A MinIO volume on that same
workstation is not an off-site backup and does not establish production RPO,
RTO, retention, capacity, key custody, object locking, or database HA.

## Local external-service boundary drill

Run the OAuth, SMTP, and provider-neutral push protocol checks without public
credentials, a server, or host-published ports:

```bash
./scripts/external-boundaries-run.sh local-boundaries
```

The script creates a uniquely named Compose project on an internal-only Docker
network. It generates independent one-run OAuth and push credentials in an
ignored mode-`0600` environment file, including a separate bearer token for the
protected metrics endpoint, builds the production release plus a non-root
standard-library Python protocol mock, and then verifies:

`compose.external-boundaries.yaml` is the only deployment file that sets
`ALLOW_INSECURE_EXTERNAL_HTTP=true`. Ordinary runtime configuration requires
HTTPS for OAuth and push endpoints so provider credentials are not sent over
plain HTTP.

1. GitHub-compatible OAuth authorization, PKCE S256, token exchange, normalized
   user lookup, state mismatch, provider rejection, one-time-code replay,
   a fresh flow after a temporary token error, and a token timeout;
2. SMTP acceptance, permanent recipient rejection without retry, one retry
   after a temporary greeting failure, a greeting timeout, and the result of
   submitting the same message twice;
3. the disabled default push boundary plus HTTP success, permanent rejection,
   one temporary retry, replay deduplication, and deduplication after an
   ambiguous timeout using the same idempotency key;
4. a fresh PostGIS database, current migrations, and two Oban worker replicas;
   request acceptance and new-chat domain transactions enqueue stable
   user-recipient events, replay is deduplicated before HTTP, an injected
   temporary chat delivery fails its first Oban attempt and completes on its
   second, and private message text is absent from the push payload.

The evidence JSON and mock state contain counters, booleans, normalized
identity fields, payload digests, run-scoped user/event identifiers, and the
privacy-safe notification metadata asserted by the drill. The script fails if
any generated secret appears in retained evidence. Its trap validates exact
Compose labels, removes only that project, volumes and one-run images, and
deletes the temporary credential file. Failure logs are retained under the
same ignored evidence directory.

This drill uses the application's real Assent/Req and Swoosh/gen_smtp clients,
but the providers are local. It therefore verifies client-side protocol and
product integration, not GitHub, SMTP-provider, FCM, or APNs availability.
SMTP permits duplicate delivery after ambiguous outcomes, so the result
explicitly makes no exactly-once claim. Push currently targets a stable
`user:<uuid>` recipient; selecting a provider, registering device tokens, and
resolving that user to devices remain deployment/provider work.

The implementation follows the configured adapter interfaces and protocol
semantics documented by
[Assent 0.3.1](https://hexdocs.pm/assent/0.3.1/Assent.HTTPAdapter.html),
[GitHub OAuth](https://docs.github.com/en/apps/oauth-apps/building-oauth-apps/authorizing-oauth-apps),
and [SMTP RFC 5321](https://datatracker.ietf.org/doc/html/rfc5321).

The optional `WhoNeedHelp.Email.UnisenderGoAdapter` maps the same Swoosh email
objects to UniSender Go's documented JSON contract, supplies a per-delivery
idempotence key, redacts rejected recipient addresses from returned errors, and
uses only HTTPS. The provider documents the endpoint, `X-API-KEY` header,
response shape, and one-minute/64-character idempotence-key behavior in its
[Web API reference](https://godocs.unisender.ru/web-api-ref).
Every request explicitly sets `track_links` and `track_read` to `0`; UniSender
Go otherwise enables both by default and may require account-level approval
before it honors those values.
`UNISENDER_GO_SKIP_UNSUBSCRIBE` defaults to `false` and must remain disabled
until UniSender Go support explicitly enables that transactional capability for
the account. After approval, set it to `true` to add `skip_unsubscribe=1` to the
provider request.
Verify a delivered message's raw HTML after the provider confirms the account
change and before launch.

## Isolated restore drill

Run a real restore into a uniquely named temporary database:

```bash
./scripts/restore-drill-compose.sh output/backups/compose-YYYYMMDD-HHMMSS.dump
```

The drill:

1. validates the SHA-256 manifest;
2. validates the archive table of contents;
3. creates a pristine database from `template0`;
4. restores with `pg_restore --exit-on-error`;
5. reads every restored public application table and checks the PostGIS
   library;
6. runs the current immutable release's migrations and migration-readiness check
   against only the temporary database;
7. drops only the temporary drill database and verifies that it is gone.

A trap also attempts to drop the exact temporary database if a check fails.
The source application database is never passed to `pg_restore`, `dropdb`, a
clean operation, or the drill migration runner. The drill intentionally does
not compare an older backup's row counts to the live source, because concurrent
legitimate writes or a historical archive would make that comparison invalid.

## Isolated Compose upgrade rehearsal

After creating a current custom-format backup, run the complete current release
against an isolated restored copy:

```bash
./scripts/upgrade-rehearsal-compose.sh \
  output/backups/compose-YYYYMMDD-HHMMSS.dump
```

The rehearsal validates the checksum and archive catalog, reads the public
origin configuration from ignored `.env`, and uses only the independently
generated credentials in ignored mode-`0600` `.env.e2e`. It builds a uniquely
tagged production release, creates a uniquely named Compose project and
database from `template0`, restores the archive, records application-table
counts, and then:

1. applies every current timestamped Ecto migration;
2. requires the localized category column and all 11 valid cursor indexes;
3. starts 2 web and 2 worker replicas behind the isolated Traefik instance;
4. requires the four-node BEAM cluster and cross-node PubSub probe;
5. checks the production HTTP-to-HTTPS redirect, trusted-proxy public pages,
   and both health endpoints;
6. requires an empty before/after diff for every public application table
   except the expected migration and Oban-internal tables;
7. removes and verifies removal of the exact containers, networks, database
   volume, and one-run image.

The input archive is read-only and is not copied into the evidence directory.
The ordinary Compose project, source database, public route, and running
containers are outside the generated project scope. Non-secret evidence is
retained under ignored mode-`0700`
`output/upgrade-rehearsal/<run-id>/`, with files mode `0600`.

## Clean tracked-revision deployment drill

Run:

```bash
./scripts/clean-deploy-verify.sh
```

The script archives the current tracked Git revision into a new temporary
directory. It verifies that the archive contains no local `.env`, Git
metadata, generated output, or existing E2E/load environment. It generates
independent one-run PostgreSQL, Phoenix, handover, BEAM-cookie, and metrics
secrets with mode `0600`; optional GitHub OAuth and push delivery stay
disabled.

The drill uses a unique Compose project, application image, router/service
name, internal network, PostgreSQL volume, database, and dynamic host ports.
It requires exactly two healthy web and two running worker replicas, all
tracked migrations, seeded categories, an idempotent repeated migration,
working readiness/home/registration/Mailpit routes, four connected BEAM
nodes, and a cross-replica PubSub message. It then removes the exact project
including volumes, its one-run image, and the temporary archive, and verifies
that those scoped resources are absent. Evidence is retained under
`output/portability/<run-id>/`.

This is a local Docker portability observation. It does not establish
production SMTP, OAuth, push-provider, TLS, database-HA, storage, capacity, or
jurisdictional readiness.

## Service checks

```bash
docker compose ps
curl --fail http://localhost:4010/healthz/live
curl --fail http://localhost:4010/healthz/ready
./scripts/verify-realtime-cluster.sh compose
```

`live` verifies that the web process can serve HTTP. `ready` additionally runs
`SELECT 1` through the configured Ecto repository. The cluster probe subscribes
on one connected BEAM node and broadcasts from another.

Health checks do not replace alerting, database backups, restore drills, or
application-level synthetic checks.

## Local failure and rolling-replacement drills

The isolated load project can exercise process crashes, sequential container
replacement, and a real Oban retry without touching the normal Compose project:

```bash
./scripts/load-stack-up.sh
./scripts/load-resilience-run.sh local-resilience
```

The resilience script refuses `LOAD_PROJECT=who_need_help` and verifies the
Compose project/service labels of every container before stopping it. It:

1. continuously calls readiness through the isolated Traefik route;
2. terminates the BEAM process in one web and one worker container and requires
   Docker's observed restart count to increase;
3. removes and replaces each web and worker replica one at a time;
4. waits for every configured BEAM node, then runs the cross-node PubSub probe;
5. enqueues a side-effect-free local worker that fails its first Oban attempt
   and succeeds on its second;
6. removes that exact Oban row and requires no fixture domain rows to remain.

Traefik's retry middleware is attached to the HTTP and local TLS routers. Its
attempt count is an environment input. Traefik retries transport failures and,
with the checked configuration, does not opt in to retrying non-idempotent
requests. This reduces a stale-backend window; it is not a claim of production
availability.

For the project-owned kind cluster, run:

```bash
./scripts/kind-rolling-verify.sh local-kind-rollout
```

The kind control plane is an optional verification environment, not part of
the ordinary public Compose route. Pause it without deleting its container or
Kubernetes state when the cluster is not being tested:

```bash
./scripts/kind-stop.sh
```

The next `./scripts/kind-up.sh` validates the ownership marker and kind labels,
starts the stopped control plane, then continues with image loading and chart
reconciliation.

That script requires both the kind ownership marker and the control-plane
cluster label before invoking `rollout restart`. It changes only the web and
worker Deployment pod templates. It snapshots application-table counts before
and after, continuously probes the observed Docker mapping for the chart's
NodePort, requires all four old pod UIDs to disappear, waits for the exact BEAM
peer count, and verifies cross-node PubSub. PostGIS, its hostPath, the
Kubernetes Secret, and the namespace are not recreated.

The rollout timeout, probe interval/timeout/retry count, and cluster-join
timeout are experiment inputs. They are not production SLOs or resource
requirements.

## Isolated Oban burst measurement

The worker role consumes only `maintenance` and `push`. Configure their
per-worker limits with `OBAN_MAINTENANCE_CONCURRENCY` and
`OBAN_PUSH_CONCURRENCY`; multiplying either value by the number of worker
replicas gives the configured cluster-wide concurrency for that queue.

After starting the isolated load project, run an explicitly sized experiment:

```bash
./scripts/load-stack-up.sh
./scripts/oban-burst-run.sh local-oban-burst 1000 120
```

The three required arguments are an evidence label, job count, and recorded
experiment timeout in seconds. They are not capacity thresholds. The script
refuses the ordinary `who_need_help` project, verifies the expected worker
replicas, inserts only confirmed `LocalBurstProbe` jobs with a unique run id,
records each worker's effective queue configuration and container samples,
requires every job to complete, compares domain-table row counts, and deletes
exactly its own jobs. The probe worker has no product side effects and no
product flow enqueues it.

## Kubernetes ingress isolation

`networkPolicy.enabled=true` renders the chart's ingress NetworkPolicy.
Enforcement is a property of the cluster CNI, not of the YAML object alone.
Before relying on it, verify that the target cluster uses a NetworkPolicy-
capable plugin and run positive HTTP/cluster checks plus negative blocked-port
checks there. Egress remains intentionally unrestricted until the actual
database and external-service destinations are known.

## BEAM runtime memory guard

Compose sets `ERL_ZFLAGS="+Q ${ERLANG_PORT_LIMIT}"`; Helm renders the same flag
from `app.erlangPortLimit`. Both default to `65536`, OTP's normal port-table
limit. This makes the runtime independent of an unusually large host or nested
container `nofile` limit. Validate all live replicas after deployment:

```bash
./scripts/verify-beam-runtime.sh compose
./scripts/verify-beam-runtime.sh kind
```

The command records the effective port count/limit, allocated port-table bytes,
BEAM memory, process count, cgroup memory, and RSS for each web and worker
replica. It fails when a live VM does not use the configured limit; kind mode
also requires the desired number of Ready application pods.

`65536` is a concurrency ceiling for simultaneously existing Erlang ports
(files, sockets, and drivers), not a container memory limit. Do not lower or
raise it from a RAM estimate alone. A changed value must be validated against
measured peak port usage and the target environment.

## Protected Prometheus metrics

The web role exposes Prometheus text format at `/metrics`. It requires the
independent `METRICS_TOKEN` deployment secret:

```bash
curl --fail \
  --header "Authorization: Bearer $METRICS_TOKEN" \
  http://localhost:4010/metrics
```

The endpoint returns `401` without the exact token, disables response caching,
and does not put the credential in a URL. The reporter exports cumulative HTTP
request and duration, router exception, database query and duration, WebSocket
connection, VM memory, and scheduler run-queue metrics. Cumulative durations are
integer microseconds because the selected reporter's sum accumulator is
integer-based; divide by `1_000_000` in PromQL when seconds are required.
Definitions intentionally have no request path, user, request, or event-name
labels that could create unbounded cardinality.

Metrics are local to each BEAM process. Discover and scrape every web pod or
container as a distinct target and preserve Prometheus's `instance` label. A
request through the load-balanced public route reaches only one replica and is
therefore useful as an authorization/smoke check, not as a cluster-wide
aggregate.

The isolated load project includes a local observability profile:

```bash
./scripts/load-stack-up.sh
./scripts/observability-run.sh local-observability
```

The run script refuses the staging project, validates every current web
container's Compose labels, and writes a `file_sd` target for each observed
internal IP. Prometheus reads the Bearer value from a mode-`0600` runtime file,
not a tracked config or URL. Its direct request includes the internal
`X-Forwarded-Proto: https` signal required by the application's production SSL
rewrite while preserving the target's own `instance` label.

Prometheus, Alertmanager, and Grafana are pinned by tag and digest. Their host
ports default to Docker-assigned values bound only to `127.0.0.1`; the run
prints the observed URLs. Grafana uses the random admin password generated in
ignored `.env.load`, disables anonymous signup, update checks, suggested plugin
installation, and its unused built-in alert engine. The Prometheus datasource
and ten-panel dashboard are provisioned from tracked files. The dashboard
separates all discovered web and worker replicas and includes HTTP
traffic/latency, BEAM memory and scheduler run queues, Ecto execution and
pool-wait latency, and Oban outcomes and queue wait by queue.

The verification stops exactly one scoped load web container. The
`WhoNeedHelpWebReplicaUnavailable` rule is based only on the factual
`up == 0` result; it is a local failure drill, not an invented latency,
capacity, or production SLO threshold. The script requires both firing and
resolved webhook payloads from Alertmanager, starts the same container, waits
for every direct target, and compares read-only database counts before and
after. Evidence is retained in `output/observability/` without the metrics or
Grafana secrets.

Stop only the monitoring services with:

```bash
./scripts/observability-stop.sh
```

Prometheus/Grafana/Alertmanager retention, production notification
destinations, production availability, and measured alert policies remain
deployment decisions. In Kubernetes, put the metrics token in
`existingSecret`; configure the external scraper to send it as a Bearer token.

## Rollback boundary

The release image is immutable and migrations run as a separate one-shot role.
Before a schema rollout, create and restore-test a current backup. Application
rollback and database migration rollback are separate decisions: do not run an
Ecto down migration merely because an image is rolled back. Inspect the exact
migration and compatibility boundary first.

The repository intentionally does not ship an automatic destructive production
restore command.