112 lines
4.6 KiB
Markdown
112 lines
4.6 KiB
Markdown
# Operations runbook
|
|
|
|
This runbook describes the commands that are implemented and verified in this
|
|
repository. It does not claim a production recovery point objective, recovery
|
|
time objective, retention period, storage capacity, or high-availability model;
|
|
those values require product policy and measurements from the eventual
|
|
production environment.
|
|
|
|
## Compose database backup
|
|
|
|
Create a PostgreSQL 18 custom-format archive, validate its table of contents,
|
|
and write a SHA-256 manifest:
|
|
|
|
```bash
|
|
./scripts/backup-compose.sh
|
|
```
|
|
|
|
The default destination is the ignored `output/backups/` directory. An explicit
|
|
new destination may be supplied as the only argument. The command refuses to
|
|
overwrite either an archive or its checksum manifest and writes through
|
|
temporary files before publishing the final pair. Files and newly created
|
|
directories are restricted by `umask 077`.
|
|
|
|
The archive covers the configured application database. PostgreSQL cluster
|
|
globals such as roles and tablespaces are not part of `pg_dump`; deployment
|
|
credentials and database roles must be provisioned separately from secrets.
|
|
Local backup files on the same workstation are not an off-site backup.
|
|
|
|
## Isolated restore drill
|
|
|
|
Run a real restore into a uniquely named temporary database:
|
|
|
|
```bash
|
|
./scripts/restore-drill-compose.sh output/backups/compose-YYYYMMDD-HHMMSS.dump
|
|
```
|
|
|
|
The drill:
|
|
|
|
1. validates the SHA-256 manifest;
|
|
2. validates the archive table of contents;
|
|
3. creates a pristine database from `template0`;
|
|
4. restores with `pg_restore --exit-on-error`;
|
|
5. reads every restored public application table and checks the PostGIS
|
|
library;
|
|
6. runs the current immutable release's migrations and migration-readiness check
|
|
against only the temporary database;
|
|
7. drops only the temporary drill database and verifies that it is gone.
|
|
|
|
A trap also attempts to drop the exact temporary database if a check fails.
|
|
The source application database is never passed to `pg_restore`, `dropdb`, a
|
|
clean operation, or the drill migration runner. The drill intentionally does
|
|
not compare an older backup's row counts to the live source, because concurrent
|
|
legitimate writes or a historical archive would make that comparison invalid.
|
|
|
|
## Service checks
|
|
|
|
```bash
|
|
docker compose ps
|
|
curl --fail http://localhost:4010/healthz/live
|
|
curl --fail http://localhost:4010/healthz/ready
|
|
./scripts/verify-realtime-cluster.sh compose
|
|
```
|
|
|
|
`live` verifies that the web process can serve HTTP. `ready` additionally runs
|
|
`SELECT 1` through the configured Ecto repository. The cluster probe subscribes
|
|
on one connected BEAM node and broadcasts from another.
|
|
|
|
Health checks do not replace alerting, database backups, restore drills, or
|
|
application-level synthetic checks.
|
|
|
|
## Protected Prometheus metrics
|
|
|
|
The web role exposes Prometheus text format at `/metrics`. It requires the
|
|
independent `METRICS_TOKEN` deployment secret:
|
|
|
|
```bash
|
|
curl --fail \
|
|
--header "Authorization: Bearer $METRICS_TOKEN" \
|
|
http://localhost:4010/metrics
|
|
```
|
|
|
|
The endpoint returns `401` without the exact token, disables response caching,
|
|
and does not put the credential in a URL. The reporter exports cumulative HTTP
|
|
request and duration, router exception, database query and duration, WebSocket
|
|
connection, VM memory, and scheduler run-queue metrics. Cumulative durations are
|
|
integer microseconds because the selected reporter's sum accumulator is
|
|
integer-based; divide by `1_000_000` in PromQL when seconds are required.
|
|
Definitions intentionally have no request path, user, request, or event-name
|
|
labels that could create unbounded cardinality.
|
|
|
|
Prometheus itself, durable metrics retention, alert rules, notification
|
|
destinations, and measured alert thresholds are deployment responsibilities and
|
|
are not claimed by this repository. In Kubernetes, put the token in
|
|
`existingSecret`; configure the external scraper to send it as a Bearer token.
|
|
|
|
Metrics are local to each BEAM process. Discover and scrape every web pod or
|
|
container as a distinct target and preserve Prometheus's `instance` label. A
|
|
request through the load-balanced public route reaches only one replica and is
|
|
therefore useful as an authorization/smoke check, not as a cluster-wide
|
|
aggregate.
|
|
|
|
## Rollback boundary
|
|
|
|
The release image is immutable and migrations run as a separate one-shot role.
|
|
Before a schema rollout, create and restore-test a current backup. Application
|
|
rollback and database migration rollback are separate decisions: do not run an
|
|
Ecto down migration merely because an image is rolled back. Inspect the exact
|
|
migration and compatibility boundary first.
|
|
|
|
The repository intentionally does not ship an automatic destructive production
|
|
restore command.
|