diff --git a/docs/operations.md b/docs/operations.md index d696e2c..60fbf30 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -705,7 +705,10 @@ Restic snapshot identifier, verification time, and `restore_verified: true`; it contains no database rows, account identifiers, recipient addresses, or credentials. A failed backup or failed restore never advances the heartbeat. -Install the daily user-systemd timer only after one manual `run` succeeds: +Install the daily user-systemd timer on the operator workstation only after one +manual `run` succeeds. The production host does not run this timer. The +workstation connects to production for the source dump and publishes the +restore-verified heartbeat to the independent repository host: ```bash ./scripts/install-production-backup-timer.sh diff --git a/docs/verification.md b/docs/verification.md index 69e87e9..70acc59 100644 --- a/docs/verification.md +++ b/docs/verification.md @@ -3217,3 +3217,51 @@ promoted. Mailpit containers remained healthy and public readiness remained `ready`. The remote repository, Caddy, production release, and frozen-test deployment were unchanged. + +# 2026-08-21 backup schedule and current production read-only load + +- The operator workstation's user-systemd timer + `who-need-help-production-backup.timer` was loaded, enabled, active, and + waiting. It last triggered at 00:00 EEST on 21 August and is scheduled to + trigger again at 00:00 EEST on 22 August. The production host has no backup + timer or service. This matches the reviewed design: the workstation obtains + the source dump over SSH, sends the encrypted Restic snapshot to the separate + BuyVM host, and publishes the restore-verified heartbeat there. +- The scheduled service exited `0/SUCCESS` after 4 minutes 48.415 seconds. It + measured 2.294 seconds of local CPU time and a 50.8 MiB local memory peak. + PostgreSQL dump, checksum, catalogue, full repository data check, and the + isolated restore drill passed. The completed snapshot identifier is + `afdb5c5caa1df84203fc261817a153ade57d133a9102c501db10befc48fc4b77`. +- The independent monitor uses the operator-selected stale-heartbeat threshold + of 129600 seconds, or 36 hours. The current monitor state was `up`, with + readiness, metrics, and backup freshness all passing. This threshold does not + define retention, RPO, RTO, capacity, ownership, or encryption-key custody. +- A current read-only production load probe targeted exact revision + `bb7eb58c8f14d8936cae0e968b50ae721516d213` on the 2-CPU, 4,004,224-KiB host. + For 60 seconds it ran 20 HTTP virtual users against the fixed public GET + allow-list and 20 Phoenix WebSocket virtual users sending only heartbeat + frames. It performed 11,464 HTTP requests with zero failed checks and opened + 20 of 20 sockets with 40 heartbeat replies and zero socket errors. HTTP + duration averaged 54.11 ms, p95 was 73.85 ms, and the maximum was 607.87 ms. +- During that probe, application-container CPU averaged 51.53% and peaked at + 58.29%. Container memory peaked at 207,827,763 bytes. Host available memory + remained at or above 1,352,896 KiB. Readiness returned the exact ready payload + before and after the run. Counts for users, user tokens, and rate-limit + buckets remained `[16, 7, 20]`; PostgreSQL reported no new rollback, temporary + file, temporary byte, or deadlock count. +- This is a public read-only regression measurement, not a representative pilot + traffic model. It does not cover request creation, assignments, chat writes, + location updates, authentication delivery, support submission, push + delivery, or SMTP. No capacity limit or scaling threshold is inferred from + it. The non-secret evidence is retained under + `output/performance/production-current-bb7eb58-20260821/`. +- A headed production browser check reused the existing authenticated Chrome + session. The account menu exposed the expected Account settings route. The + route redirected to the reauthentication screen and displayed the explicit + message that reauthentication is required. No account setting was changed and + no new authentication email was requested. +- Current production metrics exposed bounded purpose labels rather than email + addresses. The observed process reported two successful + `support_confirmation` deliveries, two successful `mail` queue jobs, and no + delivery exception in the selected metric set. These counters describe only + the current application process, not historical delivery volume.