docs: record final local regression

docs/local-hardening-plan.md

@@ -75,19 +75,19 @@ The goal remains open while any row lacks reproducible local evidence.
 - The isolated three-web/two-worker load profile now uses real password login,
   authenticated LiveView joins, tracking start, HTTP position writes, private
   chat writes, and explicit tracking stop. A measured 30-second run completed
-  all 1,880 authenticated chains and 240 heartbeat sockets with no functional
+  all 1,888 authenticated chains and 240 heartbeat sockets with no functional
   failure, retained no current positions after stop, passed cross-node PubSub
   and readiness, and restored every tracked table count after exact fixture
   cleanup.
 - The same isolated profile now passes deliberate web/worker BEAM crashes,
   sequential replacement of every replica, exact five-node cluster/PubSub
   checks, and an Oban job that records one failure before succeeding on its
-  second attempt. A 743-sample readiness probe observed no final HTTP failure
+  second attempt. A 758-sample readiness probe observed no final HTTP failure
   and the probe job/domain fixtures were absent afterward.
 - The project-owned kind cluster also passes a full web/worker rolling restart:
   all four pod UIDs changed, all replacements became Ready with zero restarts,
   the four-node cluster/PubSub probe passed, and database counts were unchanged.
-  The local single-node NodePort needed three reconnect attempts across 305
+  The local single-node NodePort needed two reconnect attempts across 362
   ultimately successful samples; this is recorded rather than presented as
   raw transport continuity.
 - The isolated observability profile scrapes all three web containers by direct
@@ -111,5 +111,11 @@ The goal remains open while any row lacks reproducible local evidence.
   timeout paths. It retained no credentials, published no host ports, removed
   its exact project and one-run images, and records remote push workflow
   integration as not implemented.
-- The final-regression row remains pending; this document is not a completion
+- The final regression repeated the 167-test quality/security gate, isolated
-  claim for the entire hardening goal.
+  browser 1/1 bootstrap plus 8/8 Chromium specs, Android debug/staging builds
+  and 5/5 API 37 device tests, the 50,000-row database benchmark, authenticated
+  HTTP/LiveView/WebSocket/tracking load, Compose crash/replacement and kind
+  rolling drills, alert firing/resolution, encrypted backup/restore failure
+  paths, and the external protocol boundary. Scoped fixtures and one-run
+  projects were removed, retained evidence is ignored and non-secret, and Git
+  was clean at the tested application commit before this documentation update.

docs/performance.md

@@ -142,43 +142,43 @@ assignment.
 
 | Observation | Result |
 | --- | ---: |
-| Public HTTP checks | 35,286 / 35,286 passed |
+| Public HTTP checks | 35,316 / 35,316 passed |
-| All HTTP failures | 0 / 39,062 |
+| All HTTP failures | 0 / 39,108 |
 | Phoenix heartbeat opened / replies | 240 / 240 |
 | Authenticated logins | 8 |
-| Authenticated pages / LiveView joins | 1,880 / 1,880 |
+| Authenticated pages / LiveView joins | 1,888 / 1,888 |
-| Tracking starts / position writes / stops | 1,880 / 1,880 / 1,880 |
+| Tracking starts / position writes / stops | 1,888 / 1,888 / 1,888 |
-| Private messages written | 1,880 |
+| Private messages written | 1,888 |
 | Authenticated or WebSocket errors | 0 |
-| HTTP request duration p95 | 5.30 ms |
+| HTTP request duration p95 | 5.29 ms |
-| Authenticated HTTP duration p95 | 8.18 ms |
+| Authenticated HTTP duration p95 | 8.10 ms |
-| WebSocket connect p95 | 3.55 ms |
+| WebSocket connect p95 | 3.16 ms |
 
-The database evidence counted 1,880 chat rows, 1,880 tracking sessions with
+The database evidence counted 1,888 chat rows, 1,888 tracking sessions with
-1,880 total samples, 8 login tokens, and zero current positions after explicit
+1,888 total samples, 8 login tokens, and zero current positions after explicit
 stop. Cleanup removed the 176 fixture users, 88 requests, 88 assignments,
-1,880 messages, 1,880 tracking sessions, and their audit rows. The post-cleanup
+1,888 messages, 1,888 tracking sessions, and their audit rows. The post-cleanup
 application-table diff was empty. Readiness and the five-node web/worker PubSub
 probe passed, and the run-scoped proxy/web/worker logs contained no application
 error, warning, HTTP 4xx/5xx, or database deadlock.
 
-Eleven sequential resource samples observed average CPU of 139.84%, 138.12%,
+Eleven sequential resource samples observed average CPU of 135.63%, 140.01%,
-and 141.90% for the three web containers, with maximum memory of 290.2, 309.4,
+and 136.88% for the three web containers, with maximum memory of 273.4, 271.2,
-and 292.3 MiB respectively. PostgreSQL averaged 38.23% CPU and reached
+and 272.9 MiB respectively. PostgreSQL averaged 37.55% CPU and reached
-237.1 MiB; Traefik averaged 45.53% CPU and reached 82.39 MiB. These are
+249.2 MiB; Traefik averaged 47.11% CPU and reached 79.25 MiB. These are
 short local observations without container resource limits, not minimums,
 production capacity, or autoscaling inputs.
 
 Ignored evidence:
 
-- `output/performance/auth-final-20260719i/`
+- `output/performance/final-load-20260719a/`
 
 ## Observed local resilience drills
 
 The canonical Compose drill on 2026-07-19 used the isolated 3-web/2-worker
 profile. One web and one worker BEAM process exited with status 1 and each
 container's observed restart count increased to 1. All original replicas were
-then replaced sequentially. The route returned 743 successful readiness
+then replaced sequentially. The route returned 758 successful readiness
 responses with zero final failure and responses from all three web nodes.
 After replacement, the observed cluster contained all five BEAM nodes and the
 cross-node PubSub probe passed.
@@ -189,20 +189,20 @@ was removed afterward. The load database then contained zero probe jobs, users,
 requests, messages, and tracking sessions. Run-scoped available logs contained
 no unexpected application error, warning, HTTP 4xx/5xx, or database deadlock.
 
-The canonical kind drill rolled both 2-replica Deployments from revision 17 to
+The canonical kind drill rolled both 2-replica Deployments from revision 19 to
-18 with the chart's observed `maxUnavailable=0` and `maxSurge=1`. All four pod
+20 with the chart's observed `maxUnavailable=0` and `maxSurge=1`. All four pod
 UIDs changed, all replacements were Ready with zero container restart, the
 four-node BEAM cluster and PubSub probe passed, and the application-table count
-diff was empty. Of 305 readiness samples, all ultimately returned HTTP 200.
+diff was empty. Of 362 readiness samples, all ultimately returned HTTP 200.
-Two samples needed three transport retries in total while kind's single-node
+Two samples needed one transport retry each while kind's single-node
 NodePort reset connections during endpoint changes. Those retries are retained
 in evidence rather than reported as uninterrupted raw TCP connections. This is
 a local kind observation, not a production availability guarantee.
 
 Ignored evidence:
 
-- `output/resilience/compose-resilience-canonical-20260719f/`
+- `output/resilience/final-resilience-20260719a/`
-- `output/resilience/kind-rollout-canonical-20260719c/`
+- `output/resilience/final-kind-20260719a/`
 
 ## Observed local monitoring drill
 
@@ -213,7 +213,7 @@ the induced failure. Grafana reported a healthy Prometheus datasource and
 served the tracked provisioned dashboard with four panels.
 
 The script then stopped the exact scoped container
-`who_need_help_load-web-13`. Prometheus fired
+`who_need_help_load-web-16`. Prometheus fired
 `WhoNeedHelpWebReplicaUnavailable`, and Alertmanager delivered a webhook whose
 labels identified that instance and the run. After the same container became
 healthy again, all three direct targets returned to `up`, the alert cleared,
@@ -223,7 +223,7 @@ results, not a production monitoring or notification guarantee.
 
 Ignored evidence:
 
-- `output/observability/observability-canonical-20260719c/`
+- `output/observability/final-observability-20260719a/`
 
 Stop the isolated containers without deleting their database volume:
 
@@ -263,18 +263,18 @@ Observed locally on 2026-07-19 with PostgreSQL 18.4, the canonical run created
 50,000 rows in each of users, requests, assignments, messages, reviews,
 reports, abuse signals, proposals, activities, and activity messages; it
 created 50,500 activity participants and 49,998 block rows. The isolated
-database occupied 231,225,023 bytes after the cursor migration.
+database occupied 231,241,407 bytes after the cursor migration.
 
 | Query | Baseline observed | Cursor migration observed | Selected cursor index |
 | --- | ---: | ---: | --- |
-| Urgent-help discovery | 8.437 ms | 0.149 ms | `help_requests_discovery_cursor_index` |
+| Urgent-help discovery | 8.010 ms | 0.110 ms | `help_requests_discovery_cursor_index` |
-| Requester history | 11.287 ms | 0.038 ms | `help_requests_requester_cursor_index` |
+| Requester history | 7.969 ms | 0.036 ms | `help_requests_requester_cursor_index` |
-| Activity discovery | 8.760 ms | 0.061 ms | `activities_discovery_cursor_index` |
+| Activity discovery | 8.137 ms | 0.051 ms | `activities_discovery_cursor_index` |
-| Visible reviews | 8.997 ms | 0.039 ms | `reviews_visible_cursor_index` |
+| Visible reviews | 8.264 ms | 0.052 ms | `reviews_visible_cursor_index` |
-| All reports | 9.293 ms | 0.026 ms | `reports_cursor_index` |
+| All reports | 8.021 ms | 0.025 ms | `reports_cursor_index` |
-| All category proposals | 6.776 ms | 0.028 ms | `category_proposals_cursor_index` |
+| All category proposals | 6.501 ms | 0.025 ms | `category_proposals_cursor_index` |
-| Moderation users | 5.805 ms | 0.032 ms | `users_moderation_cursor_index` |
+| Moderation users | 5.367 ms | 0.030 ms | `users_moderation_cursor_index` |
-| Blocks | 5.388 ms | 0.044 ms | `blocks_blocker_cursor_index` |
+| Blocks | 5.165 ms | 0.036 ms | `blocks_blocker_cursor_index` |
 
 These are measurements of one warm local run and must not be interpreted as an
 SLO or portable speedup. The measurement also exposed redundant candidate
@@ -287,4 +287,4 @@ BEAM; the leaderboard itself uses composite keyset pagination.
 
 Ignored evidence for the recorded run:
 
-- `output/db-scale/20260719020251-1612535/`
+- `output/db-scale/20260719054559-2339191/`

docs/verification.md

@@ -19,7 +19,7 @@ results from product limits and unknown production properties.
 | Social profiles | Manual links implemented; GitHub verification implemented and automated-tested | Manual links cannot set verification fields. The optional GitHub flow uses state, PKCE, a user-bound one-time session, unique provider ownership, and an audit record; 167 tests pass, including callback replay/state checks. The local protocol drill also performs real HTTP token/user exchanges without returning an access token to the application. | The staging operator has not supplied GitHub OAuth credentials, so the real external provider redirect/callback remains disabled and has not been browser-verified. Other providers remain manual/unverified. |
 | Voluntary thanks | Implemented as an external optional link | A helper can expose an optional link after completion; the UI states that the platform does not process the payment. | The platform does not provide payments, escrow, refunds, tax reporting, or payment guarantees. |
 | Android client | Local and public-staging clients implemented and emulator-verified | The native packages `org.whoneedhelp.mobile.debug` and `org.whoneedhelp.mobile.staging` launch the same authenticated LiveView app. Public HTTPS login, map, two-way chat, permission prompts, minimized foreground-service location updates, notification Stop, deep-link routing, and server cleanup were exercised on API 37. | Production signing, Play Store publication, verified Android App Links, unattended/background-permission tracking, and iOS are not implemented. |
-| Multiple web/worker instances | Implemented and locally failure/rollout-verified | The isolated Compose profile passed BEAM crashes and sequential replacement with 3 web/2 worker replicas, all five nodes joined, PubSub passed, and 743/743 readiness requests succeeded. The project-owned kind cluster replaced all 2 web/2 worker pod UIDs under `maxUnavailable=0`; all four replacement pods joined and PubSub passed. | Local PostGIS is a single instance. Production database HA, backups, and recovery are operator work and are not claimed complete. |
+| Multiple web/worker instances | Implemented and locally failure/rollout-verified | The isolated Compose profile passed BEAM crashes and sequential replacement with 3 web/2 worker replicas, all five nodes joined, PubSub passed, and 758/758 readiness requests succeeded. The project-owned kind cluster replaced all 2 web/2 worker pod UIDs under `maxUnavailable=0`; all four replacement pods joined and PubSub passed. | Local PostGIS is a single instance. Production database HA, backups, and recovery are operator work and are not claimed complete. |
 | Local observability | Implemented and protocol-verified | Pinned Prometheus scraped all 3 direct load web targets with a file Bearer credential; Grafana provisioned a healthy datasource and four-panel dashboard; Alertmanager delivered firing and resolved webhooks for an induced scoped replica stop. | Local delivery does not establish production retention, notification-provider reliability, on-call policy, or measured alert thresholds. |
 | Encrypted local backup | Implemented and failure-verified | Pinned Restic streamed PostgreSQL custom format into pinned local MinIO with no host plaintext dump, passed full-data checking and a fresh-database restore, rejected a corrupted repository, and published no snapshot for an interrupted upload. | The retained MinIO volume is on the same workstation; this is not off-site storage, database HA, or a production RPO/RTO/retention claim. |
 | External protocol boundaries | Implemented and locally failure-verified | The production release used its configured Assent/Req and Swoosh/gen_smtp clients against an internal-only mock; OAuth and SMTP success/rejection/retry/replay/timeout paths passed. The future HTTP push boundary passed disabled, retry, rejection and idempotency paths. | This does not verify external provider availability. Remote FCM/APNs delivery and product-workflow integration are not implemented; SMTP exactly-once delivery is not claimed. |
@@ -61,10 +61,12 @@ results from product limits and unknown production properties.
   runner then passed 5/5 instrumentation tests for denied/granted location
   permission, deep-link lifecycle, foreground location upload, notification
   Stop, and disconnected-Stop retry state. The public-staging target ran the
-  unit tests plus `lintStaging` and `assembleStaging`; both previously recorded
+  unit tests plus `lintStaging` and `assembleStaging`; both final artifacts
-  lint reports contain no errors or warnings.
+  retain the documented hashes. Evidence for the final device run is
-- Release Android guard: a release build without `WNH_BASE_URL` failed as
+  `output/android-instrumentation/20260719053610-2122819`.
-  intended; release configuration accepts only an explicit HTTPS origin.
+- Release Android guard: a staging/release build with a non-HTTPS
+  `WNH_BASE_URL` failed at its dedicated preflight; the successful staging
+  build used the explicit temporary HTTPS origin.
 - Helm lint, template rendering, server-side dry-run, rollout waits, readiness
   checks, and cross-node PubSub verification passed in the local kind cluster.
 - The hardened kind rollout then ran PostGIS as UID/GID 70 and Mailpit as
@@ -78,19 +80,21 @@ results from product limits and unknown production properties.
   one crashed web and worker BEAM process, replaced every replica sequentially,
   observed all 5 cluster nodes, passed PubSub, and completed a real Oban retry
   on attempt 2 after one recorded error. Its exact job row and all fixture
-  domain rows were absent afterward; 743 readiness samples had no final
+  domain rows were absent afterward; 758 readiness samples had no final
   failure.
 - The reproducible kind rolling drill advanced both Deployments from revision
-  17 to 18, replaced all four pod UIDs, observed all 4 BEAM nodes, passed
+  19 to 20, replaced all four pod UIDs, observed all 4 BEAM nodes, passed
-  PubSub, and left the database-count diff empty. All 305 readiness samples
+  PubSub, and left the database-count diff empty. All 362 readiness samples
-  ultimately returned 200; two samples needed three total reconnect attempts
+  ultimately returned 200; two samples needed one reconnect attempt each
   during local single-node NodePort endpoint replacement.
 - The canonical local observability drill matched all 3 generated/active
   Prometheus instance targets, checked the provisioned Grafana datasource and
   four-panel dashboard, received firing and resolved Alertmanager webhooks for
   the exact stopped/recovered web replica, restored every target to `up`, and
   left the application-table count diff empty. Its retained evidence contains
-  neither the metrics token nor the random Grafana password.
+  neither the metrics token nor the random Grafana password. Final evidence is
+  retained at
+  `output/observability/final-observability-20260719a`.
 - The observability stop command changed only the four scoped monitoring
   container states. All app/database/worker container IDs stayed unchanged and
   running, and the checked user/request/message counts were identical before
@@ -108,7 +112,8 @@ results from product limits and unknown production properties.
   secret appeared in retained evidence, and the source database count diff was
   empty. The obsolete successful-run bucket was then removed; only the final
   canonical bucket remains. Evidence is retained at
-  `output/backups-s3/backup-canonical-20260719d`.
+  `output/backups-s3/final-backup-20260719a`; the retained MinIO bucket is
+  `wnh-backup-final-backup-20260719a`.
 - The canonical external-boundary drill passed real client-side OAuth token and
   user HTTP exchanges, PKCE/state checks, provider denial, one-time-code replay,
   a fresh flow after a temporary token error, and timeout failure. It also
@@ -123,7 +128,7 @@ results from product limits and unknown production properties.
 - The committed browser suite passed its 1/1 bootstrap and all 8/8 Chromium
   specs against a fresh PostGIS volume with two web and two worker replicas on
   2026-07-19. The retained successful-run artifact directory is
-  `output/e2e/20260719012404-847451`.
+  `output/e2e/20260719053436-2083768`.
   It covered axe checks across four pages and two themes, keyboard navigation,
   three responsive widths, Activity/moderation/privacy, the two-user medicine
   handover and blind reviews, public boundaries, local raster maps, and an
@@ -156,7 +161,8 @@ Local generated evidence (ignored by Git):
 - `output/android/final-image-smoke.png`
 - `output/android/foreground-notification.png`
 - `output/android/staging-deep-link-safety.png`
-- `output/android-instrumentation/20260718225400-2694340/results.txt`
+- `output/android-instrumentation/20260719053610-2122819/results.txt`
+- `output/e2e/20260719053436-2083768/`
 
 Android artifact:
 
@@ -244,7 +250,7 @@ target-environment measurement. The isolated measurements in
 `docs/performance.md` include the earlier paced public/readiness/heartbeat
 comparison and a later 3-web/2-worker run with real authentication, LiveView,
 chat writes, tracking writes, and explicit position deletion. The latter
-completed 1,880 authenticated chains and 240 heartbeat sockets without a
+completed 1,888 authenticated chains and 240 heartbeat sockets without a
 functional failure and restored every tracked application table count after
 fixture cleanup. Neither short run found a saturation point or represents
 production traffic, so the Helm chart does not invent resource limits or an HPA
@@ -265,6 +271,17 @@ Certbot renewal timer were active. This verifies the current staging path; it
 does not make the workstation or gateway a production availability
 environment.
 
+On 2026-07-19 a new read-only check resolved the origin to `77.110.101.144`;
+HTTP redirected to HTTPS, and the homepage, Safety page, and readiness endpoint
+returned HTTP 200 with successful TLS verification. The certificate still had
+the exact SAN and the same expiry. The backing ordinary Compose project was
+healthy and its four-node PubSub probe passed. It contained 2 users, 1 help
+request, 7 messages, and 8 applied migrations. Because that public-facing data
+cannot be assumed disposable, the final local regression did not migrate or
+replace that project without separate operator approval; current source was
+instead rebuilt and exercised in the isolated Compose profiles and the
+project-owned kind cluster.
+
 After the Activity rollout, headed Chrome rendered the updated public
 navigation, followed the Activity link to the authenticated route, and received
 the expected login redirect and flash with zero console errors or warnings.
@@ -380,35 +397,41 @@ this verification window.
 
 ## Final full-stack regression
 
-After isolating the load-test Compose project from staging, the complete
+The complete local regression was repeated against application commit
-regression was repeated against the final source on 2026-07-18:
+`0ad71f6` on 2026-07-19:
 
-- `./scripts/test.sh` rebuilt the test image and passed all 150 Phoenix tests;
+- `./scripts/quality.sh` passed the full static/security/dependency gate and
-  `mix format --check-formatted` and `mix compile --warnings-as-errors` also
+  167/167 Phoenix tests. The isolated external-service drill also passed on the
-  passed.
+  same source and removed its exact one-run project and images.
-- The staging and load-test Compose configurations both rendered successfully.
+- The fresh browser project passed its 1/1 bootstrap and all 8/8 Chromium
-  Helm 4.2.3 lint and template rendering passed with the checked-in kind values.
+  scenarios, then removed its containers, two networks, and PostgreSQL volume.
-- The public root, `/healthz/live`, and `/healthz/ready` returned HTTP 200.
+- Android debug and public-staging builds repeated unit tests, lint, and APK
-  The two health responses were served by different web nodes during the
+  assembly. The network-isolated API 37 emulator passed 5/5 device tests and
-  observation. The four-node web/worker cluster also passed the cross-node
+  was removed with its one-run image. A non-HTTPS staging/release origin failed
-  Phoenix PubSub probe.
+  at the dedicated Gradle guard.
-- Headed Chrome loaded the public home and Safety pages, including all
+- The database-scale project measured 50,000 rows per large table before and
-  fingerprinted assets, and redirected an anonymous `/requests` visit to login
+  after the cursor-index migration, verified consecutive cursor pages, and
-  with the expected flash. The browser reported zero console errors and zero
+  removed its database volume and image.
-  warnings. Existing browser windows were left open.
+- The 3-web/2-worker load run passed 35,316/35,316 public checks, 39,108 HTTP
-- Both Android Docker builds repeated their unit-test, lint, and APK assembly
+  requests with zero failures, 1,888 authenticated write/tracking chains, and
-  stages. The debug and staging artifacts retained the documented SHA-256
+  240 heartbeat WebSockets. Exact fixture cleanup restored the pre-run
-  hashes, and both lint reports had no nonzero error or warning summary.
+  application-table counts and PubSub passed.
-  `apksigner` verified the staging APK's v2 generic debug signature; `aapt`
+- The Compose failure drill observed both BEAM container restarts, replaced all
-  confirmed package `org.whoneedhelp.mobile.staging`, SDK 24 through 37,
+  five replicas, passed 758 readiness samples, and completed and removed the
-  disabled cleartext, and the exact HTTPS staging host.
+  one-failure/second-attempt Oban probe. The current-image kind rollout replaced
-- The protected metrics endpoint returned HTTP 401 without a token and valid
+  all four pod UIDs under `maxUnavailable=0`, passed all 362 readiness samples,
-  Prometheus text with the generated ignored token. The checked 15-minute
+  cluster/PubSub, and an empty database-count diff.
-  web/worker/proxy log window contained no application error, warning, or SCTP
+- Prometheus matched all three current web targets; Grafana provisioning passed;
-  finding.
+  Alertmanager delivered firing and resolved webhooks around one exact replica
-- The staging database remained at 2 users, 1 help request, 1 assignment, 7
+  stop/recovery; the database-count diff was empty.
-  messages, 14 categories, 4 ended tracking sessions, 0 active tracking
+- Restic/MinIO passed encrypted stream backup, full-data check, fresh-database
-  sessions, 0 activities, 0 reports, and 0 social identities.
+  restore, corrupted-repository rejection, and interruption without a snapshot.
+  Temporary buckets/databases/containers were absent afterward, the obsolete
+  retained bucket was removed, and only the final canonical bucket remains.
+- The ordinary public-facing Compose project was checked read-only: local
+  live/readiness, four-node PubSub, public HTTPS home/Safety/readiness, DNS, and
+  TLS passed. Its existing rows and migration level were recorded without
+  changing them.
 
 ## Known work before a public production launch