docs: record final performance verification

docs/operations.md

@@ -383,7 +383,10 @@ ports default to Docker-assigned values bound only to `127.0.0.1`; the run
 prints the observed URLs. Grafana uses the random admin password generated in
 ignored `.env.load`, disables anonymous signup, update checks, suggested plugin
 installation, and its unused built-in alert engine. The Prometheus datasource
-and four-panel dashboard are provisioned from tracked files.
+and ten-panel dashboard are provisioned from tracked files. The dashboard
+separates all discovered web and worker replicas and includes HTTP
+traffic/latency, BEAM memory and scheduler run queues, Ecto execution and
+pool-wait latency, and Oban outcomes and queue wait by queue.
 
 The verification stops exactly one scoped load web container. The
 `WhoNeedHelpWebReplicaUnavailable` rule is based only on the factual

docs/performance.md

@@ -581,3 +581,44 @@ working sets and took 53.949 and 41.247 ms in this deliberately concentrated
 fixture. That is a measured future optimization target, not evidence of a
 current failure or a portable latency. Evidence is
 `output/db-scale/20260720035817-2416501`.
+
+## Final queue/runtime load replay
+
+After the queue and network changes, the same isolated 3-web/2-worker profile
+was replayed on 2026-07-20 with 40 paced public HTTP VUs, 40 Phoenix heartbeat
+VUs, and 8 authenticated mutual-aid VUs for 30 seconds.
+
+| Observation | Result |
+| --- | ---: |
+| HTTP requests / failures | 38,962 / 0 |
+| Public checks | 35,280 / 35,280 passed |
+| Authenticated page/chat/tracking chains | 1,833 |
+| Phoenix heartbeat opened / replies | 240 / 240 |
+| HTTP duration average / p95 / maximum | 1.91 / 4.90 / 179.46 ms |
+| Authenticated HTTP duration p95 | 7.77 ms |
+| Configured long-running Repo pool slots | 16 |
+| Peak PostgreSQL client / active backends | 21 / 4 |
+| Minimum observed non-reserved connection headroom | 76 |
+| Ecto average execution / queue / decode time | 313.90 / 29.75 / 0.14 µs |
+| PostgreSQL rollback / deadlock / conflict / temp files | 0 / 0 / 0 / 0 |
+
+The load generated exactly 1,833 chat rows, tracking sessions, and tracking
+samples; explicit stop left zero current positions. Cleanup removed the 176
+fixture users, 88 requests, 88 assignments, 1,833 messages, 1,833 tracking
+sessions, and their audit rows. The before/after application-table count diff
+was empty, all five BEAM nodes participated in the PubSub probe, and every load
+service remained running.
+
+Ten sequential resource samples recorded maxima of 268.5, 260.4, and
+284.3 MiB for the three web cgroups; 207.2 and 198.7 MiB for the workers;
+119.9 MiB for PostgreSQL; and 94.85 MiB for Traefik. These short samples had
+no memory limit and do not establish a steady-state plateau or server minimum.
+No captured query used temporary blocks. The highest mean SQL statement was a
+single 5.249 ms tracking-session maintenance update; the highest-total
+statement was the indexed session-token lookup at 0.081 ms mean across 7,332
+calls. No measured query result justifies another schema or pool change at
+this data size.
+
+Ignored evidence:
+
+- `output/performance/final-after-queue-20260720/`

docs/verification.md

@@ -16,11 +16,11 @@ results from product limits and unknown production properties.
 | Consent-driven live tracking | Implemented and cross-client verified | On API 37, Android started `TrackingService` as a location foreground service with a persistent Stop notification. After Home minimized the Activity, an emulator coordinate change reached PostGIS. Notification Stop removed the service, notification, active session, and raw position. | Browsers stop with the page. Android has no `ACCESS_BACKGROUND_LOCATION`, unattended start, or route history. |
 | Privacy settings | Implemented and browser-verified | The profile exposed hidden, approximate public, exact for active match, and explicit exact-public options. Blocking and current-position cleanup have automated tests. | Exact public location remains a user opt-in; legal privacy and retention text still requires jurisdiction-specific review before launch. |
 | Reputation and anti-abuse | Implemented at MVP level | Handover codes, two-party completion, double-blind reviews, unique-counterpart ranking, optional movement/proximity evidence, reports, blocks, abuse signals, and moderator audit paths have automated tests. | The system is not bot-proof and does not claim identity verification. No punitive numeric policy is enabled without measured and approved thresholds. |
-| Social profiles | Manual links implemented; optional GitHub verification implemented and automated-tested | Manual links cannot set verification fields. The optional GitHub flow uses state, PKCE, a user-bound one-time session, unique provider ownership, and an audit record; the final 172-test suite includes callback replay/state checks. The local protocol drill also performs real HTTP token/user exchanges without returning an access token to the application. | GitHub OAuth credentials are intentionally absent and are not required for registration or the help flow. The real external provider redirect/callback remains disabled and unverified; other providers remain manual/unverified. |
+| Social profiles | Manual links implemented; optional GitHub verification implemented and automated-tested | Manual links cannot set verification fields. The optional GitHub flow uses state, PKCE, a user-bound one-time session, unique provider ownership, and an audit record; the current 186-test suite includes callback replay/state checks. The local protocol drill also performs real HTTP token/user exchanges without returning an access token to the application. | GitHub OAuth credentials are intentionally absent and are not required for registration or the help flow. The real external provider redirect/callback remains disabled and unverified; other providers remain manual/unverified. |
 | Voluntary thanks | Implemented as an external optional link | A helper can expose an optional link after completion; the UI states that the platform does not process the payment. | The platform does not provide payments, escrow, refunds, tax reporting, or payment guarantees. |
 | Android client | Local and public-staging clients implemented and emulator-verified | The native packages `org.whoneedhelp.mobile.debug` and `org.whoneedhelp.mobile.staging` launch the same authenticated LiveView app. Seven lifecycle, permission, deep-link, foreground tracking, recoverable main-page failure, notification-Stop, and Activity-destruction tests passed on each of API 30, 34, and 37. The API 37 staging smoke asserted the public home and Safety DOM over HTTPS. A run-scoped Android/browser staging test passed login, private chat in both directions, foreground tracking, live marker appearance/removal, and exact cleanup. | Production signing, Play Store publication, verified Android App Links, unattended/background-permission tracking, and iOS are not implemented. |
 | Multiple web/worker instances | Implemented and locally failure/rollout-verified | The final isolated Compose drill passed BEAM crashes and sequential replacement with 3 web/2 worker replicas: all five nodes joined, PubSub passed, and 744/744 readiness requests succeeded. The project-owned kind cluster replaced all 2 web/2 worker pod UIDs under `maxUnavailable=0`; all four replacement pods joined, PubSub passed, and 363/363 samples ultimately succeeded. | Local PostGIS is a single instance. Production database HA, backups, and recovery are operator work and are not claimed complete. |
-| Local observability | Implemented and protocol-verified | Pinned Prometheus scraped all 3 direct load web targets with a file Bearer credential; Grafana provisioned a healthy datasource and four-panel dashboard; Alertmanager delivered firing and resolved webhooks for an induced scoped replica stop. | Local delivery does not establish production retention, notification-provider reliability, on-call policy, or measured alert thresholds. |
+| Local observability | Implemented and protocol-verified | Pinned Prometheus scraped the exact 3 web and 2 worker targets with a file Bearer credential; Grafana provisioned a healthy datasource and ten-panel web/worker/BEAM/Ecto/Oban dashboard; Alertmanager delivered firing and resolved webhooks for an induced scoped replica stop. | Local delivery does not establish production retention, notification-provider reliability, on-call policy, or measured alert thresholds. |
 | Encrypted local backup | Implemented and failure-verified | Pinned Restic streamed PostgreSQL custom format into pinned local MinIO with no host plaintext dump, passed full-data checking and a fresh-database restore, rejected a corrupted repository, and published no snapshot for an interrupted upload. The one-run MinIO project and volume were removed after retaining the non-secret evidence. | The drill proves the local mechanism, not off-site durability, database HA, or a production RPO/RTO/retention policy. |
 | External protocol boundaries | Implemented and locally failure-verified | The production release used its configured Assent/Req and Swoosh/gen_smtp clients against an internal-only mock; OAuth and SMTP success/rejection/retry/replay/timeout paths passed. The HTTP push boundary passed disabled, retry, rejection, timeout, and idempotency paths. Request acceptance and new-chat transactions created durable jobs processed by two Oban worker replicas; the chat event completed on Oban attempt 2 after an injected temporary failure. | This does not verify external provider availability or device delivery. FCM/APNs token registration and provider selection remain external work; SMTP exactly-once delivery is not claimed. |
 
@@ -35,7 +35,7 @@ results from product limits and unknown production properties.
 - `./scripts/quality.sh` passed ShellCheck 0.11.0, Hadolint 2.14.0 at warning
   threshold, actionlint 1.7.12, every configured Compose profile render, Helm
   lint, Trivy source/rendered-manifest scanning, xref, Credo high-priority
-  checks, Sobelow strict/private checks, Hex audit, 172 Phoenix tests, both npm
+  checks, Sobelow strict/private checks, Hex audit, 186 Phoenix tests, both npm
   audits, and the
   backup/MinIO/mc/external-mock/release-image vulnerability scans. The rendered
   Helm manifest reported zero HIGH/CRITICAL misconfigurations; the Alpine
@@ -68,10 +68,14 @@ results from product limits and unknown production properties.
   and `assembleStagingAndroidTest`; the API 37 staging smoke asserted the
   rendered `/` and `/safety` DOM from the temporary HTTPS origin with no
   observed load/TLS errors. Evidence is retained at
-  `output/android-instrumentation/api30/20260719222936-2642890`,
-  `output/android-instrumentation/api34/20260719223131-2695250`,
-  `output/android-instrumentation/api37-0/20260719224048-2695249`, and
-  `output/android-staging-smoke/20260719230137-3479426`.
+  `output/android-instrumentation/api30/20260720050057-122714`,
+  `output/android-instrumentation/api34/20260720050900-333071`,
+  `output/android-instrumentation/api37-0/20260720051712-538500`, and
+  `output/android-staging-smoke/20260720052802-802174`.
+  The final matrix used the measured 1 GiB emulator userdata partition. The
+  former 4 GiB default made API 34 fail before boot because the Docker writable
+  layer had 4.23 GiB available while the emulator attempted to create a
+  7.20 GiB partition; host memory and kernel logs showed no OOM event.
 - Release Android guard: a staging/release build with a non-HTTPS
   `WNH_BASE_URL` failed at its dedicated preflight; the successful staging
   build used the explicit temporary HTTPS origin.
@@ -95,14 +99,14 @@ results from product limits and unknown production properties.
   empty. All 363 readiness samples ultimately returned 200; two samples needed
   one reconnect attempt each during local single-node NodePort endpoint
   replacement.
-- The final local observability drill matched all 3 generated/active
+- The final local observability drill matched all 3 web and 2 worker
   Prometheus instance targets, checked the provisioned Grafana datasource and
-  four-panel dashboard, received firing and resolved Alertmanager webhooks for
+  ten-panel dashboard, received firing and resolved Alertmanager webhooks for
   the exact stopped/recovered web replica, restored every target to `up`, and
   left the application-table count diff empty. Its retained evidence contains
   neither the metrics token nor the random Grafana password. Final evidence is
   retained at
-  `output/observability/final-observability-current-20260719`.
+  `output/observability/queue-worker-metrics-fixed-20260720`.
 - The observability stop command changed only the four scoped monitoring
   container states. All app/database/worker container IDs stayed unchanged and
   running, and the checked user/request/message counts were identical before
@@ -134,10 +138,10 @@ results from product limits and unknown production properties.
   retained, no host port was published, and the exact project, volume, network,
   and one-run images were absent after cleanup. Evidence is retained at
   `output/external-boundaries/final-boundaries-20260719`.
-- The committed browser suite passed its 1/1 bootstrap and all 27/27 scenarios:
-  9 each in Chromium, Firefox, and WebKit, against a fresh PostGIS volume with
-  two web and two worker replicas on 2026-07-19. The retained successful-run
-  artifact directory is `output/e2e/20260719162432-2790071`.
+- The committed browser suite passed its 1/1 bootstrap and all 30/30 scenarios:
+  10 each in Chromium, Firefox, and WebKit, against a fresh PostGIS volume with
+  two web and two worker replicas on 2026-07-20. The retained successful-run
+  artifact directory is `output/e2e/20260720042117-3279852`.
   It covered axe checks across four pages and two themes, keyboard navigation,
   three responsive widths, Activity/moderation/privacy, the two-user medicine
   handover and blind reviews, public boundaries, local raster maps, and an
@@ -433,7 +437,7 @@ the two expected messages, one ended tracking session, and no raw current
 position. Cleanup removed only the UUIDs recorded in the run manifest; the
 before/after count diff across 19 application tables was empty. Browser,
 instrumentation, fixture verification, and cleanup all passed. Evidence is
-`output/android-browser-staging-e2e/20260719232413-3987628`.
+`output/android-browser-staging-e2e/20260720053109-872921`.
 
 The browser scenario exposed one UI defect: movement evidence was persisted and
 the marker updated, but the evidence badge remained stale until reload. The
@@ -725,7 +729,7 @@ following additional local and temporary-origin checks:
   private chat in both directions, foreground tracking, browser marker
   appearance/removal, fixture verification, and an empty 19-table cleanup
   diff. Evidence is
-  `output/android-browser-staging-e2e/20260719232413-3987628`.
+  `output/android-browser-staging-e2e/20260720053109-872921`.
 - The external-boundary drill passed OAuth, SMTP, provider-neutral HTTP push,
   product job integration, retry, rejection, timeout, replay, and
   deduplication paths with two isolated workers. Its one-run project, volume,
@@ -743,7 +747,7 @@ following additional local and temporary-origin checks:
   heartbeat sockets without a functional error. PostgreSQL observed zero
   rollbacks, deadlocks, conflicts, or temporary files. Exact fixture cleanup
   produced an empty application-table diff. Evidence is
-  `output/performance/final-load-20260720`.
+  `output/performance/final-after-queue-20260720`.
 - Prometheus exactly matched and scraped all three web targets before and after
   the induced replica stop. Grafana's provisioned Prometheus datasource
   returned `OK`; Alertmanager delivered firing and resolved webhooks for the
@@ -795,12 +799,12 @@ state through the idempotent settings page instead of replaying the token.
   links, blocks, reports, category voting and moderation. Its cleanup restored
   all 19 recorded application-table counts byte-for-byte and removed the
   run-specific Mailpit messages. Evidence is
-  `output/staging-full-e2e/20260719235055-542603`.
+  `output/staging-full-e2e/20260720042807-3462613`.
 - The fresh isolated browser matrix passed bootstrap 1/1 and 30/30 application
   scenarios, 10 each in Chromium, Firefox and WebKit, including realtime
   recovery after the serving BEAM node was restarted. The run-scoped
   containers, networks and PostgreSQL volume were removed. Evidence is
-  `output/e2e/20260719235159-569218`.
+  `output/e2e/20260720042117-3279852`.
 - The public HTTPS PWA replay passed 1/1, covering manifest and install assets,
   service-worker cache update, offline public fallback, and exclusion of
   private application pages from the offline cache. Evidence is