From 0a96d83d5825ed868bbfe10852558ebe8c7f21e9 Mon Sep 17 00:00:00 2001 From: SimpleTest Date: Mon, 20 Jul 2026 08:44:49 +0300 Subject: [PATCH] docs: record final performance verification --- docs/operations.md | 5 ++++- docs/performance.md | 41 +++++++++++++++++++++++++++++++++++++++++ docs/verification.md | 42 +++++++++++++++++++++++------------------- 3 files changed, 68 insertions(+), 20 deletions(-) diff --git a/docs/operations.md b/docs/operations.md index 17fe637..8dbfc27 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -383,7 +383,10 @@ ports default to Docker-assigned values bound only to `127.0.0.1`; the run prints the observed URLs. Grafana uses the random admin password generated in ignored `.env.load`, disables anonymous signup, update checks, suggested plugin installation, and its unused built-in alert engine. The Prometheus datasource -and four-panel dashboard are provisioned from tracked files. +and ten-panel dashboard are provisioned from tracked files. The dashboard +separates all discovered web and worker replicas and includes HTTP +traffic/latency, BEAM memory and scheduler run queues, Ecto execution and +pool-wait latency, and Oban outcomes and queue wait by queue. The verification stops exactly one scoped load web container. The `WhoNeedHelpWebReplicaUnavailable` rule is based only on the factual diff --git a/docs/performance.md b/docs/performance.md index c543bf5..a9d853a 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -581,3 +581,44 @@ working sets and took 53.949 and 41.247 ms in this deliberately concentrated fixture. That is a measured future optimization target, not evidence of a current failure or a portable latency. Evidence is `output/db-scale/20260720035817-2416501`. + +## Final queue/runtime load replay + +After the queue and network changes, the same isolated 3-web/2-worker profile +was replayed on 2026-07-20 with 40 paced public HTTP VUs, 40 Phoenix heartbeat +VUs, and 8 authenticated mutual-aid VUs for 30 seconds. + +| Observation | Result | +| --- | ---: | +| HTTP requests / failures | 38,962 / 0 | +| Public checks | 35,280 / 35,280 passed | +| Authenticated page/chat/tracking chains | 1,833 | +| Phoenix heartbeat opened / replies | 240 / 240 | +| HTTP duration average / p95 / maximum | 1.91 / 4.90 / 179.46 ms | +| Authenticated HTTP duration p95 | 7.77 ms | +| Configured long-running Repo pool slots | 16 | +| Peak PostgreSQL client / active backends | 21 / 4 | +| Minimum observed non-reserved connection headroom | 76 | +| Ecto average execution / queue / decode time | 313.90 / 29.75 / 0.14 µs | +| PostgreSQL rollback / deadlock / conflict / temp files | 0 / 0 / 0 / 0 | + +The load generated exactly 1,833 chat rows, tracking sessions, and tracking +samples; explicit stop left zero current positions. Cleanup removed the 176 +fixture users, 88 requests, 88 assignments, 1,833 messages, 1,833 tracking +sessions, and their audit rows. The before/after application-table count diff +was empty, all five BEAM nodes participated in the PubSub probe, and every load +service remained running. + +Ten sequential resource samples recorded maxima of 268.5, 260.4, and +284.3 MiB for the three web cgroups; 207.2 and 198.7 MiB for the workers; +119.9 MiB for PostgreSQL; and 94.85 MiB for Traefik. These short samples had +no memory limit and do not establish a steady-state plateau or server minimum. +No captured query used temporary blocks. The highest mean SQL statement was a +single 5.249 ms tracking-session maintenance update; the highest-total +statement was the indexed session-token lookup at 0.081 ms mean across 7,332 +calls. No measured query result justifies another schema or pool change at +this data size. + +Ignored evidence: + +- `output/performance/final-after-queue-20260720/` diff --git a/docs/verification.md b/docs/verification.md index 94f6983..e087fc4 100644 --- a/docs/verification.md +++ b/docs/verification.md @@ -16,11 +16,11 @@ results from product limits and unknown production properties. | Consent-driven live tracking | Implemented and cross-client verified | On API 37, Android started `TrackingService` as a location foreground service with a persistent Stop notification. After Home minimized the Activity, an emulator coordinate change reached PostGIS. Notification Stop removed the service, notification, active session, and raw position. | Browsers stop with the page. Android has no `ACCESS_BACKGROUND_LOCATION`, unattended start, or route history. | | Privacy settings | Implemented and browser-verified | The profile exposed hidden, approximate public, exact for active match, and explicit exact-public options. Blocking and current-position cleanup have automated tests. | Exact public location remains a user opt-in; legal privacy and retention text still requires jurisdiction-specific review before launch. | | Reputation and anti-abuse | Implemented at MVP level | Handover codes, two-party completion, double-blind reviews, unique-counterpart ranking, optional movement/proximity evidence, reports, blocks, abuse signals, and moderator audit paths have automated tests. | The system is not bot-proof and does not claim identity verification. No punitive numeric policy is enabled without measured and approved thresholds. | -| Social profiles | Manual links implemented; optional GitHub verification implemented and automated-tested | Manual links cannot set verification fields. The optional GitHub flow uses state, PKCE, a user-bound one-time session, unique provider ownership, and an audit record; the final 172-test suite includes callback replay/state checks. The local protocol drill also performs real HTTP token/user exchanges without returning an access token to the application. | GitHub OAuth credentials are intentionally absent and are not required for registration or the help flow. The real external provider redirect/callback remains disabled and unverified; other providers remain manual/unverified. | +| Social profiles | Manual links implemented; optional GitHub verification implemented and automated-tested | Manual links cannot set verification fields. The optional GitHub flow uses state, PKCE, a user-bound one-time session, unique provider ownership, and an audit record; the current 186-test suite includes callback replay/state checks. The local protocol drill also performs real HTTP token/user exchanges without returning an access token to the application. | GitHub OAuth credentials are intentionally absent and are not required for registration or the help flow. The real external provider redirect/callback remains disabled and unverified; other providers remain manual/unverified. | | Voluntary thanks | Implemented as an external optional link | A helper can expose an optional link after completion; the UI states that the platform does not process the payment. | The platform does not provide payments, escrow, refunds, tax reporting, or payment guarantees. | | Android client | Local and public-staging clients implemented and emulator-verified | The native packages `org.whoneedhelp.mobile.debug` and `org.whoneedhelp.mobile.staging` launch the same authenticated LiveView app. Seven lifecycle, permission, deep-link, foreground tracking, recoverable main-page failure, notification-Stop, and Activity-destruction tests passed on each of API 30, 34, and 37. The API 37 staging smoke asserted the public home and Safety DOM over HTTPS. A run-scoped Android/browser staging test passed login, private chat in both directions, foreground tracking, live marker appearance/removal, and exact cleanup. | Production signing, Play Store publication, verified Android App Links, unattended/background-permission tracking, and iOS are not implemented. | | Multiple web/worker instances | Implemented and locally failure/rollout-verified | The final isolated Compose drill passed BEAM crashes and sequential replacement with 3 web/2 worker replicas: all five nodes joined, PubSub passed, and 744/744 readiness requests succeeded. The project-owned kind cluster replaced all 2 web/2 worker pod UIDs under `maxUnavailable=0`; all four replacement pods joined, PubSub passed, and 363/363 samples ultimately succeeded. | Local PostGIS is a single instance. Production database HA, backups, and recovery are operator work and are not claimed complete. | -| Local observability | Implemented and protocol-verified | Pinned Prometheus scraped all 3 direct load web targets with a file Bearer credential; Grafana provisioned a healthy datasource and four-panel dashboard; Alertmanager delivered firing and resolved webhooks for an induced scoped replica stop. | Local delivery does not establish production retention, notification-provider reliability, on-call policy, or measured alert thresholds. | +| Local observability | Implemented and protocol-verified | Pinned Prometheus scraped the exact 3 web and 2 worker targets with a file Bearer credential; Grafana provisioned a healthy datasource and ten-panel web/worker/BEAM/Ecto/Oban dashboard; Alertmanager delivered firing and resolved webhooks for an induced scoped replica stop. | Local delivery does not establish production retention, notification-provider reliability, on-call policy, or measured alert thresholds. | | Encrypted local backup | Implemented and failure-verified | Pinned Restic streamed PostgreSQL custom format into pinned local MinIO with no host plaintext dump, passed full-data checking and a fresh-database restore, rejected a corrupted repository, and published no snapshot for an interrupted upload. The one-run MinIO project and volume were removed after retaining the non-secret evidence. | The drill proves the local mechanism, not off-site durability, database HA, or a production RPO/RTO/retention policy. | | External protocol boundaries | Implemented and locally failure-verified | The production release used its configured Assent/Req and Swoosh/gen_smtp clients against an internal-only mock; OAuth and SMTP success/rejection/retry/replay/timeout paths passed. The HTTP push boundary passed disabled, retry, rejection, timeout, and idempotency paths. Request acceptance and new-chat transactions created durable jobs processed by two Oban worker replicas; the chat event completed on Oban attempt 2 after an injected temporary failure. | This does not verify external provider availability or device delivery. FCM/APNs token registration and provider selection remain external work; SMTP exactly-once delivery is not claimed. | @@ -35,7 +35,7 @@ results from product limits and unknown production properties. - `./scripts/quality.sh` passed ShellCheck 0.11.0, Hadolint 2.14.0 at warning threshold, actionlint 1.7.12, every configured Compose profile render, Helm lint, Trivy source/rendered-manifest scanning, xref, Credo high-priority - checks, Sobelow strict/private checks, Hex audit, 172 Phoenix tests, both npm + checks, Sobelow strict/private checks, Hex audit, 186 Phoenix tests, both npm audits, and the backup/MinIO/mc/external-mock/release-image vulnerability scans. The rendered Helm manifest reported zero HIGH/CRITICAL misconfigurations; the Alpine @@ -68,10 +68,14 @@ results from product limits and unknown production properties. and `assembleStagingAndroidTest`; the API 37 staging smoke asserted the rendered `/` and `/safety` DOM from the temporary HTTPS origin with no observed load/TLS errors. Evidence is retained at - `output/android-instrumentation/api30/20260719222936-2642890`, - `output/android-instrumentation/api34/20260719223131-2695250`, - `output/android-instrumentation/api37-0/20260719224048-2695249`, and - `output/android-staging-smoke/20260719230137-3479426`. + `output/android-instrumentation/api30/20260720050057-122714`, + `output/android-instrumentation/api34/20260720050900-333071`, + `output/android-instrumentation/api37-0/20260720051712-538500`, and + `output/android-staging-smoke/20260720052802-802174`. + The final matrix used the measured 1 GiB emulator userdata partition. The + former 4 GiB default made API 34 fail before boot because the Docker writable + layer had 4.23 GiB available while the emulator attempted to create a + 7.20 GiB partition; host memory and kernel logs showed no OOM event. - Release Android guard: a staging/release build with a non-HTTPS `WNH_BASE_URL` failed at its dedicated preflight; the successful staging build used the explicit temporary HTTPS origin. @@ -95,14 +99,14 @@ results from product limits and unknown production properties. empty. All 363 readiness samples ultimately returned 200; two samples needed one reconnect attempt each during local single-node NodePort endpoint replacement. -- The final local observability drill matched all 3 generated/active +- The final local observability drill matched all 3 web and 2 worker Prometheus instance targets, checked the provisioned Grafana datasource and - four-panel dashboard, received firing and resolved Alertmanager webhooks for + ten-panel dashboard, received firing and resolved Alertmanager webhooks for the exact stopped/recovered web replica, restored every target to `up`, and left the application-table count diff empty. Its retained evidence contains neither the metrics token nor the random Grafana password. Final evidence is retained at - `output/observability/final-observability-current-20260719`. + `output/observability/queue-worker-metrics-fixed-20260720`. - The observability stop command changed only the four scoped monitoring container states. All app/database/worker container IDs stayed unchanged and running, and the checked user/request/message counts were identical before @@ -134,10 +138,10 @@ results from product limits and unknown production properties. retained, no host port was published, and the exact project, volume, network, and one-run images were absent after cleanup. Evidence is retained at `output/external-boundaries/final-boundaries-20260719`. -- The committed browser suite passed its 1/1 bootstrap and all 27/27 scenarios: - 9 each in Chromium, Firefox, and WebKit, against a fresh PostGIS volume with - two web and two worker replicas on 2026-07-19. The retained successful-run - artifact directory is `output/e2e/20260719162432-2790071`. +- The committed browser suite passed its 1/1 bootstrap and all 30/30 scenarios: + 10 each in Chromium, Firefox, and WebKit, against a fresh PostGIS volume with + two web and two worker replicas on 2026-07-20. The retained successful-run + artifact directory is `output/e2e/20260720042117-3279852`. It covered axe checks across four pages and two themes, keyboard navigation, three responsive widths, Activity/moderation/privacy, the two-user medicine handover and blind reviews, public boundaries, local raster maps, and an @@ -433,7 +437,7 @@ the two expected messages, one ended tracking session, and no raw current position. Cleanup removed only the UUIDs recorded in the run manifest; the before/after count diff across 19 application tables was empty. Browser, instrumentation, fixture verification, and cleanup all passed. Evidence is -`output/android-browser-staging-e2e/20260719232413-3987628`. +`output/android-browser-staging-e2e/20260720053109-872921`. The browser scenario exposed one UI defect: movement evidence was persisted and the marker updated, but the evidence badge remained stale until reload. The @@ -725,7 +729,7 @@ following additional local and temporary-origin checks: private chat in both directions, foreground tracking, browser marker appearance/removal, fixture verification, and an empty 19-table cleanup diff. Evidence is - `output/android-browser-staging-e2e/20260719232413-3987628`. + `output/android-browser-staging-e2e/20260720053109-872921`. - The external-boundary drill passed OAuth, SMTP, provider-neutral HTTP push, product job integration, retry, rejection, timeout, replay, and deduplication paths with two isolated workers. Its one-run project, volume, @@ -743,7 +747,7 @@ following additional local and temporary-origin checks: heartbeat sockets without a functional error. PostgreSQL observed zero rollbacks, deadlocks, conflicts, or temporary files. Exact fixture cleanup produced an empty application-table diff. Evidence is - `output/performance/final-load-20260720`. + `output/performance/final-after-queue-20260720`. - Prometheus exactly matched and scraped all three web targets before and after the induced replica stop. Grafana's provisioned Prometheus datasource returned `OK`; Alertmanager delivered firing and resolved webhooks for the @@ -795,12 +799,12 @@ state through the idempotent settings page instead of replaying the token. links, blocks, reports, category voting and moderation. Its cleanup restored all 19 recorded application-table counts byte-for-byte and removed the run-specific Mailpit messages. Evidence is - `output/staging-full-e2e/20260719235055-542603`. + `output/staging-full-e2e/20260720042807-3462613`. - The fresh isolated browser matrix passed bootstrap 1/1 and 30/30 application scenarios, 10 each in Chromium, Firefox and WebKit, including realtime recovery after the serving BEAM node was restarted. The run-scoped containers, networks and PostgreSQL volume were removed. Evidence is - `output/e2e/20260719235159-569218`. + `output/e2e/20260720042117-3279852`. - The public HTTPS PWA replay passed 1/1, covering manifest and install assets, service-worker cache update, offline public fallback, and exclusion of private application pages from the offline cache. Evidence is