diff --git a/docs/local-hardening-plan.md b/docs/local-hardening-plan.md index cf257a0..d5f2d93 100644 --- a/docs/local-hardening-plan.md +++ b/docs/local-hardening-plan.md @@ -75,19 +75,19 @@ The goal remains open while any row lacks reproducible local evidence. - The isolated three-web/two-worker load profile now uses real password login, authenticated LiveView joins, tracking start, HTTP position writes, private chat writes, and explicit tracking stop. A measured 30-second run completed - all 1,880 authenticated chains and 240 heartbeat sockets with no functional + all 1,888 authenticated chains and 240 heartbeat sockets with no functional failure, retained no current positions after stop, passed cross-node PubSub and readiness, and restored every tracked table count after exact fixture cleanup. - The same isolated profile now passes deliberate web/worker BEAM crashes, sequential replacement of every replica, exact five-node cluster/PubSub checks, and an Oban job that records one failure before succeeding on its - second attempt. A 743-sample readiness probe observed no final HTTP failure + second attempt. A 758-sample readiness probe observed no final HTTP failure and the probe job/domain fixtures were absent afterward. - The project-owned kind cluster also passes a full web/worker rolling restart: all four pod UIDs changed, all replacements became Ready with zero restarts, the four-node cluster/PubSub probe passed, and database counts were unchanged. - The local single-node NodePort needed three reconnect attempts across 305 + The local single-node NodePort needed two reconnect attempts across 362 ultimately successful samples; this is recorded rather than presented as raw transport continuity. - The isolated observability profile scrapes all three web containers by direct @@ -111,5 +111,11 @@ The goal remains open while any row lacks reproducible local evidence. timeout paths. It retained no credentials, published no host ports, removed its exact project and one-run images, and records remote push workflow integration as not implemented. -- The final-regression row remains pending; this document is not a completion - claim for the entire hardening goal. +- The final regression repeated the 167-test quality/security gate, isolated + browser 1/1 bootstrap plus 8/8 Chromium specs, Android debug/staging builds + and 5/5 API 37 device tests, the 50,000-row database benchmark, authenticated + HTTP/LiveView/WebSocket/tracking load, Compose crash/replacement and kind + rolling drills, alert firing/resolution, encrypted backup/restore failure + paths, and the external protocol boundary. Scoped fixtures and one-run + projects were removed, retained evidence is ignored and non-secret, and Git + was clean at the tested application commit before this documentation update. diff --git a/docs/performance.md b/docs/performance.md index 77648db..b2c5482 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -142,43 +142,43 @@ assignment. | Observation | Result | | --- | ---: | -| Public HTTP checks | 35,286 / 35,286 passed | -| All HTTP failures | 0 / 39,062 | +| Public HTTP checks | 35,316 / 35,316 passed | +| All HTTP failures | 0 / 39,108 | | Phoenix heartbeat opened / replies | 240 / 240 | | Authenticated logins | 8 | -| Authenticated pages / LiveView joins | 1,880 / 1,880 | -| Tracking starts / position writes / stops | 1,880 / 1,880 / 1,880 | -| Private messages written | 1,880 | +| Authenticated pages / LiveView joins | 1,888 / 1,888 | +| Tracking starts / position writes / stops | 1,888 / 1,888 / 1,888 | +| Private messages written | 1,888 | | Authenticated or WebSocket errors | 0 | -| HTTP request duration p95 | 5.30 ms | -| Authenticated HTTP duration p95 | 8.18 ms | -| WebSocket connect p95 | 3.55 ms | +| HTTP request duration p95 | 5.29 ms | +| Authenticated HTTP duration p95 | 8.10 ms | +| WebSocket connect p95 | 3.16 ms | -The database evidence counted 1,880 chat rows, 1,880 tracking sessions with -1,880 total samples, 8 login tokens, and zero current positions after explicit +The database evidence counted 1,888 chat rows, 1,888 tracking sessions with +1,888 total samples, 8 login tokens, and zero current positions after explicit stop. Cleanup removed the 176 fixture users, 88 requests, 88 assignments, -1,880 messages, 1,880 tracking sessions, and their audit rows. The post-cleanup +1,888 messages, 1,888 tracking sessions, and their audit rows. The post-cleanup application-table diff was empty. Readiness and the five-node web/worker PubSub probe passed, and the run-scoped proxy/web/worker logs contained no application error, warning, HTTP 4xx/5xx, or database deadlock. -Eleven sequential resource samples observed average CPU of 139.84%, 138.12%, -and 141.90% for the three web containers, with maximum memory of 290.2, 309.4, -and 292.3 MiB respectively. PostgreSQL averaged 38.23% CPU and reached -237.1 MiB; Traefik averaged 45.53% CPU and reached 82.39 MiB. These are +Eleven sequential resource samples observed average CPU of 135.63%, 140.01%, +and 136.88% for the three web containers, with maximum memory of 273.4, 271.2, +and 272.9 MiB respectively. PostgreSQL averaged 37.55% CPU and reached +249.2 MiB; Traefik averaged 47.11% CPU and reached 79.25 MiB. These are short local observations without container resource limits, not minimums, production capacity, or autoscaling inputs. Ignored evidence: -- `output/performance/auth-final-20260719i/` +- `output/performance/final-load-20260719a/` ## Observed local resilience drills The canonical Compose drill on 2026-07-19 used the isolated 3-web/2-worker profile. One web and one worker BEAM process exited with status 1 and each container's observed restart count increased to 1. All original replicas were -then replaced sequentially. The route returned 743 successful readiness +then replaced sequentially. The route returned 758 successful readiness responses with zero final failure and responses from all three web nodes. After replacement, the observed cluster contained all five BEAM nodes and the cross-node PubSub probe passed. @@ -189,20 +189,20 @@ was removed afterward. The load database then contained zero probe jobs, users, requests, messages, and tracking sessions. Run-scoped available logs contained no unexpected application error, warning, HTTP 4xx/5xx, or database deadlock. -The canonical kind drill rolled both 2-replica Deployments from revision 17 to -18 with the chart's observed `maxUnavailable=0` and `maxSurge=1`. All four pod +The canonical kind drill rolled both 2-replica Deployments from revision 19 to +20 with the chart's observed `maxUnavailable=0` and `maxSurge=1`. All four pod UIDs changed, all replacements were Ready with zero container restart, the four-node BEAM cluster and PubSub probe passed, and the application-table count -diff was empty. Of 305 readiness samples, all ultimately returned HTTP 200. -Two samples needed three transport retries in total while kind's single-node +diff was empty. Of 362 readiness samples, all ultimately returned HTTP 200. +Two samples needed one transport retry each while kind's single-node NodePort reset connections during endpoint changes. Those retries are retained in evidence rather than reported as uninterrupted raw TCP connections. This is a local kind observation, not a production availability guarantee. Ignored evidence: -- `output/resilience/compose-resilience-canonical-20260719f/` -- `output/resilience/kind-rollout-canonical-20260719c/` +- `output/resilience/final-resilience-20260719a/` +- `output/resilience/final-kind-20260719a/` ## Observed local monitoring drill @@ -213,7 +213,7 @@ the induced failure. Grafana reported a healthy Prometheus datasource and served the tracked provisioned dashboard with four panels. The script then stopped the exact scoped container -`who_need_help_load-web-13`. Prometheus fired +`who_need_help_load-web-16`. Prometheus fired `WhoNeedHelpWebReplicaUnavailable`, and Alertmanager delivered a webhook whose labels identified that instance and the run. After the same container became healthy again, all three direct targets returned to `up`, the alert cleared, @@ -223,7 +223,7 @@ results, not a production monitoring or notification guarantee. Ignored evidence: -- `output/observability/observability-canonical-20260719c/` +- `output/observability/final-observability-20260719a/` Stop the isolated containers without deleting their database volume: @@ -263,18 +263,18 @@ Observed locally on 2026-07-19 with PostgreSQL 18.4, the canonical run created 50,000 rows in each of users, requests, assignments, messages, reviews, reports, abuse signals, proposals, activities, and activity messages; it created 50,500 activity participants and 49,998 block rows. The isolated -database occupied 231,225,023 bytes after the cursor migration. +database occupied 231,241,407 bytes after the cursor migration. | Query | Baseline observed | Cursor migration observed | Selected cursor index | | --- | ---: | ---: | --- | -| Urgent-help discovery | 8.437 ms | 0.149 ms | `help_requests_discovery_cursor_index` | -| Requester history | 11.287 ms | 0.038 ms | `help_requests_requester_cursor_index` | -| Activity discovery | 8.760 ms | 0.061 ms | `activities_discovery_cursor_index` | -| Visible reviews | 8.997 ms | 0.039 ms | `reviews_visible_cursor_index` | -| All reports | 9.293 ms | 0.026 ms | `reports_cursor_index` | -| All category proposals | 6.776 ms | 0.028 ms | `category_proposals_cursor_index` | -| Moderation users | 5.805 ms | 0.032 ms | `users_moderation_cursor_index` | -| Blocks | 5.388 ms | 0.044 ms | `blocks_blocker_cursor_index` | +| Urgent-help discovery | 8.010 ms | 0.110 ms | `help_requests_discovery_cursor_index` | +| Requester history | 7.969 ms | 0.036 ms | `help_requests_requester_cursor_index` | +| Activity discovery | 8.137 ms | 0.051 ms | `activities_discovery_cursor_index` | +| Visible reviews | 8.264 ms | 0.052 ms | `reviews_visible_cursor_index` | +| All reports | 8.021 ms | 0.025 ms | `reports_cursor_index` | +| All category proposals | 6.501 ms | 0.025 ms | `category_proposals_cursor_index` | +| Moderation users | 5.367 ms | 0.030 ms | `users_moderation_cursor_index` | +| Blocks | 5.165 ms | 0.036 ms | `blocks_blocker_cursor_index` | These are measurements of one warm local run and must not be interpreted as an SLO or portable speedup. The measurement also exposed redundant candidate @@ -287,4 +287,4 @@ BEAM; the leaderboard itself uses composite keyset pagination. Ignored evidence for the recorded run: -- `output/db-scale/20260719020251-1612535/` +- `output/db-scale/20260719054559-2339191/` diff --git a/docs/verification.md b/docs/verification.md index 756de3f..6b60190 100644 --- a/docs/verification.md +++ b/docs/verification.md @@ -19,7 +19,7 @@ results from product limits and unknown production properties. | Social profiles | Manual links implemented; GitHub verification implemented and automated-tested | Manual links cannot set verification fields. The optional GitHub flow uses state, PKCE, a user-bound one-time session, unique provider ownership, and an audit record; 167 tests pass, including callback replay/state checks. The local protocol drill also performs real HTTP token/user exchanges without returning an access token to the application. | The staging operator has not supplied GitHub OAuth credentials, so the real external provider redirect/callback remains disabled and has not been browser-verified. Other providers remain manual/unverified. | | Voluntary thanks | Implemented as an external optional link | A helper can expose an optional link after completion; the UI states that the platform does not process the payment. | The platform does not provide payments, escrow, refunds, tax reporting, or payment guarantees. | | Android client | Local and public-staging clients implemented and emulator-verified | The native packages `org.whoneedhelp.mobile.debug` and `org.whoneedhelp.mobile.staging` launch the same authenticated LiveView app. Public HTTPS login, map, two-way chat, permission prompts, minimized foreground-service location updates, notification Stop, deep-link routing, and server cleanup were exercised on API 37. | Production signing, Play Store publication, verified Android App Links, unattended/background-permission tracking, and iOS are not implemented. | -| Multiple web/worker instances | Implemented and locally failure/rollout-verified | The isolated Compose profile passed BEAM crashes and sequential replacement with 3 web/2 worker replicas, all five nodes joined, PubSub passed, and 743/743 readiness requests succeeded. The project-owned kind cluster replaced all 2 web/2 worker pod UIDs under `maxUnavailable=0`; all four replacement pods joined and PubSub passed. | Local PostGIS is a single instance. Production database HA, backups, and recovery are operator work and are not claimed complete. | +| Multiple web/worker instances | Implemented and locally failure/rollout-verified | The isolated Compose profile passed BEAM crashes and sequential replacement with 3 web/2 worker replicas, all five nodes joined, PubSub passed, and 758/758 readiness requests succeeded. The project-owned kind cluster replaced all 2 web/2 worker pod UIDs under `maxUnavailable=0`; all four replacement pods joined and PubSub passed. | Local PostGIS is a single instance. Production database HA, backups, and recovery are operator work and are not claimed complete. | | Local observability | Implemented and protocol-verified | Pinned Prometheus scraped all 3 direct load web targets with a file Bearer credential; Grafana provisioned a healthy datasource and four-panel dashboard; Alertmanager delivered firing and resolved webhooks for an induced scoped replica stop. | Local delivery does not establish production retention, notification-provider reliability, on-call policy, or measured alert thresholds. | | Encrypted local backup | Implemented and failure-verified | Pinned Restic streamed PostgreSQL custom format into pinned local MinIO with no host plaintext dump, passed full-data checking and a fresh-database restore, rejected a corrupted repository, and published no snapshot for an interrupted upload. | The retained MinIO volume is on the same workstation; this is not off-site storage, database HA, or a production RPO/RTO/retention claim. | | External protocol boundaries | Implemented and locally failure-verified | The production release used its configured Assent/Req and Swoosh/gen_smtp clients against an internal-only mock; OAuth and SMTP success/rejection/retry/replay/timeout paths passed. The future HTTP push boundary passed disabled, retry, rejection and idempotency paths. | This does not verify external provider availability. Remote FCM/APNs delivery and product-workflow integration are not implemented; SMTP exactly-once delivery is not claimed. | @@ -61,10 +61,12 @@ results from product limits and unknown production properties. runner then passed 5/5 instrumentation tests for denied/granted location permission, deep-link lifecycle, foreground location upload, notification Stop, and disconnected-Stop retry state. The public-staging target ran the - unit tests plus `lintStaging` and `assembleStaging`; both previously recorded - lint reports contain no errors or warnings. -- Release Android guard: a release build without `WNH_BASE_URL` failed as - intended; release configuration accepts only an explicit HTTPS origin. + unit tests plus `lintStaging` and `assembleStaging`; both final artifacts + retain the documented hashes. Evidence for the final device run is + `output/android-instrumentation/20260719053610-2122819`. +- Release Android guard: a staging/release build with a non-HTTPS + `WNH_BASE_URL` failed at its dedicated preflight; the successful staging + build used the explicit temporary HTTPS origin. - Helm lint, template rendering, server-side dry-run, rollout waits, readiness checks, and cross-node PubSub verification passed in the local kind cluster. - The hardened kind rollout then ran PostGIS as UID/GID 70 and Mailpit as @@ -78,19 +80,21 @@ results from product limits and unknown production properties. one crashed web and worker BEAM process, replaced every replica sequentially, observed all 5 cluster nodes, passed PubSub, and completed a real Oban retry on attempt 2 after one recorded error. Its exact job row and all fixture - domain rows were absent afterward; 743 readiness samples had no final + domain rows were absent afterward; 758 readiness samples had no final failure. - The reproducible kind rolling drill advanced both Deployments from revision - 17 to 18, replaced all four pod UIDs, observed all 4 BEAM nodes, passed - PubSub, and left the database-count diff empty. All 305 readiness samples - ultimately returned 200; two samples needed three total reconnect attempts + 19 to 20, replaced all four pod UIDs, observed all 4 BEAM nodes, passed + PubSub, and left the database-count diff empty. All 362 readiness samples + ultimately returned 200; two samples needed one reconnect attempt each during local single-node NodePort endpoint replacement. - The canonical local observability drill matched all 3 generated/active Prometheus instance targets, checked the provisioned Grafana datasource and four-panel dashboard, received firing and resolved Alertmanager webhooks for the exact stopped/recovered web replica, restored every target to `up`, and left the application-table count diff empty. Its retained evidence contains - neither the metrics token nor the random Grafana password. + neither the metrics token nor the random Grafana password. Final evidence is + retained at + `output/observability/final-observability-20260719a`. - The observability stop command changed only the four scoped monitoring container states. All app/database/worker container IDs stayed unchanged and running, and the checked user/request/message counts were identical before @@ -108,7 +112,8 @@ results from product limits and unknown production properties. secret appeared in retained evidence, and the source database count diff was empty. The obsolete successful-run bucket was then removed; only the final canonical bucket remains. Evidence is retained at - `output/backups-s3/backup-canonical-20260719d`. + `output/backups-s3/final-backup-20260719a`; the retained MinIO bucket is + `wnh-backup-final-backup-20260719a`. - The canonical external-boundary drill passed real client-side OAuth token and user HTTP exchanges, PKCE/state checks, provider denial, one-time-code replay, a fresh flow after a temporary token error, and timeout failure. It also @@ -123,7 +128,7 @@ results from product limits and unknown production properties. - The committed browser suite passed its 1/1 bootstrap and all 8/8 Chromium specs against a fresh PostGIS volume with two web and two worker replicas on 2026-07-19. The retained successful-run artifact directory is - `output/e2e/20260719012404-847451`. + `output/e2e/20260719053436-2083768`. It covered axe checks across four pages and two themes, keyboard navigation, three responsive widths, Activity/moderation/privacy, the two-user medicine handover and blind reviews, public boundaries, local raster maps, and an @@ -156,7 +161,8 @@ Local generated evidence (ignored by Git): - `output/android/final-image-smoke.png` - `output/android/foreground-notification.png` - `output/android/staging-deep-link-safety.png` -- `output/android-instrumentation/20260718225400-2694340/results.txt` +- `output/android-instrumentation/20260719053610-2122819/results.txt` +- `output/e2e/20260719053436-2083768/` Android artifact: @@ -244,7 +250,7 @@ target-environment measurement. The isolated measurements in `docs/performance.md` include the earlier paced public/readiness/heartbeat comparison and a later 3-web/2-worker run with real authentication, LiveView, chat writes, tracking writes, and explicit position deletion. The latter -completed 1,880 authenticated chains and 240 heartbeat sockets without a +completed 1,888 authenticated chains and 240 heartbeat sockets without a functional failure and restored every tracked application table count after fixture cleanup. Neither short run found a saturation point or represents production traffic, so the Helm chart does not invent resource limits or an HPA @@ -265,6 +271,17 @@ Certbot renewal timer were active. This verifies the current staging path; it does not make the workstation or gateway a production availability environment. +On 2026-07-19 a new read-only check resolved the origin to `77.110.101.144`; +HTTP redirected to HTTPS, and the homepage, Safety page, and readiness endpoint +returned HTTP 200 with successful TLS verification. The certificate still had +the exact SAN and the same expiry. The backing ordinary Compose project was +healthy and its four-node PubSub probe passed. It contained 2 users, 1 help +request, 7 messages, and 8 applied migrations. Because that public-facing data +cannot be assumed disposable, the final local regression did not migrate or +replace that project without separate operator approval; current source was +instead rebuilt and exercised in the isolated Compose profiles and the +project-owned kind cluster. + After the Activity rollout, headed Chrome rendered the updated public navigation, followed the Activity link to the authenticated route, and received the expected login redirect and flash with zero console errors or warnings. @@ -380,35 +397,41 @@ this verification window. ## Final full-stack regression -After isolating the load-test Compose project from staging, the complete -regression was repeated against the final source on 2026-07-18: +The complete local regression was repeated against application commit +`0ad71f6` on 2026-07-19: -- `./scripts/test.sh` rebuilt the test image and passed all 150 Phoenix tests; - `mix format --check-formatted` and `mix compile --warnings-as-errors` also - passed. -- The staging and load-test Compose configurations both rendered successfully. - Helm 4.2.3 lint and template rendering passed with the checked-in kind values. -- The public root, `/healthz/live`, and `/healthz/ready` returned HTTP 200. - The two health responses were served by different web nodes during the - observation. The four-node web/worker cluster also passed the cross-node - Phoenix PubSub probe. -- Headed Chrome loaded the public home and Safety pages, including all - fingerprinted assets, and redirected an anonymous `/requests` visit to login - with the expected flash. The browser reported zero console errors and zero - warnings. Existing browser windows were left open. -- Both Android Docker builds repeated their unit-test, lint, and APK assembly - stages. The debug and staging artifacts retained the documented SHA-256 - hashes, and both lint reports had no nonzero error or warning summary. - `apksigner` verified the staging APK's v2 generic debug signature; `aapt` - confirmed package `org.whoneedhelp.mobile.staging`, SDK 24 through 37, - disabled cleartext, and the exact HTTPS staging host. -- The protected metrics endpoint returned HTTP 401 without a token and valid - Prometheus text with the generated ignored token. The checked 15-minute - web/worker/proxy log window contained no application error, warning, or SCTP - finding. -- The staging database remained at 2 users, 1 help request, 1 assignment, 7 - messages, 14 categories, 4 ended tracking sessions, 0 active tracking - sessions, 0 activities, 0 reports, and 0 social identities. +- `./scripts/quality.sh` passed the full static/security/dependency gate and + 167/167 Phoenix tests. The isolated external-service drill also passed on the + same source and removed its exact one-run project and images. +- The fresh browser project passed its 1/1 bootstrap and all 8/8 Chromium + scenarios, then removed its containers, two networks, and PostgreSQL volume. +- Android debug and public-staging builds repeated unit tests, lint, and APK + assembly. The network-isolated API 37 emulator passed 5/5 device tests and + was removed with its one-run image. A non-HTTPS staging/release origin failed + at the dedicated Gradle guard. +- The database-scale project measured 50,000 rows per large table before and + after the cursor-index migration, verified consecutive cursor pages, and + removed its database volume and image. +- The 3-web/2-worker load run passed 35,316/35,316 public checks, 39,108 HTTP + requests with zero failures, 1,888 authenticated write/tracking chains, and + 240 heartbeat WebSockets. Exact fixture cleanup restored the pre-run + application-table counts and PubSub passed. +- The Compose failure drill observed both BEAM container restarts, replaced all + five replicas, passed 758 readiness samples, and completed and removed the + one-failure/second-attempt Oban probe. The current-image kind rollout replaced + all four pod UIDs under `maxUnavailable=0`, passed all 362 readiness samples, + cluster/PubSub, and an empty database-count diff. +- Prometheus matched all three current web targets; Grafana provisioning passed; + Alertmanager delivered firing and resolved webhooks around one exact replica + stop/recovery; the database-count diff was empty. +- Restic/MinIO passed encrypted stream backup, full-data check, fresh-database + restore, corrupted-repository rejection, and interruption without a snapshot. + Temporary buckets/databases/containers were absent afterward, the obsolete + retained bucket was removed, and only the final canonical bucket remains. +- The ordinary public-facing Compose project was checked read-only: local + live/readiness, four-node PubSub, public HTTPS home/Safety/readiness, DNS, and + TLS passed. Its existing rows and migration level were recorded without + changing them. ## Known work before a public production launch