diff --git a/docs/local-hardening-plan.md b/docs/local-hardening-plan.md index 9953690..7b626f9 100644 --- a/docs/local-hardening-plan.md +++ b/docs/local-hardening-plan.md @@ -46,11 +46,12 @@ The goal remains open while any row lacks reproducible local evidence. - A forced LiveSocket transport loss while Chromium is offline exposes the reconnect alert, and restoring the network clears it and reconnects the authenticated LiveView. -- The isolated API 30/34/37 Android matrix passes six device tests per API: +- The isolated API 30/34/37 Android matrix passes seven device tests per API: missing and granted location permission, same-origin deep-link routing across Activity recreation, native foreground location upload, notification Stop with remote cleanup, a disconnected Stop request with visible retry state, - and continued uploads after Home and Activity destruction. A separate probe + continued uploads after Home and Activity destruction, and a forced + main-frame disconnect followed by native Retry and a rendered DOM. A separate probe externally kills the instrumentation process and verifies that the `START_NOT_STICKY` service and notification do not return. The suite uses an in-process loopback fixture, runs each emulator without external networking, @@ -117,9 +118,9 @@ The goal remains open while any row lacks reproducible local evidence. retained no credentials, published no host ports, and removed its exact project, volume, network, and one-run images. FCM/APNs device registration is not claimed. -- The final regression repeated the 167-test quality/security gate, isolated +- The final regression repeated the 174-test quality/security gate, isolated browser 1/1 bootstrap plus 8/8 Chromium specs, Android debug/staging builds - and 5/5 API 37 device tests, the 50,000-row database benchmark, authenticated + and 7/7 device tests on API 30, 34, and 37, the 50,000-row database benchmark, authenticated HTTP/LiveView/WebSocket/tracking load, Compose crash/replacement and kind rolling drills, alert firing/resolution, encrypted backup/restore failure paths, and the external protocol boundary. Scoped fixtures and one-run diff --git a/docs/performance.md b/docs/performance.md index 336f18a..f201ae4 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -361,3 +361,30 @@ BEAM; the leaderboard itself uses composite keyset pagination. Ignored evidence for the recorded run: - `output/db-scale/20260719054559-2339191/` + +## Current final replay + +The same threshold-free profiles were replayed on 2026-07-20 against +application commit `b96d443`. + +- The 30-second 3-web/2-worker load run used 40 public HTTP, 40 heartbeat + WebSocket, and 8 authenticated VUs. It completed 13,765 iterations and 38,826 + HTTP requests; all 35,280 checks passed and HTTP failures were 0. The + authenticated scenario completed 1,765 page/LiveView/chat/tracking + start-update-stop chains, and the heartbeat scenario opened and answered 240 + sockets. PostgreSQL recorded no rollback, deadlock, conflict, or temporary + file. The run-specific fixtures were removed and the cleanup count diff was + empty. Evidence is `output/performance/final-load-20260720`. +- The database benchmark again used 50,000 configured rows per large table on + PostgreSQL 18.4. After migration, the asserted list queries selected their + cursor indexes; for example, observed execution time changed from 8.139 ms to + 0.127 ms for urgent-help discovery and from 8.035 ms to 0.050 ms for + Activity discovery. These are single local observations, not thresholds or + portable performance claims. Evidence is + `output/db-scale/20260719233526-97476`. +- The corrected kind rolling run replaced all four application pod UIDs, + retained four Ready zero-restart replacements, passed the four-node BEAM and + PubSub probes, and produced an empty database diff. All 395 readiness samples + ultimately returned HTTP 200; two required one retry during local NodePort + endpoint replacement. Evidence is + `output/resilience/final-kind-rollout-fixed-20260720`. diff --git a/docs/verification.md b/docs/verification.md index bda669e..50f5473 100644 --- a/docs/verification.md +++ b/docs/verification.md @@ -1,6 +1,6 @@ # Who Need Help — implementation verification -Observed through 2026-07-19 in the local workspace. This report separates observed +Observed through 2026-07-20 in the local workspace. This report separates observed results from product limits and unknown production properties. ## Verified MVP capabilities @@ -700,6 +700,74 @@ unrelated application records: Evidence is `output/regression/full-public-web-quality-20260719.log`. +## Final Android and operational replay + +Application commit `b96d443` and its exact source state were subjected to the +following additional local and temporary-origin checks: + +- `./scripts/quality.sh` passed all configured source, Compose, Helm, + observability, dependency, release-image, and security gates with 174/174 + ExUnit tests. Evidence is + `output/regression/android-cross-quality-20260720.log`. +- Android debug and staging unit/lint/APK builds passed. The isolated API + 30/34/37 matrix passed 7/7 device tests per API. Public API 37 DOM smoke + passed 1/1. The Android/browser staging run passed Android magic-link login, + private chat in both directions, foreground tracking, browser marker + appearance/removal, fixture verification, and an empty 19-table cleanup + diff. Evidence is + `output/android-browser-staging-e2e/20260719232413-3987628`. +- The external-boundary drill passed OAuth, SMTP, provider-neutral HTTP push, + product job integration, retry, rejection, timeout, replay, and + deduplication paths with two isolated workers. Its one-run project, volume, + images, network, and generated secrets were removed. Evidence is + `output/external-boundaries/final-boundaries-20260720`. +- The isolated 3-web/2-worker Compose failure drill crashed one web and one + worker BEAM process, sequentially replaced every replica, rejoined all five + nodes, passed PubSub, and completed the injected Oban job on attempt 2 after + one recorded failure. All 725 readiness samples succeeded and the exact job + was deleted. Evidence is + `output/resilience/final-resilience-20260720`. +- The 30-second, 88-VU load replay completed 13,765 iterations and 38,826 HTTP + requests. All 35,280 checks passed; HTTP failures were 0. It completed 1,765 + authenticated LiveView page/chat/tracking start-update-stop chains and 240 + heartbeat sockets without a functional error. PostgreSQL observed zero + rollbacks, deadlocks, conflicts, or temporary files. Exact fixture cleanup + produced an empty application-table diff. Evidence is + `output/performance/final-load-20260720`. +- Prometheus exactly matched and scraped all three web targets before and after + the induced replica stop. Grafana's provisioned Prometheus datasource + returned `OK`; Alertmanager delivered firing and resolved webhooks for the + scoped instance. The database count diff was empty. Evidence is + `output/observability/final-observability-20260720`. +- Restic created and checked an encrypted local S3 snapshot, restored it into a + fresh database, rejected a corrupted repository, and published no snapshot + for an interrupted upload. The interrupted repository passed checking after + pruning unreferenced data; the source database diff was empty. Evidence is + `output/backups-s3/final-backup-20260720`. +- The 50,000-row database replay selected every asserted cursor index and + retained the same generated table counts before and after the index + migration. Its isolated project, volume, and image were removed. Evidence is + `output/db-scale/20260719233526-97476`. +- A clean deployment from `git archive b96d443` generated independent secrets, + applied all 10 migrations, repeated migration without changing its 10 + migration or 14 category counts, and passed HTTP, Mailpit, four-node cluster, + and PubSub checks on 2 web/2 worker replicas. Its workspace, image, project, + volumes, and networks were absent after cleanup. Evidence is + `output/portability/20260719233712-142771-ac1883`. +- The project-owned kind cluster was rebuilt with the current image. The final + rolling drill replaced all four application pod UIDs with four Ready, + zero-restart pods, rejoined all four BEAM nodes, passed PubSub, and left an + empty database diff. All 395 readiness samples ultimately succeeded; two + used one transport retry during the local single-node NodePort endpoint + replacement. Evidence is + `output/resilience/final-kind-rollout-fixed-20260720`. + +The first 2026-07-20 kind attempt exposed a race in the verification harness: +one label-based `kubectl wait` invocation retained a terminating pod in its +initial resource set. The harness now waits for old UIDs to disappear, captures +the exact four replacement pod names, and waits for those resources. The fixed +full replay above passed. + ## Known work before a public production launch - Replace the temporary staging origin with the production-owned domain and diff --git a/scripts/kind-rolling-verify.sh b/scripts/kind-rolling-verify.sh index 6ca456d..2551953 100755 --- a/scripts/kind-rolling-verify.sh +++ b/scripts/kind-rolling-verify.sh @@ -273,13 +273,6 @@ probe_pid=$! "${kube[@]}" rollout status "deployment/$WORKER_DEPLOYMENT" \ "--timeout=$ROLLOUT_TIMEOUT" >"$output_dir/worker-rollout-status.txt" -"${kube[@]}" wait \ - --for=condition=Ready \ - --timeout="$ROLLOUT_TIMEOUT" \ - pod \ - -l app.kubernetes.io/instance=who-need-help \ - >"$output_dir/pods-ready.txt" - old_pod_deadline=$((SECONDS + CLUSTER_JOIN_TIMEOUT)) while ((SECONDS < old_pod_deadline)); do @@ -310,6 +303,31 @@ if [[ "$retained_count" != "0" ]]; then exit 1 fi +mapfile -t current_application_pods < <( + "${kube[@]}" get pods \ + -l app.kubernetes.io/instance=who-need-help \ + -o json | + jq -r ' + .items[] + | select( + .metadata.labels["app.kubernetes.io/component"] == "web" or + .metadata.labels["app.kubernetes.io/component"] == "worker" + ) + | "pod/" + .metadata.name + ' +) + +if [[ "${#current_application_pods[@]}" -ne 4 ]]; then + echo "Expected four replacement application pods; observed ${#current_application_pods[@]}." >&2 + exit 1 +fi + +"${kube[@]}" wait \ + --for=condition=Ready \ + --timeout="$ROLLOUT_TIMEOUT" \ + "${current_application_pods[@]}" \ + >"$output_dir/pods-ready.txt" + deployment_snapshot "$output_dir/deployments-after.json" pod_snapshot "$output_dir/pods-after.json" database_snapshot "$output_dir/database-after.txt"