who_need_help/docs/performance.md

Performance measurement

No production capacity, minimum resource requirement, SLO, alert threshold, pool size, or autoscaling threshold is known yet. The repository therefore contains a reproducible measurement profile, not a capacity claim or blocking resource preflight.

The profile uses a separate Compose project, generated independent secrets, and a separate PostgreSQL volume. Its Traefik instance is constrained to that exact Compose project; its router, service, middleware, and Host rule are unique, so the load profile cannot join the staging router. It exercises:

• dynamic public home and Safety pages;
• database-backed readiness queries;
• Phoenix WebSocket upgrades and protocol heartbeats;
• real password login and authenticated LiveView joins;
• consent-driven tracking start, authenticated location writes, private chat writes, and explicit tracking stop/position deletion;
• the configured number of web and worker replicas;
• cross-node Phoenix PubSub after each run.

It does not render a real browser, retain a browser tracking session for an unknown real-world trip duration, or reproduce an unknown future production traffic mix. It also does not sample the k6 load-generator container itself. Its results only describe the recorded host, container versions, replica counts, database state, and experiment inputs.

Create the isolated profile

./scripts/ensure-local-load-env.sh
./scripts/load-stack-up.sh

The generated .env.load is ignored, restricted to mode 600, and contains independent PostgreSQL and application secrets. Edit its LOAD_* inputs to define a specific experiment. Values in .env.load.example are reproducible measurement points, not recommendations.

Run and compare replica counts

./scripts/load-run.sh two-web
./scripts/load-stack-up.sh 3
# Set LOAD_WEB_REPLICAS=3 in .env.load so the recorded expected topology
# matches the running topology, then:
./scripts/load-run.sh three-web

Each run writes ignored evidence below output/performance/<label>/:

• the exact inputs and Docker host observation;
• a machine-readable k6 summary and complete console report;
• periodic Docker CPU/memory/network/block-I/O/PID samples for every load stack container;
• before/after application table counts, exact authenticated-write evidence, fixture cleanup output, and a cleanup diff;
• final Compose state, readiness response, application logs, and cross-node PubSub result.

Before k6 starts, the production-compiled fixture task verifies the exact configured load database name and a deliberate confirmation marker. It creates isolated requester/helper/request/assignment pairs through the application contexts. The pair count equals the maximum simultaneous VU count across all scenarios, because k6 may reuse its global VU pool between parallel scenarios; mapping each global VU identifier directly to its own pair prevents concurrent users from sharing a tracking assignment. The random fixture password exists only in ignored mode-600 .env.load and is never written to the manifest or console.

On success and on trapped failure, cleanup deletes only the UUIDs recorded in the validated run manifest. A successful run additionally requires every tracked application-table count after cleanup to equal its pre-fixture count. In a separate negative check, replacing one helper UUID with another valid UUID made cleanup exit nonzero before mutation; all 2 users, 1 request, and 1 assignment remained. Restoring the original manifest then removed those exact rows and returned all three counts to zero.

The k6 image is pinned to version 2.1.0 by digest and runs locally with anonymous usage reporting disabled. No Grafana Cloud account, API token, OpenAI API, or usage-based service is involved.

Observed local comparison

Observed on 2026-07-18 with Docker Engine 29.6.2 on the recorded 32-CPU, 100289265664-byte Docker host. Neither application containers nor the load generator had CPU or memory limits. Both runs used 40 HTTP VUs, 40 WebSocket VUs, a 30-second duration, a 0.1-second HTTP think time, and five-second socket sessions. Each HTTP iteration requested the home page, Safety page, and database-backed readiness endpoint.

Observation	2 web replicas	3 web replicas
HTTP checks	35,160 / 35,160 passed	35,163 / 35,163 passed
HTTP failed rate	0%	0%
Observed HTTP rate	1,169.12 requests/s	1,169.42 requests/s
HTTP duration p95	2.87 ms	3.36 ms
WebSocket opened / heartbeat replies	240 / 240	240 / 240
WebSocket errors	0	0
WebSocket connect p95	46.65 ms	24.12 ms
Average CPU per web container	66.57%, 66.63%	50.25%, 49.77%, 49.18%
Maximum CPU per web container	79.80%, 79.21%	58.16%, 59.23%, 59.08%
Maximum observed memory per web container	268.3, 238.3 MiB	221.0, 223.2, 219.4 MiB

Each resource series contained 11 sequential docker stats --no-stream samples. Both before/after database diffs were empty, both cross-node PubSub probes passed, readiness remained healthy, and the captured proxy/web/worker logs contained no application error or warning. The public staging root and readiness endpoint also returned HTTP 200 while the isolated project was running.

An earlier harness smoke exposed that fixed Traefik router/service names are global to the Docker provider: starting a second Compose project temporarily made staging readiness return HTTP 404. The load project was stopped immediately and staging returned HTTP 200. The final Compose configuration parameterizes the project constraint, router/service name, Docker network, and router rule. Merged-config inspection found no shared router/service label key, the two running proxies showed different exact project constraints, and both origins returned HTTP 200 concurrently before the recorded final runs.

The nearly identical request rate is expected from the selected paced workload: 40 HTTP VUs each issue three requests and then wait 0.1 seconds, giving an approximate workload ceiling of 1,200 requests/s before response time is added. The result therefore does not show the saturation point of two replicas and does not justify an HPA threshold. The lower per-container CPU and lower WebSocket connect p95 with three replicas are observations for this run only; the slightly higher HTTP p95 also shows why one short comparison is not a production sizing study.

Ignored evidence:

• output/performance/two-web-final-20260718/
• output/performance/three-web-final-20260718/

Observed authenticated write run

Observed on 2026-07-19 on the same recorded Docker host with 3 web replicas, 2 worker replicas, 40 paced public HTTP VUs, 40 five-second Phoenix heartbeat VUs, and 8 authenticated mutual-aid VUs for 30 seconds. The fixture tool created 88 isolated pairs so every possible global VU had a unique account and assignment.

Observation	Result
Public HTTP checks	35,286 / 35,286 passed
All HTTP failures	0 / 39,062
Phoenix heartbeat opened / replies	240 / 240
Authenticated logins	8
Authenticated pages / LiveView joins	1,880 / 1,880
Tracking starts / position writes / stops	1,880 / 1,880 / 1,880
Private messages written	1,880
Authenticated or WebSocket errors	0
HTTP request duration p95	5.30 ms
Authenticated HTTP duration p95	8.18 ms
WebSocket connect p95	3.55 ms

The database evidence counted 1,880 chat rows, 1,880 tracking sessions with 1,880 total samples, 8 login tokens, and zero current positions after explicit stop. Cleanup removed the 176 fixture users, 88 requests, 88 assignments, 1,880 messages, 1,880 tracking sessions, and their audit rows. The post-cleanup application-table diff was empty. Readiness and the five-node web/worker PubSub probe passed, and the run-scoped proxy/web/worker logs contained no application error, warning, HTTP 4xx/5xx, or database deadlock.

Eleven sequential resource samples observed average CPU of 139.84%, 138.12%, and 141.90% for the three web containers, with maximum memory of 290.2, 309.4, and 292.3 MiB respectively. PostgreSQL averaged 38.23% CPU and reached 237.1 MiB; Traefik averaged 45.53% CPU and reached 82.39 MiB. These are short local observations without container resource limits, not minimums, production capacity, or autoscaling inputs.

Ignored evidence:

• output/performance/auth-final-20260719i/

Observed local resilience drills

The canonical Compose drill on 2026-07-19 used the isolated 3-web/2-worker profile. One web and one worker BEAM process exited with status 1 and each container's observed restart count increased to 1. All original replicas were then replaced sequentially. The route returned 743 successful readiness responses with zero final failure and responses from all three web nodes. After replacement, the observed cluster contained all five BEAM nodes and the cross-node PubSub probe passed.

The local Oban probe completed with state completed, attempt 2, max_attempts=2, and exactly one recorded first-attempt error. Its exact row was removed afterward. The load database then contained zero probe jobs, users, requests, messages, and tracking sessions. Run-scoped available logs contained no unexpected application error, warning, HTTP 4xx/5xx, or database deadlock.

The canonical kind drill rolled both 2-replica Deployments from revision 17 to 18 with the chart's observed maxUnavailable=0 and maxSurge=1. All four pod UIDs changed, all replacements were Ready with zero container restart, the four-node BEAM cluster and PubSub probe passed, and the application-table count diff was empty. Of 305 readiness samples, all ultimately returned HTTP 200. Two samples needed three transport retries in total while kind's single-node NodePort reset connections during endpoint changes. Those retries are retained in evidence rather than reported as uninterrupted raw TCP connections. This is a local kind observation, not a production availability guarantee.

Ignored evidence:

• output/resilience/compose-resilience-canonical-20260719f/
• output/resilience/kind-rollout-canonical-20260719c/

Observed local monitoring drill

The canonical observability run on 2026-07-19 generated one direct Prometheus target for each of the three current load web containers. The exact expected and observed instance lists matched, and all three targets were up before the induced failure. Grafana reported a healthy Prometheus datasource and served the tracked provisioned dashboard with four panels.

The script then stopped the exact scoped container who_need_help_load-web-13. Prometheus fired WhoNeedHelpWebReplicaUnavailable, and Alertmanager delivered a webhook whose labels identified that instance and the run. After the same container became healthy again, all three direct targets returned to up, the alert cleared, and a resolved webhook with the same instance/run labels arrived. The application-table count diff was empty. These are observed local protocol results, not a production monitoring or notification guarantee.

Ignored evidence:

• output/observability/observability-canonical-20260719c/

Stop the isolated containers without deleting their database volume:

./scripts/load-stack-stop.sh

The stop script reads the actual LOAD_PROJECT, refuses the staging project name, and does not delete the volume. Volume deletion is intentionally not automated.

Isolated database scale and cursor plans

Run the database-only before/after measurement with:

./scripts/db-scale-benchmark.sh

The script builds the test image and creates a uniquely named Compose project, random one-run PostgreSQL credentials, and a separate PostGIS volume. It migrates only through 20260719004249, seeds the isolated baseline, records machine-readable PostgreSQL EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) plans, applies the generated 20260719013320_add_keyset_pagination_indexes migration, and records the same queries again. It also verifies that two 25-row keyset pages exactly equal the first 50 ordered rows without duplicates or gaps. The exact project, network, volume, and image are removed by its trap.

DB_SCALE_ROWS controls the configured sample size and defaults to 50,000. That default is a reproducible experiment input, not a minimum database size, capacity claim, latency target, or production traffic model. The structural gate checks the canonical sample's chosen cursor indexes and query correctness; it does not fail on an arbitrary millisecond threshold.

Observed locally on 2026-07-19 with PostgreSQL 18.4, the canonical run created 50,000 rows in each of users, requests, assignments, messages, reviews, reports, abuse signals, proposals, activities, and activity messages; it created 50,500 activity participants and 49,998 block rows. The isolated database occupied 231,225,023 bytes after the cursor migration.

Query	Baseline observed	Cursor migration observed	Selected cursor index
Urgent-help discovery	8.437 ms	0.149 ms	help_requests_discovery_cursor_index
Requester history	11.287 ms	0.038 ms	help_requests_requester_cursor_index
Activity discovery	8.760 ms	0.061 ms	activities_discovery_cursor_index
Visible reviews	8.997 ms	0.039 ms	reviews_visible_cursor_index
All reports	9.293 ms	0.026 ms	reports_cursor_index
All category proposals	6.776 ms	0.028 ms	category_proposals_cursor_index
Moderation users	5.805 ms	0.032 ms	users_moderation_cursor_index
Blocks	5.388 ms	0.044 ms	blocks_blocker_cursor_index

These are measurements of one warm local run and must not be interpreted as an SLO or portable speedup. The measurement also exposed redundant candidate indexes and an OR-based Activity membership query. The final migration keeps the indexes PostgreSQL selected, uses partial discovery indexes for open, non-hidden records, and the application joins the existing participant membership invariant directly. Personal reputation and the leaderboard now aggregate in PostgreSQL instead of loading all completed assignments into the BEAM; the leaderboard itself uses composite keyset pagination.

Ignored evidence for the recorded run:

• output/db-scale/20260719020251-1612535/