who_need_help/docs/performance.md

# Performance measurement

No production capacity, minimum resource requirement, SLO, alert threshold,
pool size, or autoscaling threshold is known yet. The repository therefore
contains a reproducible measurement profile, not a capacity claim or blocking
resource preflight.

The profile uses a separate Compose project, generated independent secrets, and
a separate PostgreSQL volume. Its Traefik instance is constrained to that exact
Compose project; its router, service, middleware, and Host rule are unique, so
the load profile cannot join the staging router. It exercises:

- dynamic public home and Safety pages;
- database-backed readiness queries;
- Phoenix WebSocket upgrades and protocol heartbeats;
- real password login and authenticated LiveView joins;
- consent-driven tracking start, authenticated location writes, private chat
  writes, and explicit tracking stop/position deletion;
- the configured number of web and worker replicas;
- cross-node Phoenix PubSub after each run.

It does not render a real browser, retain a browser tracking session for an
unknown real-world trip duration, or reproduce an unknown future production
traffic mix. It also does not sample the k6 load-generator container itself.
Its results only describe the recorded host, container versions, replica
counts, database state, and experiment inputs.

## Create the isolated profile

```sh
./scripts/ensure-local-load-env.sh
./scripts/load-stack-up.sh
```

The generated `.env.load` is ignored, restricted to mode 600, and contains
independent PostgreSQL and application secrets. Edit its `LOAD_*` inputs to
define a specific experiment. Values in `.env.load.example` are reproducible
measurement points, not recommendations.

For a one-run duration that must not rewrite `.env.load`, pass an explicit
experiment override:

```sh
LOAD_DURATION_OVERRIDE=10m ./scripts/load-run.sh local-soak-YYYYMMDD
```

The effective value and whether it came from `.env.load` or the override are
recorded in that run's `environment.txt`.

## Run and compare replica counts

```sh
./scripts/load-run.sh two-web
./scripts/load-stack-up.sh 3
# Set LOAD_WEB_REPLICAS=3 in .env.load so the recorded expected topology
# matches the running topology, then:
./scripts/load-run.sh three-web
```

Each run writes ignored evidence below `output/performance/<label>/`:

- the exact inputs and Docker host observation;
- a machine-readable k6 summary and complete console report;
- periodic Docker CPU/memory/network/block-I/O/PID samples for every load
  stack container, plus a threshold-free CPU/RAM/PID JSON summary;
- read-only PostgreSQL database size and `pg_stat_database` snapshots before
  and after k6, plus their machine-readable counter deltas;
- before/after application table counts, exact authenticated-write evidence,
  fixture cleanup output, and a cleanup diff;
- final Compose state, readiness response, application logs, and cross-node
  PubSub result.

Before k6 starts, the production-compiled fixture task verifies the exact
configured load database name and a deliberate confirmation marker. It creates
isolated requester/helper/request/assignment pairs through the application
contexts. The pair count equals the maximum simultaneous VU count across all
scenarios, because k6 may reuse its global VU pool between parallel scenarios;
mapping each global VU identifier directly to its own pair prevents concurrent
users from sharing a tracking assignment. The random fixture password exists
only in ignored mode-600 `.env.load` and is never written to the manifest or
console.

On success and on trapped failure, cleanup deletes only the UUIDs recorded in
the validated run manifest. A successful run additionally requires every
tracked application-table count after cleanup to equal its pre-fixture count.
In a separate negative check, replacing one helper UUID with another valid UUID
made cleanup exit nonzero before mutation; all 2 users, 1 request, and 1
assignment remained. Restoring the original manifest then removed those exact
rows and returned all three counts to zero.

The k6 image is pinned to version 2.1.0 by digest and runs locally with anonymous
usage reporting disabled. No Grafana Cloud account, API token, OpenAI API, or
usage-based service is involved.

## Observed local comparison

Observed on 2026-07-18 with Docker Engine 29.6.2 on the recorded 32-CPU,
100289265664-byte Docker host. Neither application containers nor the load
generator had CPU or memory limits. Both runs used 40 HTTP VUs, 40 WebSocket
VUs, a 30-second duration, a 0.1-second HTTP think time, and five-second socket
sessions. Each HTTP iteration requested the home page, Safety page, and
database-backed readiness endpoint.

| Observation | 2 web replicas | 3 web replicas |
| --- | ---: | ---: |
| HTTP checks | 35,160 / 35,160 passed | 35,163 / 35,163 passed |
| HTTP failed rate | 0% | 0% |
| Observed HTTP rate | 1,169.12 requests/s | 1,169.42 requests/s |
| HTTP duration p95 | 2.87 ms | 3.36 ms |
| WebSocket opened / heartbeat replies | 240 / 240 | 240 / 240 |
| WebSocket errors | 0 | 0 |
| WebSocket connect p95 | 46.65 ms | 24.12 ms |
| Average CPU per web container | 66.57%, 66.63% | 50.25%, 49.77%, 49.18% |
| Maximum CPU per web container | 79.80%, 79.21% | 58.16%, 59.23%, 59.08% |
| Maximum observed memory per web container | 268.3, 238.3 MiB | 221.0, 223.2, 219.4 MiB |

Each resource series contained 11 sequential `docker stats --no-stream`
samples. Both before/after database diffs were empty, both cross-node PubSub
probes passed, readiness remained healthy, and the captured proxy/web/worker
logs contained no application error or warning. The public staging root and
readiness endpoint also returned HTTP 200 while the isolated project was
running.

An earlier harness smoke exposed that fixed Traefik router/service names are
global to the Docker provider: starting a second Compose project temporarily
made staging readiness return HTTP 404. The load project was stopped
immediately and staging returned HTTP 200. The final Compose configuration
parameterizes the project constraint, router/service name, Docker network, and
router rule. Merged-config inspection found no shared router/service label key,
the two running proxies showed different exact project constraints, and both
origins returned HTTP 200 concurrently before the recorded final runs.

The nearly identical request rate is expected from the selected paced workload:
40 HTTP VUs each issue three requests and then wait 0.1 seconds, giving an
approximate workload ceiling of 1,200 requests/s before response time is added.
The result therefore does not show the saturation point of two replicas and
does not justify an HPA threshold. The lower per-container CPU and lower
WebSocket connect p95 with three replicas are observations for this run only;
the slightly higher HTTP p95 also shows why one short comparison is not a
production sizing study.

Ignored evidence:

- `output/performance/two-web-final-20260718/`
- `output/performance/three-web-final-20260718/`

## Observed authenticated write run

Observed on 2026-07-19 on the same recorded Docker host with 3 web replicas,
2 worker replicas, 40 paced public HTTP VUs, 40 five-second Phoenix heartbeat
VUs, and 8 authenticated mutual-aid VUs for 30 seconds. The fixture tool
created 88 isolated pairs so every possible global VU had a unique account and
assignment.

| Observation | Result |
| --- | ---: |
| Public HTTP checks | 35,316 / 35,316 passed |
| All HTTP failures | 0 / 39,108 |
| Phoenix heartbeat opened / replies | 240 / 240 |
| Authenticated logins | 8 |
| Authenticated pages / LiveView joins | 1,888 / 1,888 |
| Tracking starts / position writes / stops | 1,888 / 1,888 / 1,888 |
| Private messages written | 1,888 |
| Authenticated or WebSocket errors | 0 |
| HTTP request duration p95 | 5.29 ms |
| Authenticated HTTP duration p95 | 8.10 ms |
| WebSocket connect p95 | 3.16 ms |

The database evidence counted 1,888 chat rows, 1,888 tracking sessions with
1,888 total samples, 8 login tokens, and zero current positions after explicit
stop. Cleanup removed the 176 fixture users, 88 requests, 88 assignments,
1,888 messages, 1,888 tracking sessions, and their audit rows. The post-cleanup
application-table diff was empty. Readiness and the five-node web/worker PubSub
probe passed, and the run-scoped proxy/web/worker logs contained no application
error, warning, HTTP 4xx/5xx, or database deadlock.

Eleven sequential resource samples observed average CPU of 135.63%, 140.01%,
and 136.88% for the three web containers, with maximum memory of 273.4, 271.2,
and 272.9 MiB respectively. PostgreSQL averaged 37.55% CPU and reached
249.2 MiB; Traefik averaged 47.11% CPU and reached 79.25 MiB. These are
short local observations without container resource limits, not minimums,
production capacity, or autoscaling inputs.

Ignored evidence:

- `output/performance/final-load-20260719a/`

## Observed 10-minute authenticated soak

Observed on 2026-07-19 with the same 3-web/2-worker topology and the same
40 public HTTP, 40 five-second Phoenix heartbeat, and 8 authenticated
mutual-aid VUs. The duration came from the recorded
`LOAD_DURATION_OVERRIDE=10m`; it is an experiment input, not a recommended
soak duration or capacity target.

| Observation | Result |
| --- | ---: |
| Completed / interrupted iterations | 271,072 / 0 |
| Public checks | 703,995 / 703,995 passed |
| HTTP requests / failures | 767,225 / 0 |
| HTTP duration average / p95 / maximum | 2.34 / 5.71 / 175.96 ms |
| Phoenix heartbeat opened / replies | 4,800 / 4,800 |
| Authenticated logins | 8 |
| Authenticated pages / LiveView joins | 31,607 / 31,607 |
| Tracking starts / position writes / stops | 31,607 / 31,607 / 31,607 |
| Private messages written | 31,607 |
| Authenticated or WebSocket errors | 0 |

The database evidence counted exactly those 31,607 messages, tracking
sessions, and samples, plus 8 login tokens and zero current positions after
explicit stop. Its read-only `pg_stat_database` delta recorded 2,921,283
commits, zero rollbacks, 12 block reads, 33,967,340 block hits, zero temporary
files/bytes, zero deadlocks, and zero conflicts. Database size increased by
16,678,912 bytes during the write workload. The physical database size was
recorded but was not treated as a cleanup assertion; the post-cleanup count
diff across all tracked application tables was empty. The five-node PubSub
probe passed. All seven measured app/database/proxy containers remained
running, had zero observed restarts, and were not OOM-killed. The 75,058,636
byte run-scoped log had zero matches for the recorded error, warning,
exception, crash, HTTP 4xx/5xx, and deadlock search.

Each container has 199 sequential resource samples. No container resource
limit or pass/fail resource threshold was applied.

| Container | Average / max CPU | First 10 / last 10 memory average | Max memory |
| --- | ---: | ---: | ---: |
| PostgreSQL | 61.44% / 85.30% | 236.15 / 261.81 MiB | 263.60 MiB |
| Traefik | 57.25% / 73.14% | 93.44 / 103.98 MiB | 108.50 MiB |
| web 16 | 198.67% / 281.58% | 264.24 / 376.33 MiB | 397.10 MiB |
| web 17 | 195.66% / 247.46% | 259.30 / 373.79 MiB | 406.10 MiB |
| web 18 | 195.36% / 266.74% | 265.06 / 388.06 MiB | 410.80 MiB |
| worker 10 | 1.47% / 5.09% | 201.63 / 202.64 MiB | 203.80 MiB |
| worker 11 | 2.21% / 5.56% | 199.30 / 201.05 MiB | 201.80 MiB |

The web RSS increased across most of the loaded interval, so this run does
not establish a steady-state memory plateau. A separate read-only observation
at 2026-07-19T16:06:21Z, 5 minutes 23 seconds after the k6 summary timestamp,
measured web RSS at 278.2, 269.8, and 285.6 MiB. At that point the BEAM runtime
reported 142,250,344, 147,808,440, and 147,403,872 total bytes with 613, 612,
and 612 processes. This observed post-load reduction shows that a substantial
part of the loaded RSS was released after the connections closed; it does not
prove that the application is free of a longer-term memory leak. A
production-like duration and traffic distribution remain unknown.

Ignored evidence:

- `output/performance/authenticated-soak-10m-20260719/`

## Observed local resilience drills

The canonical Compose drill on 2026-07-19 used the isolated 3-web/2-worker
profile. One web and one worker BEAM process exited with status 1 and each
container's observed restart count increased to 1. All original replicas were
then replaced sequentially. The route returned 758 successful readiness
responses with zero final failure and responses from all three web nodes.
After replacement, the observed cluster contained all five BEAM nodes and the
cross-node PubSub probe passed.

The local Oban probe completed with state `completed`, attempt `2`,
`max_attempts=2`, and exactly one recorded first-attempt error. Its exact row
was removed afterward. The load database then contained zero probe jobs, users,
requests, messages, and tracking sessions. Run-scoped available logs contained
no unexpected application error, warning, HTTP 4xx/5xx, or database deadlock.

The canonical kind drill rolled both 2-replica Deployments from revision 19 to
20 with the chart's observed `maxUnavailable=0` and `maxSurge=1`. All four pod
UIDs changed, all replacements were Ready with zero container restart, the
four-node BEAM cluster and PubSub probe passed, and the application-table count
diff was empty. Of 362 readiness samples, all ultimately returned HTTP 200.
Two samples needed one transport retry each while kind's single-node
NodePort reset connections during endpoint changes. Those retries are retained
in evidence rather than reported as uninterrupted raw TCP connections. This is
a local kind observation, not a production availability guarantee.

Ignored evidence:

- `output/resilience/final-resilience-20260719a/`
- `output/resilience/final-kind-20260719a/`

## Observed local monitoring drill

The canonical observability run on 2026-07-19 generated one direct Prometheus
target for each of the three current load web containers. The exact expected
and observed `instance` lists matched, and all three targets were `up` before
the induced failure. Grafana reported a healthy Prometheus datasource and
served the tracked provisioned dashboard with four panels.

The script then stopped the exact scoped container
`who_need_help_load-web-16`. Prometheus fired
`WhoNeedHelpWebReplicaUnavailable`, and Alertmanager delivered a webhook whose
labels identified that instance and the run. After the same container became
healthy again, all three direct targets returned to `up`, the alert cleared,
and a resolved webhook with the same instance/run labels arrived. The
application-table count diff was empty. These are observed local protocol
results, not a production monitoring or notification guarantee.

Ignored evidence:

- `output/observability/final-observability-20260719a/`

Stop the isolated containers without deleting their database volume:

```sh
./scripts/load-stack-stop.sh
```

The stop script reads the actual `LOAD_PROJECT`, refuses the staging project
name, and does not delete the volume. Volume deletion is intentionally not
automated.

## Isolated database scale and cursor plans

Run the database-only before/after measurement with:

```sh
./scripts/db-scale-benchmark.sh
```

The script builds the test image and creates a uniquely named Compose project,
random one-run PostgreSQL credentials, and a separate PostGIS volume. It
migrates only through `20260719004249`, seeds the isolated baseline, records
machine-readable PostgreSQL
`EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON)` plans, applies the generated
`20260719013320_add_keyset_pagination_indexes` migration, and records the same
queries again. It also verifies that two 25-row keyset pages exactly equal the
first 50 ordered rows without duplicates or gaps. The exact project, network,
volume, and image are removed by its trap.

`DB_SCALE_ROWS` controls the configured sample size and defaults to 50,000.
That default is a reproducible experiment input, not a minimum database size,
capacity claim, latency target, or production traffic model. The structural
gate checks the canonical sample's chosen cursor indexes and query correctness;
it does not fail on an arbitrary millisecond threshold.

Observed locally on 2026-07-19 with PostgreSQL 18.4, the canonical run created
50,000 rows in each of users, requests, assignments, messages, reviews,
reports, abuse signals, proposals, activities, and activity messages; it
created 50,500 activity participants and 49,998 block rows. The isolated
database occupied 231,241,407 bytes after the cursor migration.

| Query | Baseline observed | Cursor migration observed | Selected cursor index |
| --- | ---: | ---: | --- |
| Urgent-help discovery | 8.010 ms | 0.110 ms | `help_requests_discovery_cursor_index` |
| Requester history | 7.969 ms | 0.036 ms | `help_requests_requester_cursor_index` |
| Activity discovery | 8.137 ms | 0.051 ms | `activities_discovery_cursor_index` |
| Visible reviews | 8.264 ms | 0.052 ms | `reviews_visible_cursor_index` |
| All reports | 8.021 ms | 0.025 ms | `reports_cursor_index` |
| All category proposals | 6.501 ms | 0.025 ms | `category_proposals_cursor_index` |
| Moderation users | 5.367 ms | 0.030 ms | `users_moderation_cursor_index` |
| Blocks | 5.165 ms | 0.036 ms | `blocks_blocker_cursor_index` |

These are measurements of one warm local run and must not be interpreted as an
SLO or portable speedup. The measurement also exposed redundant candidate
indexes and an `OR`-based Activity membership query. The final migration keeps
the indexes PostgreSQL selected, uses partial discovery indexes for open,
non-hidden records, and the application joins the existing participant
membership invariant directly. Personal reputation and the leaderboard now
aggregate in PostgreSQL instead of loading all completed assignments into the
BEAM; the leaderboard itself uses composite keyset pagination.

Ignored evidence for the recorded run:

- `output/db-scale/20260719054559-2339191/`

## Current final replay

The same threshold-free profiles were replayed on 2026-07-20 against
application commit `b96d443`.

- The 30-second 3-web/2-worker load run used 40 public HTTP, 40 heartbeat
  WebSocket, and 8 authenticated VUs. It completed 13,765 iterations and 38,826
  HTTP requests; all 35,280 checks passed and HTTP failures were 0. The
  authenticated scenario completed 1,765 page/LiveView/chat/tracking
  start-update-stop chains, and the heartbeat scenario opened and answered 240
  sockets. PostgreSQL recorded no rollback, deadlock, conflict, or temporary
  file. The run-specific fixtures were removed and the cleanup count diff was
  empty. Evidence is `output/performance/final-load-20260720`.
- The database benchmark again used 50,000 configured rows per large table on
  PostgreSQL 18.4. After migration, the asserted list queries selected their
  cursor indexes; for example, observed execution time changed from 8.139 ms to
  0.127 ms for urgent-help discovery and from 8.035 ms to 0.050 ms for
  Activity discovery. These are single local observations, not thresholds or
  portable performance claims. Evidence is
  `output/db-scale/20260719233526-97476`.
- The corrected kind rolling run replaced all four application pod UIDs,
  retained four Ready zero-restart replacements, passed the four-node BEAM and
  PubSub probes, and produced an empty database diff. All 395 readiness samples
  ultimately returned HTTP 200; two required one retry during local NodePort
  endpoint replacement. Evidence is
  `output/resilience/final-kind-rollout-fixed-20260720`.