# Performance measurement

No production capacity, minimum resource requirement, SLO, alert threshold,
pool size, or autoscaling threshold is known yet. The repository therefore
contains a reproducible measurement profile, not a capacity claim or blocking
resource preflight.

## Authentication limiter micro-measurement

Observed locally on 2026-07-28 with Elixir 1.20.2 / OTP 29.0.3, PostgreSQL
18.4, Docker 29.6.2, and the current uncommitted authentication-limiter
candidate on an AMD Ryzen 9 7950X3D workstation. PostgreSQL used an isolated
tmpfs data directory and the application connected over a Docker bridge.

Each operation incremented an email bucket and an IP bucket together. A
telemetry-backed integration test confirmed that the operation executes one
PostgreSQL `INSERT ... ON CONFLICT` statement, not two independent counter
queries.

| Observation | Result |
| --- | ---: |
| Sequential operations | 1,000 |
| Sequential p50 / p95 / p99 | 155 / 237 / 273 µs |
| Concurrent operations / concurrency | 1,000 / 50 |
| Concurrent elapsed time | 107.4 ms |
| Concurrent observed throughput | 9,308 operations/s |

This isolates the limiter on a fast local tmpfs database. It does not include
SMTP delivery, account lookup, production storage latency, internet latency,
or production contention, and therefore is not a production capacity claim.
It verifies the one-statement implementation shape. It provides no evidence
that Redis is currently required or unnecessary on the production host.

## Two different load profiles

The repository deliberately separates two experiments:

1. `scripts/load-cycle.sh` is the full write-capable profile. It creates a
   unique Compose project, database, credentials, images, and fixtures. It
   exercises login, database writes, private chat, tracking, WebSockets,
   multiple web/worker replicas, SQL statements, and connection pools. Use
   this profile to find application and query bottlenecks without touching
   real users.
2. `scripts/production-readonly-load.sh` is pinned to
   `https://whoneedhelp.com`. Its compiled allowlist contains only public GET
   pages, readiness, and Phoenix WebSocket heartbeats. It cannot exercise the
   authenticated write path and therefore cannot determine write capacity or
   whether Redis is warranted.

The production runner has separate `plan` and `run` modes. Every VU, duration,
think-time, and socket value is mandatory; the script supplies no hidden load
defaults. `plan` verifies the remote checkout identity, exact commit, Compose
project, running application containers, and current host resources without
starting k6. `run` additionally requires the exact confirmation string emitted
by that verified plan. If the production commit or any experiment input
changes, the confirmation no longer matches.

Example plan only:

```sh
WNH_PRODUCTION_LOAD_HTTP_VUS=<chosen-count> \
WNH_PRODUCTION_LOAD_WS_VUS=<chosen-count> \
WNH_PRODUCTION_LOAD_DURATION=<chosen-duration> \
WNH_PRODUCTION_LOAD_HTTP_THINK_SECONDS=<chosen-seconds> \
WNH_PRODUCTION_LOAD_WS_HOLD_MS=<chosen-milliseconds> \
WNH_PRODUCTION_LOAD_WS_CONNECT_TIMEOUT_MS=<chosen-milliseconds> \
  ./scripts/production-readonly-load.sh plan
```

Do not copy a previous run's values as production limits. Choose and record a
specific experiment, inspect the printed target and scope, then use the printed
confirmation only when that production probe has been explicitly approved.

When run, evidence is written only on the operator workstation below
`output/performance/<label>/`. It includes the exact scope, machine-readable k6
summary, console report, readiness before/after, read-only database counter
snapshots, application metrics before/after, one-second host/container samples,
and a threshold-free resource summary. The sampler is stopped on success,
failure, or interruption and creates no remote file.

The host sampler reports the combined RSS of every PostgreSQL process on that
server as `host_postgres_rss_kib`. It is intentionally labelled host-wide
because multiple PostgreSQL clusters or databases may coexist; it must not be
presented as production-database-only memory.

The production profile applies pass/fail checks only to functional correctness:
HTTP 200 responses, successful WebSocket heartbeats, and zero observed request
or socket errors. It records latency, CPU, RAM, process counts, and PostgreSQL
counters but does not compare them to invented SLOs or capacity thresholds.

Read-only production inventory observed on 2026-07-28 before implementing this
runner:

- commit `3d6f9c5004f9218c55a040bc31e72b56b22d4470`;
- one healthy compact application container;
- 2 online CPUs;
- 4,004,224 KiB total memory and 1,282,436 KiB available in that sample;
- five idle application database connections;
- no Docker CPU or memory hard limit on the application container.

These values are a point-in-time observation, not a server minimum. A later
`plan` rechecks the mutable facts rather than trusting this paragraph.

## Current isolated harness verification

Observed on 2026-07-28 from the application tree at commit
`b519f902e8d30f49a3475ac07311f69722bba188`, with the uncommitted load-harness
additions documented on this page. The isolated profile used its recorded
inputs: 3 web replicas, 2 worker replicas, 40 public HTTP VUs, 40 Phoenix
WebSocket VUs, 8 authenticated mutual-aid VUs, and a 30-second duration.

| Observation | Result |
| --- | ---: |
| HTTP requests / failures | 38,222 / 0 |
| Public checks | 34,806 / 34,806 passed |
| Overall HTTP average / p95 / maximum | 2.94 / 7.03 / 182.37 ms |
| Authenticated chain average / p95 | 7.35 / 10.47 ms |
| Authenticated chat/tracking chains | 1,700 |
| WebSocket opened / heartbeat replies | 240 / 240 |
| PostgreSQL peak client / active backends | 21 / 3 |
| Configured Repo pool slots | 16 |
| Minimum observed connection headroom | 76 |
| Ecto average total / execution / queue | 257.46 / 234.79 / 21.75 µs |
| Rollbacks / deadlocks / temp files | 0 / 0 / 0 |

The SQL statement with the largest individual maximum took 8.793 ms and was a
primary-key request lookup. The highest mean was 0.761 ms across five session
token inserts. No statement used temporary blocks. This particular isolated
dataset and workload exposed no database connection exhaustion, deadlock,
temporary-file spill, or failed application operation. It does not prove that
the same result holds for production data cardinality, storage latency, or a
different request mix.

The load profile recorded exactly 1,700 messages, tracking sessions, and
tracking samples. Explicit tracking stop left zero current tracking positions.
Its cleanup diff was empty, and the lifecycle wrapper then removed all
containers, networks, named volumes, and unique image tags belonging to that
run. Evidence remains in the ignored local directory
`output/performance/load-harness-20260728/`.

The run used a 32-CPU, 100,289,273,856-byte workstation without container CPU
or RAM limits. Its latency, throughput, and CPU percentages must not be used as
production capacity evidence. The production server observed above has a
different CPU and memory topology.

## Latest local candidate measurement

Observed on 2026-07-25 at application commit
`4f2ab7d205ad1e137b07a8365e30b7cf007e2a19` with 3 web and 2 worker
replicas, 40 public HTTP VUs, 40 Phoenix WebSocket VUs, and 8 authenticated
mutual-aid VUs for 30 seconds:

| Observation | Result |
| --- | ---: |
| HTTP requests / failures | 38,240 / 0 |
| Public checks | 34,866 / 34,866 passed |
| WebSocket opened / heartbeat replies | 240 / 240 |
| Authenticated chat/tracking iterations | 1,679 |
| Overall HTTP duration average / p95 / maximum | 2.90 / 7.28 / 184.51 ms |
| Authenticated HTTP duration average / p95 | 7.63 / 10.81 ms |
| WebSocket connect p95 | 4.76 ms |
| PostgreSQL commits / rollbacks | 141,740 / 0 |
| Peak client / active backends | 21 / 5 |
| Minimum observed connection headroom | 76 |
| Ecto queries / average total / average queue | 181,796 / 0.275 / 0.022 ms |

Ten sequential resource samples observed maximum memory of 312.4, 319.8, and
331.6 MiB across web replicas; 204.2 and 204.6 MiB across workers; 140.7 MiB
for PostgreSQL; and 323.1 MiB for the proxy. The sum of each container's
individual maximum was 1,836.4 MiB. PostgreSQL recorded no deadlock, conflict,
temporary file, or lock waiter, and exact fixture cleanup restored every
tracked application-table count.

These measurements describe this specific concurrent workstation replay. They
do not establish a minimum server size, production capacity, SLO, alert
threshold, database pool size, or autoscaling policy. Machine-readable evidence
is retained under `output/performance/final-4f2ab7d`.

## Kind port-table memory investigation

On 2026-07-20, the local kind control-plane container used 9.748 GiB while its
four otherwise idle application `beam.smp` processes each had roughly
2.3 GiB RSS. Read-only runtime probes found an effective Erlang port limit of
134,217,727 and an allocated port table of 1,610,612,736 bytes in every kind
BEAM VM. The nested runtime exposed `nofile=2,147,483,584`; OTP had derived the
maximum port limit from that value.

The same release image in ordinary Compose used the normal 65,536 port limit
and a 786,432-byte port table. A separate VM probe with
`ERL_ZFLAGS="+Q 65536"` reproduced those smaller values inside the kind pod
before any rollout. Compose and Helm now set that value explicitly, and the
startup and rolling-verification scripts check the effective value in every
running web and worker VM.

On a host with more CPUs than the intended server, the optional
`compose.cpu-replay.yaml` override can pin the application, worker, database,
proxy, and supporting services to the same explicit host CPU set. The external
load generator remains outside that set:

```sh
CPU_REPLAY_CPUSET=0 \
CPU_REPLAY_WEB_CPUS=1 CPU_REPLAY_WORKER_CPUS=1 \
CPU_REPLAY_WEB_SCHEDULERS=1 CPU_REPLAY_WORKER_SCHEDULERS=1 \
  docker compose -f compose.yaml -f compose.cpu-replay.yaml up -d --wait
```

All five values are explicit experiment inputs. Pinning every stack service to
the same one-element CPU set reproduces CPU contention, but does not reproduce
an unknown server's memory, storage, network, kernel, or neighbouring
workloads. The scheduler inputs make total and online BEAM scheduler counts
match the intended replay instead of leaving offline scheduler threads sized
from the development host. None of these inputs is a production sizing
recommendation. Running ordinary `docker compose up` without the override
removes the replay constraints.

The isolated lifecycle accepts the same override without touching the ordinary
Compose project. The explicit VU and replica values below are experiment
inputs:

```sh
WNH_LOAD_CPU_REPLAY=true \
CPU_REPLAY_CPUSET=0 \
CPU_REPLAY_WEB_CPUS=1 CPU_REPLAY_WORKER_CPUS=1 \
CPU_REPLAY_WEB_SCHEDULERS=1 CPU_REPLAY_WORKER_SCHEDULERS=1 \
WNH_LOAD_WEB_REPLICAS_OVERRIDE=1 \
WNH_LOAD_WORKER_REPLICAS_OVERRIDE=1 \
WNH_LOAD_HTTP_VUS_OVERRIDE=5 \
WNH_LOAD_WS_VUS_OVERRIDE=5 \
WNH_LOAD_AUTH_VUS_OVERRIDE=1 \
LOAD_DURATION_OVERRIDE=30s \
  ./scripts/load-cycle.sh one-cpu-baseline load
```

After the Helm rollout, the unchanged kind control-plane container measured
1.796 GiB, a reduction of 7.952 GiB (81.6%). Its four application pod cgroups
measured 188.5–215.6 MiB in the first post-change sample. A subsequent
four-pod rolling replacement retained the 65,536 limit and 786,432-byte table;
the replacement cgroups measured 194.3–211.4 MiB, all four replacement pods
were Ready, cross-node PubSub passed, 184 readiness samples had no failed
sample after retry handling, and the application-table count diff was empty.
Evidence is retained under
`output/memory/20260720-port-table-fix-005420` and
`output/resilience/beam-port-limit-final-20260720`.

In the ordinary 2-web/2-worker Compose profile, the immediate post-recreation
sample measured 190.1–205.5 MiB per application container and 168.3 MiB for
PostgreSQL. These local observations do not include a target server's OS and do
not establish that a 1 GiB or 2 GiB machine is sufficient for production.

On 2026-07-21, an isolated production-compiled compact Compose run measured
198.3 MiB for the combined Phoenix+Oban container and 70.9 MiB for its fresh
PostgreSQL/PostGIS container in one no-load snapshot. An isolated split run
against a separately started external-database fixture measured 207.2–219.6
MiB for each of two web and two worker containers, 20.05 MiB for the Docker API
proxy, 41.66 MiB for Traefik, and 117.6 MiB for that database fixture. These are
short workstation observations, not minimums, limits, or capacity claims.

The profile uses a separate Compose project, generated independent secrets, and
a separate PostgreSQL volume. Its Traefik instance is constrained to that exact
Compose project; its router, service, middleware, and Host rule are unique, so
the load profile cannot join the staging router. It exercises:

- dynamic public home and Safety pages;
- database-backed readiness queries;
- Phoenix WebSocket upgrades and protocol heartbeats;
- real password login and authenticated LiveView joins;
- consent-driven tracking start, authenticated location writes, private chat
  writes, and explicit tracking stop/position deletion;
- the configured number of web and worker replicas;
- cross-node Phoenix PubSub after each run.

It does not render a real browser, retain a browser tracking session for an
unknown real-world trip duration, or reproduce an unknown future production
traffic mix. It also does not sample the k6 load-generator container itself.
Its results only describe the recorded host, container versions, replica
counts, database state, and experiment inputs.

## Create the isolated profile

For an ordinary one-run measurement, prefer the lifecycle wrapper. It creates
unique Compose and image names, then removes only that run's containers,
networks, named volumes, temporary environment, and image tags on success,
failure, or interruption:

```sh
./scripts/load-cycle.sh local-load load
./scripts/load-cycle.sh local-resilience resilience
./scripts/load-cycle.sh local-complete both
```

Evidence remains under `output/performance/` and `output/resilience/`. The
commands below are the manual diagnostic workflow; unlike the lifecycle
wrapper, it deliberately keeps the isolated database volume between commands.

```sh
./scripts/ensure-local-load-env.sh
./scripts/load-stack-up.sh
```

The generated `output/runtime/load.env` is ignored, restricted to mode 600,
and contains independent PostgreSQL and application secrets. Edit its `LOAD_*` inputs to
define a specific experiment. Values in `.env.load.example` are reproducible
measurement points, not recommendations.

For a one-run duration that must not rewrite `output/runtime/load.env`, pass an explicit
experiment override:

```sh
LOAD_DURATION_OVERRIDE=10m ./scripts/load-run.sh local-soak-YYYYMMDD
```

The effective value and whether it came from `output/runtime/load.env` or the override are
recorded in that run's `environment.txt`.

## Run and compare replica counts

```sh
./scripts/load-run.sh two-web
./scripts/load-stack-up.sh 3
# Set LOAD_WEB_REPLICAS=3 in output/runtime/load.env so the recorded expected topology
# matches the running topology, then:
./scripts/load-run.sh three-web
```

Each run writes ignored evidence below `output/performance/<label>/`:

- the exact inputs and Docker host observation;
- a machine-readable k6 summary and complete console report;
- periodic Docker CPU/memory/network/block-I/O/PID samples for every load
  stack container, plus a threshold-free CPU/RAM/PID JSON summary;
- read-only PostgreSQL database size and `pg_stat_database` snapshots before
  and after k6, plus their machine-readable counter deltas;
- `pg_stat_statements` rankings by total, mean, and maximum execution time,
  including calls, rows, buffer activity, temporary blocks, and WAL activity;
- periodic `pg_stat_activity` connection samples with configured pool slots,
  observed connection headroom, transaction state, lock waits, and named
  active wait events;
- per-web Ecto telemetry deltas separating database execution, pool queue, and
  result decode time;
- before/after application table counts, exact authenticated-write evidence,
  fixture cleanup output, and a cleanup diff;
- final Compose state, readiness response, application logs, and cross-node
  PubSub result.

The load-only PostgreSQL command preloads `pg_stat_statements` and creates the
extension only in the isolated load database. The application exposes
cumulative Ecto measurements through its authenticated metrics endpoint; the
runner snapshots every web replica before and after k6 and records the delta.
Neither mechanism changes the ordinary public Compose database.

Before k6 starts, the production-compiled fixture task verifies the exact
configured load database name and a deliberate confirmation marker. It creates
isolated requester/helper/request/assignment pairs through the application
contexts. The pair count equals the maximum simultaneous VU count across all
scenarios, because k6 may reuse its global VU pool between parallel scenarios;
mapping each global VU identifier directly to its own pair prevents concurrent
users from sharing a tracking assignment. The random fixture password exists
only in ignored mode-600 `output/runtime/load.env` and is never written to the manifest or
console.

On success and on trapped failure, cleanup deletes only the UUIDs recorded in
the validated run manifest. A successful run additionally requires every
tracked application-table count after cleanup to equal its pre-fixture count.
In a separate negative check, replacing one helper UUID with another valid UUID
made cleanup exit nonzero before mutation; all 2 users, 1 request, and 1
assignment remained. Restoring the original manifest then removed those exact
rows and returned all three counts to zero.

The k6 image is pinned to version 2.1.0 by digest and runs locally with anonymous
usage reporting disabled. No Grafana Cloud account, API token, OpenAI API, or
usage-based service is involved.

## Observed production public read-only profile

Observed on 2026-07-28 against `https://whoneedhelp.com`, after the runner
verified `/srv/who_need_help-production`, Compose project
`who_need_help_production`, and deployed commit
`3d6f9c5004f9218c55a040bc31e72b56b22d4470`. The host exposed 2 CPUs and
4,004,224 KiB total memory. The runner sent only the documented public GET
allowlist and Phoenix transport heartbeats; it did not submit forms, log in,
register, send email, open product LiveViews, or write application data.

| Experiment input / observation | Result |
| --- | ---: |
| Baseline input | 1 HTTP VU + 1 WebSocket VU, 30 seconds |
| Baseline HTTP checks / failures | 28 / 0 |
| Baseline HTTP average / p95 | 91.64 / 107.75 ms |
| Baseline WebSocket opens / heartbeat replies / errors | 6 / 6 / 0 |
| Baseline app average / max CPU | 1.26% / 4.60% |
| Five-client input | 5 HTTP VUs + 5 WebSocket VUs, 60 seconds |
| Five-client HTTP checks / failures / observed rate | 1,055 / 0 / 17.27 requests/s |
| Five-client HTTP average / p95 | 83.85 / 129.63 ms |
| Five-client WebSocket opens / heartbeat replies / errors | 20 / 20 / 0 |
| Five-client app average / max CPU | 7.37% / 10.77% |
| HTTP-only input | 20 HTTP VUs, 30 seconds |
| HTTP-only checks / failures / observed rate | 5,557 / 0 / 184.47 requests/s |
| HTTP-only average / p95 / maximum | 57.18 / 79.25 / 673.45 ms |
| HTTP-only app average / max CPU | 54.70% / 61.53% |
| HTTP-only app first-to-last memory delta | +1 MiB |
| WebSocket-only input | 20 sockets held for 60 seconds |
| WebSocket opens / heartbeat replies / errors | 20 / 40 / 0 |
| WebSocket connect average / p95 | 184.38 / 191.53 ms |
| WebSocket-only app average / max CPU | 0.79% / 3.82% |
| WebSocket-only app first-to-last memory delta | -0.6 MiB |

Readiness returned `{"status":"ready"}` before and after every valid run. The
application container remained healthy with zero restarts. The lowest observed
host `MemAvailable` across the recorded valid runs was 1,241,432 KiB. The
reported PostgreSQL RSS is explicitly the sum of every PostgreSQL process on
the shared host, not an attribution to this application.

The HTTP-only run recorded 5,562 application HTTP completions and 994 Ecto
queries. Their cumulative database execution time increased by 388,598
microseconds and pool queue time by 558,053 microseconds. The two sequential
scans of the empty `rate_limit_buckets` table were the runner's own before/after
count snapshots; the HTTP traffic added no rate-limit-table scan. Application
table counts were unchanged.

This is not a product-query capacity result. Production contained no help
requests or activities during the observation, and `/requests` and
`/activities` require an authenticated session. PostgreSQL also did not have
`pg_stat_statements` installed. Viewport clustering, authenticated LiveView
joins, chat, tracking, handover, and write throughput therefore remain covered
by the isolated profiles rather than this public production probe.

The environment measured in this historical production observation had
`RATE_LIMIT_POLICIES_JSON={}`. In that
configuration `RateLimiter.check/2` returns `:not_configured` without a database
query, so this observation neither exercises nor validates authentication
throttling. It also provides no evidence that Redis is needed. The current
candidate keeps cross-replica counters in PostgreSQL and ships a pilot
email/IP policy; its limiter overhead still requires a separate measured run.

Ignored evidence:

- `output/performance/production-baseline-20260728/`
- `output/performance/production-tier2-20260728/`
- `output/performance/production-http-db-stats-20260728/`
- `output/performance/production-ws-heartbeat-final-20260728/`

## Isolated rate-limiter observation

Observed locally on 2026-08-03 with the checked-in
`scripts/rate-limit-benchmark.sh`. The application container was limited to one
CPU; PostgreSQL reported version 18.4. Both profiles used 5,000 calls with
concurrency 20. The first profile deliberately contended on one bucket; the
second used 5,000 independent scopes.

| Profile | Successful / failed / limited | Elapsed | Observed rate | Buckets |
| --- | ---: | ---: | ---: | ---: |
| One hot scope | 5,000 / 0 / 0 | 12,769.306 ms | 391.56 operations/s | 1 |
| Distributed scopes | 5,000 / 0 / 0 | 1,312.608 ms | 3,809.21 operations/s | 5,000 |

Each run persisted exactly 5,000 increments. The task then deleted its unique
benchmark action: cleanup left zero benchmark buckets. The generated JSON is
kept in the ignored local directory
`output/rate-limit/20260802233523-2242041/`.

This observation measures the atomic counter implementation on this local
host. It is not a production capacity claim and does not justify a hard startup
guard. The much slower hot-scope result is expected contention on one database
row; ordinary email/IP policy traffic distributes attempts across many scopes.

## Observed local comparison

Observed on 2026-07-18 with Docker Engine 29.6.2 on the recorded 32-CPU,
100289265664-byte Docker host. Neither application containers nor the load
generator had CPU or memory limits. Both runs used 40 HTTP VUs, 40 WebSocket
VUs, a 30-second duration, a 0.1-second HTTP think time, and five-second socket
sessions. Each HTTP iteration requested the home page, Safety page, and
database-backed readiness endpoint.

| Observation | 2 web replicas | 3 web replicas |
| --- | ---: | ---: |
| HTTP checks | 35,160 / 35,160 passed | 35,163 / 35,163 passed |
| HTTP failed rate | 0% | 0% |
| Observed HTTP rate | 1,169.12 requests/s | 1,169.42 requests/s |
| HTTP duration p95 | 2.87 ms | 3.36 ms |
| WebSocket opened / heartbeat replies | 240 / 240 | 240 / 240 |
| WebSocket errors | 0 | 0 |
| WebSocket connect p95 | 46.65 ms | 24.12 ms |
| Average CPU per web container | 66.57%, 66.63% | 50.25%, 49.77%, 49.18% |
| Maximum CPU per web container | 79.80%, 79.21% | 58.16%, 59.23%, 59.08% |
| Maximum observed memory per web container | 268.3, 238.3 MiB | 221.0, 223.2, 219.4 MiB |

Each resource series contained 11 sequential `docker stats --no-stream`
samples. Both before/after database diffs were empty, both cross-node PubSub
probes passed, readiness remained healthy, and the captured proxy/web/worker
logs contained no application error or warning. The public staging root and
readiness endpoint also returned HTTP 200 while the isolated project was
running.

An earlier harness smoke exposed that fixed Traefik router/service names are
global to the Docker provider: starting a second Compose project temporarily
made staging readiness return HTTP 404. The load project was stopped
immediately and staging returned HTTP 200. The final Compose configuration
parameterizes the project constraint, router/service name, Docker network, and
router rule. Merged-config inspection found no shared router/service label key,
the two running proxies showed different exact project constraints, and both
origins returned HTTP 200 concurrently before the recorded final runs.

The nearly identical request rate is expected from the selected paced workload:
40 HTTP VUs each issue three requests and then wait 0.1 seconds, giving an
approximate workload ceiling of 1,200 requests/s before response time is added.
The result therefore does not show the saturation point of two replicas and
does not justify an HPA threshold. The lower per-container CPU and lower
WebSocket connect p95 with three replicas are observations for this run only;
the slightly higher HTTP p95 also shows why one short comparison is not a
production sizing study.

Ignored evidence:

- `output/performance/two-web-final-20260718/`
- `output/performance/three-web-final-20260718/`

## Observed authenticated write run

Observed on 2026-07-19 on the same recorded Docker host with 3 web replicas,
2 worker replicas, 40 paced public HTTP VUs, 40 five-second Phoenix heartbeat
VUs, and 8 authenticated mutual-aid VUs for 30 seconds. The fixture tool
created 88 isolated pairs so every possible global VU had a unique account and
assignment.

| Observation | Result |
| --- | ---: |
| Public HTTP checks | 35,316 / 35,316 passed |
| All HTTP failures | 0 / 39,108 |
| Phoenix heartbeat opened / replies | 240 / 240 |
| Authenticated logins | 8 |
| Authenticated pages / LiveView joins | 1,888 / 1,888 |
| Tracking starts / position writes / stops | 1,888 / 1,888 / 1,888 |
| Private messages written | 1,888 |
| Authenticated or WebSocket errors | 0 |
| HTTP request duration p95 | 5.29 ms |
| Authenticated HTTP duration p95 | 8.10 ms |
| WebSocket connect p95 | 3.16 ms |

The database evidence counted 1,888 chat rows, 1,888 tracking sessions with
1,888 total samples, 8 login tokens, and zero current positions after explicit
stop. Cleanup removed the 176 fixture users, 88 requests, 88 assignments,
1,888 messages, 1,888 tracking sessions, and their audit rows. The post-cleanup
application-table diff was empty. Readiness and the five-node web/worker PubSub
probe passed, and the run-scoped proxy/web/worker logs contained no application
error, warning, HTTP 4xx/5xx, or database deadlock.

Eleven sequential resource samples observed average CPU of 135.63%, 140.01%,
and 136.88% for the three web containers, with maximum memory of 273.4, 271.2,
and 272.9 MiB respectively. PostgreSQL averaged 37.55% CPU and reached
249.2 MiB; Traefik averaged 47.11% CPU and reached 79.25 MiB. These are
short local observations without container resource limits, not minimums,
production capacity, or autoscaling inputs.

Ignored evidence:

- `output/performance/final-load-20260719a/`

## Observed SQL and connection-pool run

Observed on 2026-07-20 with 3 web replicas, 2 worker replicas, PostgreSQL 18.4,
40 paced public HTTP VUs, 40 Phoenix heartbeat VUs, and 8 authenticated
mutual-aid VUs for 30 seconds.

| Observation | Result |
| --- | ---: |
| HTTP requests / failures | 38,429 / 0 |
| Public checks | 35,163 / 35,163 passed |
| HTTP duration average / p95 / maximum | 2.42 / 7.74 / 187.11 ms |
| Authenticated chat/tracking chains | 1,664 |
| Phoenix heartbeat opened / replies | 240 / 240 |
| Configured Ecto Repo pool slots | 50 |
| Peak PostgreSQL client backends | 55 |
| Minimum observed non-reserved connection headroom | 42 |
| Idle-in-transaction / lock-waiting backends | 0 / 0 |
| Ecto average execution / queue / decode time | 308.58 / 31.15 / 0.27 µs |
| PostgreSQL rollback / deadlock / conflict / temp files | 0 / 0 / 0 / 0 |

The connection sampler observed active PostgreSQL waits named
`Client:ClientRead` and `IO:WalSync`; it observed no lock wait or
idle-in-transaction backend. The Ecto queue
measurement is the application-side evidence for time waiting to check out a
Repo connection. It averaged 31.15 microseconds over 166,459 queries across
the three web replicas. This short run did not produce a connection timeout or
establish a maximum connection capacity.

The SQL ranking exposed an unbounded preload of every message attached to a
request even though the LiveView immediately loaded a separate 50-message
cursor page. In the before run that redundant statement executed 3,318 times,
returned 342,378 rows in aggregate, and consumed 392.378 ms of PostgreSQL
execution time. The request preload was removed and a regression test now
requires the chat association to remain unloaded. In the repeat run the
unbounded statement was absent; the bounded cursor statement retained its
`LIMIT`, executed 3,328 times, returned 148,512 rows, and averaged 0.122 ms.

No normalized statement in the repeat run used temporary blocks. The largest
mean statement time was 0.481 ms for six session-token inserts; the
highest-call request lookup averaged 0.031 ms over 39,936 calls. These are
observations against the small generated load dataset, not proof that every
query remains fast at arbitrary data size. The separate 50,000-row database
profile below checks the indexed list and cursor queries at a larger
cardinality.

Across the seven measured application/database/proxy containers, simultaneous
memory samples totalled 1,501.60–1,509.86 MiB during this run. This excludes
the host OS, the k6 generator, Mailpit, and optional observability services.
Therefore it is not a claim that this 3-web/2-worker topology fits a 1 GiB or
2 GiB server. It does show that the measured workload used about 1.47 GiB for
those seven containers rather than the former kind port-table failure's
roughly 10 GiB.

The run created exactly 1,664 messages, tracking sessions, and tracking
samples, plus 8 session tokens; explicit tracking stop left zero current
positions. Cleanup restored every tracked application-table count, readiness
and the five-node PubSub probe passed, and the captured application log had no
database timeout, Postgrex error, deadlock, `ERROR`, `FATAL`, or `PANIC`
match.

Ignored before/after evidence:

- `output/performance/pool-sql-final-20260720/`
- `output/performance/pool-sql-chat-fixed-canonical-20260720/`

## Observed role-specific connection pools

On 2026-07-20 the same 3-web/2-worker, 40 public HTTP VU, 40 Phoenix
heartbeat VU, and 8 authenticated mutual-aid VU profile was repeated with
four Repo connections per web replica and two per worker replica. The migrate
role also started and completed with a two-connection pool. These are measured
development defaults, not production minimums.

The 16 configured long-running Repo slots produced 38,678 HTTP requests with
zero failures, 1,760 complete authenticated page/chat/tracking chains, and 240
Phoenix heartbeat sessions. The application recorded 131,409 web-side queries
with 34.82 microseconds average pool queue time, compared with 31.15
microseconds in the earlier 10-connections-per-replica run. PostgreSQL observed
21 peak client backends, 6 peak active backends, 76 minimum non-reserved
connection headroom, no idle transaction, no lock wait, no rollback, no
deadlock, no conflict, and no temporary file.

The run-scoped application log had no match for DBConnection/Postgrex
connection errors, pool timeouts, exceptions, deadlocks, `FATAL`, or `PANIC`.
All 1,760 messages, tracking sessions, and tracking samples were recorded;
explicit stop left zero current positions, and fixture cleanup restored the
tracked table counts exactly. The result supports using web=4, worker=2, and
migrate=2 as the repository's current measured starting profile. It does not
establish saturation, an SLO, or a production capacity limit.

Ignored evidence:

- `output/performance/role-pools-4-2-20260720/`

## Observed 10-minute authenticated soak

Observed on 2026-07-19 with the same 3-web/2-worker topology and the same
40 public HTTP, 40 five-second Phoenix heartbeat, and 8 authenticated
mutual-aid VUs. The duration came from the recorded
`LOAD_DURATION_OVERRIDE=10m`; it is an experiment input, not a recommended
soak duration or capacity target.

| Observation | Result |
| --- | ---: |
| Completed / interrupted iterations | 271,072 / 0 |
| Public checks | 703,995 / 703,995 passed |
| HTTP requests / failures | 767,225 / 0 |
| HTTP duration average / p95 / maximum | 2.34 / 5.71 / 175.96 ms |
| Phoenix heartbeat opened / replies | 4,800 / 4,800 |
| Authenticated logins | 8 |
| Authenticated pages / LiveView joins | 31,607 / 31,607 |
| Tracking starts / position writes / stops | 31,607 / 31,607 / 31,607 |
| Private messages written | 31,607 |
| Authenticated or WebSocket errors | 0 |

The database evidence counted exactly those 31,607 messages, tracking
sessions, and samples, plus 8 login tokens and zero current positions after
explicit stop. Its read-only `pg_stat_database` delta recorded 2,921,283
commits, zero rollbacks, 12 block reads, 33,967,340 block hits, zero temporary
files/bytes, zero deadlocks, and zero conflicts. Database size increased by
16,678,912 bytes during the write workload. The physical database size was
recorded but was not treated as a cleanup assertion; the post-cleanup count
diff across all tracked application tables was empty. The five-node PubSub
probe passed. All seven measured app/database/proxy containers remained
running, had zero observed restarts, and were not OOM-killed. The 75,058,636
byte run-scoped log had zero matches for the recorded error, warning,
exception, crash, HTTP 4xx/5xx, and deadlock search.

Each container has 199 sequential resource samples. No container resource
limit or pass/fail resource threshold was applied.

| Container | Average / max CPU | First 10 / last 10 memory average | Max memory |
| --- | ---: | ---: | ---: |
| PostgreSQL | 61.44% / 85.30% | 236.15 / 261.81 MiB | 263.60 MiB |
| Traefik | 57.25% / 73.14% | 93.44 / 103.98 MiB | 108.50 MiB |
| web 16 | 198.67% / 281.58% | 264.24 / 376.33 MiB | 397.10 MiB |
| web 17 | 195.66% / 247.46% | 259.30 / 373.79 MiB | 406.10 MiB |
| web 18 | 195.36% / 266.74% | 265.06 / 388.06 MiB | 410.80 MiB |
| worker 10 | 1.47% / 5.09% | 201.63 / 202.64 MiB | 203.80 MiB |
| worker 11 | 2.21% / 5.56% | 199.30 / 201.05 MiB | 201.80 MiB |

The web RSS increased across most of the loaded interval, so this run does
not establish a steady-state memory plateau. A separate read-only observation
at 2026-07-19T16:06:21Z, 5 minutes 23 seconds after the k6 summary timestamp,
measured web RSS at 278.2, 269.8, and 285.6 MiB. At that point the BEAM runtime
reported 142,250,344, 147,808,440, and 147,403,872 total bytes with 613, 612,
and 612 processes. This observed post-load reduction shows that a substantial
part of the loaded RSS was released after the connections closed; it does not
prove that the application is free of a longer-term memory leak. A
production-like duration and traffic distribution remain unknown.

Ignored evidence:

- `output/performance/authenticated-soak-10m-20260719/`

## Observed local resilience drills

The canonical Compose drill on 2026-07-19 used the isolated 3-web/2-worker
profile. One web and one worker BEAM process exited with status 1 and each
container's observed restart count increased to 1. All original replicas were
then replaced sequentially. The route returned 758 successful readiness
responses with zero final failure and responses from all three web nodes.
After replacement, the observed cluster contained all five BEAM nodes and the
cross-node PubSub probe passed.

The local Oban probe completed with state `completed`, attempt `2`,
`max_attempts=2`, and exactly one recorded first-attempt error. Its exact row
was removed afterward. The load database then contained zero probe jobs, users,
requests, messages, and tracking sessions. Run-scoped available logs contained
no unexpected application error, warning, HTTP 4xx/5xx, or database deadlock.

The canonical kind drill rolled both 2-replica Deployments from revision 19 to
20 with the chart's observed `maxUnavailable=0` and `maxSurge=1`. All four pod
UIDs changed, all replacements were Ready with zero container restart, the
four-node BEAM cluster and PubSub probe passed, and the application-table count
diff was empty. Of 362 readiness samples, all ultimately returned HTTP 200.
Two samples needed one transport retry each while kind's single-node
NodePort reset connections during endpoint changes. Those retries are retained
in evidence rather than reported as uninterrupted raw TCP connections. This is
a local kind observation, not a production availability guarantee.

Ignored evidence:

- `output/resilience/final-resilience-20260719a/`
- `output/resilience/final-kind-20260719a/`

## Observed local monitoring drill

The canonical observability run on 2026-07-19 generated one direct Prometheus
target for each of the three current load web containers. The exact expected
and observed `instance` lists matched, and all three targets were `up` before
the induced failure. Grafana reported a healthy Prometheus datasource and
served the tracked provisioned dashboard with four panels.

The script then stopped the exact scoped container
`who_need_help_load-web-16`. Prometheus fired
`WhoNeedHelpWebReplicaUnavailable`, and Alertmanager delivered a webhook whose
labels identified that instance and the run. After the same container became
healthy again, all three direct targets returned to `up`, the alert cleared,
and a resolved webhook with the same instance/run labels arrived. The
application-table count diff was empty. These are observed local protocol
results, not a production monitoring or notification guarantee.

Ignored evidence:

- `output/observability/final-observability-20260719a/`

Stop the isolated containers without deleting their database volume:

```sh
./scripts/load-stack-stop.sh
```

The stop script reads the actual `LOAD_PROJECT`, refuses the staging project
name, and does not delete the volume. Volume deletion is intentionally not
automated.

## Persistent million-row discovery profile

The write-capable load lifecycle above intentionally cleans its fixtures. For
repeatable map, clustering, pagination, and responsive-UI work at high
cardinality, a second profile keeps its synthetic database:

```sh
./scripts/scale-stack-up.sh
./scripts/scale-seed.sh
./scripts/scale-verify.sh
```

The profile is pinned to Compose project `who_need_help_scale`, database
`who_need_help_scale`, and named volume
`who_need_help_scale_postgres_data`. Its generated credentials live only in
ignored `output/runtime/scale.env`. Fixture mutation additionally requires the
exact database name and `WNH_SCALE_FIXTURE_CONFIRM=persistent-local-scale`.
The seed task refuses a partially populated database instead of mixing data
sets. `scale-stack-stop.sh` stops only this Compose project and deliberately
preserves the named volume. `scale-stack-down.sh` removes only the profile's
containers and networks while also preserving that volume for an explicit
operator decision.

The canonical local fixture contains 1,000,000 requests, 1,000,000 activities,
1,000 synthetic request owners, and one scale viewer. Records are distributed
across 20 cities and include varied categories, states, urgency, hidden
locations, exact public points, and privacy radii. The recorded database size
was 1,237,898,943 bytes: 709,197,824 bytes for requests and 505,962,496 bytes
for activities including their indexes and TOAST data.

Every scale service is pinned to CPU `0` in this experiment. Three warm
application-level samples recorded the following medians:

| Viewport | Request list / map | Activity list / map | Map rows / represented records |
| --- | ---: | ---: | ---: |
| Kyiv | 171.139 / 297.860 ms | 265.442 / 700.397 ms | 1 / 49,485 |
| Europe | 23.072 / 2,302.602 ms | 1,544.453 / 4,130.415 ms | 8 / 395,878 |
| World | 56.182 / 3,975.605 ms | 3,910.191 / 7,416.714 ms | 10 / 791,753 |

Server clustering kept each measured map payload between 135 and 1,463 bytes
instead of returning hundreds of thousands of points. The broad low-zoom map
queries remain the measured optimization target: clustering bounds the
network/browser payload, but PostgreSQL still reads and groups the matching
records.

A headed Chrome check used the same persistent profile. It verified 24-card
initial pages, duplicate-free 48-card pagination, List/Map/Together layouts,
390-by-844 mobile rendering, cluster zoom, and request/activity switching.
That check exposed a functional defect: grid-cell coordinates could pan to an
empty area after selecting a cluster. Cluster coordinates now use the average
of the public points in that group while the snapped cell remains only the
grouping and stable-ID key. The full suite then exposed a second privacy
regression: an activity close to a narrow viewport could pass the expanded
index candidate envelope even when its rounded public point was outside that
viewport. The query now retains the indexed candidate filter and also requires
an exact intersection against the rounded public point. The final headed check
confirmed cluster zoom, an explicit `Search this area` refresh, 390-by-844
mobile rendering without horizontal overflow, and no application or map
console errors. The only console warning was the expected inability to
register a service worker under the local self-signed TLS certificate.

The first authenticated HTML response now renders an honest loading shell and
starts discovery only after LiveView connects. The discovery runs
asynchronously, so a disconnected client no longer leaves its LiveView blocked
inside a broad spatial query. During the headed verification, reconnects
initially caused three identical logical world queries to overlap (nine active
PostgreSQL worker processes). The application now coalesces an identical
in-flight discovery key within each BEAM node and reuses a completed result for
10 seconds, which covers the observed 4.5- and 6.5-second local reconnect
intervals. Each key includes the viewer, filters, view mode, and viewport;
PubSub mutations invalidate the corresponding key. This is a node-local
single-flight optimization, not a cluster-wide cache or a capacity guarantee.

The final headed request and activity map checks each represented 791,753
records as 10 clusters. No fresh application `500`/`503`, database pool timeout,
discovery error, or container restart was observed. Screenshots are retained
in ignored local evidence:

- `output/playwright/scale-million-request-map-desktop-stable.png`
- `output/playwright/scale-million-activity-map-desktop-stable.png`

Authenticated HTTP load can reuse a restricted browser state without a login
or fixture write:

```sh
WNH_SCALE_READONLY_VUS=<chosen-count> \
WNH_SCALE_READONLY_DURATION=<chosen-duration> \
WNH_SCALE_READONLY_THINK_SECONDS=<chosen-seconds> \
WNH_SCALE_READONLY_LABEL=<unique-label> \
  ./scripts/scale-readonly-load.sh
```

The k6 file has a compiled four-route GET allowlist for the request/activity
static LiveView shells in Europe and the world. It intentionally does not claim
to measure connected LiveView discovery: `scripts/scale-verify.sh` measures the
actual list and clustering queries separately. The runner verifies the exact
Compose project and database, uses the stack's internal network, and compares
counts for users, tokens, requests, activities, assignments, participants,
messages, and rate-limit buckets before and after.

The final 2026-07-28 one-VU, 30-second, one-CPU static-shell observation
completed 60 authenticated GETs with 0 HTTP failures and 120/120 checks.

### Keeping the fixtures in the ordinary development site

The isolated profile is useful for destructive load rehearsals. For routine
development of the actual `whoneedhelp.imalto.site` UI, the same deterministic
fixtures can instead be appended to the already running `who_need_help`
development database:

```sh
./scripts/dev-scale-seed.sh
```

This guarded path requires the running Compose project and services to be named
`who_need_help`, `who_need_help-web-1`, and `who_need_help-db-1`; verifies that
the target database is exactly `who_need_help`; and uses the separate
confirmation value `persistent-dev-scale`. It preserves ordinary development
users and records. Synthetic rows carry `structured_data.scale_fixture=true`,
use deterministic UUIDs, and are inserted with conflict handling, so rerunning
the command fills missing fixtures without duplicating them. The resulting
summary is ignored at `output/dev-scale/summary.json`.
Overall HTTP duration was 2.566 ms median, 3.196 ms average, and 8.709 ms
maximum. Every tracked table count was identical before and after. The
discovery-query medians are the independently measured values in the table
above. These are measurements of one local CPU-contention replay, not a
production SLO, capacity claim, or minimum resource requirement. Ignored
evidence is retained under `output/scale/verification.json` and
`output/performance/scale-million-onecpu-20260728-async-shell/`.

## Isolated database scale and cursor plans

Run the database-only before/after measurement with:

```sh
./scripts/db-scale-benchmark.sh
```

The script builds the test image and creates a uniquely named Compose project,
random one-run PostgreSQL credentials, and a separate PostGIS volume. It
migrates only through `20260719004249`, seeds the isolated baseline, records
machine-readable PostgreSQL
`EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON)` plans, applies the generated
`20260719013320_add_keyset_pagination_indexes` migration, and records the same
queries again. It also verifies that two 25-row keyset pages exactly equal the
first 50 ordered rows without duplicates or gaps. The exact project, network,
volume, and image are removed by its trap.

`DB_SCALE_ROWS` controls the configured sample size and defaults to 50,000.
That default is a reproducible experiment input, not a minimum database size,
capacity claim, latency target, or production traffic model. The structural
gate checks the canonical sample's chosen cursor indexes and query correctness;
it does not fail on an arbitrary millisecond threshold.

Observed locally on 2026-07-19 with PostgreSQL 18.4, the canonical run created
50,000 rows in each of users, requests, assignments, messages, reviews,
reports, abuse signals, proposals, activities, and activity messages; it
created 50,500 activity participants and 49,998 block rows. The isolated
database occupied 231,241,407 bytes after the cursor migration.

| Query | Baseline observed | Cursor migration observed | Selected cursor index |
| --- | ---: | ---: | --- |
| Urgent-help discovery | 8.010 ms | 0.110 ms | `help_requests_discovery_cursor_index` |
| Requester history | 7.969 ms | 0.036 ms | `help_requests_requester_cursor_index` |
| Activity discovery | 8.137 ms | 0.051 ms | `activities_discovery_cursor_index` |
| Visible reviews | 8.264 ms | 0.052 ms | `reviews_visible_cursor_index` |
| All reports | 8.021 ms | 0.025 ms | `reports_cursor_index` |
| All category proposals | 6.501 ms | 0.025 ms | `category_proposals_cursor_index` |
| Moderation users | 5.367 ms | 0.030 ms | `users_moderation_cursor_index` |
| Blocks | 5.165 ms | 0.036 ms | `blocks_blocker_cursor_index` |

These are measurements of one warm local run and must not be interpreted as an
SLO or portable speedup. The measurement also exposed redundant candidate
indexes and an `OR`-based Activity membership query. The final migration keeps
the indexes PostgreSQL selected, uses partial discovery indexes for open,
non-hidden records, and the application joins the existing participant
membership invariant directly. Personal reputation and the leaderboard now
aggregate in PostgreSQL instead of loading all completed assignments into the
BEAM; the leaderboard itself uses composite keyset pagination.

Ignored evidence for the recorded run:

- `output/db-scale/20260719054559-2339191/`

## Current final replay

The same threshold-free profiles were replayed on 2026-07-20 against
application commit `b96d443`.

- The 30-second 3-web/2-worker load run used 40 public HTTP, 40 heartbeat
  WebSocket, and 8 authenticated VUs. It completed 13,765 iterations and 38,826
  HTTP requests; all 35,280 checks passed and HTTP failures were 0. The
  authenticated scenario completed 1,765 page/LiveView/chat/tracking
  start-update-stop chains, and the heartbeat scenario opened and answered 240
  sockets. PostgreSQL recorded no rollback, deadlock, conflict, or temporary
  file. The run-specific fixtures were removed and the cleanup count diff was
  empty. Evidence is `output/performance/final-load-20260720`.
- The database benchmark again used 50,000 configured rows per large table on
  PostgreSQL 18.4. After migration, the asserted list queries selected their
  cursor indexes; for example, observed execution time changed from 8.139 ms to
  0.127 ms for urgent-help discovery and from 8.035 ms to 0.050 ms for
  Activity discovery. These are single local observations, not thresholds or
  portable performance claims. Evidence is
  `output/db-scale/20260719233526-97476`.
- The corrected kind rolling run replaced all four application pod UIDs,
  retained four Ready zero-restart replacements, passed the four-node BEAM and
  PubSub probes, and produced an empty database diff. All 395 readiness samples
  ultimately returned HTTP 200; two required one retry during local NodePort
  endpoint replacement. Evidence is
  `output/resilience/final-kind-rollout-fixed-20260720`.

## Runtime hardening and queue replay on 2026-07-20

The ordinary and isolated load releases were rebuilt after separating
internal cluster, ingress, and outbound networks. All web and worker node names
used internal `172.16.52.x` or `172.16.55.x` addresses respectively, every
node observed the expected peers, and cross-node PubSub passed. The
worker runtime no longer starts the unused `default` queue. Both measured load
workers reported exactly `maintenance: 2` and `push: 1`; their BEAM process
counts were 574 after the change, compared with the earlier observation of
577/578. This three-to-four-process difference is an observation, not a memory
or capacity guarantee.

The isolated Oban burst experiment inserted 1,000 side-effect-free jobs in one
batch. With two workers and configured maintenance concurrency two per worker,
all 1,000 reached `completed` in an observed 4,516 ms. Three resource samples
per measured container recorded maxima of 80.80 MiB for PostgreSQL and
218.80/186.90 MiB for the two worker cgroups. No error matched the run-scoped
logs; the database reported zero deadlocks, temporary files, or conflicts.
The script removed exactly 1,000 probe rows, verified zero remaining probe
jobs, and found byte-identical before/after row-count documents for all domain
tables. These short local samples do not establish a production throughput,
steady-state memory plateau, or minimum server size. Evidence is
`output/performance/oban-burst-1000-final-20260720`.

The database-scale harness was repaired to build and pass its own hardened
PostGIS image rather than relying on an image variable owned by another
script. Its fresh PostgreSQL 18.4 replay again used 50,000 configured rows per
large table and identical before/after table counts. Every asserted list plan
selected its intended cursor index. Observed examples were 8.806 to 0.123 ms
for urgent-help discovery, 8.348 to 0.061 ms for Activity discovery, and 9.084
to 0.048 ms for visible reviews. The aggregate leaderboard and single hot
helper reputation plans still scanned their 50,000-row completed-assignment
working sets and took 53.949 and 41.247 ms in this deliberately concentrated
fixture. That is a measured future optimization target, not evidence of a
current failure or a portable latency. Evidence is
`output/db-scale/20260720035817-2416501`.

## Final queue/runtime load replay

After the queue and network changes, the same isolated 3-web/2-worker profile
was replayed on 2026-07-20 with 40 paced public HTTP VUs, 40 Phoenix heartbeat
VUs, and 8 authenticated mutual-aid VUs for 30 seconds.

| Observation | Result |
| --- | ---: |
| HTTP requests / failures | 38,962 / 0 |
| Public checks | 35,280 / 35,280 passed |
| Authenticated page/chat/tracking chains | 1,833 |
| Phoenix heartbeat opened / replies | 240 / 240 |
| HTTP duration average / p95 / maximum | 1.91 / 4.90 / 179.46 ms |
| Authenticated HTTP duration p95 | 7.77 ms |
| Configured long-running Repo pool slots | 16 |
| Peak PostgreSQL client / active backends | 21 / 4 |
| Minimum observed non-reserved connection headroom | 76 |
| Ecto average execution / queue / decode time | 313.90 / 29.75 / 0.14 µs |
| PostgreSQL rollback / deadlock / conflict / temp files | 0 / 0 / 0 / 0 |

The load generated exactly 1,833 chat rows, tracking sessions, and tracking
samples; explicit stop left zero current positions. Cleanup removed the 176
fixture users, 88 requests, 88 assignments, 1,833 messages, 1,833 tracking
sessions, and their audit rows. The before/after application-table count diff
was empty, all five BEAM nodes participated in the PubSub probe, and every load
service remained running.

Ten sequential resource samples recorded maxima of 268.5, 260.4, and
284.3 MiB for the three web cgroups; 207.2 and 198.7 MiB for the workers;
119.9 MiB for PostgreSQL; and 94.85 MiB for Traefik. These short samples had
no memory limit and do not establish a steady-state plateau or server minimum.
No captured query used temporary blocks. The highest mean SQL statement was a
single 5.249 ms tracking-session maintenance update; the highest-total
statement was the indexed session-token lookup at 0.081 ms mean across 7,332
calls. No measured query result justifies another schema or pool change at
this data size.

Ignored evidence:

- `output/performance/final-after-queue-20260720/`