270 lines
13 KiB
Markdown
270 lines
13 KiB
Markdown
# Performance measurement
|
|
|
|
No production capacity, minimum resource requirement, SLO, alert threshold,
|
|
pool size, or autoscaling threshold is known yet. The repository therefore
|
|
contains a reproducible measurement profile, not a capacity claim or blocking
|
|
resource preflight.
|
|
|
|
The profile uses a separate Compose project, generated independent secrets, and
|
|
a separate PostgreSQL volume. Its Traefik instance is constrained to that exact
|
|
Compose project; its router, service, middleware, and Host rule are unique, so
|
|
the load profile cannot join the staging router. It exercises:
|
|
|
|
- dynamic public home and Safety pages;
|
|
- database-backed readiness queries;
|
|
- Phoenix WebSocket upgrades and protocol heartbeats;
|
|
- real password login and authenticated LiveView joins;
|
|
- consent-driven tracking start, authenticated location writes, private chat
|
|
writes, and explicit tracking stop/position deletion;
|
|
- the configured number of web and worker replicas;
|
|
- cross-node Phoenix PubSub after each run.
|
|
|
|
It does not render a real browser, retain a browser tracking session for an
|
|
unknown real-world trip duration, or reproduce an unknown future production
|
|
traffic mix. It also does not sample the k6 load-generator container itself.
|
|
Its results only describe the recorded host, container versions, replica
|
|
counts, database state, and experiment inputs.
|
|
|
|
## Create the isolated profile
|
|
|
|
```sh
|
|
./scripts/ensure-local-load-env.sh
|
|
./scripts/load-stack-up.sh
|
|
```
|
|
|
|
The generated `.env.load` is ignored, restricted to mode 600, and contains
|
|
independent PostgreSQL and application secrets. Edit its `LOAD_*` inputs to
|
|
define a specific experiment. Values in `.env.load.example` are reproducible
|
|
measurement points, not recommendations.
|
|
|
|
## Run and compare replica counts
|
|
|
|
```sh
|
|
./scripts/load-run.sh two-web
|
|
./scripts/load-stack-up.sh 3
|
|
# Set LOAD_WEB_REPLICAS=3 in .env.load so the recorded expected topology
|
|
# matches the running topology, then:
|
|
./scripts/load-run.sh three-web
|
|
```
|
|
|
|
Each run writes ignored evidence below `output/performance/<label>/`:
|
|
|
|
- the exact inputs and Docker host observation;
|
|
- a machine-readable k6 summary and complete console report;
|
|
- periodic Docker CPU/memory/network/block-I/O/PID samples for every load
|
|
stack container;
|
|
- before/after application table counts, exact authenticated-write evidence,
|
|
fixture cleanup output, and a cleanup diff;
|
|
- final Compose state, readiness response, application logs, and cross-node
|
|
PubSub result.
|
|
|
|
Before k6 starts, the production-compiled fixture task verifies the exact
|
|
configured load database name and a deliberate confirmation marker. It creates
|
|
isolated requester/helper/request/assignment pairs through the application
|
|
contexts. The pair count equals the maximum simultaneous VU count across all
|
|
scenarios, because k6 may reuse its global VU pool between parallel scenarios;
|
|
mapping each global VU identifier directly to its own pair prevents concurrent
|
|
users from sharing a tracking assignment. The random fixture password exists
|
|
only in ignored mode-600 `.env.load` and is never written to the manifest or
|
|
console.
|
|
|
|
On success and on trapped failure, cleanup deletes only the UUIDs recorded in
|
|
the validated run manifest. A successful run additionally requires every
|
|
tracked application-table count after cleanup to equal its pre-fixture count.
|
|
In a separate negative check, replacing one helper UUID with another valid UUID
|
|
made cleanup exit nonzero before mutation; all 2 users, 1 request, and 1
|
|
assignment remained. Restoring the original manifest then removed those exact
|
|
rows and returned all three counts to zero.
|
|
|
|
The k6 image is pinned to version 2.1.0 by digest and runs locally with anonymous
|
|
usage reporting disabled. No Grafana Cloud account, API token, OpenAI API, or
|
|
usage-based service is involved.
|
|
|
|
## Observed local comparison
|
|
|
|
Observed on 2026-07-18 with Docker Engine 29.6.2 on the recorded 32-CPU,
|
|
100289265664-byte Docker host. Neither application containers nor the load
|
|
generator had CPU or memory limits. Both runs used 40 HTTP VUs, 40 WebSocket
|
|
VUs, a 30-second duration, a 0.1-second HTTP think time, and five-second socket
|
|
sessions. Each HTTP iteration requested the home page, Safety page, and
|
|
database-backed readiness endpoint.
|
|
|
|
| Observation | 2 web replicas | 3 web replicas |
|
|
| --- | ---: | ---: |
|
|
| HTTP checks | 35,160 / 35,160 passed | 35,163 / 35,163 passed |
|
|
| HTTP failed rate | 0% | 0% |
|
|
| Observed HTTP rate | 1,169.12 requests/s | 1,169.42 requests/s |
|
|
| HTTP duration p95 | 2.87 ms | 3.36 ms |
|
|
| WebSocket opened / heartbeat replies | 240 / 240 | 240 / 240 |
|
|
| WebSocket errors | 0 | 0 |
|
|
| WebSocket connect p95 | 46.65 ms | 24.12 ms |
|
|
| Average CPU per web container | 66.57%, 66.63% | 50.25%, 49.77%, 49.18% |
|
|
| Maximum CPU per web container | 79.80%, 79.21% | 58.16%, 59.23%, 59.08% |
|
|
| Maximum observed memory per web container | 268.3, 238.3 MiB | 221.0, 223.2, 219.4 MiB |
|
|
|
|
Each resource series contained 11 sequential `docker stats --no-stream`
|
|
samples. Both before/after database diffs were empty, both cross-node PubSub
|
|
probes passed, readiness remained healthy, and the captured proxy/web/worker
|
|
logs contained no application error or warning. The public staging root and
|
|
readiness endpoint also returned HTTP 200 while the isolated project was
|
|
running.
|
|
|
|
An earlier harness smoke exposed that fixed Traefik router/service names are
|
|
global to the Docker provider: starting a second Compose project temporarily
|
|
made staging readiness return HTTP 404. The load project was stopped
|
|
immediately and staging returned HTTP 200. The final Compose configuration
|
|
parameterizes the project constraint, router/service name, Docker network, and
|
|
router rule. Merged-config inspection found no shared router/service label key,
|
|
the two running proxies showed different exact project constraints, and both
|
|
origins returned HTTP 200 concurrently before the recorded final runs.
|
|
|
|
The nearly identical request rate is expected from the selected paced workload:
|
|
40 HTTP VUs each issue three requests and then wait 0.1 seconds, giving an
|
|
approximate workload ceiling of 1,200 requests/s before response time is added.
|
|
The result therefore does not show the saturation point of two replicas and
|
|
does not justify an HPA threshold. The lower per-container CPU and lower
|
|
WebSocket connect p95 with three replicas are observations for this run only;
|
|
the slightly higher HTTP p95 also shows why one short comparison is not a
|
|
production sizing study.
|
|
|
|
Ignored evidence:
|
|
|
|
- `output/performance/two-web-final-20260718/`
|
|
- `output/performance/three-web-final-20260718/`
|
|
|
|
## Observed authenticated write run
|
|
|
|
Observed on 2026-07-19 on the same recorded Docker host with 3 web replicas,
|
|
2 worker replicas, 40 paced public HTTP VUs, 40 five-second Phoenix heartbeat
|
|
VUs, and 8 authenticated mutual-aid VUs for 30 seconds. The fixture tool
|
|
created 88 isolated pairs so every possible global VU had a unique account and
|
|
assignment.
|
|
|
|
| Observation | Result |
|
|
| --- | ---: |
|
|
| Public HTTP checks | 35,286 / 35,286 passed |
|
|
| All HTTP failures | 0 / 39,062 |
|
|
| Phoenix heartbeat opened / replies | 240 / 240 |
|
|
| Authenticated logins | 8 |
|
|
| Authenticated pages / LiveView joins | 1,880 / 1,880 |
|
|
| Tracking starts / position writes / stops | 1,880 / 1,880 / 1,880 |
|
|
| Private messages written | 1,880 |
|
|
| Authenticated or WebSocket errors | 0 |
|
|
| HTTP request duration p95 | 5.30 ms |
|
|
| Authenticated HTTP duration p95 | 8.18 ms |
|
|
| WebSocket connect p95 | 3.55 ms |
|
|
|
|
The database evidence counted 1,880 chat rows, 1,880 tracking sessions with
|
|
1,880 total samples, 8 login tokens, and zero current positions after explicit
|
|
stop. Cleanup removed the 176 fixture users, 88 requests, 88 assignments,
|
|
1,880 messages, 1,880 tracking sessions, and their audit rows. The post-cleanup
|
|
application-table diff was empty. Readiness and the five-node web/worker PubSub
|
|
probe passed, and the run-scoped proxy/web/worker logs contained no application
|
|
error, warning, HTTP 4xx/5xx, or database deadlock.
|
|
|
|
Eleven sequential resource samples observed average CPU of 139.84%, 138.12%,
|
|
and 141.90% for the three web containers, with maximum memory of 290.2, 309.4,
|
|
and 292.3 MiB respectively. PostgreSQL averaged 38.23% CPU and reached
|
|
237.1 MiB; Traefik averaged 45.53% CPU and reached 82.39 MiB. These are
|
|
short local observations without container resource limits, not minimums,
|
|
production capacity, or autoscaling inputs.
|
|
|
|
Ignored evidence:
|
|
|
|
- `output/performance/auth-final-20260719i/`
|
|
|
|
## Observed local resilience drills
|
|
|
|
The canonical Compose drill on 2026-07-19 used the isolated 3-web/2-worker
|
|
profile. One web and one worker BEAM process exited with status 1 and each
|
|
container's observed restart count increased to 1. All original replicas were
|
|
then replaced sequentially. The route returned 743 successful readiness
|
|
responses with zero final failure and responses from all three web nodes.
|
|
After replacement, the observed cluster contained all five BEAM nodes and the
|
|
cross-node PubSub probe passed.
|
|
|
|
The local Oban probe completed with state `completed`, attempt `2`,
|
|
`max_attempts=2`, and exactly one recorded first-attempt error. Its exact row
|
|
was removed afterward. The load database then contained zero probe jobs, users,
|
|
requests, messages, and tracking sessions. Run-scoped available logs contained
|
|
no unexpected application error, warning, HTTP 4xx/5xx, or database deadlock.
|
|
|
|
The canonical kind drill rolled both 2-replica Deployments from revision 17 to
|
|
18 with the chart's observed `maxUnavailable=0` and `maxSurge=1`. All four pod
|
|
UIDs changed, all replacements were Ready with zero container restart, the
|
|
four-node BEAM cluster and PubSub probe passed, and the application-table count
|
|
diff was empty. Of 305 readiness samples, all ultimately returned HTTP 200.
|
|
Two samples needed three transport retries in total while kind's single-node
|
|
NodePort reset connections during endpoint changes. Those retries are retained
|
|
in evidence rather than reported as uninterrupted raw TCP connections. This is
|
|
a local kind observation, not a production availability guarantee.
|
|
|
|
Ignored evidence:
|
|
|
|
- `output/resilience/compose-resilience-canonical-20260719f/`
|
|
- `output/resilience/kind-rollout-canonical-20260719c/`
|
|
|
|
Stop the isolated containers without deleting their database volume:
|
|
|
|
```sh
|
|
./scripts/load-stack-stop.sh
|
|
```
|
|
|
|
The stop script reads the actual `LOAD_PROJECT`, refuses the staging project
|
|
name, and does not delete the volume. Volume deletion is intentionally not
|
|
automated.
|
|
|
|
## Isolated database scale and cursor plans
|
|
|
|
Run the database-only before/after measurement with:
|
|
|
|
```sh
|
|
./scripts/db-scale-benchmark.sh
|
|
```
|
|
|
|
The script builds the test image and creates a uniquely named Compose project,
|
|
random one-run PostgreSQL credentials, and a separate PostGIS volume. It
|
|
migrates only through `20260719004249`, seeds the isolated baseline, records
|
|
machine-readable PostgreSQL
|
|
`EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON)` plans, applies the generated
|
|
`20260719013320_add_keyset_pagination_indexes` migration, and records the same
|
|
queries again. It also verifies that two 25-row keyset pages exactly equal the
|
|
first 50 ordered rows without duplicates or gaps. The exact project, network,
|
|
volume, and image are removed by its trap.
|
|
|
|
`DB_SCALE_ROWS` controls the configured sample size and defaults to 50,000.
|
|
That default is a reproducible experiment input, not a minimum database size,
|
|
capacity claim, latency target, or production traffic model. The structural
|
|
gate checks the canonical sample's chosen cursor indexes and query correctness;
|
|
it does not fail on an arbitrary millisecond threshold.
|
|
|
|
Observed locally on 2026-07-19 with PostgreSQL 18.4, the canonical run created
|
|
50,000 rows in each of users, requests, assignments, messages, reviews,
|
|
reports, abuse signals, proposals, activities, and activity messages; it
|
|
created 50,500 activity participants and 49,998 block rows. The isolated
|
|
database occupied 231,225,023 bytes after the cursor migration.
|
|
|
|
| Query | Baseline observed | Cursor migration observed | Selected cursor index |
|
|
| --- | ---: | ---: | --- |
|
|
| Urgent-help discovery | 8.437 ms | 0.149 ms | `help_requests_discovery_cursor_index` |
|
|
| Requester history | 11.287 ms | 0.038 ms | `help_requests_requester_cursor_index` |
|
|
| Activity discovery | 8.760 ms | 0.061 ms | `activities_discovery_cursor_index` |
|
|
| Visible reviews | 8.997 ms | 0.039 ms | `reviews_visible_cursor_index` |
|
|
| All reports | 9.293 ms | 0.026 ms | `reports_cursor_index` |
|
|
| All category proposals | 6.776 ms | 0.028 ms | `category_proposals_cursor_index` |
|
|
| Moderation users | 5.805 ms | 0.032 ms | `users_moderation_cursor_index` |
|
|
| Blocks | 5.388 ms | 0.044 ms | `blocks_blocker_cursor_index` |
|
|
|
|
These are measurements of one warm local run and must not be interpreted as an
|
|
SLO or portable speedup. The measurement also exposed redundant candidate
|
|
indexes and an `OR`-based Activity membership query. The final migration keeps
|
|
the indexes PostgreSQL selected, uses partial discovery indexes for open,
|
|
non-hidden records, and the application joins the existing participant
|
|
membership invariant directly. Personal reputation and the leaderboard now
|
|
aggregate in PostgreSQL instead of loading all completed assignments into the
|
|
BEAM; the leaderboard itself uses composite keyset pagination.
|
|
|
|
Ignored evidence for the recorded run:
|
|
|
|
- `output/db-scale/20260719020251-1612535/`
|