176 lines
8.1 KiB
Markdown
176 lines
8.1 KiB
Markdown
# Performance measurement
|
|
|
|
No production capacity, minimum resource requirement, SLO, alert threshold,
|
|
pool size, or autoscaling threshold is known yet. The repository therefore
|
|
contains a reproducible measurement profile, not a capacity claim or blocking
|
|
resource preflight.
|
|
|
|
The profile uses a separate Compose project, generated independent secrets, and
|
|
a separate PostgreSQL volume. Its Traefik instance is constrained to that exact
|
|
Compose project; its router, service, middleware, and Host rule are unique, so
|
|
the load profile cannot join the staging router. It exercises:
|
|
|
|
- dynamic public home and Safety pages;
|
|
- database-backed readiness queries;
|
|
- Phoenix WebSocket upgrades and protocol heartbeats;
|
|
- the configured number of web and worker replicas;
|
|
- cross-node Phoenix PubSub after each run.
|
|
|
|
It does not authenticate users, create requests, send chat messages, update
|
|
locations, render a real browser, or reproduce an unknown future production
|
|
traffic mix. It also does not sample the k6 load-generator container itself.
|
|
Its results only describe the recorded host, container versions, replica
|
|
counts, database state, and experiment inputs.
|
|
|
|
## Create the isolated profile
|
|
|
|
```sh
|
|
./scripts/ensure-local-load-env.sh
|
|
./scripts/load-stack-up.sh
|
|
```
|
|
|
|
The generated `.env.load` is ignored, restricted to mode 600, and contains
|
|
independent PostgreSQL and application secrets. Edit its `LOAD_*` inputs to
|
|
define a specific experiment. Values in `.env.load.example` are reproducible
|
|
measurement points, not recommendations.
|
|
|
|
## Run and compare replica counts
|
|
|
|
```sh
|
|
./scripts/load-run.sh two-web
|
|
./scripts/load-stack-up.sh 3
|
|
# Set LOAD_WEB_REPLICAS=3 in .env.load so the recorded expected topology
|
|
# matches the running topology, then:
|
|
./scripts/load-run.sh three-web
|
|
```
|
|
|
|
Each run writes ignored evidence below `output/performance/<label>/`:
|
|
|
|
- the exact inputs and Docker host observation;
|
|
- a machine-readable k6 summary and complete console report;
|
|
- periodic Docker CPU/memory/network/block-I/O/PID samples for every load
|
|
stack container;
|
|
- before/after application table counts and their diff;
|
|
- final Compose state, readiness response, application logs, and cross-node
|
|
PubSub result.
|
|
|
|
The k6 image is pinned to version 2.1.0 by digest and runs locally with anonymous
|
|
usage reporting disabled. No Grafana Cloud account, API token, OpenAI API, or
|
|
usage-based service is involved.
|
|
|
|
## Observed local comparison
|
|
|
|
Observed on 2026-07-18 with Docker Engine 29.6.2 on the recorded 32-CPU,
|
|
100289265664-byte Docker host. Neither application containers nor the load
|
|
generator had CPU or memory limits. Both runs used 40 HTTP VUs, 40 WebSocket
|
|
VUs, a 30-second duration, a 0.1-second HTTP think time, and five-second socket
|
|
sessions. Each HTTP iteration requested the home page, Safety page, and
|
|
database-backed readiness endpoint.
|
|
|
|
| Observation | 2 web replicas | 3 web replicas |
|
|
| --- | ---: | ---: |
|
|
| HTTP checks | 35,160 / 35,160 passed | 35,163 / 35,163 passed |
|
|
| HTTP failed rate | 0% | 0% |
|
|
| Observed HTTP rate | 1,169.12 requests/s | 1,169.42 requests/s |
|
|
| HTTP duration p95 | 2.87 ms | 3.36 ms |
|
|
| WebSocket opened / heartbeat replies | 240 / 240 | 240 / 240 |
|
|
| WebSocket errors | 0 | 0 |
|
|
| WebSocket connect p95 | 46.65 ms | 24.12 ms |
|
|
| Average CPU per web container | 66.57%, 66.63% | 50.25%, 49.77%, 49.18% |
|
|
| Maximum CPU per web container | 79.80%, 79.21% | 58.16%, 59.23%, 59.08% |
|
|
| Maximum observed memory per web container | 268.3, 238.3 MiB | 221.0, 223.2, 219.4 MiB |
|
|
|
|
Each resource series contained 11 sequential `docker stats --no-stream`
|
|
samples. Both before/after database diffs were empty, both cross-node PubSub
|
|
probes passed, readiness remained healthy, and the captured proxy/web/worker
|
|
logs contained no application error or warning. The public staging root and
|
|
readiness endpoint also returned HTTP 200 while the isolated project was
|
|
running.
|
|
|
|
An earlier harness smoke exposed that fixed Traefik router/service names are
|
|
global to the Docker provider: starting a second Compose project temporarily
|
|
made staging readiness return HTTP 404. The load project was stopped
|
|
immediately and staging returned HTTP 200. The final Compose configuration
|
|
parameterizes the project constraint, router/service name, Docker network, and
|
|
router rule. Merged-config inspection found no shared router/service label key,
|
|
the two running proxies showed different exact project constraints, and both
|
|
origins returned HTTP 200 concurrently before the recorded final runs.
|
|
|
|
The nearly identical request rate is expected from the selected paced workload:
|
|
40 HTTP VUs each issue three requests and then wait 0.1 seconds, giving an
|
|
approximate workload ceiling of 1,200 requests/s before response time is added.
|
|
The result therefore does not show the saturation point of two replicas and
|
|
does not justify an HPA threshold. The lower per-container CPU and lower
|
|
WebSocket connect p95 with three replicas are observations for this run only;
|
|
the slightly higher HTTP p95 also shows why one short comparison is not a
|
|
production sizing study.
|
|
|
|
Ignored evidence:
|
|
|
|
- `output/performance/two-web-final-20260718/`
|
|
- `output/performance/three-web-final-20260718/`
|
|
|
|
Stop the isolated containers without deleting their database volume:
|
|
|
|
```sh
|
|
./scripts/load-stack-stop.sh
|
|
```
|
|
|
|
The stop script reads the actual `LOAD_PROJECT`, refuses the staging project
|
|
name, and does not delete the volume. Volume deletion is intentionally not
|
|
automated.
|
|
|
|
## Isolated database scale and cursor plans
|
|
|
|
Run the database-only before/after measurement with:
|
|
|
|
```sh
|
|
./scripts/db-scale-benchmark.sh
|
|
```
|
|
|
|
The script builds the test image and creates a uniquely named Compose project,
|
|
random one-run PostgreSQL credentials, and a separate PostGIS volume. It
|
|
migrates only through `20260719004249`, seeds the isolated baseline, records
|
|
machine-readable PostgreSQL
|
|
`EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON)` plans, applies the generated
|
|
`20260719013320_add_keyset_pagination_indexes` migration, and records the same
|
|
queries again. It also verifies that two 25-row keyset pages exactly equal the
|
|
first 50 ordered rows without duplicates or gaps. The exact project, network,
|
|
volume, and image are removed by its trap.
|
|
|
|
`DB_SCALE_ROWS` controls the configured sample size and defaults to 50,000.
|
|
That default is a reproducible experiment input, not a minimum database size,
|
|
capacity claim, latency target, or production traffic model. The structural
|
|
gate checks the canonical sample's chosen cursor indexes and query correctness;
|
|
it does not fail on an arbitrary millisecond threshold.
|
|
|
|
Observed locally on 2026-07-19 with PostgreSQL 18.4, the canonical run created
|
|
50,000 rows in each of users, requests, assignments, messages, reviews,
|
|
reports, abuse signals, proposals, activities, and activity messages; it
|
|
created 50,500 activity participants and 49,998 block rows. The isolated
|
|
database occupied 231,225,023 bytes after the cursor migration.
|
|
|
|
| Query | Baseline observed | Cursor migration observed | Selected cursor index |
|
|
| --- | ---: | ---: | --- |
|
|
| Urgent-help discovery | 8.437 ms | 0.149 ms | `help_requests_discovery_cursor_index` |
|
|
| Requester history | 11.287 ms | 0.038 ms | `help_requests_requester_cursor_index` |
|
|
| Activity discovery | 8.760 ms | 0.061 ms | `activities_discovery_cursor_index` |
|
|
| Visible reviews | 8.997 ms | 0.039 ms | `reviews_visible_cursor_index` |
|
|
| All reports | 9.293 ms | 0.026 ms | `reports_cursor_index` |
|
|
| All category proposals | 6.776 ms | 0.028 ms | `category_proposals_cursor_index` |
|
|
| Moderation users | 5.805 ms | 0.032 ms | `users_moderation_cursor_index` |
|
|
| Blocks | 5.388 ms | 0.044 ms | `blocks_blocker_cursor_index` |
|
|
|
|
These are measurements of one warm local run and must not be interpreted as an
|
|
SLO or portable speedup. The measurement also exposed redundant candidate
|
|
indexes and an `OR`-based Activity membership query. The final migration keeps
|
|
the indexes PostgreSQL selected, uses partial discovery indexes for open,
|
|
non-hidden records, and the application joins the existing participant
|
|
membership invariant directly. Personal reputation and the leaderboard now
|
|
aggregate in PostgreSQL instead of loading all completed assignments into the
|
|
BEAM; the leaderboard itself uses composite keyset pagination.
|
|
|
|
Ignored evidence for the recorded run:
|
|
|
|
- `output/db-scale/20260719020251-1612535/`
|