test: measure database load and pool pressure
This commit is contained in:
parent
fdaef2d685
commit
eacfdb2451
16
README.md
16
README.md
|
|
@ -231,6 +231,22 @@ The same external-service protocol drill used by CI can be run independently:
|
||||||
./scripts/external-boundaries-run.sh local-boundaries
|
./scripts/external-boundaries-run.sh local-boundaries
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Run the isolated HTTP/WebSocket/authenticated chat/tracking load profile with
|
||||||
|
resource, PostgreSQL statement, database-connection, and Ecto pool-wait
|
||||||
|
measurements:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./scripts/load-stack-up.sh
|
||||||
|
./scripts/load-run.sh local-load
|
||||||
|
./scripts/load-stack-stop.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
The profile has its own generated mode-`0600` environment, Compose project,
|
||||||
|
and PostgreSQL volume. It does not use or mutate the ordinary public Compose
|
||||||
|
database. Exact inputs and threshold-free evidence are retained below
|
||||||
|
`output/performance/local-load/`; see
|
||||||
|
[Performance measurement](docs/performance.md) for scope and interpretation.
|
||||||
|
|
||||||
The cursor-pagination database benchmark also creates a one-run Compose
|
The cursor-pagination database benchmark also creates a one-run Compose
|
||||||
project, random database credentials, and a separate PostgreSQL volume:
|
project, random database credentials, and a separate PostgreSQL volume:
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,15 @@
|
||||||
services:
|
services:
|
||||||
|
db:
|
||||||
|
# Query-level statistics are enabled only for the isolated load profile.
|
||||||
|
# PostgreSQL requires pg_stat_statements in shared_preload_libraries and a
|
||||||
|
# server restart; the ordinary Compose database is not changed.
|
||||||
|
command:
|
||||||
|
- postgres
|
||||||
|
- -c
|
||||||
|
- shared_preload_libraries=pg_stat_statements
|
||||||
|
- -c
|
||||||
|
- compute_query_id=on
|
||||||
|
|
||||||
proxy:
|
proxy:
|
||||||
networks:
|
networks:
|
||||||
edge:
|
edge:
|
||||||
|
|
|
||||||
|
|
@ -97,11 +97,24 @@ Each run writes ignored evidence below `output/performance/<label>/`:
|
||||||
stack container, plus a threshold-free CPU/RAM/PID JSON summary;
|
stack container, plus a threshold-free CPU/RAM/PID JSON summary;
|
||||||
- read-only PostgreSQL database size and `pg_stat_database` snapshots before
|
- read-only PostgreSQL database size and `pg_stat_database` snapshots before
|
||||||
and after k6, plus their machine-readable counter deltas;
|
and after k6, plus their machine-readable counter deltas;
|
||||||
|
- `pg_stat_statements` rankings by total, mean, and maximum execution time,
|
||||||
|
including calls, rows, buffer activity, temporary blocks, and WAL activity;
|
||||||
|
- periodic `pg_stat_activity` connection samples with configured pool slots,
|
||||||
|
observed connection headroom, transaction state, lock waits, and named
|
||||||
|
active wait events;
|
||||||
|
- per-web Ecto telemetry deltas separating database execution, pool queue, and
|
||||||
|
result decode time;
|
||||||
- before/after application table counts, exact authenticated-write evidence,
|
- before/after application table counts, exact authenticated-write evidence,
|
||||||
fixture cleanup output, and a cleanup diff;
|
fixture cleanup output, and a cleanup diff;
|
||||||
- final Compose state, readiness response, application logs, and cross-node
|
- final Compose state, readiness response, application logs, and cross-node
|
||||||
PubSub result.
|
PubSub result.
|
||||||
|
|
||||||
|
The load-only PostgreSQL command preloads `pg_stat_statements` and creates the
|
||||||
|
extension only in the isolated load database. The application exposes
|
||||||
|
cumulative Ecto measurements through its authenticated metrics endpoint; the
|
||||||
|
runner snapshots every web replica before and after k6 and records the delta.
|
||||||
|
Neither mechanism changes the ordinary public Compose database.
|
||||||
|
|
||||||
Before k6 starts, the production-compiled fixture task verifies the exact
|
Before k6 starts, the production-compiled fixture task verifies the exact
|
||||||
configured load database name and a deliberate confirmation marker. It creates
|
configured load database name and a deliberate confirmation marker. It creates
|
||||||
isolated requester/helper/request/assignment pairs through the application
|
isolated requester/helper/request/assignment pairs through the application
|
||||||
|
|
@ -217,6 +230,71 @@ Ignored evidence:
|
||||||
|
|
||||||
- `output/performance/final-load-20260719a/`
|
- `output/performance/final-load-20260719a/`
|
||||||
|
|
||||||
|
## Observed SQL and connection-pool run
|
||||||
|
|
||||||
|
Observed on 2026-07-20 with 3 web replicas, 2 worker replicas, PostgreSQL 18.4,
|
||||||
|
40 paced public HTTP VUs, 40 Phoenix heartbeat VUs, and 8 authenticated
|
||||||
|
mutual-aid VUs for 30 seconds.
|
||||||
|
|
||||||
|
| Observation | Result |
|
||||||
|
| --- | ---: |
|
||||||
|
| HTTP requests / failures | 38,429 / 0 |
|
||||||
|
| Public checks | 35,163 / 35,163 passed |
|
||||||
|
| HTTP duration average / p95 / maximum | 2.42 / 7.74 / 187.11 ms |
|
||||||
|
| Authenticated chat/tracking chains | 1,664 |
|
||||||
|
| Phoenix heartbeat opened / replies | 240 / 240 |
|
||||||
|
| Configured Ecto Repo pool slots | 50 |
|
||||||
|
| Peak PostgreSQL client backends | 55 |
|
||||||
|
| Minimum observed non-reserved connection headroom | 42 |
|
||||||
|
| Idle-in-transaction / lock-waiting backends | 0 / 0 |
|
||||||
|
| Ecto average execution / queue / decode time | 308.58 / 31.15 / 0.27 µs |
|
||||||
|
| PostgreSQL rollback / deadlock / conflict / temp files | 0 / 0 / 0 / 0 |
|
||||||
|
|
||||||
|
The connection sampler observed active PostgreSQL waits named
|
||||||
|
`Client:ClientRead` and `IO:WalSync`; it observed no lock wait or
|
||||||
|
idle-in-transaction backend. The Ecto queue
|
||||||
|
measurement is the application-side evidence for time waiting to check out a
|
||||||
|
Repo connection. It averaged 31.15 microseconds over 166,459 queries across
|
||||||
|
the three web replicas. This short run did not produce a connection timeout or
|
||||||
|
establish a maximum connection capacity.
|
||||||
|
|
||||||
|
The SQL ranking exposed an unbounded preload of every message attached to a
|
||||||
|
request even though the LiveView immediately loaded a separate 50-message
|
||||||
|
cursor page. In the before run that redundant statement executed 3,318 times,
|
||||||
|
returned 342,378 rows in aggregate, and consumed 392.378 ms of PostgreSQL
|
||||||
|
execution time. The request preload was removed and a regression test now
|
||||||
|
requires the chat association to remain unloaded. In the repeat run the
|
||||||
|
unbounded statement was absent; the bounded cursor statement retained its
|
||||||
|
`LIMIT`, executed 3,328 times, returned 148,512 rows, and averaged 0.122 ms.
|
||||||
|
|
||||||
|
No normalized statement in the repeat run used temporary blocks. The largest
|
||||||
|
mean statement time was 0.481 ms for six session-token inserts; the
|
||||||
|
highest-call request lookup averaged 0.031 ms over 39,936 calls. These are
|
||||||
|
observations against the small generated load dataset, not proof that every
|
||||||
|
query remains fast at arbitrary data size. The separate 50,000-row database
|
||||||
|
profile below checks the indexed list and cursor queries at a larger
|
||||||
|
cardinality.
|
||||||
|
|
||||||
|
Across the seven measured application/database/proxy containers, simultaneous
|
||||||
|
memory samples totalled 1,501.60–1,509.86 MiB during this run. This excludes
|
||||||
|
the host OS, the k6 generator, Mailpit, and optional observability services.
|
||||||
|
Therefore it is not a claim that this 3-web/2-worker topology fits a 1 GiB or
|
||||||
|
2 GiB server. It does show that the measured workload used about 1.47 GiB for
|
||||||
|
those seven containers rather than the former kind port-table failure's
|
||||||
|
roughly 10 GiB.
|
||||||
|
|
||||||
|
The run created exactly 1,664 messages, tracking sessions, and tracking
|
||||||
|
samples, plus 8 session tokens; explicit tracking stop left zero current
|
||||||
|
positions. Cleanup restored every tracked application-table count, readiness
|
||||||
|
and the five-node PubSub probe passed, and the captured application log had no
|
||||||
|
database timeout, Postgrex error, deadlock, `ERROR`, `FATAL`, or `PANIC`
|
||||||
|
match.
|
||||||
|
|
||||||
|
Ignored before/after evidence:
|
||||||
|
|
||||||
|
- `output/performance/pool-sql-final-20260720/`
|
||||||
|
- `output/performance/pool-sql-chat-fixed-canonical-20260720/`
|
||||||
|
|
||||||
## Observed 10-minute authenticated soak
|
## Observed 10-minute authenticated soak
|
||||||
|
|
||||||
Observed on 2026-07-19 with the same 3-web/2-worker topology and the same
|
Observed on 2026-07-19 with the same 3-web/2-worker topology and the same
|
||||||
|
|
|
||||||
|
|
@ -110,7 +110,7 @@ defmodule WhoNeedHelp.Help do
|
||||||
|> Repo.preload([
|
|> Repo.preload([
|
||||||
:category,
|
:category,
|
||||||
requester: :social_identities,
|
requester: :social_identities,
|
||||||
assignment: [helper: :social_identities, messages: :sender]
|
assignment: [helper: :social_identities]
|
||||||
])
|
])
|
||||||
end
|
end
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -60,6 +60,31 @@ defmodule WhoNeedHelpWeb.Telemetry do
|
||||||
end,
|
end,
|
||||||
description: "Cumulative database query duration"
|
description: "Cumulative database query duration"
|
||||||
),
|
),
|
||||||
|
sum("who_need_help.database.query.execution.duration.microseconds.total",
|
||||||
|
event_name: [:who_need_help, :repo, :query],
|
||||||
|
measurement: fn measurements ->
|
||||||
|
System.convert_time_unit(measurements.query_time, :native, :microsecond)
|
||||||
|
end,
|
||||||
|
description: "Cumulative database execution duration"
|
||||||
|
),
|
||||||
|
sum("who_need_help.database.query.queue.duration.microseconds.total",
|
||||||
|
event_name: [:who_need_help, :repo, :query],
|
||||||
|
measurement: fn measurements ->
|
||||||
|
measurements
|
||||||
|
|> Map.get(:queue_time, 0)
|
||||||
|
|> System.convert_time_unit(:native, :microsecond)
|
||||||
|
end,
|
||||||
|
description: "Cumulative time waiting for a database connection"
|
||||||
|
),
|
||||||
|
sum("who_need_help.database.query.decode.duration.microseconds.total",
|
||||||
|
event_name: [:who_need_help, :repo, :query],
|
||||||
|
measurement: fn measurements ->
|
||||||
|
measurements
|
||||||
|
|> Map.get(:decode_time, 0)
|
||||||
|
|> System.convert_time_unit(:native, :microsecond)
|
||||||
|
end,
|
||||||
|
description: "Cumulative database result decoding duration"
|
||||||
|
),
|
||||||
counter("who_need_help.websocket.connections.total",
|
counter("who_need_help.websocket.connections.total",
|
||||||
event_name: [:phoenix, :socket_connected],
|
event_name: [:phoenix, :socket_connected],
|
||||||
measurement: :duration,
|
measurement: :duration,
|
||||||
|
|
|
||||||
|
|
@ -122,8 +122,123 @@ chmod 700 "$ROOT/output" "$ROOT/output/performance" "$output_dir"
|
||||||
run_started_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
run_started_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||||
running_marker="$output_dir/.sampling"
|
running_marker="$output_dir/.sampling"
|
||||||
resource_log="$output_dir/docker-stats.jsonl"
|
resource_log="$output_dir/docker-stats.jsonl"
|
||||||
|
connection_log="$output_dir/database-connections.jsonl"
|
||||||
touch "$running_marker"
|
touch "$running_marker"
|
||||||
|
|
||||||
|
database_psql() {
|
||||||
|
# The variables below are intentionally expanded inside the database container.
|
||||||
|
# shellcheck disable=SC2016
|
||||||
|
"${compose[@]}" exec -T db sh -c \
|
||||||
|
'psql --no-psqlrc --quiet --tuples-only --no-align --set ON_ERROR_STOP=1 \
|
||||||
|
--username "$POSTGRES_USER" --dbname "$POSTGRES_DB"'
|
||||||
|
}
|
||||||
|
|
||||||
|
prepare_statement_stats() {
|
||||||
|
local preloaded
|
||||||
|
preloaded=$(database_psql <<'SQL'
|
||||||
|
SELECT current_setting('shared_preload_libraries')
|
||||||
|
~ '(^|,)pg_stat_statements(,|$)';
|
||||||
|
SQL
|
||||||
|
)
|
||||||
|
|
||||||
|
if [[ "$preloaded" != "t" ]]; then
|
||||||
|
echo "pg_stat_statements is not present in shared_preload_libraries." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
database_psql >"$output_dir/statement-statistics-setup.txt" <<'SQL'
|
||||||
|
SELECT current_setting('shared_preload_libraries');
|
||||||
|
CREATE EXTENSION IF NOT EXISTS pg_stat_statements;
|
||||||
|
SQL
|
||||||
|
}
|
||||||
|
|
||||||
|
reset_statement_stats() {
|
||||||
|
database_psql >"$output_dir/statement-statistics-reset.txt" <<'SQL'
|
||||||
|
SELECT pg_stat_statements_reset();
|
||||||
|
SQL
|
||||||
|
}
|
||||||
|
|
||||||
|
snapshot_statement_stats() {
|
||||||
|
local destination=$1
|
||||||
|
|
||||||
|
database_psql >"$destination" <<'SQL'
|
||||||
|
WITH statements AS (
|
||||||
|
SELECT
|
||||||
|
queryid::text AS query_id,
|
||||||
|
regexp_replace(query, '[[:space:]]+', ' ', 'g') AS query,
|
||||||
|
calls,
|
||||||
|
rows,
|
||||||
|
round(total_exec_time::numeric, 3) AS total_exec_time_ms,
|
||||||
|
round(mean_exec_time::numeric, 3) AS mean_exec_time_ms,
|
||||||
|
round(max_exec_time::numeric, 3) AS max_exec_time_ms,
|
||||||
|
shared_blks_hit,
|
||||||
|
shared_blks_read,
|
||||||
|
shared_blks_dirtied,
|
||||||
|
shared_blks_written,
|
||||||
|
temp_blks_read,
|
||||||
|
temp_blks_written,
|
||||||
|
wal_records,
|
||||||
|
wal_fpi,
|
||||||
|
wal_bytes
|
||||||
|
FROM pg_stat_statements
|
||||||
|
WHERE dbid = (SELECT oid FROM pg_database WHERE datname = current_database())
|
||||||
|
AND query NOT LIKE '%pg_stat_statements%'
|
||||||
|
AND query NOT LIKE '%pg_stat_activity%'
|
||||||
|
AND query NOT LIKE '%pg_stat_database%'
|
||||||
|
),
|
||||||
|
top_total AS (
|
||||||
|
SELECT *
|
||||||
|
FROM statements
|
||||||
|
ORDER BY total_exec_time_ms DESC, query_id
|
||||||
|
LIMIT 25
|
||||||
|
),
|
||||||
|
top_mean AS (
|
||||||
|
SELECT *
|
||||||
|
FROM statements
|
||||||
|
ORDER BY mean_exec_time_ms DESC, query_id
|
||||||
|
LIMIT 25
|
||||||
|
),
|
||||||
|
top_max AS (
|
||||||
|
SELECT *
|
||||||
|
FROM statements
|
||||||
|
ORDER BY max_exec_time_ms DESC, query_id
|
||||||
|
LIMIT 25
|
||||||
|
)
|
||||||
|
SELECT jsonb_pretty(
|
||||||
|
jsonb_build_object(
|
||||||
|
'captured_at', clock_timestamp(),
|
||||||
|
'measurement', 'pg_stat_statements since the pre-load reset',
|
||||||
|
'statement_count', (SELECT count(*) FROM statements),
|
||||||
|
'statistics_info', (SELECT to_jsonb(info) FROM pg_stat_statements_info AS info),
|
||||||
|
'top_by_total_execution_time',
|
||||||
|
(SELECT coalesce(
|
||||||
|
jsonb_agg(
|
||||||
|
to_jsonb(item)
|
||||||
|
ORDER BY total_exec_time_ms DESC, query_id
|
||||||
|
),
|
||||||
|
'[]'::jsonb
|
||||||
|
) FROM top_total AS item),
|
||||||
|
'top_by_mean_execution_time',
|
||||||
|
(SELECT coalesce(
|
||||||
|
jsonb_agg(
|
||||||
|
to_jsonb(item)
|
||||||
|
ORDER BY mean_exec_time_ms DESC, query_id
|
||||||
|
),
|
||||||
|
'[]'::jsonb
|
||||||
|
) FROM top_mean AS item),
|
||||||
|
'top_by_max_execution_time',
|
||||||
|
(SELECT coalesce(
|
||||||
|
jsonb_agg(
|
||||||
|
to_jsonb(item)
|
||||||
|
ORDER BY max_exec_time_ms DESC, query_id
|
||||||
|
),
|
||||||
|
'[]'::jsonb
|
||||||
|
) FROM top_max AS item)
|
||||||
|
)
|
||||||
|
);
|
||||||
|
SQL
|
||||||
|
}
|
||||||
|
|
||||||
snapshot_database() {
|
snapshot_database() {
|
||||||
local destination=$1
|
local destination=$1
|
||||||
|
|
||||||
|
|
@ -246,6 +361,271 @@ summarize_resources() {
|
||||||
/output/resource-summary.json
|
/output/resource-summary.json
|
||||||
}
|
}
|
||||||
|
|
||||||
|
sample_database_connections() {
|
||||||
|
database_psql >>"$connection_log" <<'SQL'
|
||||||
|
WITH activity AS (
|
||||||
|
SELECT *
|
||||||
|
FROM pg_stat_activity
|
||||||
|
WHERE backend_type = 'client backend'
|
||||||
|
AND pid <> pg_backend_pid()
|
||||||
|
),
|
||||||
|
active_wait_events AS (
|
||||||
|
SELECT
|
||||||
|
concat_ws(':', wait_event_type, wait_event) AS event,
|
||||||
|
count(*) AS backends
|
||||||
|
FROM activity
|
||||||
|
WHERE state = 'active'
|
||||||
|
AND wait_event IS NOT NULL
|
||||||
|
GROUP BY wait_event_type, wait_event
|
||||||
|
)
|
||||||
|
SELECT json_build_object(
|
||||||
|
'captured_at', clock_timestamp(),
|
||||||
|
'max_connections', current_setting('max_connections')::integer,
|
||||||
|
'reserved_connections',
|
||||||
|
current_setting('reserved_connections')::integer,
|
||||||
|
'superuser_reserved_connections',
|
||||||
|
current_setting('superuser_reserved_connections')::integer,
|
||||||
|
'client_backends',
|
||||||
|
count(*),
|
||||||
|
'active_backends',
|
||||||
|
count(*) FILTER (WHERE state = 'active'),
|
||||||
|
'idle_backends',
|
||||||
|
count(*) FILTER (WHERE state = 'idle'),
|
||||||
|
'idle_in_transaction_backends',
|
||||||
|
count(*) FILTER (WHERE state = 'idle in transaction'),
|
||||||
|
'active_waiting_backends',
|
||||||
|
count(*) FILTER (
|
||||||
|
WHERE state = 'active'
|
||||||
|
AND wait_event IS NOT NULL
|
||||||
|
),
|
||||||
|
'lock_waiting_backends',
|
||||||
|
count(*) FILTER (WHERE wait_event_type = 'Lock'),
|
||||||
|
'active_wait_events',
|
||||||
|
(
|
||||||
|
SELECT coalesce(json_object_agg(event, backends), '{}'::json)
|
||||||
|
FROM active_wait_events
|
||||||
|
),
|
||||||
|
'longest_idle_in_transaction_ms',
|
||||||
|
coalesce(
|
||||||
|
max(
|
||||||
|
extract(epoch FROM clock_timestamp() - xact_start) * 1000
|
||||||
|
) FILTER (
|
||||||
|
WHERE state = 'idle in transaction'
|
||||||
|
),
|
||||||
|
0
|
||||||
|
),
|
||||||
|
'longest_active_query_ms',
|
||||||
|
coalesce(
|
||||||
|
max(
|
||||||
|
extract(epoch FROM clock_timestamp() - query_start) * 1000
|
||||||
|
) FILTER (
|
||||||
|
WHERE state = 'active'
|
||||||
|
),
|
||||||
|
0
|
||||||
|
)
|
||||||
|
)
|
||||||
|
FROM activity;
|
||||||
|
SQL
|
||||||
|
}
|
||||||
|
|
||||||
|
summarize_database_connections() {
|
||||||
|
jq -s \
|
||||||
|
--argjson pool_size "$POOL_SIZE" \
|
||||||
|
--argjson app_replicas "$((LOAD_WEB_REPLICAS + LOAD_WORKER_REPLICAS))" '
|
||||||
|
. as $samples |
|
||||||
|
{
|
||||||
|
schema_version: 1,
|
||||||
|
measurement: "read-only pg_stat_activity samples during the k6 run",
|
||||||
|
thresholds_applied: false,
|
||||||
|
sample_count: length,
|
||||||
|
configured_app_replicas: $app_replicas,
|
||||||
|
configured_pool_size_per_replica: $pool_size,
|
||||||
|
configured_repo_pool_slots: ($app_replicas * $pool_size),
|
||||||
|
max_connections: (map(.max_connections) | max),
|
||||||
|
reserved_connections: (map(.reserved_connections) | max),
|
||||||
|
superuser_reserved_connections:
|
||||||
|
(map(.superuser_reserved_connections) | max),
|
||||||
|
peak_client_backends: (map(.client_backends) | max),
|
||||||
|
peak_active_backends: (map(.active_backends) | max),
|
||||||
|
peak_idle_backends: (map(.idle_backends) | max),
|
||||||
|
peak_idle_in_transaction_backends:
|
||||||
|
(map(.idle_in_transaction_backends) | max),
|
||||||
|
peak_active_waiting_backends:
|
||||||
|
(map(.active_waiting_backends) | max),
|
||||||
|
peak_lock_waiting_backends: (map(.lock_waiting_backends) | max),
|
||||||
|
observed_active_wait_events:
|
||||||
|
([
|
||||||
|
$samples[]
|
||||||
|
| .active_wait_events
|
||||||
|
| to_entries[]
|
||||||
|
]
|
||||||
|
| group_by(.key)
|
||||||
|
| map({
|
||||||
|
event: .[0].key,
|
||||||
|
peak_backends: (map(.value) | max),
|
||||||
|
observed_samples: length
|
||||||
|
})),
|
||||||
|
longest_observed_idle_in_transaction_ms:
|
||||||
|
(map(.longest_idle_in_transaction_ms | tonumber) | max),
|
||||||
|
longest_observed_active_query_ms:
|
||||||
|
(map(.longest_active_query_ms | tonumber) | max),
|
||||||
|
minimum_observed_connection_headroom:
|
||||||
|
((map(
|
||||||
|
.max_connections -
|
||||||
|
.reserved_connections -
|
||||||
|
.superuser_reserved_connections
|
||||||
|
) | min) -
|
||||||
|
(map(.client_backends) | max))
|
||||||
|
}
|
||||||
|
' "$connection_log" >"$output_dir/database-connections-summary.json"
|
||||||
|
}
|
||||||
|
|
||||||
|
prometheus_value() {
|
||||||
|
local metric=$1
|
||||||
|
|
||||||
|
awk -v metric="$metric" '$1 == metric { print $2; found = 1 }
|
||||||
|
END { if (!found) print 0 }'
|
||||||
|
}
|
||||||
|
|
||||||
|
snapshot_web_metrics() {
|
||||||
|
local destination=$1
|
||||||
|
local temporary
|
||||||
|
temporary=$(mktemp "${TMPDIR:-/tmp}/wnh-load-metrics.XXXXXX")
|
||||||
|
: >"$temporary"
|
||||||
|
|
||||||
|
for container in "${web_containers[@]}"; do
|
||||||
|
local scrape node queries total execution queue decode
|
||||||
|
scrape=$(
|
||||||
|
docker exec "$container" sh -c \
|
||||||
|
'curl --fail --silent --show-error \
|
||||||
|
--header "Authorization: Bearer $METRICS_TOKEN" \
|
||||||
|
--header "X-Forwarded-Proto: https" \
|
||||||
|
--header "Host: $PHX_HOST" \
|
||||||
|
http://127.0.0.1:4000/metrics'
|
||||||
|
)
|
||||||
|
node=$(
|
||||||
|
docker exec "$container" /app/bin/who_need_help rpc \
|
||||||
|
'IO.puts(JSON.encode!(%{node: to_string(node())}))' |
|
||||||
|
tail -n 1 |
|
||||||
|
jq -r '.node'
|
||||||
|
)
|
||||||
|
queries=$(
|
||||||
|
printf '%s\n' "$scrape" |
|
||||||
|
prometheus_value who_need_help_database_queries_total
|
||||||
|
)
|
||||||
|
total=$(
|
||||||
|
printf '%s\n' "$scrape" |
|
||||||
|
prometheus_value who_need_help_database_query_duration_microseconds_total
|
||||||
|
)
|
||||||
|
execution=$(
|
||||||
|
printf '%s\n' "$scrape" |
|
||||||
|
prometheus_value \
|
||||||
|
who_need_help_database_query_execution_duration_microseconds_total
|
||||||
|
)
|
||||||
|
queue=$(
|
||||||
|
printf '%s\n' "$scrape" |
|
||||||
|
prometheus_value \
|
||||||
|
who_need_help_database_query_queue_duration_microseconds_total
|
||||||
|
)
|
||||||
|
decode=$(
|
||||||
|
printf '%s\n' "$scrape" |
|
||||||
|
prometheus_value \
|
||||||
|
who_need_help_database_query_decode_duration_microseconds_total
|
||||||
|
)
|
||||||
|
|
||||||
|
jq -cn \
|
||||||
|
--arg container "$container" \
|
||||||
|
--arg node "$node" \
|
||||||
|
--argjson queries "$queries" \
|
||||||
|
--argjson total_duration_us "$total" \
|
||||||
|
--argjson execution_duration_us "$execution" \
|
||||||
|
--argjson queue_duration_us "$queue" \
|
||||||
|
--argjson decode_duration_us "$decode" \
|
||||||
|
'{
|
||||||
|
container: $container,
|
||||||
|
node: $node,
|
||||||
|
queries: $queries,
|
||||||
|
total_duration_us: $total_duration_us,
|
||||||
|
execution_duration_us: $execution_duration_us,
|
||||||
|
queue_duration_us: $queue_duration_us,
|
||||||
|
decode_duration_us: $decode_duration_us
|
||||||
|
}' >>"$temporary"
|
||||||
|
done
|
||||||
|
|
||||||
|
jq -s 'sort_by(.node)' "$temporary" >"$destination"
|
||||||
|
rm -f "$temporary"
|
||||||
|
}
|
||||||
|
|
||||||
|
summarize_web_metrics() {
|
||||||
|
local before=$1
|
||||||
|
local after=$2
|
||||||
|
|
||||||
|
jq -n \
|
||||||
|
--slurpfile before "$before" \
|
||||||
|
--slurpfile after "$after" '
|
||||||
|
[
|
||||||
|
$after[0][] as $after_row
|
||||||
|
| ($before[0][] | select(.node == $after_row.node)) as $before_row
|
||||||
|
| {
|
||||||
|
node: $after_row.node,
|
||||||
|
queries: ($after_row.queries - $before_row.queries),
|
||||||
|
total_duration_us:
|
||||||
|
($after_row.total_duration_us - $before_row.total_duration_us),
|
||||||
|
execution_duration_us:
|
||||||
|
($after_row.execution_duration_us -
|
||||||
|
$before_row.execution_duration_us),
|
||||||
|
queue_duration_us:
|
||||||
|
($after_row.queue_duration_us - $before_row.queue_duration_us),
|
||||||
|
decode_duration_us:
|
||||||
|
($after_row.decode_duration_us - $before_row.decode_duration_us)
|
||||||
|
}
|
||||||
|
| . + {
|
||||||
|
average_total_duration_us:
|
||||||
|
(if .queries > 0 then .total_duration_us / .queries else 0 end),
|
||||||
|
average_execution_duration_us:
|
||||||
|
(if .queries > 0 then .execution_duration_us / .queries else 0 end),
|
||||||
|
average_queue_duration_us:
|
||||||
|
(if .queries > 0 then .queue_duration_us / .queries else 0 end),
|
||||||
|
average_decode_duration_us:
|
||||||
|
(if .queries > 0 then .decode_duration_us / .queries else 0 end)
|
||||||
|
}
|
||||||
|
] as $nodes
|
||||||
|
| {
|
||||||
|
schema_version: 1,
|
||||||
|
measurement:
|
||||||
|
"per-web Ecto telemetry counter deltas around the k6 run",
|
||||||
|
thresholds_applied: false,
|
||||||
|
nodes: $nodes,
|
||||||
|
cluster: (
|
||||||
|
reduce $nodes[] as $node
|
||||||
|
({
|
||||||
|
queries: 0,
|
||||||
|
total_duration_us: 0,
|
||||||
|
execution_duration_us: 0,
|
||||||
|
queue_duration_us: 0,
|
||||||
|
decode_duration_us: 0
|
||||||
|
};
|
||||||
|
.queries += $node.queries |
|
||||||
|
.total_duration_us += $node.total_duration_us |
|
||||||
|
.execution_duration_us += $node.execution_duration_us |
|
||||||
|
.queue_duration_us += $node.queue_duration_us |
|
||||||
|
.decode_duration_us += $node.decode_duration_us)
|
||||||
|
| . + {
|
||||||
|
average_total_duration_us:
|
||||||
|
(if .queries > 0 then .total_duration_us / .queries else 0 end),
|
||||||
|
average_execution_duration_us:
|
||||||
|
(if .queries > 0
|
||||||
|
then .execution_duration_us / .queries else 0 end),
|
||||||
|
average_queue_duration_us:
|
||||||
|
(if .queries > 0 then .queue_duration_us / .queries else 0 end),
|
||||||
|
average_decode_duration_us:
|
||||||
|
(if .queries > 0 then .decode_duration_us / .queries else 0 end)
|
||||||
|
}
|
||||||
|
)
|
||||||
|
}
|
||||||
|
' >"$output_dir/ecto-query-metrics-summary.json"
|
||||||
|
}
|
||||||
|
|
||||||
run_fixture_tool() {
|
run_fixture_tool() {
|
||||||
local action=$1
|
local action=$1
|
||||||
|
|
||||||
|
|
@ -317,6 +697,7 @@ sample_resources() {
|
||||||
docker stats --no-stream --format '{{json .}}' "${measured_containers[@]}" |
|
docker stats --no-stream --format '{{json .}}' "${measured_containers[@]}" |
|
||||||
jq -c --arg observed_at "$observed_at" '. + {ObservedAt: $observed_at}' \
|
jq -c --arg observed_at "$observed_at" '. + {ObservedAt: $observed_at}' \
|
||||||
>>"$resource_log"
|
>>"$resource_log"
|
||||||
|
sample_database_connections
|
||||||
sleep 1
|
sleep 1
|
||||||
done
|
done
|
||||||
}
|
}
|
||||||
|
|
@ -378,11 +759,14 @@ trap cleanup_on_exit EXIT HUP INT TERM
|
||||||
uname -a
|
uname -a
|
||||||
} >"$output_dir/environment.txt"
|
} >"$output_dir/environment.txt"
|
||||||
|
|
||||||
|
prepare_statement_stats
|
||||||
snapshot_database "$output_dir/database-pre-fixtures.txt"
|
snapshot_database "$output_dir/database-pre-fixtures.txt"
|
||||||
run_fixture_tool prepare >"$output_dir/fixture-prepare.log" 2>&1
|
run_fixture_tool prepare >"$output_dir/fixture-prepare.log" 2>&1
|
||||||
fixtures_prepared=true
|
fixtures_prepared=true
|
||||||
snapshot_database "$output_dir/database-before-load.txt"
|
snapshot_database "$output_dir/database-before-load.txt"
|
||||||
snapshot_database_metrics "$output_dir/database-metrics-before-load.json"
|
snapshot_database_metrics "$output_dir/database-metrics-before-load.json"
|
||||||
|
reset_statement_stats
|
||||||
|
snapshot_web_metrics "$output_dir/ecto-query-metrics-before.json"
|
||||||
sample_resources &
|
sample_resources &
|
||||||
sampler_pid=$!
|
sampler_pid=$!
|
||||||
|
|
||||||
|
|
@ -419,6 +803,12 @@ set -e
|
||||||
cleanup_sampler
|
cleanup_sampler
|
||||||
sampler_pid=
|
sampler_pid=
|
||||||
summarize_resources
|
summarize_resources
|
||||||
|
summarize_database_connections
|
||||||
|
snapshot_web_metrics "$output_dir/ecto-query-metrics-after.json"
|
||||||
|
summarize_web_metrics \
|
||||||
|
"$output_dir/ecto-query-metrics-before.json" \
|
||||||
|
"$output_dir/ecto-query-metrics-after.json"
|
||||||
|
snapshot_statement_stats "$output_dir/statement-statistics.json"
|
||||||
|
|
||||||
snapshot_database "$output_dir/database-after-load.txt"
|
snapshot_database "$output_dir/database-after-load.txt"
|
||||||
snapshot_database_metrics "$output_dir/database-metrics-after-load.json"
|
snapshot_database_metrics "$output_dir/database-metrics-after-load.json"
|
||||||
|
|
|
||||||
|
|
@ -85,6 +85,11 @@ REHEARSAL_IMAGE=who-need-help:rehearsal-render \
|
||||||
-f compose.yaml -f compose.upgrade-rehearsal.yaml config --quiet
|
-f compose.yaml -f compose.upgrade-rehearsal.yaml config --quiet
|
||||||
docker compose --env-file .env.load.example \
|
docker compose --env-file .env.load.example \
|
||||||
-f compose.yaml -f compose.load.yaml config --quiet
|
-f compose.yaml -f compose.load.yaml config --quiet
|
||||||
|
test "$(
|
||||||
|
docker compose --env-file .env.load.example \
|
||||||
|
-f compose.yaml -f compose.load.yaml config --format json |
|
||||||
|
jq -r '.services.db.command | join(" ")'
|
||||||
|
)" = "postgres -c shared_preload_libraries=pg_stat_statements -c compute_query_id=on"
|
||||||
mkdir -p "$scan_dir/observability-runtime/prometheus" \
|
mkdir -p "$scan_dir/observability-runtime/prometheus" \
|
||||||
"$scan_dir/observability-runtime/grafana"
|
"$scan_dir/observability-runtime/grafana"
|
||||||
OBSERVABILITY_RUNTIME_DIR="$scan_dir/observability-runtime" \
|
OBSERVABILITY_RUNTIME_DIR="$scan_dir/observability-runtime" \
|
||||||
|
|
|
||||||
38
scripts/rotate-load-metrics-token.sh
Executable file
38
scripts/rotate-load-metrics-token.sh
Executable file
|
|
@ -0,0 +1,38 @@
|
||||||
|
#!/bin/sh
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
ROOT=$(CDPATH='' cd -- "$(dirname -- "$0")/.." && pwd)
|
||||||
|
ENV_FILE="$ROOT/.env.load"
|
||||||
|
|
||||||
|
if [ ! -f "$ENV_FILE" ]; then
|
||||||
|
echo "Missing $ENV_FILE. Run scripts/ensure-local-load-env.sh first." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
for command in openssl perl; do
|
||||||
|
if ! command -v "$command" >/dev/null 2>&1; then
|
||||||
|
echo "Required command is unavailable: $command" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
new_metrics_token=$(openssl rand -hex 32)
|
||||||
|
temporary=$(mktemp "${ENV_FILE}.rotate.XXXXXX")
|
||||||
|
trap 'rm -f "$temporary"' EXIT HUP INT TERM
|
||||||
|
chmod 600 "$temporary"
|
||||||
|
|
||||||
|
NEW_METRICS_TOKEN=$new_metrics_token \
|
||||||
|
perl -pe '
|
||||||
|
s/^METRICS_TOKEN=.*/METRICS_TOKEN=$ENV{NEW_METRICS_TOKEN}/;
|
||||||
|
' "$ENV_FILE" >"$temporary"
|
||||||
|
|
||||||
|
if cmp -s "$ENV_FILE" "$temporary"; then
|
||||||
|
echo "METRICS_TOKEN is missing from $ENV_FILE." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
mv "$temporary" "$ENV_FILE"
|
||||||
|
chmod 600 "$ENV_FILE"
|
||||||
|
trap - EXIT HUP INT TERM
|
||||||
|
|
||||||
|
echo "The isolated load-profile metrics token was rotated without printing it."
|
||||||
|
|
@ -83,6 +83,21 @@ defmodule WhoNeedHelp.MutualAidFlowTest do
|
||||||
Messaging.send_message(outsider_scope, assignment, %{"body" => "not allowed"})
|
Messaging.send_message(outsider_scope, assignment, %{"body" => "not allowed"})
|
||||||
end
|
end
|
||||||
|
|
||||||
|
test "request details do not preload the unbounded chat history", context do
|
||||||
|
{:ok, request} = Help.create_request(context.requester_scope, context.request_attrs)
|
||||||
|
{:ok, assignment} = Help.accept_request(context.helper_scope, request.id)
|
||||||
|
|
||||||
|
{:ok, _message} =
|
||||||
|
Messaging.send_message(context.helper_scope, assignment, %{"body" => "Bounded separately"})
|
||||||
|
|
||||||
|
loaded_request = Help.get_request!(request.id)
|
||||||
|
|
||||||
|
assert %Ecto.Association.NotLoaded{} = loaded_request.assignment.messages
|
||||||
|
|
||||||
|
assert [%{body: "Bounded separately"}] =
|
||||||
|
Messaging.list_messages(context.requester_scope, loaded_request.assignment)
|
||||||
|
end
|
||||||
|
|
||||||
test "request discovery uses stable cursor pages without gaps or duplicates", context do
|
test "request discovery uses stable cursor pages without gaps or duplicates", context do
|
||||||
requests =
|
requests =
|
||||||
for offset <- 1..5 do
|
for offset <- 1..5 do
|
||||||
|
|
|
||||||
|
|
@ -20,6 +20,7 @@ defmodule WhoNeedHelpWeb.MetricsControllerTest do
|
||||||
|
|
||||||
test "exports Prometheus metrics for the configured bearer token", %{conn: conn} do
|
test "exports Prometheus metrics for the configured bearer token", %{conn: conn} do
|
||||||
get(conn, "/healthz/live")
|
get(conn, "/healthz/live")
|
||||||
|
WhoNeedHelp.Repo.query!("SELECT 1")
|
||||||
|
|
||||||
conn =
|
conn =
|
||||||
conn
|
conn
|
||||||
|
|
@ -36,5 +37,8 @@ defmodule WhoNeedHelpWeb.MetricsControllerTest do
|
||||||
assert body =~ "# TYPE who_need_help_http_requests_total counter"
|
assert body =~ "# TYPE who_need_help_http_requests_total counter"
|
||||||
assert body =~ "who_need_help_http_requests_total "
|
assert body =~ "who_need_help_http_requests_total "
|
||||||
assert body =~ "who_need_help_http_request_duration_microseconds_total "
|
assert body =~ "who_need_help_http_request_duration_microseconds_total "
|
||||||
|
assert body =~ "who_need_help_database_query_execution_duration_microseconds_total "
|
||||||
|
assert body =~ "who_need_help_database_query_queue_duration_microseconds_total "
|
||||||
|
assert body =~ "who_need_help_database_query_decode_duration_microseconds_total "
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue
Block a user