simpletest/who_need_help
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test: measure database load and pool pressure

Browse Source
This commit is contained in:
SimpleTest 2026-07-20 04:36:20 +03:00
parent fdaef2d685
commit eacfdb2451
10 changed files with 583 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

16
README.md
View File

@ -231,6 +231,22 @@ The same external-service protocol drill used by CI can be run independently:
./scripts/external-boundaries-run.sh local-boundaries
```
Run the isolated HTTP/WebSocket/authenticated chat/tracking load profile with
resource, PostgreSQL statement, database-connection, and Ecto pool-wait
measurements:
```bash
./scripts/load-stack-up.sh
./scripts/load-run.sh local-load
./scripts/load-stack-stop.sh
```
The profile has its own generated mode-`0600` environment, Compose project,
and PostgreSQL volume. It does not use or mutate the ordinary public Compose
database. Exact inputs and threshold-free evidence are retained below
`output/performance/local-load/`; see
[Performance measurement](docs/performance.md) for scope and interpretation.
The cursor-pagination database benchmark also creates a one-run Compose
project, random database credentials, and a separate PostgreSQL volume:

11
compose.load.yaml
View File

@ -1,4 +1,15 @@
services:
db:
# Query-level statistics are enabled only for the isolated load profile.
# PostgreSQL requires pg_stat_statements in shared_preload_libraries and a
# server restart; the ordinary Compose database is not changed.
command:
- postgres
- -c
- shared_preload_libraries=pg_stat_statements
- -c
- compute_query_id=on
proxy:
networks:
edge:

78
docs/performance.md
View File

@ -97,11 +97,24 @@ Each run writes ignored evidence below `output/performance/<label>/`:
stack container, plus a threshold-free CPU/RAM/PID JSON summary;
- read-only PostgreSQL database size and `pg_stat_database` snapshots before
and after k6, plus their machine-readable counter deltas;
- `pg_stat_statements` rankings by total, mean, and maximum execution time,
including calls, rows, buffer activity, temporary blocks, and WAL activity;
- periodic `pg_stat_activity` connection samples with configured pool slots,
observed connection headroom, transaction state, lock waits, and named
active wait events;
- per-web Ecto telemetry deltas separating database execution, pool queue, and
result decode time;
- before/after application table counts, exact authenticated-write evidence,
fixture cleanup output, and a cleanup diff;
- final Compose state, readiness response, application logs, and cross-node
PubSub result.
The load-only PostgreSQL command preloads `pg_stat_statements` and creates the
extension only in the isolated load database. The application exposes
cumulative Ecto measurements through its authenticated metrics endpoint; the
runner snapshots every web replica before and after k6 and records the delta.
Neither mechanism changes the ordinary public Compose database.
Before k6 starts, the production-compiled fixture task verifies the exact
configured load database name and a deliberate confirmation marker. It creates
isolated requester/helper/request/assignment pairs through the application
@ -217,6 +230,71 @@ Ignored evidence:
- `output/performance/final-load-20260719a/`
## Observed SQL and connection-pool run
Observed on 2026-07-20 with 3 web replicas, 2 worker replicas, PostgreSQL 18.4,
40 paced public HTTP VUs, 40 Phoenix heartbeat VUs, and 8 authenticated
mutual-aid VUs for 30 seconds.
| Observation | Result |
| --- | ---: |
| HTTP requests / failures | 38,429 / 0 |
| Public checks | 35,163 / 35,163 passed |
| HTTP duration average / p95 / maximum | 2.42 / 7.74 / 187.11 ms |
| Authenticated chat/tracking chains | 1,664 |
| Phoenix heartbeat opened / replies | 240 / 240 |
| Configured Ecto Repo pool slots | 50 |
| Peak PostgreSQL client backends | 55 |
| Minimum observed non-reserved connection headroom | 42 |
| Idle-in-transaction / lock-waiting backends | 0 / 0 |
| Ecto average execution / queue / decode time | 308.58 / 31.15 / 0.27 µs |
| PostgreSQL rollback / deadlock / conflict / temp files | 0 / 0 / 0 / 0 |
The connection sampler observed active PostgreSQL waits named
`Client:ClientRead` and `IO:WalSync`; it observed no lock wait or
idle-in-transaction backend. The Ecto queue
measurement is the application-side evidence for time waiting to check out a
Repo connection. It averaged 31.15 microseconds over 166,459 queries across
the three web replicas. This short run did not produce a connection timeout or
establish a maximum connection capacity.
The SQL ranking exposed an unbounded preload of every message attached to a
request even though the LiveView immediately loaded a separate 50-message
cursor page. In the before run that redundant statement executed 3,318 times,
returned 342,378 rows in aggregate, and consumed 392.378 ms of PostgreSQL
execution time. The request preload was removed and a regression test now
requires the chat association to remain unloaded. In the repeat run the
unbounded statement was absent; the bounded cursor statement retained its
`LIMIT`, executed 3,328 times, returned 148,512 rows, and averaged 0.122 ms.
No normalized statement in the repeat run used temporary blocks. The largest
mean statement time was 0.481 ms for six session-token inserts; the
highest-call request lookup averaged 0.031 ms over 39,936 calls. These are
observations against the small generated load dataset, not proof that every
query remains fast at arbitrary data size. The separate 50,000-row database
profile below checks the indexed list and cursor queries at a larger
cardinality.
Across the seven measured application/database/proxy containers, simultaneous
memory samples totalled 1,501.60–1,509.86 MiB during this run. This excludes
the host OS, the k6 generator, Mailpit, and optional observability services.
Therefore it is not a claim that this 3-web/2-worker topology fits a 1 GiB or
2 GiB server. It does show that the measured workload used about 1.47 GiB for
those seven containers rather than the former kind port-table failure's
roughly 10 GiB.
The run created exactly 1,664 messages, tracking sessions, and tracking
samples, plus 8 session tokens; explicit tracking stop left zero current
positions. Cleanup restored every tracked application-table count, readiness
and the five-node PubSub probe passed, and the captured application log had no
database timeout, Postgrex error, deadlock, `ERROR`, `FATAL`, or `PANIC`
match.
Ignored before/after evidence:
- `output/performance/pool-sql-final-20260720/`
- `output/performance/pool-sql-chat-fixed-canonical-20260720/`
## Observed 10-minute authenticated soak
Observed on 2026-07-19 with the same 3-web/2-worker topology and the same

2
lib/who_need_help/help.ex
View File

@ -110,7 +110,7 @@ defmodule WhoNeedHelp.Help do
|> Repo.preload([
:category,
requester: :social_identities,
assignment: [helper: :social_identities, messages: :sender]
assignment: [helper: :social_identities]
])
end

25
lib/who_need_help_web/telemetry.ex
View File

@ -60,6 +60,31 @@ defmodule WhoNeedHelpWeb.Telemetry do
end,
description: "Cumulative database query duration"
),
sum("who_need_help.database.query.execution.duration.microseconds.total",
event_name: [:who_need_help, :repo, :query],
measurement: fn measurements ->
System.convert_time_unit(measurements.query_time, :native, :microsecond)
end,
description: "Cumulative database execution duration"
),
sum("who_need_help.database.query.queue.duration.microseconds.total",
event_name: [:who_need_help, :repo, :query],
measurement: fn measurements ->
measurements
|> Map.get(:queue_time, 0)
|> System.convert_time_unit(:native, :microsecond)
end,
description: "Cumulative time waiting for a database connection"
),
sum("who_need_help.database.query.decode.duration.microseconds.total",
event_name: [:who_need_help, :repo, :query],
measurement: fn measurements ->
measurements
|> Map.get(:decode_time, 0)
|> System.convert_time_unit(:native, :microsecond)
end,
description: "Cumulative database result decoding duration"
),
counter("who_need_help.websocket.connections.total",
event_name: [:phoenix, :socket_connected],
measurement: :duration,

390
scripts/load-run.sh
View File

@ -122,8 +122,123 @@ chmod 700 "$ROOT/output" "$ROOT/output/performance" "$output_dir"
run_started_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)
running_marker="$output_dir/.sampling"
resource_log="$output_dir/docker-stats.jsonl"
connection_log="$output_dir/database-connections.jsonl"
touch "$running_marker"
database_psql() {
# The variables below are intentionally expanded inside the database container.
# shellcheck disable=SC2016
"${compose[@]}" exec -T db sh -c \
'psql --no-psqlrc --quiet --tuples-only --no-align --set ON_ERROR_STOP=1 \
--username "$POSTGRES_USER" --dbname "$POSTGRES_DB"'
}
prepare_statement_stats() {
local preloaded
preloaded=$(database_psql <<'SQL'
SELECT current_setting('shared_preload_libraries')
~ '(^|,)pg_stat_statements(,|$)';
SQL
)
if [[ "$preloaded" != "t" ]]; then
echo "pg_stat_statements is not present in shared_preload_libraries." >&2
exit 1
fi
database_psql >"$output_dir/statement-statistics-setup.txt" <<'SQL'
SELECT current_setting('shared_preload_libraries');
CREATE EXTENSION IF NOT EXISTS pg_stat_statements;
SQL
}
reset_statement_stats() {
database_psql >"$output_dir/statement-statistics-reset.txt" <<'SQL'
SELECT pg_stat_statements_reset();
SQL
}
snapshot_statement_stats() {
local destination=$1
database_psql >"$destination" <<'SQL'
WITH statements AS (
SELECT
queryid::text AS query_id,
regexp_replace(query, '[[:space:]]+', ' ', 'g') AS query,
calls,
rows,
round(total_exec_time::numeric, 3) AS total_exec_time_ms,
round(mean_exec_time::numeric, 3) AS mean_exec_time_ms,
round(max_exec_time::numeric, 3) AS max_exec_time_ms,
shared_blks_hit,
shared_blks_read,
shared_blks_dirtied,
shared_blks_written,
temp_blks_read,
temp_blks_written,
wal_records,
wal_fpi,
wal_bytes
FROM pg_stat_statements
WHERE dbid = (SELECT oid FROM pg_database WHERE datname = current_database())
AND query NOT LIKE '%pg_stat_statements%'
AND query NOT LIKE '%pg_stat_activity%'
AND query NOT LIKE '%pg_stat_database%'
),
top_total AS (
SELECT *
FROM statements
ORDER BY total_exec_time_ms DESC, query_id
LIMIT 25
),
top_mean AS (
SELECT *
FROM statements
ORDER BY mean_exec_time_ms DESC, query_id
LIMIT 25
),
top_max AS (
SELECT *
FROM statements
ORDER BY max_exec_time_ms DESC, query_id
LIMIT 25
)
SELECT jsonb_pretty(
jsonb_build_object(
'captured_at', clock_timestamp(),
'measurement', 'pg_stat_statements since the pre-load reset',
'statement_count', (SELECT count(*) FROM statements),
'statistics_info', (SELECT to_jsonb(info) FROM pg_stat_statements_info AS info),
'top_by_total_execution_time',
(SELECT coalesce(
jsonb_agg(
to_jsonb(item)
ORDER BY total_exec_time_ms DESC, query_id
),
'[]'::jsonb
) FROM top_total AS item),
'top_by_mean_execution_time',
(SELECT coalesce(
jsonb_agg(
to_jsonb(item)
ORDER BY mean_exec_time_ms DESC, query_id
),
'[]'::jsonb
) FROM top_mean AS item),
'top_by_max_execution_time',
(SELECT coalesce(
jsonb_agg(
to_jsonb(item)
ORDER BY max_exec_time_ms DESC, query_id
),
'[]'::jsonb
) FROM top_max AS item)
)
);
SQL
}
snapshot_database() {
local destination=$1
@ -246,6 +361,271 @@ summarize_resources() {
/output/resource-summary.json
}
sample_database_connections() {
database_psql >>"$connection_log" <<'SQL'
WITH activity AS (
SELECT *
FROM pg_stat_activity
WHERE backend_type = 'client backend'
AND pid <> pg_backend_pid()
),
active_wait_events AS (
SELECT
concat_ws(':', wait_event_type, wait_event) AS event,
count(*) AS backends
FROM activity
WHERE state = 'active'
AND wait_event IS NOT NULL
GROUP BY wait_event_type, wait_event
)
SELECT json_build_object(
'captured_at', clock_timestamp(),
'max_connections', current_setting('max_connections')::integer,
'reserved_connections',
current_setting('reserved_connections')::integer,
'superuser_reserved_connections',
current_setting('superuser_reserved_connections')::integer,
'client_backends',
count(*),
'active_backends',
count(*) FILTER (WHERE state = 'active'),
'idle_backends',
count(*) FILTER (WHERE state = 'idle'),
'idle_in_transaction_backends',
count(*) FILTER (WHERE state = 'idle in transaction'),
'active_waiting_backends',
count(*) FILTER (
WHERE state = 'active'
AND wait_event IS NOT NULL
),
'lock_waiting_backends',
count(*) FILTER (WHERE wait_event_type = 'Lock'),
'active_wait_events',
(
SELECT coalesce(json_object_agg(event, backends), '{}'::json)
FROM active_wait_events
),
'longest_idle_in_transaction_ms',
coalesce(
max(
extract(epoch FROM clock_timestamp() - xact_start) * 1000
) FILTER (
WHERE state = 'idle in transaction'
),
0
),
'longest_active_query_ms',
coalesce(
max(
extract(epoch FROM clock_timestamp() - query_start) * 1000
) FILTER (
WHERE state = 'active'
),
0
)
)
FROM activity;
SQL
}
summarize_database_connections() {
jq -s \
--argjson pool_size "$POOL_SIZE" \
--argjson app_replicas "$((LOAD_WEB_REPLICAS + LOAD_WORKER_REPLICAS))" '
. as $samples |
{
schema_version: 1,
measurement: "read-only pg_stat_activity samples during the k6 run",
thresholds_applied: false,
sample_count: length,
configured_app_replicas: $app_replicas,
configured_pool_size_per_replica: $pool_size,
configured_repo_pool_slots: ($app_replicas * $pool_size),
max_connections: (map(.max_connections) | max),
reserved_connections: (map(.reserved_connections) | max),
superuser_reserved_connections:
(map(.superuser_reserved_connections) | max),
peak_client_backends: (map(.client_backends) | max),
peak_active_backends: (map(.active_backends) | max),
peak_idle_backends: (map(.idle_backends) | max),
peak_idle_in_transaction_backends:
(map(.idle_in_transaction_backends) | max),
peak_active_waiting_backends:
(map(.active_waiting_backends) | max),
peak_lock_waiting_backends: (map(.lock_waiting_backends) | max),
observed_active_wait_events:
([
$samples[]
| .active_wait_events
| to_entries[]
]
| group_by(.key)
| map({
event: .[0].key,
peak_backends: (map(.value) | max),
observed_samples: length
})),
longest_observed_idle_in_transaction_ms:
(map(.longest_idle_in_transaction_ms | tonumber) | max),
longest_observed_active_query_ms:
(map(.longest_active_query_ms | tonumber) | max),
minimum_observed_connection_headroom:
((map(
.max_connections -
.reserved_connections -
.superuser_reserved_connections
) | min) -
(map(.client_backends) | max))
}
' "$connection_log" >"$output_dir/database-connections-summary.json"
}
prometheus_value() {
local metric=$1
awk -v metric="$metric" '$1 == metric { print $2; found = 1 }
END { if (!found) print 0 }'
}
snapshot_web_metrics() {
local destination=$1
local temporary
temporary=$(mktemp "${TMPDIR:-/tmp}/wnh-load-metrics.XXXXXX")
: >"$temporary"
for container in "${web_containers[@]}"; do
local scrape node queries total execution queue decode
scrape=$(
docker exec "$container" sh -c \
'curl --fail --silent --show-error \
--header "Authorization: Bearer $METRICS_TOKEN" \
--header "X-Forwarded-Proto: https" \
--header "Host: $PHX_HOST" \
http://127.0.0.1:4000/metrics'
)
node=$(
docker exec "$container" /app/bin/who_need_help rpc \
'IO.puts(JSON.encode!(%{node: to_string(node())}))' |
tail -n 1 |
jq -r '.node'
)
queries=$(
printf '%s\n' "$scrape" |
prometheus_value who_need_help_database_queries_total
)
total=$(
printf '%s\n' "$scrape" |
prometheus_value who_need_help_database_query_duration_microseconds_total
)
execution=$(
printf '%s\n' "$scrape" |
prometheus_value \
who_need_help_database_query_execution_duration_microseconds_total
)
queue=$(
printf '%s\n' "$scrape" |
prometheus_value \
who_need_help_database_query_queue_duration_microseconds_total
)
decode=$(
printf '%s\n' "$scrape" |
prometheus_value \
who_need_help_database_query_decode_duration_microseconds_total
)
jq -cn \
--arg container "$container" \
--arg node "$node" \
--argjson queries "$queries" \
--argjson total_duration_us "$total" \
--argjson execution_duration_us "$execution" \
--argjson queue_duration_us "$queue" \
--argjson decode_duration_us "$decode" \
'{
container: $container,
node: $node,
queries: $queries,
total_duration_us: $total_duration_us,
execution_duration_us: $execution_duration_us,
queue_duration_us: $queue_duration_us,
decode_duration_us: $decode_duration_us
}' >>"$temporary"
done
jq -s 'sort_by(.node)' "$temporary" >"$destination"
rm -f "$temporary"
}
summarize_web_metrics() {
local before=$1
local after=$2
jq -n \
--slurpfile before "$before" \
--slurpfile after "$after" '
[
$after[0][] as $after_row
| ($before[0][] | select(.node == $after_row.node)) as $before_row
| {
node: $after_row.node,
queries: ($after_row.queries - $before_row.queries),
total_duration_us:
($after_row.total_duration_us - $before_row.total_duration_us),
execution_duration_us:
($after_row.execution_duration_us -
$before_row.execution_duration_us),
queue_duration_us:
($after_row.queue_duration_us - $before_row.queue_duration_us),
decode_duration_us:
($after_row.decode_duration_us - $before_row.decode_duration_us)
}
| . + {
average_total_duration_us:
(if .queries > 0 then .total_duration_us / .queries else 0 end),
average_execution_duration_us:
(if .queries > 0 then .execution_duration_us / .queries else 0 end),
average_queue_duration_us:
(if .queries > 0 then .queue_duration_us / .queries else 0 end),
average_decode_duration_us:
(if .queries > 0 then .decode_duration_us / .queries else 0 end)
}
] as $nodes
| {
schema_version: 1,
measurement:
"per-web Ecto telemetry counter deltas around the k6 run",
thresholds_applied: false,
nodes: $nodes,
cluster: (
reduce $nodes[] as $node
({
queries: 0,
total_duration_us: 0,
execution_duration_us: 0,
queue_duration_us: 0,
decode_duration_us: 0
};
.queries += $node.queries |
.total_duration_us += $node.total_duration_us |
.execution_duration_us += $node.execution_duration_us |
.queue_duration_us += $node.queue_duration_us |
.decode_duration_us += $node.decode_duration_us)
| . + {
average_total_duration_us:
(if .queries > 0 then .total_duration_us / .queries else 0 end),
average_execution_duration_us:
(if .queries > 0
then .execution_duration_us / .queries else 0 end),
average_queue_duration_us:
(if .queries > 0 then .queue_duration_us / .queries else 0 end),
average_decode_duration_us:
(if .queries > 0 then .decode_duration_us / .queries else 0 end)
}
)
}
' >"$output_dir/ecto-query-metrics-summary.json"
}
run_fixture_tool() {
local action=$1
@ -317,6 +697,7 @@ sample_resources() {
docker stats --no-stream --format '{{json .}}' "${measured_containers[@]}" |
jq -c --arg observed_at "$observed_at" '. + {ObservedAt: $observed_at}' \
>>"$resource_log"
sample_database_connections
sleep 1
done
}
@ -378,11 +759,14 @@ trap cleanup_on_exit EXIT HUP INT TERM
uname -a
} >"$output_dir/environment.txt"
prepare_statement_stats
snapshot_database "$output_dir/database-pre-fixtures.txt"
run_fixture_tool prepare >"$output_dir/fixture-prepare.log" 2>&1
fixtures_prepared=true
snapshot_database "$output_dir/database-before-load.txt"
snapshot_database_metrics "$output_dir/database-metrics-before-load.json"
reset_statement_stats
snapshot_web_metrics "$output_dir/ecto-query-metrics-before.json"
sample_resources &
sampler_pid=$!
@ -419,6 +803,12 @@ set -e
cleanup_sampler
sampler_pid=
summarize_resources
summarize_database_connections
snapshot_web_metrics "$output_dir/ecto-query-metrics-after.json"
summarize_web_metrics \
"$output_dir/ecto-query-metrics-before.json" \
"$output_dir/ecto-query-metrics-after.json"
snapshot_statement_stats "$output_dir/statement-statistics.json"
snapshot_database "$output_dir/database-after-load.txt"
snapshot_database_metrics "$output_dir/database-metrics-after-load.json"

5
scripts/quality.sh
View File

@ -85,6 +85,11 @@ REHEARSAL_IMAGE=who-need-help:rehearsal-render \
-f compose.yaml -f compose.upgrade-rehearsal.yaml config --quiet
docker compose --env-file .env.load.example \
-f compose.yaml -f compose.load.yaml config --quiet
test "$(
docker compose --env-file .env.load.example \
-f compose.yaml -f compose.load.yaml config --format json |
jq -r '.services.db.command | join(" ")'
)" = "postgres -c shared_preload_libraries=pg_stat_statements -c compute_query_id=on"
mkdir -p "$scan_dir/observability-runtime/prometheus" \
"$scan_dir/observability-runtime/grafana"
OBSERVABILITY_RUNTIME_DIR="$scan_dir/observability-runtime" \

38
scripts/rotate-load-metrics-token.sh Executable file
View File

@ -0,0 +1,38 @@
#!/bin/sh
set -eu
ROOT=$(CDPATH='' cd -- "$(dirname -- "$0")/.." && pwd)
ENV_FILE="$ROOT/.env.load"
if [ ! -f "$ENV_FILE" ]; then
echo "Missing $ENV_FILE. Run scripts/ensure-local-load-env.sh first." >&2
exit 1
fi
for command in openssl perl; do
if ! command -v "$command" >/dev/null 2>&1; then
echo "Required command is unavailable: $command" >&2
exit 1
fi
done
new_metrics_token=$(openssl rand -hex 32)
temporary=$(mktemp "${ENV_FILE}.rotate.XXXXXX")
trap 'rm -f "$temporary"' EXIT HUP INT TERM
chmod 600 "$temporary"
NEW_METRICS_TOKEN=$new_metrics_token \
perl -pe '
s/^METRICS_TOKEN=.*/METRICS_TOKEN=$ENV{NEW_METRICS_TOKEN}/;
' "$ENV_FILE" >"$temporary"
if cmp -s "$ENV_FILE" "$temporary"; then
echo "METRICS_TOKEN is missing from $ENV_FILE." >&2
exit 1
fi
mv "$temporary" "$ENV_FILE"
chmod 600 "$ENV_FILE"
trap - EXIT HUP INT TERM
echo "The isolated load-profile metrics token was rotated without printing it."

15
test/who_need_help/mutual_aid_flow_test.exs
View File

@ -83,6 +83,21 @@ defmodule WhoNeedHelp.MutualAidFlowTest do
Messaging.send_message(outsider_scope, assignment, %{"body" => "not allowed"})
end
test "request details do not preload the unbounded chat history", context do
{:ok, request} = Help.create_request(context.requester_scope, context.request_attrs)
{:ok, assignment} = Help.accept_request(context.helper_scope, request.id)
{:ok, _message} =
Messaging.send_message(context.helper_scope, assignment, %{"body" => "Bounded separately"})
loaded_request = Help.get_request!(request.id)
assert %Ecto.Association.NotLoaded{} = loaded_request.assignment.messages
assert [%{body: "Bounded separately"}] =
Messaging.list_messages(context.requester_scope, loaded_request.assignment)
end
test "request discovery uses stable cursor pages without gaps or duplicates", context do
requests =
for offset <- 1..5 do

4
test/who_need_help_web/controllers/metrics_controller_test.exs
View File

@ -20,6 +20,7 @@ defmodule WhoNeedHelpWeb.MetricsControllerTest do
test "exports Prometheus metrics for the configured bearer token", %{conn: conn} do
get(conn, "/healthz/live")
WhoNeedHelp.Repo.query!("SELECT 1")
conn =
conn
@ -36,5 +37,8 @@ defmodule WhoNeedHelpWeb.MetricsControllerTest do
assert body =~ "# TYPE who_need_help_http_requests_total counter"
assert body =~ "who_need_help_http_requests_total "
assert body =~ "who_need_help_http_request_duration_microseconds_total "
assert body =~ "who_need_help_database_query_execution_duration_microseconds_total "
assert body =~ "who_need_help_database_query_queue_duration_microseconds_total "
assert body =~ "who_need_help_database_query_decode_duration_microseconds_total "
end
end