Record production read-only load results

docs/performance.md

@@ -348,6 +348,74 @@ The k6 image is pinned to version 2.1.0 by digest and runs locally with anonymou
 usage reporting disabled. No Grafana Cloud account, API token, OpenAI API, or
 usage-based service is involved.
 
+## Observed production public read-only profile
+
+Observed on 2026-07-28 against `https://whoneedhelp.com`, after the runner
+verified `/srv/who_need_help-production`, Compose project
+`who_need_help_production`, and deployed commit
+`3d6f9c5004f9218c55a040bc31e72b56b22d4470`. The host exposed 2 CPUs and
+4,004,224 KiB total memory. The runner sent only the documented public GET
+allowlist and Phoenix transport heartbeats; it did not submit forms, log in,
+register, send email, open product LiveViews, or write application data.
+
+| Experiment input / observation | Result |
+| --- | ---: |
+| Baseline input | 1 HTTP VU + 1 WebSocket VU, 30 seconds |
+| Baseline HTTP checks / failures | 28 / 0 |
+| Baseline HTTP average / p95 | 91.64 / 107.75 ms |
+| Baseline WebSocket opens / heartbeat replies / errors | 6 / 6 / 0 |
+| Baseline app average / max CPU | 1.26% / 4.60% |
+| Five-client input | 5 HTTP VUs + 5 WebSocket VUs, 60 seconds |
+| Five-client HTTP checks / failures / observed rate | 1,055 / 0 / 17.27 requests/s |
+| Five-client HTTP average / p95 | 83.85 / 129.63 ms |
+| Five-client WebSocket opens / heartbeat replies / errors | 20 / 20 / 0 |
+| Five-client app average / max CPU | 7.37% / 10.77% |
+| HTTP-only input | 20 HTTP VUs, 30 seconds |
+| HTTP-only checks / failures / observed rate | 5,557 / 0 / 184.47 requests/s |
+| HTTP-only average / p95 / maximum | 57.18 / 79.25 / 673.45 ms |
+| HTTP-only app average / max CPU | 54.70% / 61.53% |
+| HTTP-only app first-to-last memory delta | +1 MiB |
+| WebSocket-only input | 20 sockets held for 60 seconds |
+| WebSocket opens / heartbeat replies / errors | 20 / 40 / 0 |
+| WebSocket connect average / p95 | 184.38 / 191.53 ms |
+| WebSocket-only app average / max CPU | 0.79% / 3.82% |
+| WebSocket-only app first-to-last memory delta | -0.6 MiB |
+
+Readiness returned `{"status":"ready"}` before and after every valid run. The
+application container remained healthy with zero restarts. The lowest observed
+host `MemAvailable` across the recorded valid runs was 1,241,432 KiB. The
+reported PostgreSQL RSS is explicitly the sum of every PostgreSQL process on
+the shared host, not an attribution to this application.
+
+The HTTP-only run recorded 5,562 application HTTP completions and 994 Ecto
+queries. Their cumulative database execution time increased by 388,598
+microseconds and pool queue time by 558,053 microseconds. The two sequential
+scans of the empty `rate_limit_buckets` table were the runner's own before/after
+count snapshots; the HTTP traffic added no rate-limit-table scan. Application
+table counts were unchanged.
+
+This is not a product-query capacity result. Production contained no help
+requests or activities during the observation, and `/requests` and
+`/activities` require an authenticated session. PostgreSQL also did not have
+`pg_stat_statements` installed. Viewport clustering, authenticated LiveView
+joins, chat, tracking, handover, and write throughput therefore remain covered
+by the isolated profiles rather than this public production probe.
+
+The deployed environment had `RATE_LIMIT_POLICIES_JSON={}`. In that
+configuration `RateLimiter.check/2` returns `:not_configured` without a database
+query, so this observation neither exercises nor validates authentication
+throttling. It also provides no evidence that Redis is needed. The current
+candidate keeps cross-replica counters in PostgreSQL and requires explicit
+email- and IP-scoped policies before public authentication traffic is
+protected.
+
+Ignored evidence:
+
+- `output/performance/production-baseline-20260728/`
+- `output/performance/production-tier2-20260728/`
+- `output/performance/production-http-db-stats-20260728/`
+- `output/performance/production-ws-heartbeat-final-20260728/`
+
 ## Observed local comparison
 
 Observed on 2026-07-18 with Docker Engine 29.6.2 on the recorded 32-CPU,

load/k6/production-readonly.js

@@ -61,6 +61,10 @@ const duration = required("DURATION");
 const httpThinkSeconds = nonNegativeNumber("HTTP_THINK_SECONDS");
 const websocketHoldMs = positiveInteger("WS_HOLD_MS");
 const websocketConnectTimeoutMs = positiveInteger("WS_CONNECT_TIMEOUT_MS");
+// Phoenix 1.8's browser client sends a heartbeat every 30 seconds. Keep the
+// production probe aligned with that behavior so a long-lived socket is not
+// mistaken for an idle client and closed by the server.
+const websocketHeartbeatIntervalMs = 30000;
 const websocketUrl =
   baseUrl.replace(/^http/, "ws") + "/live/websocket?vsn=2.0.0";
 
@@ -140,9 +144,34 @@ export function phoenixWebsocket() {
       "User-Agent": "WhoNeedHelp-production-readonly-load/1",
     },
   });
-  const ref = `${__VU}-${__ITER}`;
   let opened = false;
   let closeTimer;
+  let heartbeatTimer;
+  let heartbeatSequence = 0;
+  let pendingHeartbeatRef;
+
+  const sendHeartbeat = () => {
+    pendingHeartbeatRef = `${__VU}-${__ITER}-${heartbeatSequence}`;
+    heartbeatSequence += 1;
+    socket.send(
+      JSON.stringify([null, pendingHeartbeatRef, "phoenix", "heartbeat", {}]),
+    );
+  };
+
+  const scheduleHeartbeat = () => {
+    const nextHeartbeatAtMs =
+      heartbeatSequence * websocketHeartbeatIntervalMs;
+
+    if (nextHeartbeatAtMs >= websocketHoldMs) {
+      return;
+    }
+
+    heartbeatTimer = setTimeout(() => {
+      sendHeartbeat();
+      scheduleHeartbeat();
+    }, websocketHeartbeatIntervalMs);
+  };
+
   const connectTimer = setTimeout(() => {
     if (!opened) {
       websocketErrors.add(1);
@@ -154,7 +183,8 @@ export function phoenixWebsocket() {
     opened = true;
     clearTimeout(connectTimer);
     websocketOpened.add(1);
-    socket.send(JSON.stringify([null, ref, "phoenix", "heartbeat", {}]));
+    sendHeartbeat();
+    scheduleHeartbeat();
     closeTimer = setTimeout(() => socket.close(), websocketHoldMs);
   });
 
@@ -169,7 +199,7 @@ export function phoenixWebsocket() {
 
     if (
       Array.isArray(frame) &&
-      frame[1] === ref &&
+      frame[1] === pendingHeartbeatRef &&
       frame[2] === "phoenix" &&
       frame[3] === "phx_reply" &&
       frame[4]?.status === "ok"
@@ -190,5 +220,9 @@ export function phoenixWebsocket() {
     if (closeTimer !== undefined) {
       clearTimeout(closeTimer);
     }
+
+    if (heartbeatTimer !== undefined) {
+      clearTimeout(heartbeatTimer);
+    }
   });
 }

scripts/production-readonly-load.sh

@@ -263,10 +263,34 @@ docker exec "$container" /app/bin/who_need_help rpc '
         (SELECT count(*) FROM users_tokens),
         (SELECT count(*) FROM rate_limit_buckets)
       """).rows
+      table_stats = Repo.query!("""
+      SELECT relname, n_live_tup, n_dead_tup, seq_scan, idx_scan
+      FROM pg_stat_user_tables
+      WHERE relname = ANY($1)
+      ORDER BY relname
+      """, [[
+        "activities",
+        "activity_participants",
+        "assignments",
+        "blocks",
+        "categories",
+        "help_requests",
+        "rate_limit_buckets",
+        "users",
+        "users_tokens"
+      ]]).rows
       IO.puts(Jason.encode!(%{
         captured_at: DateTime.utc_now(),
         pg_stat_database: database,
         counts: counts,
+        table_stats: table_stats,
+        table_stats_columns: [
+          "relname",
+          "n_live_tup",
+          "n_dead_tup",
+          "seq_scan",
+          "idx_scan"
+        ],
         tables: ["users", "users_tokens", "rate_limit_buckets"]
       }))
     '
@@ -403,7 +427,7 @@ jq -e \
     (
       (metric("wnh_readonly_websocket_errors").count // 0) == 0 and
       metric("wnh_readonly_websocket_opened").count > 0 and
-      metric("wnh_readonly_websocket_opened").count ==
+      metric("wnh_readonly_websocket_opened").count <=
         metric("wnh_readonly_heartbeat_replies").count
     )
   )