fix: make kind rollout verification race-free
This commit is contained in:
parent
b96d443a96
commit
a25a56142b
|
|
@ -46,11 +46,12 @@ The goal remains open while any row lacks reproducible local evidence.
|
|||
- A forced LiveSocket transport loss while Chromium is offline exposes the
|
||||
reconnect alert, and restoring the network clears it and reconnects the
|
||||
authenticated LiveView.
|
||||
- The isolated API 30/34/37 Android matrix passes six device tests per API:
|
||||
- The isolated API 30/34/37 Android matrix passes seven device tests per API:
|
||||
missing and granted location permission, same-origin deep-link routing across
|
||||
Activity recreation, native foreground location upload, notification Stop
|
||||
with remote cleanup, a disconnected Stop request with visible retry state,
|
||||
and continued uploads after Home and Activity destruction. A separate probe
|
||||
continued uploads after Home and Activity destruction, and a forced
|
||||
main-frame disconnect followed by native Retry and a rendered DOM. A separate probe
|
||||
externally kills the instrumentation process and verifies that the
|
||||
`START_NOT_STICKY` service and notification do not return. The suite uses an
|
||||
in-process loopback fixture, runs each emulator without external networking,
|
||||
|
|
@ -117,9 +118,9 @@ The goal remains open while any row lacks reproducible local evidence.
|
|||
retained no credentials, published no host ports, and removed its exact
|
||||
project, volume, network, and one-run images. FCM/APNs device registration is
|
||||
not claimed.
|
||||
- The final regression repeated the 167-test quality/security gate, isolated
|
||||
- The final regression repeated the 174-test quality/security gate, isolated
|
||||
browser 1/1 bootstrap plus 8/8 Chromium specs, Android debug/staging builds
|
||||
and 5/5 API 37 device tests, the 50,000-row database benchmark, authenticated
|
||||
and 7/7 device tests on API 30, 34, and 37, the 50,000-row database benchmark, authenticated
|
||||
HTTP/LiveView/WebSocket/tracking load, Compose crash/replacement and kind
|
||||
rolling drills, alert firing/resolution, encrypted backup/restore failure
|
||||
paths, and the external protocol boundary. Scoped fixtures and one-run
|
||||
|
|
|
|||
|
|
@ -361,3 +361,30 @@ BEAM; the leaderboard itself uses composite keyset pagination.
|
|||
Ignored evidence for the recorded run:
|
||||
|
||||
- `output/db-scale/20260719054559-2339191/`
|
||||
|
||||
## Current final replay
|
||||
|
||||
The same threshold-free profiles were replayed on 2026-07-20 against
|
||||
application commit `b96d443`.
|
||||
|
||||
- The 30-second 3-web/2-worker load run used 40 public HTTP, 40 heartbeat
|
||||
WebSocket, and 8 authenticated VUs. It completed 13,765 iterations and 38,826
|
||||
HTTP requests; all 35,280 checks passed and HTTP failures were 0. The
|
||||
authenticated scenario completed 1,765 page/LiveView/chat/tracking
|
||||
start-update-stop chains, and the heartbeat scenario opened and answered 240
|
||||
sockets. PostgreSQL recorded no rollback, deadlock, conflict, or temporary
|
||||
file. The run-specific fixtures were removed and the cleanup count diff was
|
||||
empty. Evidence is `output/performance/final-load-20260720`.
|
||||
- The database benchmark again used 50,000 configured rows per large table on
|
||||
PostgreSQL 18.4. After migration, the asserted list queries selected their
|
||||
cursor indexes; for example, observed execution time changed from 8.139 ms to
|
||||
0.127 ms for urgent-help discovery and from 8.035 ms to 0.050 ms for
|
||||
Activity discovery. These are single local observations, not thresholds or
|
||||
portable performance claims. Evidence is
|
||||
`output/db-scale/20260719233526-97476`.
|
||||
- The corrected kind rolling run replaced all four application pod UIDs,
|
||||
retained four Ready zero-restart replacements, passed the four-node BEAM and
|
||||
PubSub probes, and produced an empty database diff. All 395 readiness samples
|
||||
ultimately returned HTTP 200; two required one retry during local NodePort
|
||||
endpoint replacement. Evidence is
|
||||
`output/resilience/final-kind-rollout-fixed-20260720`.
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# Who Need Help — implementation verification
|
||||
|
||||
Observed through 2026-07-19 in the local workspace. This report separates observed
|
||||
Observed through 2026-07-20 in the local workspace. This report separates observed
|
||||
results from product limits and unknown production properties.
|
||||
|
||||
## Verified MVP capabilities
|
||||
|
|
@ -700,6 +700,74 @@ unrelated application records:
|
|||
Evidence is
|
||||
`output/regression/full-public-web-quality-20260719.log`.
|
||||
|
||||
## Final Android and operational replay
|
||||
|
||||
Application commit `b96d443` and its exact source state were subjected to the
|
||||
following additional local and temporary-origin checks:
|
||||
|
||||
- `./scripts/quality.sh` passed all configured source, Compose, Helm,
|
||||
observability, dependency, release-image, and security gates with 174/174
|
||||
ExUnit tests. Evidence is
|
||||
`output/regression/android-cross-quality-20260720.log`.
|
||||
- Android debug and staging unit/lint/APK builds passed. The isolated API
|
||||
30/34/37 matrix passed 7/7 device tests per API. Public API 37 DOM smoke
|
||||
passed 1/1. The Android/browser staging run passed Android magic-link login,
|
||||
private chat in both directions, foreground tracking, browser marker
|
||||
appearance/removal, fixture verification, and an empty 19-table cleanup
|
||||
diff. Evidence is
|
||||
`output/android-browser-staging-e2e/20260719232413-3987628`.
|
||||
- The external-boundary drill passed OAuth, SMTP, provider-neutral HTTP push,
|
||||
product job integration, retry, rejection, timeout, replay, and
|
||||
deduplication paths with two isolated workers. Its one-run project, volume,
|
||||
images, network, and generated secrets were removed. Evidence is
|
||||
`output/external-boundaries/final-boundaries-20260720`.
|
||||
- The isolated 3-web/2-worker Compose failure drill crashed one web and one
|
||||
worker BEAM process, sequentially replaced every replica, rejoined all five
|
||||
nodes, passed PubSub, and completed the injected Oban job on attempt 2 after
|
||||
one recorded failure. All 725 readiness samples succeeded and the exact job
|
||||
was deleted. Evidence is
|
||||
`output/resilience/final-resilience-20260720`.
|
||||
- The 30-second, 88-VU load replay completed 13,765 iterations and 38,826 HTTP
|
||||
requests. All 35,280 checks passed; HTTP failures were 0. It completed 1,765
|
||||
authenticated LiveView page/chat/tracking start-update-stop chains and 240
|
||||
heartbeat sockets without a functional error. PostgreSQL observed zero
|
||||
rollbacks, deadlocks, conflicts, or temporary files. Exact fixture cleanup
|
||||
produced an empty application-table diff. Evidence is
|
||||
`output/performance/final-load-20260720`.
|
||||
- Prometheus exactly matched and scraped all three web targets before and after
|
||||
the induced replica stop. Grafana's provisioned Prometheus datasource
|
||||
returned `OK`; Alertmanager delivered firing and resolved webhooks for the
|
||||
scoped instance. The database count diff was empty. Evidence is
|
||||
`output/observability/final-observability-20260720`.
|
||||
- Restic created and checked an encrypted local S3 snapshot, restored it into a
|
||||
fresh database, rejected a corrupted repository, and published no snapshot
|
||||
for an interrupted upload. The interrupted repository passed checking after
|
||||
pruning unreferenced data; the source database diff was empty. Evidence is
|
||||
`output/backups-s3/final-backup-20260720`.
|
||||
- The 50,000-row database replay selected every asserted cursor index and
|
||||
retained the same generated table counts before and after the index
|
||||
migration. Its isolated project, volume, and image were removed. Evidence is
|
||||
`output/db-scale/20260719233526-97476`.
|
||||
- A clean deployment from `git archive b96d443` generated independent secrets,
|
||||
applied all 10 migrations, repeated migration without changing its 10
|
||||
migration or 14 category counts, and passed HTTP, Mailpit, four-node cluster,
|
||||
and PubSub checks on 2 web/2 worker replicas. Its workspace, image, project,
|
||||
volumes, and networks were absent after cleanup. Evidence is
|
||||
`output/portability/20260719233712-142771-ac1883`.
|
||||
- The project-owned kind cluster was rebuilt with the current image. The final
|
||||
rolling drill replaced all four application pod UIDs with four Ready,
|
||||
zero-restart pods, rejoined all four BEAM nodes, passed PubSub, and left an
|
||||
empty database diff. All 395 readiness samples ultimately succeeded; two
|
||||
used one transport retry during the local single-node NodePort endpoint
|
||||
replacement. Evidence is
|
||||
`output/resilience/final-kind-rollout-fixed-20260720`.
|
||||
|
||||
The first 2026-07-20 kind attempt exposed a race in the verification harness:
|
||||
one label-based `kubectl wait` invocation retained a terminating pod in its
|
||||
initial resource set. The harness now waits for old UIDs to disappear, captures
|
||||
the exact four replacement pod names, and waits for those resources. The fixed
|
||||
full replay above passed.
|
||||
|
||||
## Known work before a public production launch
|
||||
|
||||
- Replace the temporary staging origin with the production-owned domain and
|
||||
|
|
|
|||
|
|
@ -273,13 +273,6 @@ probe_pid=$!
|
|||
"${kube[@]}" rollout status "deployment/$WORKER_DEPLOYMENT" \
|
||||
"--timeout=$ROLLOUT_TIMEOUT" >"$output_dir/worker-rollout-status.txt"
|
||||
|
||||
"${kube[@]}" wait \
|
||||
--for=condition=Ready \
|
||||
--timeout="$ROLLOUT_TIMEOUT" \
|
||||
pod \
|
||||
-l app.kubernetes.io/instance=who-need-help \
|
||||
>"$output_dir/pods-ready.txt"
|
||||
|
||||
old_pod_deadline=$((SECONDS + CLUSTER_JOIN_TIMEOUT))
|
||||
|
||||
while ((SECONDS < old_pod_deadline)); do
|
||||
|
|
@ -310,6 +303,31 @@ if [[ "$retained_count" != "0" ]]; then
|
|||
exit 1
|
||||
fi
|
||||
|
||||
mapfile -t current_application_pods < <(
|
||||
"${kube[@]}" get pods \
|
||||
-l app.kubernetes.io/instance=who-need-help \
|
||||
-o json |
|
||||
jq -r '
|
||||
.items[]
|
||||
| select(
|
||||
.metadata.labels["app.kubernetes.io/component"] == "web" or
|
||||
.metadata.labels["app.kubernetes.io/component"] == "worker"
|
||||
)
|
||||
| "pod/" + .metadata.name
|
||||
'
|
||||
)
|
||||
|
||||
if [[ "${#current_application_pods[@]}" -ne 4 ]]; then
|
||||
echo "Expected four replacement application pods; observed ${#current_application_pods[@]}." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
"${kube[@]}" wait \
|
||||
--for=condition=Ready \
|
||||
--timeout="$ROLLOUT_TIMEOUT" \
|
||||
"${current_application_pods[@]}" \
|
||||
>"$output_dir/pods-ready.txt"
|
||||
|
||||
deployment_snapshot "$output_dir/deployments-after.json"
|
||||
pod_snapshot "$output_dir/pods-after.json"
|
||||
database_snapshot "$output_dir/database-after.txt"
|
||||
|
|
|
|||
Loading…
Reference in New Issue
Block a user