Skip to main content

Production launch evidence checklist

Production traffic is blocked until every required item below links to immutable evidence reviewed by the named operations and security owners. Store artifacts in the approved evidence repository; this document defines the manifest but intentionally does not claim that external drills or reviews have occurred. Use Railway deployment for the release procedure, Observability for SLO evidence, and Incident response for exercised runbooks.

Release identity

  • Reviewed commit SHA is protected and signed according to repository policy.
  • GitHub testnet, production, and no-wait production-rollback environments accept only protected main; their Railway tokens are environment-scoped and not repository-wide.
  • Keeper, API, indexer, and market-data @sha256 image references equal the run-scoped image identity artifact published by successful exact-commit container CI. The separately released Cloudflare frontend is bound to its own immutable build artifact and deployment ID; the retained frontend container is qualification evidence only, not a Railway production service.
  • The previous-image set is bound to its successful container workflow run ID; validation and the isolated rollback job independently re-download and compare that run-scoped identity artifact.
  • SPDX SBOMs, maximum-mode BuildKit provenance, dependency scan, and container scan are retained.
  • Both Railway environments match the service profiles; each signing role has exactly two distinct sealed keys with matching on-chain registration. Relayer, automation, funding oracle, depth/resolution attester, and guardian each run two replicas sharing their role’s two-key pool. Revenue deliberately runs one replica with its two-key pool because distribution may be delayed safely. The single auditor and single Ponder indexer are keyless, as are the two API and engine replicas. The two stateless market-data replicas are also keyless and expose independently scoped read-only feed credentials. The Cloudflare frontend is outside Railway. The indexer has no Redis or coordination database access.
  • Production runtime variables, CSP, allowed relay origin, RPC quorum independence, role separation, PostgreSQL grants, Redis authentication, and telemetry redaction have been peer reviewed.

Automated verification

  • The exact commit passed workspace tests, typechecks, builds, lint, Railway-schema validation, Docker builds, runtime smoke checks, and the operations-asset tests.
  • The scheduled market-data upstream workflow retained a fresh successful testnet qualification, including HIP-4 metadata discovery and a valid two-sided order-book sample.
  • The pinned OSV-Scanner dependency gate and pinned Grype container gates found no High or Critical vulnerability in the exact keeper, API, indexer, market-data, or retained frontend-fallback image.
  • Contract security CI passed Forge tests and coverage, the no-High/no-new-Medium Slither gate, the configured 50,000-call Echidna campaign, and every Halmos matrix job.
  • The testnet-qualification environment reviewer compared both security-review IDs and artifact hashes with the independent reviewers’ authenticated systems of record. Report hashes prove retained-byte integrity, not external authorship.
  • Candidate-bound k6-intents.js evidence proves a 5,000 burst plus 1,000 intents/second for at least 60 seconds: exactly one durable row per unique hash, no dropped iteration, duplicate-storm idempotency, and p99 durable acknowledgement below 250 ms. Signed settlement smoke evidence links sampled intents to canonical per-child receipts.

Recovery and resilience evidence

  • A continuous 24-hour testnet soak completed on the candidate images with actual container kills, Redis restarts, PostgreSQL failover, live RPC partition, delayed/ambiguous receipts, injected reorg, and unauthorized-write probes; no job loss, nonce collision, or unauthorized write occurred.
  • PostgreSQL PITR restored to a sibling service and passed schema, grants, audit immutability, idempotency, signer-nonce, transaction-lineage, and chain-quorum reconciliation checks.
  • Indexer PostgreSQL PITR restored to an isolated sibling; a full replay from the earliest deployment block matched market, position, settlement, funding, and event-identity samples, including an injected reorg rollback, before its stable views were promoted.
  • Redis was flushed, restarted, and rebuilt solely from PostgreSQL intent/outbox truth; generation count and checksum matched and no trigger or settlement was duplicated.
  • Engine/relayer replica kills during claim and batching preserved fencing, FIFO priority, and per-child reconciliation; PostgreSQL, Redis, and each RPC failure mode failed closed.
  • Signer, intent-key, and journal-key rotation/retirement gates passed with retained audit-event IDs.
  • RPC outage, nonce gap, stale oracle, mass liquidation, database failover, reorg, compromise, and deployment rollback runbooks were exercised by someone other than their author.
  • The workflow qualified a candidate by digest on testnet, rejected an injected production SLO failure, automatically restored every production runtime service to its previous digest, and retained testnet, production, and rollback evidence.

Independent approval

  • Independent smart-contract review completed with no open Critical/High findings; accepted Medium findings have a named owner, rationale, compensating controls, and deadline.
  • Independent operational-plane review completed against security-review-scope.md and the actual Railway deployment, not only repository configuration.
  • The single signerless auditor independently reconciled PostgreSQL intents/reservations/checkpoints, Redis generation/checksum, jobs, transactions, and canonical receipts with no critical finding.
  • Operations owner, security owner, protocol owner, and incident commander signed the evidence manifest with UTC timestamp, commit SHA, keeper/API/indexer/market-data digests, all Railway deployment IDs, and the separately released frontend artifact digest plus Cloudflare deployment ID.

Evidence record

Use one row per artifact. Hash reports before uploading and never include secrets, wallet private keys, bearer tokens, raw session signatures, or signed transaction bytes.
Last modified on July 23, 2026