​

Keeper incident response

Every incident starts by naming an incident commander, setting OPERATOR_ID, preserving the current deployment ID and config fingerprint, and opening the Tayho Keeper dashboard. Never delete Redis jobs or transaction rows manually. PostgreSQL remains authoritative. For every drill or incident, retain the UTC timeline, operator identity, commit SHA, immutable keeper/frontend image digests, Railway deployment IDs, dashboard snapshot, relevant trace IDs, operator-command output, and the final audit-event IDs. Redact secrets and signed transaction bytes. The incident commander owns containment and evidence. The service owner diagnoses and executes approved recovery; security owns key compromise and protocol-integrity decisions; the release owner owns deployment rollback. See Observability for signal triage and the Security model for authority boundaries.

​

RPC outage

1. Determine whether the write RPC, one read voter, or quorum is unavailable. Compare endpoint circuit state, latency, chain ID, latest head, pending nonce, and receipt agreement.
2. If the read quorum is lost, drain all signing lanes. Do not lower the quorum or add two endpoints from the same provider to restore readiness cosmetically.
3. If only the write RPC failed, leave transaction journal entries intact and switch RPC_URL to an independently operated, chain-verified endpoint. The reconciler must resolve every ambiguous broadcast before the lane allocates another nonce.
4. Restore one replica per affected role, verify quorum head and receipt canonicality, then resume the second replicas. Retain endpoint/provider evidence and the recovery-time measurement.

​

Database failover

1. Drain signing lanes if the database is reachable; otherwise let readiness fail closed and do not start replacement workers against an unverified database.
2. Promote or restore the Railway Postgres service according to the provider procedure. Confirm TLS, restricted runtime identity, schema version, migration lock state, and point-in-time target.
3. Before workers resume, compare nonterminal job/transaction lineages and every signer nonce with the independent RPC quorum. Run the duplicate-enqueue and restricted-write probes.
4. Resume one replica per role, observe the SLO gate, then restore the second replicas. Record actual RTO and RPO and attach the restore/failover evidence to the launch manifest.

​

Nonce gap

1. Run bun run jobs tx-pending and inspect every lineage around the first missing nonce with bun run jobs tx-show <transaction-id>.
2. Compare the database reservation, deterministic transaction hashes, and pending/latest nonce on every quorum voter. Never send a filler transaction or edit signer_nonces manually.
3. If signed ciphertext exists, let reconciliation rebroadcast the identical bytes or its bounded same-nonce replacement. If allocation is unsigned, only the built-in tail-release path may abandon it.
4. Drain the whole signer lane if the source cannot be proven. Resume only after the gap is mined, reverted, safely released, or escalated for a signed operational-security decision.

​

Stale oracle

1. Confirm timestamp, L1-head liveness, BBO validity, depth attestation, and whether the condition is isolated to one market or the attester/RPC plane.
2. Verify the on-chain market is paused. If automation has not paused it, use the authorized guardian procedure; do not substitute a manual price or relax staleness/depth limits during the incident.
3. Stop new intents durably with OPERATOR_ID=<incident> bun run jobs intent-market-pause <chain-id> <market-id> <reason>. Existing reduce-only and protective automation continues; do not stop it by scaling the engine down.
4. Resume only after independent price/depth comparison, a fresh attestation, and a canary cycle. Record the exact stale interval and positions affected.

​

Mass liquidation

1. Distinguish a genuine market move from stale/crossed books, bad scaling, attester compromise, or duplicated discovery. Pause the market immediately if data integrity is uncertain.
2. Inspect liquidation queue age, block-share limits, simulations, vault buffer, queued USDC claims, signer balances, and RPC acceptance. Never bypass per-item simulation or batch gas caps.
3. If the move is genuine, keep bounded permissionless batches running and scale discovery separately from execution. Do not raise exposure, maintenance, or batching limits during the event.
4. Reconcile every submitted batch and child result before replay. After stabilization, account for liquidations, claims, haircut, TAYHO issuance, and error-budget consumption against chain receipts.

​

Funding observation outage

1. Confirm whether impact, index, Redis sampling, attester authorization, RPC publication, or automation advancement is the failing stage. Keep exits available; do not bypass freshness to open risk.
2. Compare raw samples from both independent origins and inspect window coverage/gaps. Never synthesize a missing interval from a later price, reuse one source as both inputs, or manually change the formula.
3. If either source or a funding key may be compromised, have a guardian pause affected markets, revoke both funding attesters, preserve Redis/provider evidence, and rotate keys plus the affected source.
4. Resume with testnet/canary publication, verify the auditor’s recomputed rate/delta/reserve and vault-wide reserve reconciliation, then unpause. Missed intervals remain uncharged.

​

Deployment rollback

1. Stop promotion immediately. Preserve the failed environment JSONL, deployment IDs, image digests, migration deployment, dashboard state, and first failing trace.
2. Any promotion failure, timeout, cancellation, runner loss, failed canary, or failed evidence upload activates the separate rollback job through its environment-scoped token. It restores every production runtime service to the supplied last-known-good digests and retains independent rollback evidence. Verify those deployment digests; never rebuild or use a mutable tag.
3. If automation itself failed, use railway environment edit ... source.image <previous@sha256> for each touched service while signing roles remain drained. Database migrations must be forward-compatible or have a separately reviewed corrective migration; never downgrade schema by hand.
4. Run the canary gate against the restored services, reconcile nonterminal transactions, then record the rollback with OPERATOR_ID=<incident> bun run jobs deployment-record <digest>.

​

Job failure burn

1. Identify the affected role and outcome labels. Compare queue age, RPC health, signer balance, database health, and the latest deployment marker.
2. Drain the affected instance with bun run jobs drain <instance-id> 300.
3. Inspect dead jobs with bun run jobs dead-list and one lineage with bun run jobs job-show <job-id> plus bun run jobs tx-show <transaction-id>.
4. Replay only after the dependency is healthy: bun run jobs dead-replay <job-id>.
5. If burn continues, roll Railway back to the last healthy deployment. Record the rollback.

​

Queue backlog

1. Check PostgreSQL outbox age, BullMQ depth, Redis memory, and worker heartbeats.
2. Do not raise concurrency for a signing role while it has a nonterminal transaction.
3. Scale discovery separately from execution. Add replicas only after confirming database pool headroom and RPC limits.
4. If Redis was lost, keep workers drained, restore Redis, let the outbox republish, verify idempotency keys, then resume one canary replica before the rest.

​

Stuck transaction

1. Run bun run jobs tx-pending, then bun run jobs tx-show <transaction-id>.
2. Compare pending/latest nonces and every configured RPC receipt. Never create a fresh transaction for the same job or manually advance the database nonce.
3. Let the reconciler rebroadcast the exact encrypted bytes or produce bounded EIP-1559 replacements. If replacement budget is exhausted, drain the signer and escalate.
4. Resume only after the journal lineage is mined, reverted, superseded, or explicitly abandoned before signing.

​

Signer balance

1. Confirm the address is a worker role and not an owner, treasury, or oracle identity.
2. Fund only the minimum operating amount from the approved treasury procedure.
3. Verify tayho_signer_balance_wei recovers and readiness becomes healthy.
4. Record the funding transaction in the incident timeline.

​

Journal-key rotation

1. Add the new key to TX_JOURNAL_KEYS while retaining every old key; set the new active ID.
2. Deploy one canary replica and confirm it writes the new key ID and decrypts older lineages.
3. Deploy all replicas. Run bun run jobs key-status.
4. Only after OPERATOR_ID=... bun run jobs key-retire-check <old-id> succeeds may the old key be removed. The command refuses active keys, live lineages, and terminal ciphertext still inside JOURNAL_KEY_RETENTION_SECONDS (30 days minimum). It also requires a fresh independent-RPC quorum head, proves every affected lineage is beyond TX_REORG_LOOKBACK_BLOCKS, and compares every persisted receipt-block hash with a majority-agreed canonical block hash.
5. Redeploy, verify reconciliation, and retain the audit record. No remote signer adapter is used.

​

Signer rotation

1. Register the replacement signer on-chain without removing the old signer.
2. Run OPERATOR_ID=... bun run jobs signer-drain <old-address> 120; both replicas keep serving from the other key while PostgreSQL prevents new allocations to the old lane.
3. Run OPERATOR_ID=... bun run jobs signer-retire-check <old-address> to prove every terminal lineage is outside the retention and reorg window against an independent-RPC quorum. Within 15 minutes of that successful audited proof, run OPERATOR_ID=... bun run jobs signer-pool-rotate <chain-id> <environment:role> <old-address> <new-address>, replace the old entry in the service-scoped SIGNER_KEYS secret, and immediately roll both replicas. Old replicas fail pool allocation after the fenced handoff; new replicas must match the durable two-address membership. Readiness verifies both keys’ balance, narrow authorization, and identity separation.
4. Re-run OPERATOR_ID=... bun run jobs signer-retire-check <old-address> before on-chain removal. It rejects live lineages and terminal lineages inside SIGNER_RETIRE_FINALITY_SECONDS (24 hours minimum), then applies the same independent-RPC finality proof used by journal-key retirement.
5. Remove the old signer on-chain only after the command succeeds, then archive the local key.

​

Protocol audit finding

1. Open the finding with bun run jobs audit-list open and bun run jobs audit-show <finding-id>. Record the pinned block/hash, check, subject, expected/observed values, evidence, trace context, and linked audit job before changing protocol or worker state.
2. Treat every critical result as real until independently disproved. For reserve, solvency, permission, topology, code-hash, or projection findings, pause new engine intake and drain the relevant signing lanes. Pause affected markets through the guardian procedure when user funds or price integrity may be at risk.
3. Compare the pinned state through all three RPC providers. Inspect the relevant contract storage, receipts, database job/outbox/transaction metadata, and event projection. Never resolve a finding merely to clear an alert.
4. Record ownership with OPERATOR_ID=<incident> bun run jobs audit-ack <finding-id>. A suppression requires a reason and expires within 24 hours: OPERATOR_ID=<incident> bun run jobs audit-suppress <finding-id> <seconds> <reason>. Suppressed findings remain in history but leave the actionable Prometheus gauge on the next audit; expiry automatically reopens them.
5. After remediation, run OPERATOR_ID=<incident> bun run jobs audit-recheck. Resolve manually only when the evidence is understood; any failing audit reopens it automatically. Retain the finding, audit run, operator events, remediation transaction hashes, and review decision.

​

Auditor stale

1. Confirm the single tayho-auditor Railway instance, /readyz, Prometheus scrape, PostgreSQL, Redis/BullMQ, and all three independent RPC providers. The auditor has no signer and must never be made healthy by adding a private key or weakening quorum.
2. Inspect auditor jobs with bun run jobs job-show <job-id>, dead work with bun run jobs dead-list, and the dashboard’s audit age, duration, RPC-disagreement, and projection-lag panels. Replay only after the dependency is healthy.
3. On a checkpoint hash mismatch, let the automatic reorg rewind run. If a full reconciliation keeps reporting projection drift, preserve evidence and run OPERATOR_ID=<incident> bun run jobs audit-rebuild-projection <chain-id> <deployment-block>. With the auditor stopped, supersede obsolete scheduled audits using OPERATOR_ID=<incident> bun run jobs audit-backlog-supersede <chain-id>, clear only that BullMQ role queue, then schedule one audit-recheck before restarting the singleton. After the fresh rebuild starts, audit-backlog-supersede <chain-id> dead-only can close the retained historical delivery failures without touching the active audit; attempt and audit-event evidence remains durable. The deployment block must precede initial worker registration.
4. Restart the one Railway instance only after confirming there is no duplicate auditor deployment. PostgreSQL idempotency, BullMQ fencing, and the durable event checkpoint make restart safe. Do not promote a release or resume drained traffic until a fresh pinned audit completes with no critical findings and projection lag is within the configured bound.

​

Engine replica loss

1. Confirm both tayho-engine Railway replicas, /readyz, Prometheus targets, PostgreSQL, Redis, and the independent RPC quorum. The engine is signerless; never add a key to restore readiness.
2. Preserve request/intent trace IDs and check intake acknowledgement, validation age, checkpoint lag, trigger backlog, index outbox lag, and PostgreSQL connection saturation.
3. Restart only the missing replica using the reviewed digest. Fencing prevents duplicate market work; do not clear leases, Redis keys, or checkpoints manually.
4. Resume promotion only after two replicas are scraped for ten minutes and the intent auditor reports zero projection/reservation drift.

​

Intent intake burn

1. Separate client rejection from rate-limit pressure, dependency unavailability, and PostgreSQL commit latency using tayho_intent_requests_total and the acknowledgement histogram.
2. Inspect affected state with bun run jobs intent-list and bun run jobs intent-show <chain-id> <hash>. A Received response is durable receipt, never proof of execution.
3. If one market is abusive, pause only its intake with intent-market-pause; do not raise global limits during an incident. Existing automatic exits continue to run.
4. Resume with OPERATOR_ID=<incident> bun run jobs intent-market-resume <chain-id> <market-id> only after database and rate-limit headroom recover. Retain the pause/resume audit-event IDs.

​

Intent engine lag

1. Compare validation age, index-outbox lag, Redis checksum drift, checkpoint block/hash, trigger backlog, trigger-to-queue latency, RPC disagreement, BullMQ depth, and relayer batch utilization.
2. PostgreSQL is authoritative. If Redis drift is confirmed, preserve both summaries and run OPERATOR_ID=<incident> REDIS_URL=... bun run jobs intent-index-rebuild <chain-id>; never edit keys.
3. Revalidate only an eligible funding-blocked intent with OPERATOR_ID=<incident> bun run jobs intent-revalidate <chain-id> <order-hash>.
4. For RPC disagreement, keep the market paused at its durable checkpoint until independent providers agree. Never lower quorum or manually advance a checkpoint.

​

Intent reservation drift

1. Treat the auditor finding as critical. Inspect the finding, intent lifecycle/events, account funding snapshot, held reservations, OCO siblings, and linked settlement job without decrypting payloads.
2. Pause intake for affected markets. Do not edit reserved totals or release reservations manually.
3. Rebuild Redis only if its checksum differs; reservation truth is PostgreSQL and must be repaired by a reviewed forward migration or replay of the fenced lifecycle operation.
4. Run a fresh full audit and retain before/after counts, checksums, trace IDs, and audit events.

​

Funding blocked intents

1. Check Permit2 allowance/expiry, wallet balance, session remaining spend, position ownership, and the pinned validation block. Funding blocks are soft-reservation outcomes, not engine failures.
2. Do not spend or reserve collateral on behalf of a trader. After funds return, use automatic refresh or the audited intent-revalidate command for a specific order.
3. If growth is systemic, pause new opening-intent intake while protective closes continue, and inspect frontend/session-cap guidance before resuming.

​

Intent key rotation

1. Add the new local key to the shared engine/relayer INTENT_KEYRING, set INTENT_ACTIVE_KEY_ID, and deploy one engine and one relayer canary while retaining all prior keys.
2. Run bun run jobs intent-key-status; verify new ciphertext uses the active ID and old executable intents still validate and settle.
3. Deploy both replicas. OPERATOR_ID=<change> bun run jobs intent-key-retire-check <old-id> must report zero executable-intent blockers and refuse the active key before it may be removed.
4. Remove the key from both roles in one controlled rollout and retain the command and deployment audit events. No remote KMS/HSM/MPC adapter is introduced.

​

Reorg

The reconciler verifies canonical block hashes inside the configured lookback. An orphaned receipt reopens the durable job, fences stale execution, and rebroadcasts the canonical lineage. Page if reorg count is nonzero; pause affected markets if finality assumptions no longer hold. Never mark a job complete from one RPC response.

​

Indexer lag or failure

1. Check the single tayho-indexer writer’s /health, /ready, /status, JSON logs, and Prometheus target. Compare ponder_sync_block_timestamp with ponder_indexing_timestamp to separate RPC ingestion lag from slow handler/database work.
2. For RPC errors, preserve provider/error labels and verify both configured HTTP origins independently. Rotate or replace the failing endpoint; do not change PONDER_START_BLOCK, skip a block range, or point production at an unqualified single provider.
3. For database errors, stop release promotion and restore the separately owned tayho_indexer logical database. Do not point Ponder at the keeper’s railway logical database and do not edit projection rows manually.
4. A reorg counter increase is evidence that Ponder automatically rewound its transaction log. Compare /status with the independent auditor and wait for /ready; never replay handlers or delete a deployment schema by hand.
5. If a deployment remains in backfill, keep the prior Railway deployment serving. Inspect RPC limits, database latency below 50 ms, pool saturation, and handler duration. Roll back the indexer image if the candidate cannot reach realtime within the release window.
6. If the active projection is corrupt, restore the indexer database to an isolated sibling, start the reviewed image from the original earliest deployment block, and compare market/position/funding counts and sampled event identities against chain plus the independent auditor before switching traffic. The engine and trading workers do not depend on Ponder for execution and must not be made to trust a hurried projection repair.

​

PostgreSQL restore drill

1. In testnet, choose a target time in Railway Postgres PITR history. Restore the coordination and indexer logical databases to isolated sibling services.
2. Point one isolated auditor at the restored auditor-scoped DATABASE_URL; never overwrite production.
3. Run migrations, verify the expected schema version, constraints, worker history, audit events, outbox backlog, signer nonces, and every nonterminal transaction hash against chain quorum.
4. Prove that duplicate enqueue and transaction reservation remain fenced.
5. Destroy the isolated fork only after evidence is retained. Repeat monthly and before launch.

​

API degradation

1. Confirm both tayho-api replicas, their /readyz responses, Prometheus targets, current image digests, and the Cloudflare-to-Railway path. Read the readiness breakdown for engine, indexer, RPC, and Redis. Never add database, signer, journal, or encryption credentials to restore it.
2. Split failures by source in tayho_api_upstream_requests_total. For engine, preserve the request ID and upstream trace ID and follow the intent-engine runbook. For indexer, follow the indexer runbook. A failed intent POST was not retried by the API; clients may repeat the same signed intent safely and receive its idempotent order-hash acknowledgement.
3. Compare the failing route and status class, then call the private upstream readiness endpoint from the API environment. Do not extend write timeouts or add automatic POST retries during an incident.
4. Roll only one API replica to the last-known-good qualified digest, observe it, then restore the second. Rotate ENGINE_INTERNAL_TOKEN on both API and engine services together if compromise is suspected; mismatched credentials intentionally fail closed.

​

Realtime degradation

1. Check tayho_api_realtime_pump_total, tayho_api_websocket_connections, rejected protocol messages, and slow_consumer deliveries by replica. Verify Redis latency and stream length with XLEN tayho:realtime:v1; never delete the stream during diagnosis.
2. If the indexer pump fails, confirm one replica can renew tayho:realtime:v1:lease:indexer-pump and reach the private indexer feed. A leader loss is safe: another replica takes the expiring lease and event IDs deduplicate overlap.
3. If Redis is unavailable, REST snapshots remain the recovery path but readiness intentionally drains API replicas. Restore Redis, allow the pump to resume from its persisted cursor, and test a WebSocket snapshot plus reconnect replay before returning traffic.
4. Slow consumers are closed with a retryable code. Do not increase buffers during an incident; clients reconnect with resumeFrom, or accept resync_required and a fresh snapshot.

​

Market data degradation

1. Check both tayho-market-data replicas, /readyz, the upstream circuit gauge, and failures split by book versus metadata. Preserve the selected asset, request ID, upstream timestamp, and image digest; never log bearer credentials or raw authorization headers.
2. Query Hyperliquid outcomeMeta and the affected l2Book directly. Confirm the outcome is still active, USDC quoted, two-sided, monotonic, and inside the configured freshness window. A thin, crossed, empty, or stale book is a valid fail-closed result, not a reason to relax validation.
3. If only one replica is unhealthy, drain and roll it to the last-known-good qualified digest. If the shared upstream is unhealthy, leave funding/depth publication stopped; exits remain available and guardian staleness/depth controls protect affected markets.
4. Resolution must remain pending unless an explicit final side is available from the governed source. Never infer a result from a price near zero/one, manually patch an observation, or reuse a funding token for depth/resolution.
5. After recovery, run the HIP-4 upstream qualification workflow, confirm fresh authenticated samples from all four routes, then observe one complete 15-minute funding interval before clearing the incident.

​

Security compromise

Drain the affected role, revoke it on-chain, rotate its local signer, rotate journal encryption keys if ciphertext confidentiality may be affected, and rotate database/Redis/RPC credentials. Preserve audit events, deployment metadata, logs, traces, and the affected image digest. Do not replay dead letters until the new signer passes readiness and separation checks. For obsolete revenue skim dead letters that never acquired transaction lineage, use the audited periodic-backlog-supersede revenue skim command. It fails closed if any candidate has transaction lineage; never use it for user orders or transaction-producing retries.