> ## Documentation Index > Fetch the complete documentation index at: https://docs.tayho.io/llms.txt > Use this file to discover all available pages before exploring further. # Coordination operations > Operate and recover Tayho PostgreSQL, BullMQ, signer nonces, transaction journals, and durable jobs. # Coordination Operations TAYHO uses PostgreSQL through Drizzle as its durable system of record and Redis through BullMQ as its delivery plane. PostgreSQL owns correctness: canonical idempotency keys, jobs, the transactional outbox, execution fencing, attempt history, worker state, signer nonces, and encrypted signed transactions. Redis owns ephemeral queue mechanics: priorities, delayed delivery, retries, lock renewal, concurrency, and stalled-job recovery. This is at-least-once execution. Neither PostgreSQL nor Redis makes blockchain effects exactly once. The current BullMQ execution token fences stale database transitions, and every attempt revalidates current on-chain state before signing. The transaction journal persists deterministic signed bytes before broadcast so ambiguous retries rebroadcast the same sender/nonce/transaction. See [Data ownership](/architecture/data-ownership) for the authority model, [Observability](/operations/observability) for the signals that lead here, and [Incident response](/operations/incident-response) for symptom-driven procedures. ## Production topology * PostgreSQL 16 or newer, multi-AZ, TLS, encrypted storage/backups, point-in-time recovery, and a tested regional failover procedure. * Redis 7 or newer, TLS, automatic failover, persistence appropriate to the provider, and `maxmemory-policy noeviction`. BullMQ queues must not share an eviction-prone cache cluster. * Two replicas per signing role, spread across failure domains. BullMQ distributes jobs; redundant scanners independently discover work and PostgreSQL deduplicates it. The deliberately single, keyless auditor is restarted by Railway and pages when its audit-success SLO becomes stale. * `DATABASE_POOL_SIZE` is per process. Keep the aggregate below the database or pooler's safe connection limit. * Separate migration and runtime identities. Runtime receives DML access to operational `tayho_*` tables and no schema privileges; audit history grants only `SELECT` and `INSERT`, never `UPDATE`, `DELETE`, or `TRUNCATE`. * Database, Redis, signer, and journal credentials come from a secret manager, never images or manifests. `DATABASE_URL` and `REDIS_URL` are an inseparable pair. A process rejects a configuration that provides only one. With `INFRASTRUCTURE_REQUIRED=true`, it also refuses local in-memory mode. Signing roles require `TX_JOURNAL_KEYS`, a JSON object of versioned 32-byte `0x`-prefixed secrets, and `TX_JOURNAL_ACTIVE_KEY_ID`. Rows bind ciphertext to that key ID with AES-GCM associated data. Apply generated Drizzle migrations as a separately authorized deployment step: ```bash theme={null} cd apps/keeper MIGRATION_DATABASE_URL='postgresql://...' \ DATABASE_PRIVILEGES_REQUIRED=true \ RUNTIME_DATABASE_URL='' \ AUDITOR_DATABASE_URL='' \ ENGINE_DATABASE_URL='' \ RELAYER_DATABASE_URL='' bun run migrate ``` The migration command serializes with a PostgreSQL advisory lock, creates/updates the runtime login, generates PostgreSQL's SCRAM verifier locally so the plaintext password never enters a SQL statement, forces non-superuser/non-owner attributes, rejects inherited role memberships, applies DML/default privileges, and re-revokes audit and signer-history mutation plus schema-metadata writes. Qualification connects as that runtime login and proves those attributes and denials. The auditor has a separate restricted Drizzle store. Row-level policies permit shared-state reads but restrict all worker/job/outbox/attempt mutations to auditor-owned rows; journal ciphertext is not selectable. PostgreSQL durably records pinned audit runs, finding lifecycle, bounded evidence, operator actions, raw chain events, derived active subjects, and a named projection checkpoint. Redis/BullMQ only delivers idempotent `protocol-audit` work. A crash after persistence loads the already committed run instead of recomputing mutable operational inputs; a reorg rewinds and deterministically rebuilds the event projection. Hourly full snapshots compare projected positions, orders, withdrawal claims, and permission identities with enumerated chain state. The auditor never loads a signer, reads journal ciphertext, or submits a transaction. Use `bun run db:generate` after an intentional schema change and commit both the SQL migration and Drizzle snapshot. CI can run `bun run db:check`. Never generate a migration during application startup. Readiness fails if PostgreSQL, Redis, or the expected schema version is unavailable. ## Durable enqueue and delivery Ordinary discovery inserts the canonical job and one outbox row in the same PostgreSQL transaction. The publisher repeatedly sends due outbox rows to BullMQ using a stable `(job UUID, delivery generation)` ID, then schedules a later delivery check. A crash at any point is safe: * before commit, neither row exists; * after commit but before publish, another publisher retries the outbox row; * after publish but before acknowledgement, BullMQ's stable delivery ID deduplicates the retry. Every delivery-eligible PostgreSQL job is rechecked at `QUEUE_RECONCILE_SECONDS`. If Redis lost the queue entry, the same delivery ID is recreated. If BullMQ has a terminal record while PostgreSQL is still non-terminal, PostgreSQL atomically advances the delivery generation and publishes a fresh queue record. Publishers never remove a BullMQ record, avoiding remove/re-add races across replicas. BullMQ applies bounded exponential retry and owns delivery locks and stalled-job recovery. When a worker starts an attempt it writes BullMQ's execution token into PostgreSQL. Complete/fail updates require that current token, worker, and attempt. A resumed stale handler therefore cannot overwrite the replacement handler's state. Relevant controls are `QUEUE_PREFIX`, `QUEUE_CONCURRENCY`, `QUEUE_LOCK_SECONDS`, `QUEUE_RECONCILE_SECONDS`, `JOB_MAX_ATTEMPTS`, and `OUTBOX_POLL_MILLISECONDS`. Size `QUEUE_LOCK_SECONDS` above normal event-loop stalls and RPC bursts; BullMQ renews active locks automatically. Alert on stalled jobs, retry rate, oldest outbox age, queue age, dead jobs, and processing age. The public signerless engine uses atomic Redis counters keyed by a one-way client-identity hash. Submission and intent-status budgets are separate but shared across all engine replicas; status polling is throttled before it can query PostgreSQL. If Redis is unavailable, these routes fail closed with `503` rather than bypassing the limiter. Validated session-signed intents first enter `pending_batch` without an individual outbox row. Any relayer replica may atomically claim a FIFO prefix into one `queued` parent and transition the children to `batched`; only that parent receives an outbox row. The claim and outbox insert share a Drizzle transaction, so replicas cannot double-submit a child. A 100ms window bounds latency, while individual simulation plus a safety buffer sets a bounded on-chain gas stipend for each child; the coordinator then chooses the largest prefix that fits the live block share and absolute cap. This prevents one gas-griefing child from consuming its siblings' execution budget. The item ceiling is a final guardrail. Unbatchable children are terminally quarantined so FIFO progress cannot stall. Successful receipt logs carry each child index and are reconciled transactionally to the original job; missing or duplicate result logs fail closed. Dead-letter replay of the parent restores its children to `batched` before redelivery. ## Transaction and nonce recovery Each durable job owns one `(chain, signer, nonce, call_hash)` lineage. An optimistic Drizzle transaction advances the signer nonce once. The worker signs locally, AES-GCM encrypts and persists the raw signed transaction and deterministic hash, and only then broadcasts it. A retry resumes: * `allocated`: sign at the reserved nonce; * `signed`: decrypt and broadcast the identical bytes; * `broadcast`: reconcile the deterministic hash and receipt before rebroadcasting; * `replaced`: retain the prior same-nonce signed attempt while following the active fee replacement; * `mined` or `reverted`: terminal and immutable; * `superseded`: the quorum-confirmed account nonce advanced past the lineage but no journal hash had a receipt; the job is dead-lettered for manual investigation and the nonce lane is unblocked; * `abandoned`: an unsigned allocation failed deterministic re-simulation; its tail nonce was safely released before the job completed as not actionable. The ordinary delivery attempt budget never discards a job after it owns a journal row. If one BullMQ delivery generation exhausts its attempts, the outbox creates another generation and keeps reconciling the same nonce and signed bytes. This deliberately fails closed instead of stranding a signer nonce. Every signing process also runs reconciliation at `TX_RECONCILE_SECONDS`. It compares pending and latest network nonces with the Drizzle nonce fence, rebroadcasts a signed crash-window transaction, and finalizes receipts only after `TX_CONFIRMATIONS`. Terminal receipts inside `TX_REORG_LOOKBACK_BLOCKS` are checked against a quorum-agreed canonical block hash; an orphan atomically revives the highest-fee lineage member as `signed`, fences any in-flight executor, returns the job to `retry`, restores batch children, and advances the outbox generation. After `TX_STUCK_SECONDS`, the active EIP-1559 transaction may be replaced at the same nonce. Each replacement must raise both fee fields by at least `TX_FEE_BUMP_BASIS_POINTS`, is capped by `TX_MAX_FEE_PER_GAS_WEI`, and cannot exceed `TX_MAX_REPLACEMENTS`. The write is sent only through `RPC_URL`. Ordinary reads use the health-scored, hedged `RPC_READ_URLS` pool; receipt finalization, confirmed nonce consumption, and canonical block checks require a majority of configured endpoints to agree. Live local signers fail configuration unless three distinct endpoint hosts are present, giving 2-of-3 agreement and one-provider fault tolerance; duplicate-host quorum voters are rejected. Endpoint circuits reopen after `RPC_CIRCUIT_COOLDOWN_MILLISECONDS`. Inspect ambiguous work without exposing encrypted bytes: ```bash theme={null} bun run jobs tx-pending bun run jobs tx-show 00000000-0000-0000-0000-000000000000 ``` Never delete or manually advance a nonce row while a linked transaction is non-terminal. A timeout does not prove rejection. Keep every relevant `TX_JOURNAL_KEYS` entry available until `OPERATOR_ID=... bun run jobs key-retire-check ` proves no active transaction lineage uses it, the non-reducible incident-retention window has elapsed, and every affected lineage has a terminal receipt at least `TX_REORG_LOOKBACK_BLOCKS` behind a conservative head returned by a majority of three or more independently hosted RPC endpoints. The gate also quorum-reads each receipt block and requires its canonical hash to match the hash persisted in the journal, catching receipts orphaned while all workers were offline. ## Dead jobs and replay After the configured attempts, a job without a transaction journal reaches `dead`. Journaled jobs continue reconciliation as described above. Fix the cause and verify current on-chain state before replaying a dead job: ```bash theme={null} bun run jobs dead-list bun run jobs dead-replay 00000000-0000-0000-0000-000000000000 ``` Replay preserves the canonical idempotency key and history, resets the attempt budget, and creates a new outbox event. The executor still re-simulates before submission. ## Graceful drain and key rotation Before terminating or rotating a signing worker, mark it draining and wait for its processing count to reach zero: ```bash theme={null} bun run jobs drain tayho-automation-1 120 ``` `drain INSTANCE_ID` is replica-local and leaves the sibling serving. To rotate one key, use `bun run jobs signer-drain ADDRESS 120`; that persists the signer-lane fence, stops new allocations to that key across both replicas, and waits for its active journal lineage to settle. Then use `signer-retire-check OLD_ADDRESS` for the independent-RPC retention/finality proof. Within 15 minutes, use `signer-pool-rotate CHAIN_ID ENVIRONMENT:ROLE OLD_ADDRESS NEW_ADDRESS` for the audited membership handoff before rolling the shared `SIGNER_KEYS` secret. The runtime also pauses BullMQ intake and waits for active handlers on shutdown. If the operator command times out, investigate the durable transaction journal before enabling a replacement signer. Use `bun run jobs undrain INSTANCE_ID` only to cancel a drain for the same healthy process. Relayer, automation, funding-oracle, depth/resolution-attester, and guardian services each have two replicas and exactly two independently rotatable signer identities; both replicas load both identities. Revenue deliberately has one delay-safe replica and the same two-identity pool. A PostgreSQL-locked round-robin cursor skips busy or draining lanes, and each nonce stream is durable. An unsigned `allocated` row is deliberately bound to one signer, while any eligible replica can finish signing it. On-chain role membership remains authoritative. ## Failure handling * PostgreSQL unavailable: stop accepting durable work and fail readiness. Existing Redis delivery may retry, but must not bypass durable state. * Redis unavailable: canonical jobs continue to be safely recorded in PostgreSQL; outbox rows remain pending until delivery recovers. Fail readiness to remove the replica from service. * Worker crash: BullMQ detects the stalled delivery and hands it to another replica; execution-token fencing rejects late completion from the old handler. * Database promotion: keep workers unready until the writer is available and schema version matches. Reconcile non-terminal transaction hashes and pending/latest signer nonces before resuming. * Redis failover or total queue loss: the delivery reconciler recreates every non-terminal job from PostgreSQL. Terminal BullMQ records receive a new fenced delivery generation without changing the canonical business idempotency key. ## Backup, restore, and retention 1. Continuously archive PostgreSQL WAL and take regular full snapshots. 2. Run monthly isolated restores and verify migration version, constraints, outbox backlog, and transaction journal counts. 3. Measure RPO/RTO in drills; do not infer them from vendor configuration. 4. After restore, compare every non-terminal journal hash with chain receipts and pending/latest signer nonces before allowing submissions. 5. Test Redis failover independently. PostgreSQL remains the recovery authority; never treat a Redis snapshot as the only copy of business state. Retain completed jobs and attempts online for at least 30 days, dead jobs until explicit resolution, schema migrations permanently, and transaction rows through finality plus the incident-response window. Reap stale worker rows only when no processing job references them. Archive rather than mutate audit history.