Skip to main content

​
Coordination Operations

TAYHO uses PostgreSQL through Drizzle as its durable system of record and Redis through BullMQ as its delivery plane. PostgreSQL owns correctness: canonical idempotency keys, jobs, the transactional outbox, execution fencing, attempt history, worker state, signer nonces, and encrypted signed transactions. Redis owns ephemeral queue mechanics: priorities, delayed delivery, retries, lock renewal, concurrency, and stalled-job recovery. This is at-least-once execution. Neither PostgreSQL nor Redis makes blockchain effects exactly once. The current BullMQ execution token fences stale database transitions, and every attempt revalidates current on-chain state before signing. The transaction journal persists deterministic signed bytes before broadcast so ambiguous retries rebroadcast the same sender/nonce/transaction. See Data ownership for the authority model, Observability for the signals that lead here, and Incident response for symptom-driven procedures.

​
Production topology

  • PostgreSQL 16 or newer, multi-AZ, TLS, encrypted storage/backups, point-in-time recovery, and a tested regional failover procedure.
  • Redis 7 or newer, TLS, automatic failover, persistence appropriate to the provider, and maxmemory-policy noeviction. BullMQ queues must not share an eviction-prone cache cluster.
  • Two replicas per signing role, spread across failure domains. BullMQ distributes jobs; redundant scanners independently discover work and PostgreSQL deduplicates it. The deliberately single, keyless auditor is restarted by Railway and pages when its audit-success SLO becomes stale.
  • DATABASE_POOL_SIZE is per process. Keep the aggregate below the database or pooler’s safe connection limit.
  • Separate migration and runtime identities. Runtime receives DML access to operational tayho_* tables and no schema privileges; audit history grants only SELECT and INSERT, never UPDATE, DELETE, or TRUNCATE.
  • Database, Redis, signer, and journal credentials come from a secret manager, never images or manifests.
DATABASE_URL and REDIS_URL are an inseparable pair. A process rejects a configuration that provides only one. With INFRASTRUCTURE_REQUIRED=true, it also refuses local in-memory mode. Signing roles require TX_JOURNAL_KEYS, a JSON object of versioned 32-byte 0x-prefixed secrets, and TX_JOURNAL_ACTIVE_KEY_ID. Rows bind ciphertext to that key ID with AES-GCM associated data. Apply generated Drizzle migrations as a separately authorized deployment step:
The migration command serializes with a PostgreSQL advisory lock, creates/updates the runtime login, generates PostgreSQL’s SCRAM verifier locally so the plaintext password never enters a SQL statement, forces non-superuser/non-owner attributes, rejects inherited role memberships, applies DML/default privileges, and re-revokes audit and signer-history mutation plus schema-metadata writes. Qualification connects as that runtime login and proves those attributes and denials. The auditor has a separate restricted Drizzle store. Row-level policies permit shared-state reads but restrict all worker/job/outbox/attempt mutations to auditor-owned rows; journal ciphertext is not selectable. PostgreSQL durably records pinned audit runs, finding lifecycle, bounded evidence, operator actions, raw chain events, derived active subjects, and a named projection checkpoint. Redis/BullMQ only delivers idempotent protocol-audit work. A crash after persistence loads the already committed run instead of recomputing mutable operational inputs; a reorg rewinds and deterministically rebuilds the event projection. Hourly full snapshots compare projected positions, orders, withdrawal claims, and permission identities with enumerated chain state. The auditor never loads a signer, reads journal ciphertext, or submits a transaction. Use bun run db:generate after an intentional schema change and commit both the SQL migration and Drizzle snapshot. CI can run bun run db:check. Never generate a migration during application startup. Readiness fails if PostgreSQL, Redis, or the expected schema version is unavailable.

​
Durable enqueue and delivery

Ordinary discovery inserts the canonical job and one outbox row in the same PostgreSQL transaction. The publisher repeatedly sends due outbox rows to BullMQ using a stable (job UUID, delivery generation) ID, then schedules a later delivery check. A crash at any point is safe:
  • before commit, neither row exists;
  • after commit but before publish, another publisher retries the outbox row;
  • after publish but before acknowledgement, BullMQ’s stable delivery ID deduplicates the retry.
Every delivery-eligible PostgreSQL job is rechecked at QUEUE_RECONCILE_SECONDS. If Redis lost the queue entry, the same delivery ID is recreated. If BullMQ has a terminal record while PostgreSQL is still non-terminal, PostgreSQL atomically advances the delivery generation and publishes a fresh queue record. Publishers never remove a BullMQ record, avoiding remove/re-add races across replicas. BullMQ applies bounded exponential retry and owns delivery locks and stalled-job recovery. When a worker starts an attempt it writes BullMQ’s execution token into PostgreSQL. Complete/fail updates require that current token, worker, and attempt. A resumed stale handler therefore cannot overwrite the replacement handler’s state. Relevant controls are QUEUE_PREFIX, QUEUE_CONCURRENCY, QUEUE_LOCK_SECONDS, QUEUE_RECONCILE_SECONDS, JOB_MAX_ATTEMPTS, and OUTBOX_POLL_MILLISECONDS. Size QUEUE_LOCK_SECONDS above normal event-loop stalls and RPC bursts; BullMQ renews active locks automatically. Alert on stalled jobs, retry rate, oldest outbox age, queue age, dead jobs, and processing age. The public signerless engine uses atomic Redis counters keyed by a one-way client-identity hash. Submission and intent-status budgets are separate but shared across all engine replicas; status polling is throttled before it can query PostgreSQL. If Redis is unavailable, these routes fail closed with 503 rather than bypassing the limiter. Validated session-signed intents first enter pending_batch without an individual outbox row. Any relayer replica may atomically claim a FIFO prefix into one queued parent and transition the children to batched; only that parent receives an outbox row. The claim and outbox insert share a Drizzle transaction, so replicas cannot double-submit a child. A 100ms window bounds latency, while individual simulation plus a safety buffer sets a bounded on-chain gas stipend for each child; the coordinator then chooses the largest prefix that fits the live block share and absolute cap. This prevents one gas-griefing child from consuming its siblings’ execution budget. The item ceiling is a final guardrail. Unbatchable children are terminally quarantined so FIFO progress cannot stall. Successful receipt logs carry each child index and are reconciled transactionally to the original job; missing or duplicate result logs fail closed. Dead-letter replay of the parent restores its children to batched before redelivery.

​
Transaction and nonce recovery

Each durable job owns one (chain, signer, nonce, call_hash) lineage. An optimistic Drizzle transaction advances the signer nonce once. The worker signs locally, AES-GCM encrypts and persists the raw signed transaction and deterministic hash, and only then broadcasts it. A retry resumes:
  • allocated: sign at the reserved nonce;
  • signed: decrypt and broadcast the identical bytes;
  • broadcast: reconcile the deterministic hash and receipt before rebroadcasting;
  • replaced: retain the prior same-nonce signed attempt while following the active fee replacement;
  • mined or reverted: terminal and immutable;
  • superseded: the quorum-confirmed account nonce advanced past the lineage but no journal hash had a receipt; the job is dead-lettered for manual investigation and the nonce lane is unblocked;
  • abandoned: an unsigned allocation failed deterministic re-simulation; its tail nonce was safely released before the job completed as not actionable.
The ordinary delivery attempt budget never discards a job after it owns a journal row. If one BullMQ delivery generation exhausts its attempts, the outbox creates another generation and keeps reconciling the same nonce and signed bytes. This deliberately fails closed instead of stranding a signer nonce. Every signing process also runs reconciliation at TX_RECONCILE_SECONDS. It compares pending and latest network nonces with the Drizzle nonce fence, rebroadcasts a signed crash-window transaction, and finalizes receipts only after TX_CONFIRMATIONS. Terminal receipts inside TX_REORG_LOOKBACK_BLOCKS are checked against a quorum-agreed canonical block hash; an orphan atomically revives the highest-fee lineage member as signed, fences any in-flight executor, returns the job to retry, restores batch children, and advances the outbox generation. After TX_STUCK_SECONDS, the active EIP-1559 transaction may be replaced at the same nonce. Each replacement must raise both fee fields by at least TX_FEE_BUMP_BASIS_POINTS, is capped by TX_MAX_FEE_PER_GAS_WEI, and cannot exceed TX_MAX_REPLACEMENTS. The write is sent only through RPC_URL. Ordinary reads use the health-scored, hedged RPC_READ_URLS pool; receipt finalization, confirmed nonce consumption, and canonical block checks require a majority of configured endpoints to agree. Live local signers fail configuration unless three distinct endpoint hosts are present, giving 2-of-3 agreement and one-provider fault tolerance; duplicate-host quorum voters are rejected. Endpoint circuits reopen after RPC_CIRCUIT_COOLDOWN_MILLISECONDS. Inspect ambiguous work without exposing encrypted bytes:
Never delete or manually advance a nonce row while a linked transaction is non-terminal. A timeout does not prove rejection. Keep every relevant TX_JOURNAL_KEYS entry available until OPERATOR_ID=... bun run jobs key-retire-check <key-id> proves no active transaction lineage uses it, the non-reducible incident-retention window has elapsed, and every affected lineage has a terminal receipt at least TX_REORG_LOOKBACK_BLOCKS behind a conservative head returned by a majority of three or more independently hosted RPC endpoints. The gate also quorum-reads each receipt block and requires its canonical hash to match the hash persisted in the journal, catching receipts orphaned while all workers were offline.

​
Dead jobs and replay

After the configured attempts, a job without a transaction journal reaches dead. Journaled jobs continue reconciliation as described above. Fix the cause and verify current on-chain state before replaying a dead job:
Replay preserves the canonical idempotency key and history, resets the attempt budget, and creates a new outbox event. The executor still re-simulates before submission.

​
Graceful drain and key rotation

Before terminating or rotating a signing worker, mark it draining and wait for its processing count to reach zero:
drain INSTANCE_ID is replica-local and leaves the sibling serving. To rotate one key, use bun run jobs signer-drain ADDRESS 120; that persists the signer-lane fence, stops new allocations to that key across both replicas, and waits for its active journal lineage to settle. Then use signer-retire-check OLD_ADDRESS for the independent-RPC retention/finality proof. Within 15 minutes, use signer-pool-rotate CHAIN_ID ENVIRONMENT:ROLE OLD_ADDRESS NEW_ADDRESS for the audited membership handoff before rolling the shared SIGNER_KEYS secret. The runtime also pauses BullMQ intake and waits for active handlers on shutdown. If the operator command times out, investigate the durable transaction journal before enabling a replacement signer. Use bun run jobs undrain INSTANCE_ID only to cancel a drain for the same healthy process. Relayer, automation, funding-oracle, depth/resolution-attester, and guardian services each have two replicas and exactly two independently rotatable signer identities; both replicas load both identities. Revenue deliberately has one delay-safe replica and the same two-identity pool. A PostgreSQL-locked round-robin cursor skips busy or draining lanes, and each nonce stream is durable. An unsigned allocated row is deliberately bound to one signer, while any eligible replica can finish signing it. On-chain role membership remains authoritative.

​
Failure handling

  • PostgreSQL unavailable: stop accepting durable work and fail readiness. Existing Redis delivery may retry, but must not bypass durable state.
  • Redis unavailable: canonical jobs continue to be safely recorded in PostgreSQL; outbox rows remain pending until delivery recovers. Fail readiness to remove the replica from service.
  • Worker crash: BullMQ detects the stalled delivery and hands it to another replica; execution-token fencing rejects late completion from the old handler.
  • Database promotion: keep workers unready until the writer is available and schema version matches. Reconcile non-terminal transaction hashes and pending/latest signer nonces before resuming.
  • Redis failover or total queue loss: the delivery reconciler recreates every non-terminal job from PostgreSQL. Terminal BullMQ records receive a new fenced delivery generation without changing the canonical business idempotency key.

​
Backup, restore, and retention

  1. Continuously archive PostgreSQL WAL and take regular full snapshots.
  2. Run monthly isolated restores and verify migration version, constraints, outbox backlog, and transaction journal counts.
  3. Measure RPO/RTO in drills; do not infer them from vendor configuration.
  4. After restore, compare every non-terminal journal hash with chain receipts and pending/latest signer nonces before allowing submissions.
  5. Test Redis failover independently. PostgreSQL remains the recovery authority; never treat a Redis snapshot as the only copy of business state.
Retain completed jobs and attempts online for at least 30 days, dead jobs until explicit resolution, schema migrations permanently, and transaction rows through finality plus the incident-response window. Reap stale worker rows only when no processing job references them. Archive rather than mutate audit history.
Last modified on July 23, 2026