​

TAYHO Enterprise Trading Operations Roadmap

​

Vision

TAYHO’s operational plane is an active-active, horizontally scalable trading system that can survive process, host, RPC, signer, and database failovers without duplicate economic actions, nonce collisions, silent job loss, or privilege collapse. Primary users are traders relying on one-click and automatic orders. Secondary users are the operations, security, and governance teams running TAYHO. The design keeps the existing Bun, Effect, viem, Foundry, and Solidity stack, and assumes no backward compatibility because the protocol has not been deployed.

​

Stack

​

Failure and Delivery Semantics

• Jobs are at least once. Economic idempotency is established by unique job keys, on-chain state checks, contract nonces/order hashes, and transaction reconciliation.
• A queue lock is not economic ownership. Every executor revalidates chain state immediately before signing.
• A transaction is persisted before broadcast with a deterministic sender/nonce identity and is reconciled after crashes before that nonce can be reused.
• Each active-active signing service has two independently registered addresses, and both replicas load the same two-key pool behind the durable nonce coordinator.
• Privileged functions remain narrow. Redundancy must not turn operational workers into owners.
• Readiness fails closed when coordination, identity, chain linkage, or signer health is unknown.

​

Data Model

​

worker_instances and worker_instance_signers — Phase 1

worker_instances records role, lifecycle, drain, build, and heartbeat state. worker_instance_signers records the independently managed signer set for that instance.

​

jobs — Phase 1

id uuid primary key, role text, kind text, idempotency_key text unique, payload jsonb, state text, priority integer, available_at, processing_by, processing_token, attempt_count, max_attempts, last_error, tx_hash bytea nullable, created_at, updated_at, completed_at nullable.

​

job_attempts — Phase 1

Attempt audit: id, job_id, attempt, instance_id, execution_token, started_at, finished_at, outcome, error_class, error_detail, tx_hash.

​

signer_nonces — Phase 3

One row per (chain_id, signer_address) with next_nonce, observed_pending_nonce, locked_by, lock_expires_at, and reconciliation timestamps.

​

transactions — Phase 3

id, chain_id, signer_address, nonce, job_id, call_hash, raw_tx_ciphertext, tx_hash, replacement_of, state, fee fields, broadcast/mined/finalized timestamps, receipt block/hash, and a unique (chain_id, signer_address, nonce) constraint.

​

worker_signer_history — Phase 4

Snapshot of on-chain operational signer-set changes for deployment and incident audit.

​

signer_lanes — Phase 4

One durable admission fence per normalized signer address. Drain, registration, and job start lock this row so a restart or sibling replica cannot reopen a signer while rotation is in progress.

​

Relationships

• A worker instance processes many jobs, but a BullMQ execution token fences one live delivery.
• A job has many immutable attempts and at most one canonical successful transaction.
• A signer has a serialized durable nonce stream even when several replicas submit work.
• A signer lane keeps drain intent durable and serializes worker admission across replicas.
• On-chain signer sets are authoritative; database history is an audit projection only.

​

Phase 1 — Durable Coordination and Idempotency ✅

Completed: 2026-07-13 Goal: Multiple replicas can discover the same work without losing or concurrently executing the same logical job.

​

Tasks

• Add versioned PostgreSQL migrations for instances, jobs, and attempts.
• Implement typed Drizzle persistence with enqueue, claim, complete, retry, fail, heartbeat, stale-instance reaping, transaction journal, and transactional outbox operations.
• Use BullMQ for delivery, exponential retry, lock renewal, priorities, concurrency, and stalled recovery.
• Enforce unique idempotency keys and fencing-token ownership on every state transition.
• Add exponential backoff, maximum attempts, and a dead-letter state.
• Add a memory implementation for deterministic unit and property tests.
• Add readiness checks for database connectivity and migration version when coordination is enabled.
• Document PostgreSQL/Redis failover, queue locking, backup, retention, and dead-letter replay.

​

Definition of Done

• Two simulated replicas enqueue the same key and produce one job.
• Only the current execution token can mutate a processing job.
• A stalled BullMQ delivery becomes reclaimable; a stale executor cannot complete it.
• Poison jobs reach dead-letter state and can be explicitly replayed.
• Existing single-process mode remains usable for local dry runs.

​

Phase 2 — Active-Active Discovery and Execution ✅

Completed: 2026-07-13 Goal: Relayer and automation replicas run concurrently with deterministic ownership and failover.

​

Tasks

• Separate discovery from execution: scanners persist canonical jobs; BullMQ delivers them to executors.
• Define canonical idempotency keys for orders, pruning, liquidation, TWAP, freeze, resolution, queue processing, guardian actions, and revenue distribution.
• Keep discovery redundant instead of adding static sharding that could become a correctness boundary; PostgreSQL deduplicates the inexpensive duplicate discoveries.
• Revalidate all jobs against current chain state before simulation and broadcast.
• Let BullMQ renew locks during long operations and fence late database completions.
• Add content-bound HTTP relay idempotency and durable accepted/processing/mined responses.
• Add graceful drain: stop queue intake and wait for current processing before shutdown.

​

Definition of Done

• Killing an executor during every lifecycle stage produces no lost logical job.
• Duplicate discovery and relay submissions converge on one idempotency record.
• A standby takes over after BullMQ declares a delivery stalled without operator action.

​

Phase 3 — Transaction and Signer Reliability ✅

Goal: Broadcasts survive ambiguous RPC responses and process crashes without nonce reuse.

​

Tasks

• Keep role-isolated local signers; remote KMS/HSM/MPC adapters are explicitly out of scope.
• Persist an AES-GCM-encrypted signed transaction, deterministic hash, call hash, and sender nonce before broadcast.
• Serialize nonce allocation with database row locks and unique (chain, signer, nonce) fencing.
• Reconcile pending/latest nonces and receipts on startup and continuously.
• Add stuck-transaction fee replacement with bounded escalation and replacement lineage.
• Add configurable confirmations/finality and reorg rollback handling.
• Add RPC endpoint pools with health scoring, circuit breakers, hedged reads, and single-provider writes.
• Redact calldata/signatures/signed bytes while preserving stable hash correlation identifiers.
• Add a versioned local journal keyring, per-row key IDs, signer history, audit evidence, and retirement gates that refuse active transaction lineage.

​

Definition of Done

• Crash-before-broadcast, crash-after-broadcast, timeout-after-acceptance, replacement, and reorg tests pass.
• No (chain, signer, nonce) can represent two unrelated calls.
• Each environment keeps every raw role key isolated to its assigned service and coordinated replicas.

​

Phase 4 — Multi-Key Roles and Bounded On-Chain Batching ✅

Goal: Throughput and redundancy scale horizontally without broadening protocol privilege.

​

Tasks

• Replace singleton relayer/automation registration with bounded enumerable signer sets.
• Keep guardian multi-key; support active/passive and explicitly authorized active-active policies.
• Split funding and depth/resolution attestation into separately authorized bounded two-key roles.
• Keep owner and treasury as separate multisig/smart accounts in deployment validation.
• Use a bounded distributor set for redundant revenue workers.
• Deploy a narrow permissionless maintenance batcher for order execute/prune, liquidation, and TWAP.
• Use per-item best-effort results and bounded array sizes; keep batch size configurable.
• Keep unrelated trader relay transactions isolated; never couple user failures in a generic multicall.

​

Definition of Done

• Removing one signer drains it without downtime or accepting new processing.
• Batch fuzz tests prove one bad item cannot revert successful independent items.
• Runtime readiness verifies signer-set membership and cross-role separation after rotations.

​

Phase 5 — Observability, SLOs, and Incident Operations ✅

Goal: Operators detect, diagnose, and safely recover failures before traders report them.

​

Tasks

• Emit bounded-label Prometheus metrics for queue/outbox state, processing, attempts, dead letters, RPC health, pending and receipt latency, transaction states, reorgs, and signer balances.
• Add durable OpenTelemetry trace propagation from relay request through job, transaction, and receipt.
• Define SLOs and multi-window burn-rate alerts for job success and execution/automation latency.
• Add durable audit events for worker/configuration changes, deployments, operator actions, and DLQ replay.
• Add operator CLI commands for drain, replay, cancellation, inspection, deployment evidence, and safe rotation.
• Write runbooks for RPC outage, DB failover, signer loss, nonce gap, reorg, stuck transaction, compromised key, mass liquidation, stale oracle, and rollback.

​

Definition of Done

• Every alert links to a tested runbook and relevant dashboard.
• A transaction can be traced from request/job to finalized receipt without exposing secrets.

​

Phase 6 — Deployment and Resilience Qualification

Goal: The reference deployment withstands planned and unplanned infrastructure failures.

​

Tasks

• Build minimal non-root images with immutable digests and SBOM/provenance generation.
• Add Railway-native testnet and production config-as-code with two replicas per runtime service, health gates, rolling deployment overlap, drain windows, a least-privilege serialized migration job, testnet qualification, and production digest rollback.
• Add a single-writer Ponder service with deployment-isolated schemas, stable views, redundant RPC reads, a dedicated database, bounded queries, Prometheus alerts, and replay/PITR runbooks.
• Add managed PostgreSQL requirements: high availability, PITR, encrypted backups, connection pooling, migration locks, and restore drills.
• Add Railway sealed-variable wiring without provider credentials in Git.
• Add a signed-relayer load harness plus a repeatable real Postgres/Redis adapter soak with forced datastore restarts and deterministic RPC/reorg/crash-window fault suites.
• Execute and retain the 24-hour testnet qualification with actual worker-container kills, live RPC partitions, delayed receipts, injected chain reorgs, and unauthorized-write probes.
• Add deployment canaries and automatic rollback gates based on readiness and SLOs, with immutable previous-image inputs and retained JSONL release evidence.
• Complete an independent smart-contract and operational-plane security review before launch.

​

Definition of Done

• Recovery objectives are measured, not assumed, and recorded in the operations guide.
• A 24-hour soak with injected failures has no lost jobs, nonce collisions, or unauthorized writes.
• Launch checklist and rollback drill are signed off by operations and security owners.

​

Build Order

​

Deliberately Not Building

• A generic privileged multicall contract: too much authority and failure coupling.
• “Exactly once” marketing claims: distributed execution is at least once; effects are made idempotent.
• A shared general-purpose oracle authority: funding and depth/resolution use separate narrow roles, sources, signer sets, and contract allowlists.
• Shared EOAs across unsynchronized replicas: every production signer is independently coordinated.
• A custom persistence layer or raw SQL in application services: Drizzle keeps schema and queries typed.
• Provider-specific KMS or Kubernetes credentials in the repository.

​

Progress

• Current phase: launch qualification evidence collection.
• Phases 1–5 are implemented. Environment-native Railway deployment assets, security CI, testnet qualification and production rollback gates, and qualification harnesses are ready; the actual 24-hour testnet run, restore/failover evidence, and independent security reviews remain launch gates.