Skip to main content

​
Railway deployment

Tayho is Railway-first and intentionally has no Terraform layer. Repository JSON profiles define the reviewed service shape. Releases use CI-built GHCR images pinned by digest; Railway must not rebuild production because that would create a different artifact from the SBOM and provenance subject. This guide is paired with the launch evidence checklist, coordination recovery, and observability guide.

​
Environment and service shape

Use one Railway project with two isolated environments: testnet and production. Each environment has one PostgreSQL service containing separately owned railway coordination and tayho_indexer logical databases, plus Redis, domains, chain configuration, telemetry credentials, transaction journal keyring, and signer keys. Never reference a database, Redis instance, or signing secret across environments. Create the same services in both environments: The replica counts below are the qualified production topology. Development normally runs tayho-api, tayho-engine, and tayho-market-data with one replica each; temporarily raise them to two only for coordination, rolling-replacement, and failover qualification. The frontend is built and hosted on Cloudflare, outside Railway. It receives only the public API origin. The retained frontend container profile is a local/test fallback, not a production service. Railway balances requests between replicas and performs rolling replacement within each service. The two replicas of a signing service share the same service-scoped pool of exactly two signers. A PostgreSQL-locked cursor selects an available signer and the durable nonce journal serializes each key independently. Registration and reservation both compare against the durable two-address pool, so mismatched replica secrets fail closed. Different roles and environments never share signer keys. The migration service alone receives the schema-owner credential. The profiles are checked against Railway’s published schema in CI. Every Railway service must use its matching profile; there is intentionally no root railway.json because a root config would override the distinct commands and replica policies of every service in this shared monorepo. Production uses immutable Docker Image sources, while development may use source uploads.

​
Immutable images and release gate

.github/workflows/keeper-container.yml publishes:
All images are non-root Bun containers, use tini, and carry maximum-mode BuildKit provenance plus an SPDX SBOM. GitHub artifact attestations are intentionally not required. The workflow gates the lockfile through an OSV-Scanner image pinned by digest and gates every image with a commit-pinned Anchore action and explicit Grype version; High or Critical findings fail the candidate workflow. A run-scoped release-image-identity-<commit> artifact records every published digest; the Railway release downloads it from the successful exact-commit container run and rejects candidate inputs that do not match. Private GHCR images need a read-only Railway registry credential. Disable mutable-tag automatic updates. Run Railway testnet-qualified release with candidate and explicit last-known-good keeper, indexer, API, and market-data digests, the successful container workflow run ID that published all four last-known-good digests, plus the Railway project ID and exact testnet environment ID recorded in the qualification report binding. The frontend is released independently to Cloudflare and is never part of this Railway workflow. For the first deployment only, set initial_release=true and leave all previous-image and previous-container-run inputs empty. The protected preflight refuses that mode if any production runtime service has a successful deployment on a different image, so it cannot be reused as a rollback bypass. A retry may resume services already successful on the exact candidate digest. Keep public production DNS disabled until the first production gate succeeds. The workflow:
  1. verifies immutable inputs, exact-commit container CI, and contract-security CI;
  2. runs the migration and candidate on testnet;
  3. requires testnet readiness, replica topology, SLO telemetry, and a fresh signed order with a canonical mined outcome;
  4. verifies every production runtime service is still on the supplied rollback digest;
  5. pauses at the protected production GitHub environment;
  6. runs the production migration, then rolls the runtime services and repeats the SLO and signed-order gates; and
  7. restores all production runtime services to the supplied previous digests after any armed production failure, timeout, cancellation, runner loss, or evidence-upload failure.
Migrations are forward-only and are not rolled back automatically. After the initial launch, every rolling migration must use an expand/contract shape that remains compatible with both supplied runtime digests for the release window. This transient production safety invariant is not support for the removed duplicate-service configuration or any other pre-launch legacy interface. An initial release has no previous runtime to restore: a failed gate leaves public DNS disabled while operators remediate or apply a reviewed corrective migration. The workflow records deployment IDs and signed-order correlation evidence as 90-day artifacts. Create these GitHub environments:
  • testnet-qualification: restrict to protected main and require an independent reviewer who compares both external review IDs and artifact hashes with the reviewers’ authenticated systems of record before approval; hashes alone do not authenticate report authorship.
  • testnet: restrict to protected main; no approval wait.
  • production: require reviewers, prevent self-review, and restrict to protected main.
  • production-rollback: restrict to protected main, but do not add a reviewer wait that could prevent automatic recovery.
Set QUALIFICATION_REVIEWER_POLICY_JSON as an organization/repository variable, or identically in the protected testnet-qualification and testnet environments. It is the external trust root for the two independent reviews and contains separately governed reviewerIdentity and organization values for contract and operational. These are normalized and matched against the hashed reports, and the contract and operational identities must differ. Never source this policy from the evidence archive or a workflow-dispatch input. The protected testnet-qualification environment also receives a read-only/scoped RAILWAY_TOKEN so the governed workflow can verify every report-bound deployment ID and immutable image directly; do not use an operator-supplied deployment export as that trust source. Store the following environment-scoped secrets independently in both testnet and production:
Store only RAILWAY_ROLLBACK_TOKEN in production-rollback. Use Railway environment tokens, never personal or workspace tokens, and do not keep them as repository-wide secrets. Railway does not provide the per-service action allowlist needed here: the rollback token can modify service image and BUILD_VERSION within production. Compensate with branch-restricted GitHub environments, no human reuse, secret scanning, Railway audit monitoring, and immediate post-use alerts. CANARY_SIGNED_ORDER must be single-use, small, and carry a fresh session nonce and deadline. Never store a trader or session private key in GitHub. Prometheus must scrape every private replica directly. Operational series retain only bounded role and domain labels; tayho_worker_info carries build version, deployment ID, and environment. Candidate queries join those series by Prometheus instance and job labels to avoid per-deployment histogram cardinality.

​
Shared runtime variables

Use Railway reference variables for private links and seal every credential:
Production and qualified testnet releases keep RPC_ALLOW_SINGLE_ENDPOINT=false and use the write RPC plus at least two independently hosted read RPCs. A development-only deployment may explicitly set DEPLOYMENT_ENVIRONMENT=development, CHAIN_ID=998, and RPC_ALLOW_SINGLE_ENDPOINT=true; the runtime rejects that flag in every other environment or on any other chain. This degraded mode is for initial integration against the basic HyperEVM testnet RPC and must not be promoted to production.

​
Development observability plane

Deploy tayho-observability from deploy/railway/observability.json, attach one persistent Railway volume at /data, and expose only Grafana on port 3000. The pinned Grafana LGTM base image runs a Tayho-specific entrypoint containing only the OpenTelemetry Collector, Prometheus, Tempo, Loki, and Grafana for development. Pyroscope is not started or provisioned as a datasource. The collector receives OTLP/HTTP on the private port 4318, scrapes the application metrics endpoints, and loads the checked-in SLO rules and Tayho dashboard. Set a sealed, randomly generated GF_SECURITY_ADMIN_PASSWORD, disable anonymous Grafana access, and use Railway reference variables for every private hostname and metrics token; never copy their resolved values into source control. Every HTTP service receives OTEL_EXPORTER_OTLP_ENDPOINT=http://${{tayho-observability.RAILWAY_PRIVATE_DOMAIN}}:4318 and every keeper receives the equivalent signal-specific endpoints ending in /v1/traces and /v1/logs. The shared @tayho/otel-logs bridge exports redacted, trace-correlated application records; the Ponder launcher also redacts dependency output before both Railway stdout and OTLP export. Railway’s native Log Explorer remains an independent incident fallback while Loki is the correlated log backend in Grafana. The single-container LGTM topology is intentionally confined to development: qualified testnet and production must use independently scalable or managed collectors, metrics, trace, log, dashboard, and alerting backends with replicated durable storage and off-platform retention. The signerless tayho-indexer uses the separately owned tayho_indexer logical database on the environment’s primary Railway PostgreSQL service and receives only:
When using Hyperliquid’s basic public RPC in the explicitly degraded development-only single-RPC mode on HyperEVM testnet, set PONDER_RPC_REQUESTS_PER_MINUTE=60. The indexer wraps only that basic RPC in a paced transport and caps eth_getLogs ranges at 50 blocks, staying below Hyperliquid’s published limit of 100 EVM JSON-RPC requests per minute. Managed development RPCs do not inherit that default, but may opt into conservative provider-specific pacing with PONDER_RPC_REQUESTS_PER_MINUTE and PONDER_ETH_GET_LOGS_BLOCK_RANGE, plus PONDER_RPC_MAX_IN_FLIGHT to bound slow-call buildup; set PONDER_RPC_TIMEOUT_MS high enough for Ponder’s bounded internal queue to drain without expiring before a network slot is acquired. The development default is 300 seconds; the custom transport also coalesces compatible per-event and per-contract eth_getLogs calls by block range over a 100 ms microbatch, then demultiplexes the response locally so provider limits do not change event semantics. For the current development rollout, put the credential-bearing Alchemy HyperEVM testnet URL first and https://rpcs.chain.link/hyperevm/testnet second. Do not configure Tatum. One successful origin response satisfies a read; the fallback is used only when the primary is unavailable or unhealthy. The 10,000 RPM and 10,000-block values are valid only for managed development origins whose plan supports them. The basic rpc.hyperliquid-testnet.xyz endpoint retains its 100 RPM and 1,000-block hard ceilings. Production continues to require redundant origins and provider capacity sized for the indexer workload. It receives no REDIS_URL, signer keys, intent keyring, transaction-journal keyring, keeper database credential, or oracle source credential. The deployment command uses the Railway deployment ID as Ponder’s isolated schema and updates the stable tayho_indexer views schema only after backfill. Keep one writer replica. Add read-only ponder serve replicas only if measured query load later contends with indexing. Include the private indexer /ready endpoint in CANARY_READINESS_URLS and scrape its /metrics; alert on disconnects, RPC failures, reorgs, handler errors, database errors, and the completed-timestamp lag. The two stateless tayho-api replicas receive only:
Prefix Railway private domains with http://. The API receives Redis, its isolated API-control database role, and independent read-only RPC URLs for replay, cross-replica fan-out, quotes, simulation, and risk. It receives no keeper/indexer database role, signer, intent-keyring, transaction-journal keyring, receipt secret, or writable RPC credential. REALTIME_STREAM_KEY must equal ${QUEUE_PREFIX}:realtime:v1 on the engine; with the default QUEUE_PREFIX=tayho, keep the shown default. Only tayho-api receives a public API domain. Mount the same sealed ENGINE_INTERNAL_TOKEN into tayho-engine; never reuse the metrics token for it. The two signerless tayho-market-data replicas receive no database, Redis, RPC, contract address, worker signer, or journal key:
Give the service no Railway public domain. The oracle and attester reach it through the private Railway origin. Testnet mode explicitly permits the shared origin; production uses separately governed HTTPS providers and keeps independent mode. Every signing service additionally receives one sealed SIGNER_KEYS JSON object containing two named private keys, for example {"primary":"0x...","secondary":"0x..."}. Railway injects that same service variable into both replicas. Register all twelve derived production signer addresses in their narrow on-chain sets before starting workers; testnet uses twelve different addresses. Never reuse owner, treasury, or cross-role identities. The auditor and frontend receive no worker signer or journal key. The auditor uses its own restricted database login; the frontend receives no database credential. Both engine replicas receive the same versioned INTENT_KEYRING and INTENT_ACTIVE_KEY_ID, plus a shared ENGINE_RECEIPT_SECRET and the API-only ENGINE_INTERNAL_TOKEN. They receive no SIGNER_KEYS and no transaction-journal keyring. Both relayer replicas receive the same intent keyring so they can decrypt a referenced envelope only immediately before simulation/submission. The relayer has no Railway public domain; only its private /livez, /readyz, and protected /metrics listener is reachable inside the environment. The two tayho-automation replicas run the permissionless funding updater and submit only after the current observation exists. The separate tayho-oracle replicas receive two funding-only keys plus FUNDING_IMPACT_SOURCE_URL, FUNDING_INDEX_SOURCE_URL, distinct FUNDING_IMPACT_SOURCE_BEARER_TOKEN / FUNDING_INDEX_SOURCE_BEARER_TOKEN, FUNDING_SOURCE_MODE, FUNDING_SAMPLE_INTERVAL_SECONDS, and FUNDING_SOURCE_MAX_AGE_SECONDS. The separate tayho-attester replicas share a different two-key pool registered in ORACLE_ATTESTERS and receive ORACLE_DEPTH_SOURCE_URL, ORACLE_RESOLUTION_SOURCE_URL, ORACLE_ATTEST_INTERVAL_SECONDS, ORACLE_SOURCE_MAX_AGE_SECONDS, ORACLE_SOURCE_TIMEOUT_MILLISECONDS, and distinct service-scoped ORACLE_DEPTH_SOURCE_BEARER_TOKEN / ORACLE_RESOLUTION_SOURCE_BEARER_TOKEN secrets plus ORACLE_SOURCE_MODE. Credentials are sent only in Authorization headers; never place them in URLs, logs, or evidence. Each producer’s source origins must differ; production requires HTTPS. Every submitted depth/resolution fact includes the source observation and configured expiry, and the contract rejects expired, future-skewed, or non-monotonic facts. Funding, depth/resolution, owner, and treasury keys are pairwise isolated. The single tayho-auditor instance receives the configured contract addresses, three independently hosted RPC endpoints, AUDITOR_EXPECTED_SIGNERS with exactly two distinct public addresses for each signing role, AUDITOR_EXPECTED_CODE_HASHES for Core, Funding, Lens, Maintenance, Oracle, Orders, and Vault, and the expected COLLATERAL_ADDRESS. Use the bounded audit settings in apps/keeper/env/auditor.env.example; never give the auditor SIGNER_KEYS, INTENT_KEYRING, or TX_JOURNAL_KEYS. It receives its restricted PostgreSQL login and Redis URL only to compare the derived intent generation/count/checksum with PostgreSQL; it cannot decrypt executable payloads. Set AUDITOR_INDEXER_URL to the indexer’s Railway private URL. Ponder in that service is the sole consumer of eth_getLogs; the auditor reads its projection and retains independent quorum checks only for the pinned block and contract state. This removes audit backfill pressure from the auditor’s RPC pool while preserving an independent chain-state verification boundary. Operator access runs the compiled jobs command with the migration/operator database identity, a named OPERATOR_ID, and short-lived Railway shell access. Use intent-list/intent-show for redacted inspection, intent-market-pause/intent-market-resume for durable market intake controls, intent-revalidate only for funding-blocked orders, intent-index-rebuild for fenced Redis rebuilds, and intent-key-status/intent-key-retire-check for local key rotation. Every mutation appends an audit event; operators cannot create order terms, signatures, cancellation authority, or arbitrary calls. Frontend runtime variables are TAYHO_ENVIRONMENT, TAYHO_CORE_ADDRESS, TAYHO_ORDERS_ADDRESS, TAYHO_API_URL, TAYHO_API_KEY, HYPEREVM_RPC_URL, and HYPEREVM_WS_URL. TAYHO_API_URL is https://api.testnet.tayho.io in development and TAYHO_API_KEY is a publishable, origin-restricted key. The Bun server generates no-store /config.js, so the same digest works in both environments. The Elysia API receives the public origin allowlist; the engine accepts application traffic only from the API service credential.

​
Development public SDK and relayed-action rollout

This sequence is development-only. It does not publish @tayho/sdk or deploy production traffic. Record the deployment identity, migration outputs, key IDs (never key secrets), smoke result, and service deployment IDs in one rollout record.
  1. Pause public action traffic by setting API_ACTION_INTAKE_ENABLED=false. Existing receipt reads, realtime tickets, signed cancellation, queued work, and transaction reconciliation remain live.
  2. From packages/contracts, deploy the changed contracts with forge script script/DeployTestnet.s.sol:DeployTestnet --rpc-url "$ALCHEMY_HYPEREVM_TESTNET_RPC" --broadcast. Capture the complete JSON deployment identity; do not copy individual addresses between releases.
  3. Run bun run --cwd packages/protocol gen:abi, then require clean protocol ABI/action tests. Put that exact DEPLOYMENT_IDENTITY_JSON into API, indexer, engine, relayer, and every other keeper.
  4. Apply the API control migration first with bun run --cwd apps/api db:migrate. Apply keeper migrations with the schema-owner credential using bun run --cwd apps/keeper migrate. Confirm 0013_relayed_actions and the API-key migration are present before any new runtime starts.
  5. Roll the engine and relayer with the same INTENT_KEYRING, active key ID, receipt secret, transaction-journal keyring, queue prefix, deployment identity, and redundant RPC set. Never retire an encryption or journal key while an accepted action or transaction still references it.
  6. Roll the API with its dedicated control-database role, API_KEY_PEPPER, Redis, internal engine token, and the same deployment identity. Keep intake paused. Require API /livez and /readyz, engine /readyz/internal with its bearer token, relayer private readiness, indexer /ready, and all database/Redis/RPC dependency checks to pass.
  7. Create the smoke project and key from a short-lived Railway shell. Plaintext is displayed once:
  8. With intake still paused, run the repository gates and confirm the SDK rejects the smoke write with intake_paused while config, status reads, realtime ticket issuance, and cancellation remain available. Then set API_ACTION_INTAKE_ENABLED=true, roll only the API, and run bun test apps/keeper/test/public-actions-smoke.test.ts followed by a funded development-wallet smoke using the built SDK. The funded smoke must cover session authorize/revoke, market and conditional orders, order cancellation, add-margin, vault deposit/withdraw, stake/claim/unstake, receipt tracking through confirmed/indexed, and quota exhaustion in a second isolated project.
  9. Enable application traffic only after metrics show no validation rejection anomaly, queue age is draining, relayer signer nonces reconcile, transaction replacement is bounded, and indexer lag is below the development threshold.
Failure qualification must be recorded before traffic enablement: restart API, Redis, engine, and relayer independently; submit the same prepared action twice; reject an expired signature and a wrong-chain SDK RPC; revoke the smoke key; exhaust only the second project’s quota; saturate and then drain the pending queue; fail the Alchemy primary to prove Chainlink fallback; exercise a stuck-fee replacement; and delay the indexer while confirming engine receipt tracking remains available. The focused smoke, API admission, RPC pool, transaction reliability, realtime replay, and indexer config tests are the deterministic evidence for those cases; the funded smoke supplies the onchain effect.

​
Rollback and incident procedures

  • Runtime or SDK regression: set API_ACTION_INTAKE_ENABLED=false and roll the API first. Do not revoke receipt capabilities, delete action rows/jobs, purge Redis, or stop relayer reconciliation. Let journaled nonces reach confirmed, replaced, reverted, or explicitly abandoned terminal state. Restore last-known-good runtime images only after their schema compatibility is confirmed.
  • Contract/deployment mismatch: keep intake paused. Restore one complete prior DEPLOYMENT_IDENTITY_JSON across every service together with its matching ABI/runtime images, or complete a fresh coherent deployment. Never mix individual old/new addresses. Readiness must fail until chain ID, bytecode hashes, and deployment ID agree.
  • Key compromise: revoke or rotate only the affected key with the CLI and a recorded reason. Suspending a project stops all of its access, so prefer key revocation when receipt tracking for other project keys must remain available. Issue the replacement once and update origin/CIDR rules.
  • Queue saturation: keep new intake paused, preserve PostgreSQL and Redis, inspect aggregate queue age/depth and signer lanes, and scale only after verifying one canonical outbox delivery per action. Drain with the existing relayer/journal; never bulk-delete jobs or reset signer nonces.
  • RPC outage: keep Alchemy first and Chainlink second for development. Open the circuit on the failing origin, verify pinned-block chain identity, and let retryable validation fail closed. Never bypass pre-submit simulation or add Tatum as an unqualified emergency origin.
  • Indexer delay: continue serving receipt-authorized engine lifecycle up to confirmed; mark indexed only after the indexer checkpoint catches up. Do not replay eth_getLogs from API, auditor, SDK, or relayer—the indexer remains the sole log scanner.

​
PostgreSQL identity and recovery

tayho_migrator owns the schema and is used only by tayho-migration through MIGRATION_DATABASE_URL. tayho_runtime gets only non-relayer signing-worker table/sequence DML. On audit and signer history tables it receives SELECT and INSERT, never UPDATE, DELETE, or TRUNCATE; equivalent default privileges apply to later migrations. tayho_engine can accept immutable encrypted intent terms, maintain validation/reservation/index state and checkpoints, and insert only reference-only relayer execute-intent or cancel-intent jobs. It cannot read the transaction journal, create arbitrary jobs, change accepted signed terms, mutate intent events, or create schema objects. tayho_relayer can read encrypted intent envelopes, operate relayer batches and transaction journal rows, append settlement events, and update only the explicit intent/reservation settlement columns. It cannot accept intents or modify ciphertext, hashes, trader terms, or historical events. PostgreSQL grants and RLS enforce both boundaries. tayho_auditor is a separate login. It can inspect the coordination state required by protocol invariants and operate auditor jobs, findings, evidence, and the durable chain-event projection. PostgreSQL row-level policies allow it to mutate only rows whose role is auditor; signing-role jobs, outbox rows, attempts, and worker registrations remain read-only. Its transaction access is column-limited and excludes encrypted signed bytes. The migration job applies and verifies grants idempotently while holding a PostgreSQL advisory lock. Set DATABASE_PRIVILEGES_REQUIRED=true and inject RUNTIME_DATABASE_URL, AUDITOR_DATABASE_URL, ENGINE_DATABASE_URL, and RELAYER_DATABASE_URL as Railway references to the corresponding runtime service URLs. The role and its independent sealed password are read from each URL; this avoids a second manually synchronized copy of every credential. No long-running service receives MIGRATION_DATABASE_URL. Enable PostgreSQL PITR, daily and weekly backups, and connection pooling before traffic. Perform a PITR sibling-restore drill for both logical databases monthly. The coordination and Ponder databases remain separately owned durability domains even though they share one PostgreSQL service; Redis/BullMQ is reconstructable. After Redis loss, drain workers, restore Redis, allow the outbox to republish stable delivery IDs, verify idempotency, resume one replica per signing role, observe, then restore the second signing replicas. Restart the single auditor after PostgreSQL and Redis agree; its durable outbox and event checkpoint resume automatically.

​
Release order

  1. Verify review approval, digests, SBOM, provenance, exact-commit CI, and launch manifest.
  2. For a new contract deployment, complete and independently verify the EOA-to-multisig handoff in packages/contracts/README.md. Attach the deployment identity plus prepare, complete, and verify evidence to the release record. This operator-only workflow runs from a protected workstation or runner; its EOA key and handoff variables must never be installed on any Railway service.
  3. Ensure testnet and production signer sets, environment-scoped secrets, and observability targets are current.
  4. Dispatch the release workflow. For the one-time bootstrap, select initial_release with empty previous-image and previous-container-run inputs and keep production DNS disabled. It qualifies testnet before the protected production approval.
  5. The testnet job must verify candidate-bound k6-intents.js evidence (5,000 burst, at least 60 seconds at 1,000/s, duplicate storm, no drops, p99 acknowledgement below 250 ms) before protected production approval. Verify production smoke evidence links request/trace ID to intent/hash, validation/trigger, reference-only child and batch job, signer transaction lineage, and canonical per-child receipt.
  6. Record the release with OPERATOR_ID=<release-id> bun run jobs deployment-record <digest> and retain the canary artifacts in the permanent launch evidence store.
  7. Retire old signers and journal keys only after their finality and retention gates pass.
Railway health checks guard activation, not continuous health. Prometheus burn-rate alerts and the incident runbooks remain authoritative after deployment. Never use railway up for production: it uploads source and creates an artifact that is not the reviewed provenance subject.
Last modified on July 23, 2026