> ## Documentation Index > Fetch the complete documentation index at: https://docs.tayho.io/llms.txt > Use this file to discover all available pages before exploring further. # Railway deployment > Deploy, qualify, roll back, and recover the Tayho Railway service topology. # Railway deployment Tayho is Railway-first and intentionally has no Terraform layer. Repository JSON profiles define the reviewed service shape. Releases use CI-built GHCR images pinned by digest; Railway must not rebuild production because that would create a different artifact from the SBOM and provenance subject. This guide is paired with the [launch evidence checklist](/operations/launch-checklist), [coordination recovery](/operations/coordination), and [observability guide](/operations/observability). ## Environment and service shape Use one Railway project with two isolated environments: `testnet` and `production`. Each environment has one PostgreSQL service containing separately owned `railway` coordination and `tayho_indexer` logical databases, plus Redis, domains, chain configuration, telemetry credentials, transaction journal keyring, and signer keys. Never reference a database, Redis instance, or signing secret across environments. Create the same services in both environments: The replica counts below are the qualified production topology. Development normally runs `tayho-api`, `tayho-engine`, and `tayho-market-data` with one replica each; temporarily raise them to two only for coordination, rolling-replacement, and failover qualification. | Railway service | Config profile | Replicas | Exposure | | ------------------- | --------------------------------- | -------: | --------------------------------------- | | `tayho-api` | `deploy/railway/api.json` | 2 | Public, signerless Elysia API | | `tayho-market-data` | `deploy/railway/market-data.json` | 2 | Private, signerless HIP-4 normalization | | `tayho-engine` | `deploy/railway/engine.json` | 2 | Private, signerless | | `tayho-relayer` | `deploy/railway/relayer.json` | 2 | Private, signer-bearing | | `tayho-automation` | `deploy/railway/automation.json` | 2 | Private | | `tayho-oracle` | `deploy/railway/oracle.json` | 2 | Private, funding-attester only | | `tayho-attester` | `deploy/railway/attester.json` | 2 | Private, depth/resolution only | | `tayho-guardian` | `deploy/railway/guardian.json` | 2 | Private | | `tayho-revenue` | `deploy/railway/revenue.json` | 1 | Private; delay-tolerant | | `tayho-indexer` | `deploy/railway/indexer.json` | 1 | Private, signerless, on-chain only | | `tayho-auditor` | `deploy/railway/auditor.json` | 1 | Private, keyless, and read-only | | `tayho-migration` | `deploy/railway/migration.json` | one-shot | Private | The frontend is built and hosted on Cloudflare, outside Railway. It receives only the public API origin. The retained frontend container profile is a local/test fallback, not a production service. Railway balances requests between replicas and performs rolling replacement within each service. The two replicas of a signing service share the same service-scoped pool of exactly two signers. A PostgreSQL-locked cursor selects an available signer and the durable nonce journal serializes each key independently. Registration and reservation both compare against the durable two-address pool, so mismatched replica secrets fail closed. Different roles and environments never share signer keys. The migration service alone receives the schema-owner credential. The profiles are checked against Railway's published schema in CI. Every Railway service must use its matching profile; there is intentionally no root `railway.json` because a root config would override the distinct commands and replica policies of every service in this shared monorepo. Production uses immutable Docker Image sources, while development may use source uploads. ```mermaid theme={null} flowchart TB GH[GitHub release workflow] --> T[Testnet environment] T --> Q[Testnet SLO and signed-order gate] Q -->|approved| P[Production environment] subgraph E[Each isolated Railway environment] FE[Cloudflare-hosted frontend] --> Q[tayho-api x2] Q --> E[tayho-engine x2] Q --> I Q --> RPC[RPC pool] Q --> RD E --> PG E --> RD E --> R[tayho-relayer x2] R --> PG[(PostgreSQL)] A[tayho-automation x2] --> RD[(Redis / BullMQ)] O[tayho-oracle x2] --> RD D[tayho-attester x2] --> RD M[tayho-market-data x2] --> H[Hyperliquid info API] O --> M D --> M G[tayho-guardian x2] --> RD V[tayho-revenue x1] --> RD I[tayho-indexer x1] --> IDB[(tayho_indexer logical database)] IDB --> PG I --> RPC[Independent read RPCs] R --> RD A --> PG O --> PG D --> PG G --> PG V --> PG U[tayho-auditor x1] --> PG U --> RD J[tayho-migration one-shot] --> PG end T -. separate data and secrets .- P ``` ## Immutable images and release gate `.github/workflows/keeper-container.yml` publishes: ```text theme={null} ghcr.io///keeper@sha256: ghcr.io///indexer@sha256: ghcr.io///api@sha256: ghcr.io///market-data@sha256: ``` All images are non-root Bun containers, use `tini`, and carry maximum-mode BuildKit provenance plus an SPDX SBOM. GitHub artifact attestations are intentionally not required. The workflow gates the lockfile through an OSV-Scanner image pinned by digest and gates every image with a commit-pinned Anchore action and explicit Grype version; High or Critical findings fail the candidate workflow. A run-scoped `release-image-identity-` artifact records every published digest; the Railway release downloads it from the successful exact-commit container run and rejects candidate inputs that do not match. Private GHCR images need a read-only Railway registry credential. Disable mutable-tag automatic updates. Run `Railway testnet-qualified release` with candidate and explicit last-known-good keeper, indexer, API, and market-data digests, the successful container workflow run ID that published all four last-known-good digests, plus the Railway project ID and exact testnet environment ID recorded in the qualification report binding. The frontend is released independently to Cloudflare and is never part of this Railway workflow. For the first deployment only, set `initial_release=true` and leave all previous-image and previous-container-run inputs empty. The protected preflight refuses that mode if any production runtime service has a successful deployment on a different image, so it cannot be reused as a rollback bypass. A retry may resume services already successful on the exact candidate digest. Keep public production DNS disabled until the first production gate succeeds. The workflow: 1. verifies immutable inputs, exact-commit container CI, and contract-security CI; 2. runs the migration and candidate on testnet; 3. requires testnet readiness, replica topology, SLO telemetry, and a fresh signed order with a canonical mined outcome; 4. verifies every production runtime service is still on the supplied rollback digest; 5. pauses at the protected `production` GitHub environment; 6. runs the production migration, then rolls the runtime services and repeats the SLO and signed-order gates; and 7. restores all production runtime services to the supplied previous digests after any armed production failure, timeout, cancellation, runner loss, or evidence-upload failure. Migrations are forward-only and are not rolled back automatically. After the initial launch, every rolling migration must use an expand/contract shape that remains compatible with both supplied runtime digests for the release window. This transient production safety invariant is not support for the removed duplicate-service configuration or any other pre-launch legacy interface. An initial release has no previous runtime to restore: a failed gate leaves public DNS disabled while operators remediate or apply a reviewed corrective migration. The workflow records deployment IDs and signed-order correlation evidence as 90-day artifacts. Create these GitHub environments: * `testnet-qualification`: restrict to protected `main` and require an independent reviewer who compares both external review IDs and artifact hashes with the reviewers' authenticated systems of record before approval; hashes alone do not authenticate report authorship. * `testnet`: restrict to protected `main`; no approval wait. * `production`: require reviewers, prevent self-review, and restrict to protected `main`. * `production-rollback`: restrict to protected `main`, but do not add a reviewer wait that could prevent automatic recovery. Set `QUALIFICATION_REVIEWER_POLICY_JSON` as an organization/repository variable, or identically in the protected `testnet-qualification` and `testnet` environments. It is the external trust root for the two independent reviews and contains separately governed `reviewerIdentity` and `organization` values for `contract` and `operational`. These are normalized and matched against the hashed reports, and the contract and operational identities must differ. Never source this policy from the evidence archive or a workflow-dispatch input. The protected `testnet-qualification` environment also receives a read-only/scoped `RAILWAY_TOKEN` so the governed workflow can verify every report-bound deployment ID and immutable image directly; do not use an operator-supplied deployment export as that trust source. Store the following environment-scoped secrets independently in both `testnet` and `production`: ```text theme={null} RAILWAY_TOKEN CANARY_READINESS_URLS PROMETHEUS_URL PROMETHEUS_BEARER_TOKEN CANARY_API_URL CANARY_API_ORIGIN CANARY_SIGNED_ORDER ``` Store only `RAILWAY_ROLLBACK_TOKEN` in `production-rollback`. Use Railway environment tokens, never personal or workspace tokens, and do not keep them as repository-wide secrets. Railway does not provide the per-service action allowlist needed here: the rollback token can modify service image and `BUILD_VERSION` within production. Compensate with branch-restricted GitHub environments, no human reuse, secret scanning, Railway audit monitoring, and immediate post-use alerts. `CANARY_SIGNED_ORDER` must be single-use, small, and carry a fresh session nonce and deadline. Never store a trader or session private key in GitHub. Prometheus must scrape every private replica directly. Operational series retain only bounded role and domain labels; `tayho_worker_info` carries build version, deployment ID, and environment. Candidate queries join those series by Prometheus `instance` and `job` labels to avoid per-deployment histogram cardinality. ## Shared runtime variables Use Railway reference variables for private links and seal every credential: ```text theme={null} DATABASE_URL= REDIS_URL=${{Redis.REDIS_URL}} INFRASTRUCTURE_REQUIRED=true DEPLOYMENT_ENVIRONMENT= RPC_ALLOW_SINGLE_ENDPOINT=false KEEPER_ROLE= PORT=8080 RPC_URL= RPC_READ_URLS= DEPLOYMENT_IDENTITY_JSON= TX_JOURNAL_KEYS={"2026-07-a":"0x..."} TX_JOURNAL_ACTIVE_KEY_ID=2026-07-a BUILD_VERSION=sha256: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=/v1/traces OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=/v1/logs OTEL_EXPORTER_OTLP_HEADERS=authorization=Bearer%20 METRICS_BEARER_TOKEN= ``` Production and qualified testnet releases keep `RPC_ALLOW_SINGLE_ENDPOINT=false` and use the write RPC plus at least two independently hosted read RPCs. A development-only deployment may explicitly set `DEPLOYMENT_ENVIRONMENT=development`, `CHAIN_ID=998`, and `RPC_ALLOW_SINGLE_ENDPOINT=true`; the runtime rejects that flag in every other environment or on any other chain. This degraded mode is for initial integration against the basic HyperEVM testnet RPC and must not be promoted to production. ## Development observability plane Deploy `tayho-observability` from `deploy/railway/observability.json`, attach one persistent Railway volume at `/data`, and expose only Grafana on port `3000`. The pinned Grafana LGTM base image runs a Tayho-specific entrypoint containing only the OpenTelemetry Collector, Prometheus, Tempo, Loki, and Grafana for development. Pyroscope is not started or provisioned as a datasource. The collector receives OTLP/HTTP on the private port `4318`, scrapes the application metrics endpoints, and loads the checked-in SLO rules and Tayho dashboard. Set a sealed, randomly generated `GF_SECURITY_ADMIN_PASSWORD`, disable anonymous Grafana access, and use Railway reference variables for every private hostname and metrics token; never copy their resolved values into source control. Every HTTP service receives `OTEL_EXPORTER_OTLP_ENDPOINT=http://${{tayho-observability.RAILWAY_PRIVATE_DOMAIN}}:4318` and every keeper receives the equivalent signal-specific endpoints ending in `/v1/traces` and `/v1/logs`. The shared `@tayho/otel-logs` bridge exports redacted, trace-correlated application records; the Ponder launcher also redacts dependency output before both Railway stdout and OTLP export. Railway's native Log Explorer remains an independent incident fallback while Loki is the correlated log backend in Grafana. The single-container LGTM topology is intentionally confined to development: qualified testnet and production must use independently scalable or managed collectors, metrics, trace, log, dashboard, and alerting backends with replicated durable storage and off-platform retention. The signerless `tayho-indexer` uses the separately owned `tayho_indexer` logical database on the environment's primary Railway PostgreSQL service and receives only: ```text theme={null} DATABASE_URL= PONDER_RPC_URLS= PONDER_WS_URL= PONDER_POLLING_INTERVAL_MS=1000 PONDER_DATABASE_POOL_MAX=20 PONDER_TELEMETRY_DISABLED=true PONDER_LOG_LEVEL=info DEPLOYMENT_IDENTITY_JSON= # Development managed-RPC profile only: PONDER_RPC_REQUESTS_PER_MINUTE=10000 PONDER_ETH_GET_LOGS_BLOCK_RANGE=10000 PONDER_RPC_MAX_IN_FLIGHT=4 PONDER_RPC_TIMEOUT_MS=300000 ``` When using Hyperliquid's basic public RPC in the explicitly degraded development-only single-RPC mode on HyperEVM testnet, set `PONDER_RPC_REQUESTS_PER_MINUTE=60`. The indexer wraps only that basic RPC in a paced transport and caps `eth_getLogs` ranges at 50 blocks, staying below Hyperliquid's published limit of 100 EVM JSON-RPC requests per minute. Managed development RPCs do not inherit that default, but may opt into conservative provider-specific pacing with `PONDER_RPC_REQUESTS_PER_MINUTE` and `PONDER_ETH_GET_LOGS_BLOCK_RANGE`, plus `PONDER_RPC_MAX_IN_FLIGHT` to bound slow-call buildup; set `PONDER_RPC_TIMEOUT_MS` high enough for Ponder's bounded internal queue to drain without expiring before a network slot is acquired. The development default is 300 seconds; the custom transport also coalesces compatible per-event and per-contract `eth_getLogs` calls by block range over a 100 ms microbatch, then demultiplexes the response locally so provider limits do not change event semantics. For the current development rollout, put the credential-bearing Alchemy HyperEVM testnet URL first and `https://rpcs.chain.link/hyperevm/testnet` second. Do not configure Tatum. One successful origin response satisfies a read; the fallback is used only when the primary is unavailable or unhealthy. The 10,000 RPM and 10,000-block values are valid only for managed development origins whose plan supports them. The basic `rpc.hyperliquid-testnet.xyz` endpoint retains its 100 RPM and 1,000-block hard ceilings. Production continues to require redundant origins and provider capacity sized for the indexer workload. It receives no `REDIS_URL`, signer keys, intent keyring, transaction-journal keyring, keeper database credential, or oracle source credential. The deployment command uses the Railway deployment ID as Ponder's isolated schema and updates the stable `tayho_indexer` views schema only after backfill. Keep one writer replica. Add read-only `ponder serve` replicas only if measured query load later contends with indexing. Include the private indexer `/ready` endpoint in `CANARY_READINESS_URLS` and scrape its `/metrics`; alert on disconnects, RPC failures, reorgs, handler errors, database errors, and the completed-timestamp lag. The two stateless `tayho-api` replicas receive only: ```text theme={null} PORT=8080 API_ALLOWED_ORIGINS=http://localhost:5173,https://app.tayho.io,https://develop.tayho.io,https://next.tayho.io,https://app-381.pages.dev,https://develop.app-381.pages.dev,https://next.app-381.pages.dev ENGINE_URL=${{tayho-engine.RAILWAY_PRIVATE_DOMAIN}} INDEXER_URL=${{tayho-indexer.RAILWAY_PRIVATE_DOMAIN}} ENGINE_INTERNAL_TOKEN= METRICS_BEARER_TOKEN= API_UPSTREAM_TIMEOUT_MS=2000 API_READ_RETRY_COUNT=1 DEPLOYMENT_IDENTITY_JSON= API_RPC_URLS= API_SIMULATION_ACCOUNT= REDIS_URL=${{Redis.REDIS_URL}} REALTIME_STREAM_KEY=tayho:realtime:v1 API_REALTIME_POLL_MS=500 API_WS_MAX_SUBSCRIPTIONS=32 API_WS_MAX_CONNECTIONS=10000 API_WS_REPLAY_LIMIT=1000 API_WS_HEARTBEAT_SECONDS=30 API_REQUESTS_PER_MINUTE=600 API_WS_CONNECTIONS_PER_MINUTE=30 API_WS_MESSAGES_PER_MINUTE=120 API_TRUSTED_PROXY_HOPS=1 API_DATABASE_URL= API_DATABASE_POOL_SIZE=4 API_KEY_PEPPER= API_KEY_POSITIVE_CACHE_SECONDS=3 API_REALTIME_TICKET_TTL_SECONDS=30 API_ACTION_INTAKE_ENABLED=true BUILD_VERSION=sha256: OTEL_EXPORTER_OTLP_ENDPOINT= ``` Prefix Railway private domains with `http://`. The API receives Redis, its isolated API-control database role, and independent read-only RPC URLs for replay, cross-replica fan-out, quotes, simulation, and risk. It receives no keeper/indexer database role, signer, intent-keyring, transaction-journal keyring, receipt secret, or writable RPC credential. `REALTIME_STREAM_KEY` must equal `${QUEUE_PREFIX}:realtime:v1` on the engine; with the default `QUEUE_PREFIX=tayho`, keep the shown default. Only `tayho-api` receives a public API domain. Mount the same sealed `ENGINE_INTERNAL_TOKEN` into `tayho-engine`; never reuse the metrics token for it. The two signerless `tayho-market-data` replicas receive no database, Redis, RPC, contract address, worker signer, or journal key: ```text theme={null} PORT=8080 BUILD_VERSION=sha256: MARKET_DATA_SOURCE_MODE=testnet HYPERLIQUID_INFO_URL=https://api.hyperliquid-testnet.xyz/info HYPERLIQUID_TIMEOUT_MILLISECONDS=5000 HYPERLIQUID_BOOK_CACHE_MILLISECONDS=500 HYPERLIQUID_META_CACHE_MILLISECONDS=30000 HYPERLIQUID_FAILURE_THRESHOLD=3 HYPERLIQUID_CIRCUIT_COOLDOWN_MILLISECONDS=10000 MARKET_DATA_IMPACT_NOTIONAL_WAD=1000000000000000000000 MARKET_DATA_DEPTH_BAND_BPS=100 MARKET_DATA_MAX_OBSERVATION_AGE_SECONDS=15 MARKET_DATA_IMPACT_TOKEN= MARKET_DATA_INDEX_TOKEN= MARKET_DATA_DEPTH_TOKEN= MARKET_DATA_RESOLUTION_TOKEN= METRICS_BEARER_TOKEN= ``` Give the service no Railway public domain. The oracle and attester reach it through the private Railway origin. Testnet mode explicitly permits the shared origin; production uses separately governed HTTPS providers and keeps independent mode. Every signing service additionally receives one sealed `SIGNER_KEYS` JSON object containing two named private keys, for example `{"primary":"0x...","secondary":"0x..."}`. Railway injects that same service variable into both replicas. Register all twelve derived production signer addresses in their narrow on-chain sets before starting workers; testnet uses twelve different addresses. Never reuse owner, treasury, or cross-role identities. The auditor and frontend receive no worker signer or journal key. The auditor uses its own restricted database login; the frontend receives no database credential. Both engine replicas receive the same versioned `INTENT_KEYRING` and `INTENT_ACTIVE_KEY_ID`, plus a shared `ENGINE_RECEIPT_SECRET` and the API-only `ENGINE_INTERNAL_TOKEN`. They receive no `SIGNER_KEYS` and no transaction-journal keyring. Both relayer replicas receive the same intent keyring so they can decrypt a referenced envelope only immediately before simulation/submission. The relayer has no Railway public domain; only its private `/livez`, `/readyz`, and protected `/metrics` listener is reachable inside the environment. The two `tayho-automation` replicas run the permissionless funding updater and submit only after the current observation exists. The separate `tayho-oracle` replicas receive two funding-only keys plus `FUNDING_IMPACT_SOURCE_URL`, `FUNDING_INDEX_SOURCE_URL`, distinct `FUNDING_IMPACT_SOURCE_BEARER_TOKEN` / `FUNDING_INDEX_SOURCE_BEARER_TOKEN`, `FUNDING_SOURCE_MODE`, `FUNDING_SAMPLE_INTERVAL_SECONDS`, and `FUNDING_SOURCE_MAX_AGE_SECONDS`. The separate `tayho-attester` replicas share a different two-key pool registered in `ORACLE_ATTESTERS` and receive `ORACLE_DEPTH_SOURCE_URL`, `ORACLE_RESOLUTION_SOURCE_URL`, `ORACLE_ATTEST_INTERVAL_SECONDS`, `ORACLE_SOURCE_MAX_AGE_SECONDS`, `ORACLE_SOURCE_TIMEOUT_MILLISECONDS`, and distinct service-scoped `ORACLE_DEPTH_SOURCE_BEARER_TOKEN` / `ORACLE_RESOLUTION_SOURCE_BEARER_TOKEN` secrets plus `ORACLE_SOURCE_MODE`. Credentials are sent only in Authorization headers; never place them in URLs, logs, or evidence. Each producer's source origins must differ; production requires HTTPS. Every submitted depth/resolution fact includes the source observation and configured expiry, and the contract rejects expired, future-skewed, or non-monotonic facts. Funding, depth/resolution, owner, and treasury keys are pairwise isolated. The single `tayho-auditor` instance receives the configured contract addresses, three independently hosted RPC endpoints, `AUDITOR_EXPECTED_SIGNERS` with exactly two distinct public addresses for each signing role, `AUDITOR_EXPECTED_CODE_HASHES` for Core, Funding, Lens, Maintenance, Oracle, Orders, and Vault, and the expected `COLLATERAL_ADDRESS`. Use the bounded audit settings in `apps/keeper/env/auditor.env.example`; never give the auditor `SIGNER_KEYS`, `INTENT_KEYRING`, or `TX_JOURNAL_KEYS`. It receives its restricted PostgreSQL login and Redis URL only to compare the derived intent generation/count/checksum with PostgreSQL; it cannot decrypt executable payloads. Set `AUDITOR_INDEXER_URL` to the indexer's Railway private URL. Ponder in that service is the sole consumer of `eth_getLogs`; the auditor reads its projection and retains independent quorum checks only for the pinned block and contract state. This removes audit backfill pressure from the auditor's RPC pool while preserving an independent chain-state verification boundary. Operator access runs the compiled `jobs` command with the migration/operator database identity, a named `OPERATOR_ID`, and short-lived Railway shell access. Use `intent-list`/`intent-show` for redacted inspection, `intent-market-pause`/`intent-market-resume` for durable market intake controls, `intent-revalidate` only for funding-blocked orders, `intent-index-rebuild` for fenced Redis rebuilds, and `intent-key-status`/`intent-key-retire-check` for local key rotation. Every mutation appends an audit event; operators cannot create order terms, signatures, cancellation authority, or arbitrary calls. Frontend runtime variables are `TAYHO_ENVIRONMENT`, `TAYHO_CORE_ADDRESS`, `TAYHO_ORDERS_ADDRESS`, `TAYHO_API_URL`, `TAYHO_API_KEY`, `HYPEREVM_RPC_URL`, and `HYPEREVM_WS_URL`. `TAYHO_API_URL` is `https://api.testnet.tayho.io` in development and `TAYHO_API_KEY` is a publishable, origin-restricted key. The Bun server generates no-store `/config.js`, so the same digest works in both environments. The Elysia API receives the public origin allowlist; the engine accepts application traffic only from the API service credential. ## Development public SDK and relayed-action rollout This sequence is development-only. It does not publish `@tayho/sdk` or deploy production traffic. Record the deployment identity, migration outputs, key IDs (never key secrets), smoke result, and service deployment IDs in one rollout record. 1. Pause public action traffic by setting `API_ACTION_INTAKE_ENABLED=false`. Existing receipt reads, realtime tickets, signed cancellation, queued work, and transaction reconciliation remain live. 2. From `packages/contracts`, deploy the changed contracts with `forge script script/DeployTestnet.s.sol:DeployTestnet --rpc-url "$ALCHEMY_HYPEREVM_TESTNET_RPC" --broadcast`. Capture the complete JSON deployment identity; do not copy individual addresses between releases. 3. Run `bun run --cwd packages/protocol gen:abi`, then require clean protocol ABI/action tests. Put that exact `DEPLOYMENT_IDENTITY_JSON` into API, indexer, engine, relayer, and every other keeper. 4. Apply the API control migration first with `bun run --cwd apps/api db:migrate`. Apply keeper migrations with the schema-owner credential using `bun run --cwd apps/keeper migrate`. Confirm `0013_relayed_actions` and the API-key migration are present before any new runtime starts. 5. Roll the engine and relayer with the same `INTENT_KEYRING`, active key ID, receipt secret, transaction-journal keyring, queue prefix, deployment identity, and redundant RPC set. Never retire an encryption or journal key while an accepted action or transaction still references it. 6. Roll the API with its dedicated control-database role, `API_KEY_PEPPER`, Redis, internal engine token, and the same deployment identity. Keep intake paused. Require API `/livez` and `/readyz`, engine `/readyz/internal` with its bearer token, relayer private readiness, indexer `/ready`, and all database/Redis/RPC dependency checks to pass. 7. Create the smoke project and key from a short-lived Railway shell. Plaintext is displayed once: ```bash theme={null} bun run --cwd apps/api api-keys -- project create \ --name development-sdk-smoke --environment development \ --read-rpm 600 --live-rpm 120 --realtime-rpm 30 --submission-rpm 60 \ --pending-actions 25 --per-transaction-gas 1500000 \ --daily-sponsored-gas 20000000 --actor "$TAYHO_OPERATOR" bun run --cwd apps/api api-keys -- key create \ --project "$PROJECT_ID" --kind publishable \ --scopes read,realtime,actions:submit \ --origins https://development-app.example --actor "$TAYHO_OPERATOR" ``` 8. With intake still paused, run the repository gates and confirm the SDK rejects the smoke write with `intake_paused` while config, status reads, realtime ticket issuance, and cancellation remain available. Then set `API_ACTION_INTAKE_ENABLED=true`, roll only the API, and run `bun test apps/keeper/test/public-actions-smoke.test.ts` followed by a funded development-wallet smoke using the built SDK. The funded smoke must cover session authorize/revoke, market and conditional orders, order cancellation, add-margin, vault deposit/withdraw, stake/claim/unstake, receipt tracking through confirmed/indexed, and quota exhaustion in a second isolated project. 9. Enable application traffic only after metrics show no validation rejection anomaly, queue age is draining, relayer signer nonces reconcile, transaction replacement is bounded, and indexer lag is below the development threshold. Failure qualification must be recorded before traffic enablement: restart API, Redis, engine, and relayer independently; submit the same prepared action twice; reject an expired signature and a wrong-chain SDK RPC; revoke the smoke key; exhaust only the second project's quota; saturate and then drain the pending queue; fail the Alchemy primary to prove Chainlink fallback; exercise a stuck-fee replacement; and delay the indexer while confirming engine receipt tracking remains available. The focused smoke, API admission, RPC pool, transaction reliability, realtime replay, and indexer config tests are the deterministic evidence for those cases; the funded smoke supplies the onchain effect. ### Rollback and incident procedures * **Runtime or SDK regression:** set `API_ACTION_INTAKE_ENABLED=false` and roll the API first. Do not revoke receipt capabilities, delete action rows/jobs, purge Redis, or stop relayer reconciliation. Let journaled nonces reach confirmed, replaced, reverted, or explicitly abandoned terminal state. Restore last-known-good runtime images only after their schema compatibility is confirmed. * **Contract/deployment mismatch:** keep intake paused. Restore one complete prior `DEPLOYMENT_IDENTITY_JSON` across every service together with its matching ABI/runtime images, or complete a fresh coherent deployment. Never mix individual old/new addresses. Readiness must fail until chain ID, bytecode hashes, and deployment ID agree. * **Key compromise:** revoke or rotate only the affected key with the CLI and a recorded reason. Suspending a project stops all of its access, so prefer key revocation when receipt tracking for other project keys must remain available. Issue the replacement once and update origin/CIDR rules. * **Queue saturation:** keep new intake paused, preserve PostgreSQL and Redis, inspect aggregate queue age/depth and signer lanes, and scale only after verifying one canonical outbox delivery per action. Drain with the existing relayer/journal; never bulk-delete jobs or reset signer nonces. * **RPC outage:** keep Alchemy first and Chainlink second for development. Open the circuit on the failing origin, verify pinned-block chain identity, and let retryable validation fail closed. Never bypass pre-submit simulation or add Tatum as an unqualified emergency origin. * **Indexer delay:** continue serving receipt-authorized engine lifecycle up to `confirmed`; mark `indexed` only after the indexer checkpoint catches up. Do not replay `eth_getLogs` from API, auditor, SDK, or relayer—the indexer remains the sole log scanner. ## PostgreSQL identity and recovery `tayho_migrator` owns the schema and is used only by `tayho-migration` through `MIGRATION_DATABASE_URL`. `tayho_runtime` gets only non-relayer signing-worker table/sequence DML. On audit and signer history tables it receives `SELECT` and `INSERT`, never `UPDATE`, `DELETE`, or `TRUNCATE`; equivalent default privileges apply to later migrations. `tayho_engine` can accept immutable encrypted intent terms, maintain validation/reservation/index state and checkpoints, and insert only reference-only relayer `execute-intent` or `cancel-intent` jobs. It cannot read the transaction journal, create arbitrary jobs, change accepted signed terms, mutate intent events, or create schema objects. `tayho_relayer` can read encrypted intent envelopes, operate relayer batches and transaction journal rows, append settlement events, and update only the explicit intent/reservation settlement columns. It cannot accept intents or modify ciphertext, hashes, trader terms, or historical events. PostgreSQL grants and RLS enforce both boundaries. `tayho_auditor` is a separate login. It can inspect the coordination state required by protocol invariants and operate auditor jobs, findings, evidence, and the durable chain-event projection. PostgreSQL row-level policies allow it to mutate only rows whose role is `auditor`; signing-role jobs, outbox rows, attempts, and worker registrations remain read-only. Its transaction access is column-limited and excludes encrypted signed bytes. The migration job applies and verifies grants idempotently while holding a PostgreSQL advisory lock. Set `DATABASE_PRIVILEGES_REQUIRED=true` and inject `RUNTIME_DATABASE_URL`, `AUDITOR_DATABASE_URL`, `ENGINE_DATABASE_URL`, and `RELAYER_DATABASE_URL` as Railway references to the corresponding runtime service URLs. The role and its independent sealed password are read from each URL; this avoids a second manually synchronized copy of every credential. No long-running service receives `MIGRATION_DATABASE_URL`. Enable PostgreSQL PITR, daily and weekly backups, and connection pooling before traffic. Perform a PITR sibling-restore drill for both logical databases monthly. The coordination and Ponder databases remain separately owned durability domains even though they share one PostgreSQL service; Redis/BullMQ is reconstructable. After Redis loss, drain workers, restore Redis, allow the outbox to republish stable delivery IDs, verify idempotency, resume one replica per signing role, observe, then restore the second signing replicas. Restart the single auditor after PostgreSQL and Redis agree; its durable outbox and event checkpoint resume automatically. ## Release order 1. Verify review approval, digests, SBOM, provenance, exact-commit CI, and launch manifest. 2. For a new contract deployment, complete and independently verify the EOA-to-multisig handoff in `packages/contracts/README.md`. Attach the deployment identity plus prepare, complete, and verify evidence to the release record. This operator-only workflow runs from a protected workstation or runner; its EOA key and handoff variables must never be installed on any Railway service. 3. Ensure testnet and production signer sets, environment-scoped secrets, and observability targets are current. 4. Dispatch the release workflow. For the one-time bootstrap, select `initial_release` with empty previous-image and previous-container-run inputs and keep production DNS disabled. It qualifies testnet before the protected production approval. 5. The testnet job must verify candidate-bound `k6-intents.js` evidence (5,000 burst, at least 60 seconds at 1,000/s, duplicate storm, no drops, p99 acknowledgement below 250 ms) before protected production approval. Verify production smoke evidence links request/trace ID to intent/hash, validation/trigger, reference-only child and batch job, signer transaction lineage, and canonical per-child receipt. 6. Record the release with `OPERATOR_ID= bun run jobs deployment-record ` and retain the canary artifacts in the permanent launch evidence store. 7. Retire old signers and journal keys only after their finality and retention gates pass. Railway health checks guard activation, not continuous health. Prometheus burn-rate alerts and the incident runbooks remain authoritative after deployment. Never use `railway up` for production: it uploads source and creates an artifact that is not the reviewed provenance subject.