​

Observability and SLOs

Tayho uses Prometheus metrics, OpenTelemetry traces/logs, and a provisioned Grafana dashboard suite. The checked-in assets are executable release inputs:

• alert and recording rules: deploy/observability/prometheus-rules.yml;
• dashboards: overview, intents, execution-chain, data-plane, and auditor;
• OTLP collection: deploy/observability/otelcol-config.yml;
• scrape and datasource configuration under deploy/observability; and
• shared instrumentation in packages/http-telemetry and packages/otel-logs.

The Railway tayho-observability profile is the development/qualification plane: one pinned OTel-LGTM service with repository-provisioned immutable dashboards. Production must use a separately qualified durable metrics/logs/traces backend with equivalent rules, retention, access controls, and evidence. Do not treat one development replica as production HA.

​

Signal contract

​

Metrics

Labels are bounded enums/route templates, never wallet addresses, hashes, job IDs, request IDs, signer addresses, or error messages. Counter rates show throughput/failure. Gauges such as “oldest age,” “last success,” and queue depth show current blockage. Histograms establish user-visible and internal latency.

​

Traces

W3C trace context crosses public request, private upstream, durable job, transaction attempt, and receipt reconciliation. Start from x-request-id, find the API span, then follow the recorded trace/span identifiers on the job and transaction. A missing downstream span is itself a boundary signal; do not substitute raw signed payload logging.

​

Logs

Logs are structured OTLP records with service, role, build version, severity, request/trace correlation, and bounded outcome fields. Redaction removes credentials, RPC URL paths/query values, calldata, signatures, signed transaction bytes, encrypted payloads, and receipt capabilities.

​

Dashboard workflow

1. Start at Tayho Overview (tayho-overview) for replica presence and active alerts.
2. Use Tayho Intents for intake, validation, trigger, reservation, and settlement.
3. Use Tayho Execution Chain for queue, outbox, signer, RPC, nonce, and receipt behavior.
4. Use Tayho Data Plane for API, Ponder, market data, and realtime.
5. Use Tayho Auditor for check freshness, findings, projection, and quorum disagreement.

All dashboards refresh every 15 seconds and cross-link to the other four. UI edits are not the source of truth; change the checked-in JSON and validate it through operations-assets tests.

​

SLOs

Alert rule expressions—not prose—are authoritative. Threshold changes require an evidence-backed rule/test/dashboard/runbook change in one review.

​

Alert ownership and runbooks

Every alert has a dashboard_uid and a checked incident-response anchor. The operations asset test fails when either is missing. page means immediate on-call investigation. ticket means bounded follow-up unless correlated signals raise severity. Never silence a protocol-integrity alert solely because user traffic looks healthy.

​

Readiness versus monitoring

/livez proves a process loop is alive. /readyz (or indexer /ready) proves the dependencies and authority required to receive traffic. Readiness is deliberately stricter for chain identity, deployment linkage, credentials, Redis/PostgreSQL, signer roles, live RPC, and indexer-pump freshness. Do not turn a recurring dependency failure into green readiness by removing the check. Drain or pause the affected authority, restore the dependency, and retain receipt/status paths where the runbook explicitly permits it.

​

Evidence collection

For releases and incidents retain:

• UTC window, environment, commit and immutable image digests;
• affected service/role/replica and alert names;
• dashboard snapshots or exported queries;
• request/trace IDs and safe transaction/job hashes;
• before/after readiness and SLO state;
• operator commands and audit-event IDs; and
• recovery/rollback decision and reviewer.

Do not retain API keys, bearer values, resolved secret URLs, receipt capabilities, signatures, calldata, encrypted terms, journal ciphertext, or private key material in evidence. Sources: deploy/observability, keeper metrics, and API metrics.