> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tayho.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability and SLOs

> Tayho metrics, dashboards, traces, logs, readiness, alert ownership, and evidence collection.

# Observability and SLOs

Tayho uses Prometheus metrics, OpenTelemetry traces/logs, and a provisioned Grafana dashboard suite.
The checked-in assets are executable release inputs:

* alert and recording rules: `deploy/observability/prometheus-rules.yml`;
* dashboards: `overview`, `intents`, `execution-chain`, `data-plane`, and `auditor`;
* OTLP collection: `deploy/observability/otelcol-config.yml`;
* scrape and datasource configuration under `deploy/observability`; and
* shared instrumentation in `packages/http-telemetry` and `packages/otel-logs`.

The Railway `tayho-observability` profile is the development/qualification plane: one pinned
OTel-LGTM service with repository-provisioned immutable dashboards. Production must use a separately
qualified durable metrics/logs/traces backend with equivalent rules, retention, access controls, and
evidence. Do not treat one development replica as production HA.

## Signal contract

### Metrics

Labels are bounded enums/route templates, never wallet addresses, hashes, job IDs, request IDs,
signer addresses, or error messages.

| Area            | Primary signals                                                                               |
| --------------- | --------------------------------------------------------------------------------------------- |
| API             | Request totals by route/status class, upstream outcome/latency, authentication/quota outcomes |
| Realtime        | Active connections, message outcomes, pump freshness/failures, slow-consumer delivery         |
| Jobs            | Started/completed/attempts, queue depth/age, processing age, dead jobs, outbox age            |
| Transactions    | Attempts, pending/receipt latency, oldest nonterminal age, reorgs, signer balance             |
| Intents/actions | Intake/acknowledgement, validation, index drift, trigger/settlement age, terminal outcomes    |
| Auditor         | Run status/duration, last success, open findings, projection lag, RPC disagreement            |
| Indexer         | Ponder realtime state, indexing timestamp, RPC/database errors, reorgs                        |
| Market data     | Replica info, upstream/circuit outcomes, observation validity                                 |

Counter rates show throughput/failure. Gauges such as “oldest age,” “last success,” and queue depth
show current blockage. Histograms establish user-visible and internal latency.

### Traces

W3C trace context crosses public request, private upstream, durable job, transaction attempt, and
receipt reconciliation. Start from `x-request-id`, find the API span, then follow the recorded
trace/span identifiers on the job and transaction. A missing downstream span is itself a boundary
signal; do not substitute raw signed payload logging.

### Logs

Logs are structured OTLP records with service, role, build version, severity, request/trace
correlation, and bounded outcome fields. Redaction removes credentials, RPC URL paths/query values,
calldata, signatures, signed transaction bytes, encrypted payloads, and receipt capabilities.

## Dashboard workflow

1. Start at `Tayho Overview` (`tayho-overview`) for replica presence and active alerts.
2. Use `Tayho Intents` for intake, validation, trigger, reservation, and settlement.
3. Use `Tayho Execution Chain` for queue, outbox, signer, RPC, nonce, and receipt behavior.
4. Use `Tayho Data Plane` for API, Ponder, market data, and realtime.
5. Use `Tayho Auditor` for check freshness, findings, projection, and quorum disagreement.

All dashboards refresh every 15 seconds and cross-link to the other four. UI edits are not the
source of truth; change the checked-in JSON and validate it through operations-assets tests.

## SLOs

| SLO                          | Measurement                                                  |                 Initial threshold in rules | Owner            |
| ---------------------------- | ------------------------------------------------------------ | -----------------------------------------: | ---------------- |
| Job availability             | Error outcomes / completed jobs                              |  Multi-window burn against 99.9% objective | Keeper/on-call   |
| Queue latency                | p99 durable queue age                                        |                                  2 seconds | Keeper/on-call   |
| User order latency           | p99 accepted-to-terminal                                     |                                  5 seconds | Trading/on-call  |
| Intent intake                | Rejected/overloaded/unavailable rate and acknowledgement p99 | Rule-defined burn / 250 ms acknowledgement | Engine/API       |
| Transaction finality         | Oldest nonterminal transaction                               |                                120 seconds | Relayer          |
| Automation discovery         | Time since last successful discovery                         |                                120 seconds | Automation       |
| Audit availability/integrity | Last success, missing replica, lag, disagreements/findings   |                              Rule-specific | Security/on-call |
| Indexer freshness            | Realtime state and chain-time lag                            |                              Rule-specific | Data/on-call     |
| API/realtime availability    | Replicas, request burn, upstream/pump failures               |                              Rule-specific | API/on-call      |

Alert rule expressions—not prose—are authoritative. Threshold changes require an evidence-backed
rule/test/dashboard/runbook change in one review.

## Alert ownership and runbooks

Every alert has a `dashboard_uid` and a checked incident-response anchor. The operations asset test
fails when either is missing.

| Category           | Alerts include                                                                                           | Primary owner | Runbook                                                                                                                                      |
| ------------------ | -------------------------------------------------------------------------------------------------------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| Job/queue/outbox   | Availability burns, latency, retry rate, processing stall, dead letter                                   | Keeper        | [Job failure burn](/operations/incident-response#job-failure-burn), [Queue backlog](/operations/incident-response#queue-backlog)             |
| Transaction/signer | Stuck transaction, signer balance                                                                        | Relayer       | [Stuck transaction](/operations/incident-response#stuck-transaction), [Signer balance](/operations/incident-response#signer-balance)         |
| Auditor            | Findings, stale/missing process, projection lag, RPC disagreement                                        | Security      | [Protocol audit finding](/operations/incident-response#protocol-audit-finding), [Auditor stale](/operations/incident-response#auditor-stale) |
| Engine/intents     | Missing replica, intake burn, validation/index/trigger/reconciliation lag, reservation/funding/key drift | Engine        | [Intent intake burn](/operations/incident-response#intent-intake-burn), [Intent engine lag](/operations/incident-response#intent-engine-lag) |
| Market data        | Missing replica, circuit open, upstream/observation failures                                             | Oracle/data   | [Market-data degradation](/operations/incident-response#market-data-degradation)                                                             |
| Indexer            | Missing/not realtime/lag, RPC/DB errors, reorg                                                           | Data          | [Indexer lag or failure](/operations/incident-response#indexer-lag-or-failure)                                                               |
| API                | Missing replica, burn, upstream failure                                                                  | API           | [API degradation](/operations/incident-response#api-degradation)                                                                             |
| Realtime           | Stalled/failing pump, slow consumers                                                                     | API           | [Realtime degradation](/operations/incident-response#realtime-degradation)                                                                   |

`page` means immediate on-call investigation. `ticket` means bounded follow-up unless correlated
signals raise severity. Never silence a protocol-integrity alert solely because user traffic looks
healthy.

## Readiness versus monitoring

`/livez` proves a process loop is alive. `/readyz` (or indexer `/ready`) proves the dependencies and
authority required to receive traffic. Readiness is deliberately stricter for chain identity,
deployment linkage, credentials, Redis/PostgreSQL, signer roles, live RPC, and indexer-pump
freshness.

Do not turn a recurring dependency failure into green readiness by removing the check. Drain or
pause the affected authority, restore the dependency, and retain receipt/status paths where the
runbook explicitly permits it.

## Evidence collection

For releases and incidents retain:

* UTC window, environment, commit and immutable image digests;
* affected service/role/replica and alert names;
* dashboard snapshots or exported queries;
* request/trace IDs and safe transaction/job hashes;
* before/after readiness and SLO state;
* operator commands and audit-event IDs; and
* recovery/rollback decision and reviewer.

Do not retain API keys, bearer values, resolved secret URLs, receipt capabilities, signatures,
calldata, encrypted terms, journal ciphertext, or private key material in evidence.

Sources:
[`deploy/observability`](https://github.com/Tay-Ho/app/tree/develop/deploy/observability),
[`keeper metrics`](https://github.com/Tay-Ho/app/blob/develop/apps/keeper/src/operations/metrics.ts),
and [`API metrics`](https://github.com/Tay-Ho/app/blob/develop/apps/api/src/metrics.ts).