Observability

In short: every step the agent took, what it cost, and how long it took — recorded as it happens and readable afterwards. When someone asks "why did it do that," this is the answer.

Every run produces a step tree: an ordered record of what the agent did, what it cost, and how long each part took. It's written to Postgres as the run happens, which is what makes it survive the process that produced it.

The step record

class StepRecord(BaseModel):
    run_id: str
    idx: int
    type: StepType        # llm | tool | guardrail | memory | hitl | eval | planning
    name: str
    input: dict           # post-redaction
    output: dict          # post-redaction
    ok: bool
    error_code: str | None
    tokens_in: int
    tokens_out: int
    cost_usd: float
    latency_ms: int

Inputs and outputs are stored post-redaction — the trace can never contain PII the model wasn't allowed to see either. That single invariant is why the live stream can be fed from step records without a second redaction pass.

Cost accounting

Per step, including prompt-cache reads, which most tooling gets wrong:

observability:
  pricing:
    claude-sonnet-4-5:
      input_per_1m: 3.00
      output_per_1m: 15.00
      cache_read_per_1m: 0.30

Cache reads are billed at a fraction of input tokens. Counting them as normal input can overstate a cached run's cost by an order of magnitude — and cached runs are exactly the ones you're likely to have a lot of.

Export

OTEL — the finished step tree exported as a span tree:

observability:
  otel_endpoint: https://otlp.example.com/v1/traces
  otel_headers:
    authorization: Bearer ${OTEL_TOKEN}

LangSmith — a first-class exporter, not an OTLP shim:

observability:
  langsmith:
    api_key: ${LANGSMITH_API_KEY}
    project: agentfast-prod

When both are configured, LangSmith wins. Export is fire-and-forget by contract — telemetry failing never takes down a run.

The canonical run record

One telemetry record per run, assembled at run_end:

curl localhost:8321/api/runs/run_abc/telemetry

Totals, per-step-type breakdowns, and the outcome — the thing to ship to a warehouse when you want to ask "what did support agents cost last week" without replaying every step.

Watching a run happen

The dashboard renders the step tree live, and streaming pushes the same steps over SSE as they're written. Same data, two latencies: the stream is a view, Postgres is the record. If they ever disagree, Postgres is right.