---
title: Telemetry
description: "Trace A2 with OpenTelemetry, and the spans it reports."
---

```ts server/orders.ts
import { createServer } from 'experimental-a2/server'
import { otel } from 'experimental-a2/otel'
import { orders } from '@/contracts'

export const ordersServer = createServer({
  contract: orders,
  telemetry: otel(),
  // ...store, scheduler, and consumers
})
```

With any OpenTelemetry setup registered (on Vercel, `@vercel/otel`), one trace
shows the request, `a2.append`, the inline `a2.drain`, an `a2.event` span per
consumer run, and the appends those runs make. Failures mark their `a2.event`
span, and `a2.event.outcome = dead_lettered` is the attribute to alert on.
Agent servers add `a2.ai.*` spans.

## `experimental-a2/otel`

```ts
otel(options?: { tracer?: Tracer; tracerName?: string }): A2Telemetry
```

OpenTelemetry instrumentation for a server (`@opentelemetry/api` is a peer
dependency). Pass it as `telemetry`. Spans nest through the active context, so
the happy path lands in one trace. `A2Telemetry` is a one-method interface:
anything with `span(name, attributes, fn)` works, and `otel()` is the adapter
A2 ships.

## The span catalogue

This catalogue is a contract: renames and additions are breaking for
dashboards and are called out in the package's `CHANGELOG.md`. All spans carry
`a2.contract` and `a2.session_id`. Core spans describe the durable runtime;
`a2.ai.*` spans describe the AI runtime's consumers.

| Layer | Span | Wraps |
| ----- | ---- | ----- |
| Core | `a2.append` | validation + the store write (one batch) |
| Core | `a2.drain` | one drain pass, including the consumer runs it starts |
| Core | `a2.event` | one claimed head, including the consumer's run and its completion |
| Core | `a2.claim` | one `claimAvailable` store operation |
| Core | `a2.commit` | one `completeAttempt` or `failAttempt` store operation |
| Core | `a2.state` | one `state()` read: snapshot-plus-tail load + reducer fold |
| Core | `a2.snapshot` | one background batch of checkpoint writes |
| Core | `a2.redrive` | one `session.redrive()`: the store operation and starting the session's wakeup |
| Core | `a2.consume` | one `session.consume()`, including the drains it runs and the waits between them |
| AI | `a2.ai.control` | applying an input/work report/session-close decision and preparing its returned events |
| AI | `a2.ai.generate` | one generation run, including Core calls and model/SDK work |
| AI | `a2.ai.context` | constructing model messages, including tool prompt reconstruction when uncached |
| AI | `a2.ai.compaction` | automatic summarization or the custom compaction callback |
| AI | `a2.ai.tool` | one durable tool run, including application execution |
| AI | `a2.ai.ack` | waiting for a persisted work report's canonical events and control receipt |

Durations include nested work: `a2.event` contains its AI run and later
commit, AI runs call Core, and `a2.state` includes reducer code.
Acknowledgements overlap control processing, claims can cover several events,
and snapshots run in the background, so summing spans, or subtracting summed
children, does not yield exclusive Core or AI overhead. Model and application
I/O is included; its child spans have the detail.

Attributes arrive in two waves: **at start** (passed to `span()`) and
**mid-span** (via `handle.setAttribute`, for outcomes and counts known only
after the work ran). A recorder must merge both; the ones you alert on are all
mid-span.

| Attribute             | Span        | When  | Values                                                |
| --------------------- | ----------- | ----- | ----------------------------------------------------- |
| `a2.append.source`    | `a2.append` | start | `external` \| `consumer`                               |
| `a2.append.mode`      | `a2.append` | start | `inline` \| `dispatch`                                |
| `a2.append.types`     | `a2.append` | start | comma-joined event types                              |
| `a2.append.count`     | `a2.append` | start | batch size                                            |
| `a2.append.armed`     | `a2.append` | mid   | `false` when the scheduler arm failed and this append degraded to append-driven healing |
| `a2.drain.outcome`    | `a2.drain`  | mid   | `settled` \| `busy` \| `stalled` \| `failed`          |
| `a2.drain.processed`  | `a2.drain`  | mid   | events processed this pass                            |
| `a2.claim.outcome` | `a2.claim` | mid | `claimed` \| `busy` \| `settled` |
| `a2.claim.count` | `a2.claim` | mid | number of claimed events, or zero |
| `a2.commit.operation` | `a2.commit` | start | `complete` \| `fail` |
| `a2.commit.index` | `a2.commit` | start | event log position |
| `a2.commit.attempt` | `a2.commit` | start | durable 1-based attempt ordinal |
| `a2.commit.outcome` | `a2.commit` | mid | `completed` \| `failed` \| `dead_lettered` \| `superseded` |
| `a2.event.type`       | `a2.event`  | start | the event's type                                      |
| `a2.event.index`      | `a2.event`  | start | log position                                          |
| `a2.event.id`         | `a2.event`  | start | event id                                              |
| `a2.event.consumer`   | `a2.event`  | start | the consumer running the event                        |
| `a2.event.attempt`    | `a2.event`  | start | same durable 1-based ordinal as `ctx.attempt`          |
| `a2.event.lane`       | `a2.event`  | start | stored lane value; absent for concurrent unlaned work  |
| `a2.event.handled`    | `a2.event`  | start | normally `true`; `false` when a claimed head's consumer is not registered |
| `a2.event.age_ms`    | `a2.event`  | start | milliseconds since the event was appended, when its run starts: how far behind its consumer is in time |
| `a2.event.lag`       | `a2.event`  | start | events appended after this one when it was claimed: how far behind its consumer is in the log. Absent when the store doesn't report its head |
| `a2.event.outcome`    | `a2.event`  | mid   | `processed` \| `failed` \| `dead_lettered` \| `superseded` \| `surrendered` |
| `a2.event.aborted`    | `a2.event`  | mid   | `true` when `abortOn` fired during the run            |
| `a2.state.reducer`    | `a2.state`  | start | the reducer's name                                    |
| `a2.state.snapshot`   | `a2.state`  | mid   | `hit` \| `miss` \| `rejected` (schema guard discarded it) \| `memory` (served from the server's in-memory state) |
| `a2.state.folded`     | `a2.state`  | mid   | events folded past the snapshot                       |
| `a2.state.index`      | `a2.state`  | mid   | the frontier the returned state reflects              |
| `a2.snapshot.reducer` | `a2.snapshot` | start | the reducer's name                                  |
| `a2.snapshot.index`   | `a2.snapshot` | start | greatest checkpoint index in the write batch       |
| `a2.snapshot.count`   | `a2.snapshot` | start | number of checkpoints in the write batch           |
| `a2.consume.consumer` | `a2.consume` | start | the consumer being consumed |
| `a2.consume.through` | `a2.consume` | mid | the index it runs through |
| `a2.consume.index` | `a2.consume` | mid | the index it handled every event through |
| `a2.consume.drains` | `a2.consume` | mid | drains it ran to get there |
| `a2.redrive.index` | `a2.redrive` | start | log position of the event being redriven |
| `a2.redrive.consumer` | `a2.redrive` | start | the consumer the event is dead for |
| `a2.redrive.action` | `a2.redrive` | start | `retry` \| `skip` |
| `a2.redrive.redriven` | `a2.redrive` | mid | `true` when the event was dead-lettered and is now resolved |
| `a2.redrive.armed` | `a2.redrive` | mid | `false` when the scheduler arm failed and the wakeup degraded to append-driven healing |
| `a2.ai.control.type` | `a2.ai.control` | start | `ai.control.requested` \| `ai.work.reported` \| `ai.session.closed` |
| `a2.ai.control.id` | `a2.ai.control` | start | triggering event id |
| `a2.ai.control.attempt` | `a2.ai.control` | start | durable 1-based attempt ordinal |
| `a2.ai.generate.request_id` | `a2.ai.generate` | start | generation request event id |
| `a2.ai.generate.attempt` | `a2.ai.generate` | start | durable 1-based attempt ordinal |
| `a2.ai.context.request_id` | `a2.ai.context` | start | generation request event id |
| `a2.ai.context.phase` | `a2.ai.context` | start | `generation` \| `refresh` \| `tool` |
| `a2.ai.compaction.request_id` | `a2.ai.compaction` | start | generation request event id |
| `a2.ai.compaction.generation_id` | `a2.ai.compaction` | start | generation attempt id |
| `a2.ai.compaction.policy` | `a2.ai.compaction` | start | `automatic` \| `custom` |
| `a2.ai.tool.request_id` | `a2.ai.tool` | start | originating generation request event id |
| `a2.ai.tool.call_id` | `a2.ai.tool` | start | tool call id |
| `a2.ai.tool.name` | `a2.ai.tool` | start | tool name |
| `a2.ai.tool.attempt` | `a2.ai.tool` | start | durable 1-based attempt ordinal |
| `a2.ai.ack.request_id` | `a2.ai.ack` | start | reporting worker event id |
| `a2.ai.ack.report_id` | `a2.ai.ack` | start | work report event id, matching the control span's id |

- Consumer lag: group `a2.event` spans by `a2.event.consumer`. `a2.event.lag`
  is how many events the session had appended after the one it claimed, and
  `a2.event.age_ms` how old that event was when its run started. A consumer
  that keeps up keeps both near zero; a rising `a2.event.lag` means it is
  falling behind its log.
- Drain outcomes: `settled` means nothing actionable is left; `busy` means live
  per-event claims or a failed event's retry hold remain; `stalled` means a
  caught failure in this drain needs a later retry. `failed` appears only on
  scheduler deliveries: the drain threw after arming its successor watchdog,
  the span records the error, and that successor retries the session.
- A failing consumer run marks its `a2.event` span with the exception and
  error status. Alert on `a2.event.outcome = dead_lettered`. A dead-lettered event
  blocks only later events in its lane, until `session.redrive()` resolves it;
  other lanes stay eligible.
- `superseded` means a stale completion or failure did not change the log.