Trace a single turn end to end

Open the trace view for a slow or unexpected turn. Find the slowest span. Confirm the cost vs the time taken.

The goal

A clear answer to "why did this turn take 12 seconds and cost 3 cents?". The trace shows you exactly which spans took the time and which model calls happened.

Steps

1. Find the turn.

   Chat panel -> click the timestamp on a message -> "View trace". Or navigate directly: /agents/{agentId}/trace/{threadId} and scroll to the turn.

2. Read the waterfall.

   Top span: the turn root (turn.run). Children:

   • prompt.assemble (with cache-hit attributes).
   • model.call (with costCents, token counts, finish reason).
   • tool.call.start -> tool.call.execute -> tool.call.result (one per tool call).
   • memory.recall and memory.write if the agent uses them.

3. Find the slowest.

   Sort by duration. The slowest span is usually one of:

   • A long tool call (entity backend was slow).
   • A long model call (large prompt or large output).
   • A retrieval round-trip (slow embedding + search).

4. Drill in.

   Click the slow span. Attributes show what happened. For tool calls, link out to the audit log entry to see decrypted args.

5. Cross-link to the run.

   If the turn spawned a BGO, the trace has a bgo.spawn span. Click it; jump to the Runs page for the underlying engine run.

Verify

• Sum of model.call costCents matches the cost in the Costs view (small drift acceptable).
• Total turn duration matches what you saw in the chat panel.

Common findings

• Many small model calls vs one big one usually means a sub-agent loop. Inspect the agent's subAgentConfig.
• Repeated memory.recall spans mean the model is calling recall multiple times per turn. Tune the recall block in Prompts.
• A tool with intermittent slow times shows as a wide variance in duration; instrument inside the entity backend.

Next steps

• Debug a sudden cost spike.
• Recover a stuck run.