Troubleshoot direct OTel observability

Use this guide to verify telemetry ingestion and diagnose problems with agent telemetry sent to Agent 365 directly over OTLP. It is scoped to the direct OTel path - if you're using the Agent 365 SDK or the Microsoft OpenTelemetry Distro, see those guides instead. For wire-level limits, error codes, and silent drop conditions, see Limits and drop conditions.

Verifying ingestion

A 200 OK is not proof of ingestion. Some drop conditions return 200 with partialSuccess: null and your data simply doesn't appear (see Limits and drop conditions). Always verify your first runs:

  1. Check HTTP status. 200 → proceed. 4xx → see Common pitfalls.
  2. Parse partialSuccess. null means the API accepted everything. Anything else means at least some spans were dropped at the per-span filter.
  3. Wait ~5 minutes, then run the Defender advanced-hunting query below.
  4. No row? Use the decision tree under No data in Defender.

Defender advanced-hunting query

The canonical lookup (joining on the agent identity you sent):

let agentIdToFind = "YOUR-AGENT-APP-ID-HERE";
CloudAppEvents
| where Timestamp > ago(1d)
| where ActionType in ("InvokeAgent", "InferenceCall", "ExecuteToolBySDK", "ExecuteToolByGateway", "ExecuteToolByMCPServer")
| extend resData = parse_json(tostring(RawEventData))
| extend AgentId = resData.AgentId
| extend TargetAgentId = resData.TargetAgentId
| extend AlternateId = resData.PlatformTargetAgentId
| where AgentId == agentIdToFind or TargetAgentId == agentIdToFind or AlternateId == agentIdToFind
| project Timestamp, ActionType, resData
| order by Timestamp desc

For the full list of surfaces (Defender agent-activity views, Microsoft 365 admin center, Microsoft Purview) and what each one needs, see Where your data shows up.

No data in Defender

  • partialSuccess.rejectedSpans == totalSpans → all your spans had a bad gen_ai.operation.name. Fix: use one of invoke_agent, execute_tool, chat, output_messages (it is chat, not inference).
  • 200 with partialSuccess: null but no Defender row after 5 min → no user in the customer tenant has a Microsoft 365 E7 or Microsoft Agent 365 license assigned (Limits and drop conditions, drop condition 2). Fix: confirm at least one user in the tenant has the license assigned (not just present in the tenant); otherwise contact the Agent 365 onboarding team.
  • Spans appear but the run tree is broken / some children orphaned → missing parentSpanId, different traceId, or gen_ai.conversation.id not set on every span. Fix: review Span hierarchy and run grouping.

Common pitfalls

Symptom Most likely cause Fix
401 Unauthorized Wrong aud on token. Use 9b975845-388f-4429-889e-eab1ef63949c (or api://9b975845-...).
403 Forbidden, missing role / scope Token doesn't carry Agent365.Observability.OtelWrite. Onboard your Microsoft Entra app to the role (S2S) or scope (delegated) per Scopes and consent. For S2S, the token must be acquired with <resource>/.default.
403 Forbidden, agent identity mismatch {agentId} in URL ≠ appid / azp of token, or a span carries a gen_ai.agent.id that doesn't match the authenticated agent. The route agentId must be the appId of the calling app. For blueprint-derived identities, that's the agent identity appId, not the blueprint appId. Ensure every span's gen_ai.agent.id matches.
200 OK but partialSuccess.rejectedSpans == totalSpans All spans had a bad gen_ai.operation.name. Use one of invoke_agent, execute_tool, chat, output_messages. It is chat, not inference.
200 OK with partialSuccess: null but no data appears in Defender No user in the customer tenant has a Microsoft 365 E7 or Microsoft Agent 365 license assigned (Limits and drop conditions). Confirm at least one user in the tenant has a Microsoft 365 E7 or Microsoft Agent 365 license assigned (the SKU being present isn't enough). Verify with the KQL in Verifying ingestion; if nothing lands after 5 min, contact the Agent 365 team.
Spans appear in CloudAppEvents but the run is missing from Defender agent-activity views and from the Microsoft 365 admin center The run has no invoke_agent span. Both surfaces key off invoke_agent. Emit exactly one invoke_agent span at the root of every run; make chat / execute_tool / output_messages children of it via parentSpanId.
Run tree is broken / tool spans appear orphaned Missing parentSpanId or different traceId on child spans. See Span hierarchy and run grouping. Every non-root span sets parentSpanId and shares the run's traceId.
Tool spans show empty ChannelName / ConversationId in queries Channel / conversation not set on the tool span, and the parent invoke_agent wasn't in the same OTLP request. Set microsoft.channel.name and gen_ai.conversation.id on every span.
413 Payload Too Large Request body > 1 MB. Split the spans across multiple requests.
429 Too Many Requests Rate limit hit. Honor Retry-After: 1 and back off with jitter.
Agent appears unidentified in dashboards gen_ai.agent.id is empty or not a GUID. Use the agent's Entra appId. If the agent has no Entra registration, see Picking values.

Next steps

  • Last updated on