Integrate agent observability using direct OTel

This guide walks you end-to-end through sending agent telemetry to Agent 365 directly over OpenTelemetry (OTLP/HTTP+JSON). Before starting, read Agent 365 observability concepts to understand the model, the auth flows, and the surfaces your data lands in.

Prerequisites

Ensure the following configurations are in place before any telemetry flows.

Authentication recipes

All four recipes use the standard Microsoft Entra token endpoint:

The recipes below show raw HTTP for clarity. In production, prefer Microsoft.Identity.Web or another MSAL library, which handles token refresh and caching.

Which recipe do I need?

S2S, Standard Microsoft Entra app

One POST to the tenant's token endpoint with grant_type=client_credentials. Authenticate the app by using a client secret, a certificate (signed JWT assertion), or a managed identity or federated credential.

POST https://login.microsoftonline.com/{your-tenant-id}/oauth2/v2.0/token
Content-Type: application/x-www-form-urlencoded

client_id={your-app-id}
&scope=9b975845-388f-4429-889e-eab1ef63949c%2F.default
&client_secret={secret}
&grant_type=client_credentials

The returned token has appid/azp = {your-app-id}, roles containing Agent365.Observability.OtelWrite, and aud = 9b975845-.... Use it on the /observabilityService/.../traces route.

For certificate-based authentication, replace client_secret={secret} with client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer&client_assertion={signed-jwt}.

S2S, Blueprint-derived agent identity

Agent identities don't have credentials of their own. The agent identity blueprint holds the credentials (managed identity FIC, certificate, or client secret) and mints tokens on behalf of its child agent identities through a two-step exchange. For more information, see autonomous app OAuth flow.

1. The blueprint authenticates and gets a federated identity exchange token T1:

   • {blueprint-credential} is the blueprint's MSI token, certificate-signed JWT, or secret exchange-token assertion - per blueprint configuration.

   POST https://login.microsoftonline.com/{your-tenant-id}/oauth2/v2.0/token
   Content-Type: application/x-www-form-urlencoded
   
   client_id={blueprint-app-id}
   &scope=api%3A%2F%2FAzureADTokenExchange%2F.default
   &fmi_path={agent-identity-app-id}
   &client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer
   &client_assertion={blueprint-credential}
   &grant_type=client_credentials
   

2. The agent identity exchanges T1 for the Agent 365 Observability resource token:

   POST https://login.microsoftonline.com/{your-tenant-id}/oauth2/v2.0/token
   Content-Type: application/x-www-form-urlencoded
   
   client_id={agent-identity-app-id}
   &scope=9b975845-388f-4429-889e-eab1ef63949c%2F.default
   &client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer
   &client_assertion={T1}
   &grant_type=client_credentials
   

   • The returned token has appid/azp = {agent-identity-app-id}, roles containing Agent365.Observability.OtelWrite, and aud = 9b975845-....
   • Use this token on the /observabilityService/.../traces route.
   • The URL {agentId} is the agent identity appId, not the blueprint appId.

OBO, Standard Microsoft Entra app

Receive the user's incoming token Tc from your upstream caller (Bearer or PFAT), then exchange it:

POST https://login.microsoftonline.com/{your-tenant-id}/oauth2/v2.0/token
Content-Type: application/x-www-form-urlencoded

client_id={your-app-id}
&scope=9b975845-388f-4429-889e-eab1ef63949c%2FAgent365.Observability.OtelWrite
&client_secret={secret}
&grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer
&assertion={Tc}
&requested_token_use=on_behalf_of

For certificate auth, replace client_secret={secret} with the same client_assertion_type + client_assertion pair as in S2S.

The returned token has appid/azp = {your-app-id}, scp containing Agent365.Observability.OtelWrite, and aud = 9b975845-.... Use it on the /observability/.../traces route. A refresh token is returned alongside; cache and reuse it instead of rerunning the exchange on every call.

OBO, Blueprint-derived agent identity (including AI teammate)

There are three main steps to the on-behalf-of flow. For more information, see Agent OAuth flows: On behalf of flow.

1. Receive the user token Tc. For an AI teammate, this token represents the agent's own user account; otherwise, it represents the human caller.

2. The blueprint authenticates and gets T1, the same as the S2S blueprint-derived agent identity flow.

3. The agent identity exchanges T1 and Tc for a delegated resource token:

   POST https://login.microsoftonline.com/{your-tenant-id}/oauth2/v2.0/token
   Content-Type: application/x-www-form-urlencoded
   
   client_id={agent-identity-app-id}
   &scope=9b975845-388f-4429-889e-eab1ef63949c%2FAgent365.Observability.OtelWrite
   &client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer
   &client_assertion={T1}
   &grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer
   &assertion={Tc}
   &requested_token_use=on_behalf_of
   

The returned token has appid/azp = {agent-identity-app-id}, scp containing Agent365.Observability.OtelWrite, and represents the agent's user. Use it on the /observability/.../traces route. The URL {agentId} is the agent identity appId, not the blueprint appId. A refresh token is returned alongside; cache and reuse it.

Required claims on the returned token

S2S route (/observabilityService/...) - app-only token:

Delegated route (/observability/...) - user-delegated token (Bearer or PFAT):

The delegated route accepts both Bearer and MSAuth1.0 PFAT tokens. Direct callers should use Bearer. If you don't know which one you have, use Bearer.

Endpoints

Two routes; pick by how your service authenticates, not by what the user is doing:

POST https://agent365.svc.cloud.microsoft/observabilityService/tenants/{tenantId}/otlp/agents/{agentId}/traces?api-version=1   # S2S
POST https://agent365.svc.cloud.microsoft/observability/tenants/{tenantId}/otlp/agents/{agentId}/traces?api-version=1          # OBO

Headers:

Authorization: Bearer <token>      # or MSAuth1.0 ... for delegated PFAT
Content-Type: application/json

URL parameters

• {tenantId} - the customer tenant GUID. The server treats this as authoritative; if your spans set microsoft.tenant.id and it disagrees, the request is rejected.
• {agentId} - the calling application's appId (also the OAuth client_id). For blueprint-derived identities, this is the agent identity appId, not the blueprint appId. Must equal the appid / azp claim of your token.
• api-version=1 - required.

Request body encoding

The body is the standard OTLP/HTTP+JSON shape: an ExportTraceServiceRequest with resourceSpans → scopeSpans → spans. Keep in mind the following details:

• traceId (16 bytes) and spanId (8 bytes) are sent as lowercase hex strings.
• startTimeUnixNano / endTimeUnixNano are strings holding Unix epoch nanoseconds.
• kind is the integer OTLP enum value (for example 1 for INTERNAL); status.code is the integer enum (for example 1 for OK, 2 for ERROR).
• All attribute values are sent as stringValue.

Response shape

A successful call returns 200 OK:

{ "partialSuccess": null }

If some spans were rejected by the per-span filter:

{
  "partialSuccess": {
    "rejectedSpans": 2,
    "errorMessage": "Dropped 2 non-A365 span(s) ..."
  }
}

Field names are camelCase on the wire. Always check partialSuccess: a 200 with all your spans rejected is a real outcome you must surface. Limits and drop conditions lists the silent drop cases where a 200 returns with partialSuccess: null despite no data appearing downstream.

Smallest possible request

The simplest end-to-end test sends a single invoke_agent span. This span is the smallest body that lands in Microsoft Defender.

Step 1. Get a Bearer token. For S2S, use client credentials with scope 9b975845-388f-4429-889e-eab1ef63949c/.default (see Authentication recipes for the full recipe).

Step 2. POST a single span:

TOKEN="$(./get-token.sh)"
TENANT_ID="<customer-tenant-guid>"
AGENT_ID="<your-agent-app-id>"

curl -i -X POST \
  "https://agent365.svc.cloud.microsoft/observabilityService/tenants/${TENANT_ID}/otlp/agents/${AGENT_ID}/traces?api-version=1" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  --data @- <<EOF
{
  "resourceSpans": [{
    "scopeSpans": [{
      "scope": { "name": "my-instrumentation", "version": "1.0.0" },
      "spans": [{
        "traceId": "0102030405060708090a0b0c0d0e0f10",
        "spanId":  "1111111111111111",
        "parentSpanId": "",
        "name": "invoke_agent",
        "kind": 1,
        "startTimeUnixNano": "1736175600000000000",
        "endTimeUnixNano":   "1736175601500000000",
        "status": { "code": 1 },
        "attributes": [
          { "key": "gen_ai.operation.name", "value": { "stringValue": "invoke_agent" } },
          { "key": "gen_ai.agent.id",       "value": { "stringValue": "${AGENT_ID}" } },
          { "key": "gen_ai.agent.name",     "value": { "stringValue": "MyAgent" } },
          { "key": "microsoft.a365.agent.blueprint.id", "value": { "stringValue": "${AGENT_ID}" } },
          { "key": "gen_ai.conversation.id","value": { "stringValue": "conv-001" } },
          { "key": "microsoft.channel.name","value": { "stringValue": "web" } },
          { "key": "user.id",               "value": { "stringValue": "<entra-user-objectid>" } },
          { "key": "client.address",        "value": { "stringValue": "10.1.2.80" } },
          { "key": "server.address",        "value": { "stringValue": "myagent.example.com" } },
          { "key": "server.port",           "value": { "stringValue": "443" } },
          { "key": "gen_ai.input.messages", "value": { "stringValue": "[{\"role\":\"user\",\"content\":\"hi\"}]" } },
          { "key": "gen_ai.output.messages","value": { "stringValue": "[{\"role\":\"assistant\",\"content\":\"hello\"}]" } }
        ]
      }]
    }]
  }]
}
EOF

Step 3. Expect 200 OK with this body:

{ "partialSuccess": null }

Step 4. Confirm the data actually landed. A 200 OK is not proof of ingestion; Verifying ingestion walks through the verification flow. To POST a saved body file instead, replace --data @- <<EOF ... EOF with --data @./otlp-request.json.

Agent run example

A user on Microsoft Teams asks "What's the weather in Seattle?". Your agent calls a GetWeather function, asks an LLM to format the answer, and replies. That single run is four spans:

graph TD
    A["<b>invoke_agent</b> · spanId=A · parentSpanId=∅<br/><i>root - the run itself</i>"]
    B["<b>chat</b> · spanId=B · parentSpanId=A<br/><i>LLM picks the tool / formats reply</i>"]
    C["<b>execute_tool</b> · spanId=C · parentSpanId=A<br/><i>the GetWeather call</i>"]
    D["<b>output_messages</b> · spanId=D · parentSpanId=A<br/><i>final reply emitted to the user</i>"]
    A --> B
    A --> C
    A --> D

Run-wide attributes set on every span:

Span A: invoke_agent (root)

{
  "traceId": "0102030405060708090a0b0c0d0e0f10",
  "spanId": "1111111111111111",
  "parentSpanId": "",
  "name": "invoke_agent",
  "kind": 1,
  "startTimeUnixNano": "1736175600000000000",
  "endTimeUnixNano":   "1736175601500000000",
  "status": { "code": 1 },
  "attributes": [
    { "key": "gen_ai.operation.name",   "value": { "stringValue": "invoke_agent" } },
    { "key": "gen_ai.execution.type",   "value": { "stringValue": "HumanToAgent" } },
    { "key": "gen_ai.input.messages",   "value": { "stringValue": "[{\"role\":\"user\",\"content\":\"What's the weather in Seattle?\"}]" } },
    { "key": "gen_ai.output.messages",  "value": { "stringValue": "[{\"role\":\"assistant\",\"content\":\"It's 65F and partly cloudy in Seattle.\"}]" } },
    { "key": "user.email",              "value": { "stringValue": "alice@contoso.com" } }
    /* plus all the run-wide attributes listed above */
  ]
}

Span B: chat (LLM call)

{
  "traceId": "0102030405060708090a0b0c0d0e0f10",
  "spanId": "2222222222222222",
  "parentSpanId": "1111111111111111",
  "name": "chat",
  "kind": 1,
  "startTimeUnixNano": "1736175600200000000",
  "endTimeUnixNano":   "1736175600900000000",
  "status": { "code": 1 },
  "attributes": [
    { "key": "gen_ai.operation.name",      "value": { "stringValue": "chat" } },
    { "key": "gen_ai.request.model",       "value": { "stringValue": "gpt-4o" } },
    { "key": "gen_ai.provider.name",       "value": { "stringValue": "openai" } },
    { "key": "gen_ai.usage.input_tokens",  "value": { "stringValue": "42" } },
    { "key": "gen_ai.usage.output_tokens", "value": { "stringValue": "23" } }
    /* plus all the run-wide attributes */
  ]
}

Span C: execute_tool

{
  "traceId": "0102030405060708090a0b0c0d0e0f10",
  "spanId": "3333333333333333",
  "parentSpanId": "1111111111111111",
  "name": "execute_tool",
  "kind": 1,
  "startTimeUnixNano": "1736175600950000000",
  "endTimeUnixNano":   "1736175601200000000",
  "status": { "code": 1 },
  "attributes": [
    { "key": "gen_ai.operation.name",      "value": { "stringValue": "execute_tool" } },
    { "key": "gen_ai.tool.name",           "value": { "stringValue": "GetWeather" } },
    { "key": "gen_ai.tool.type",           "value": { "stringValue": "function" } },
    { "key": "gen_ai.tool.call.id",        "value": { "stringValue": "call-001" } },
    { "key": "gen_ai.tool.call.arguments", "value": { "stringValue": "{\"location\":\"Seattle\"}" } },
    { "key": "gen_ai.tool.call.result",    "value": { "stringValue": "{\"tempF\":65,\"condition\":\"partly cloudy\"}" } }
    /* plus all the run-wide attributes */
  ]
}

Span D: output_messages

{
  "traceId": "0102030405060708090a0b0c0d0e0f10",
  "spanId": "4444444444444444",
  "parentSpanId": "1111111111111111",
  "name": "output_messages",
  "kind": 1,
  "startTimeUnixNano": "1736175601400000000",
  "endTimeUnixNano":   "1736175601500000000",
  "status": { "code": 1 },
  "attributes": [
    { "key": "gen_ai.operation.name",  "value": { "stringValue": "output_messages" } },
    { "key": "gen_ai.output.messages", "value": { "stringValue": "[{\"role\":\"assistant\",\"content\":\"It's 65F and partly cloudy in Seattle.\"}]" } }
    /* plus all the run-wide attributes */
  ]
}

Sending telemetry

Using an OTel SDK

Most partners send traces through an OTel SDK rather than hand-rolled HTTP. The SDK handles batching, retry, and the OTLP/HTTP+JSON encoding for you. Set the exporter endpoint and inject the Authorization header.

The exporter endpoint is the route URL itself, including the query string:

https://agent365.svc.cloud.microsoft/observabilityService/tenants/{tenantId}/otlp/agents/{agentId}/traces?api-version=1

(Use /observability/... instead of /observabilityService/... for the delegated route.)

Python

from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter(
    endpoint="https://agent365.svc.cloud.microsoft/observabilityService/tenants/{tenantId}/otlp/agents/{agentId}/traces?api-version=1",
    headers={"Authorization": f"Bearer {token}"},
)

Package: opentelemetry-exporter-otlp-proto-http.

Node.js / TypeScript

import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";

const exporter = new OTLPTraceExporter({
  url: "https://agent365.svc.cloud.microsoft/observabilityService/tenants/{tenantId}/otlp/agents/{agentId}/traces?api-version=1",
  headers: { Authorization: `Bearer ${token}` },
});

Package: @opentelemetry/exporter-trace-otlp-http.

.NET

using OpenTelemetry.Exporter;

services.AddOpenTelemetry().WithTracing(b => b
    .AddOtlpExporter(o =>
    {
        o.Endpoint = new Uri("https://agent365.svc.cloud.microsoft/observabilityService/tenants/{tenantId}/otlp/agents/{agentId}/traces?api-version=1");
        o.Headers = $"Authorization=Bearer {token}";
        o.Protocol = OtlpExportProtocol.HttpJson;
    }));

Package: OpenTelemetry.Exporter.OpenTelemetryProtocol.

Manual HTTP

If you can't or don't want to use an OTel SDK, build the OTLP/HTTP+JSON request yourself and POST it. The body shape is defined by the OpenTelemetry OTLP/HTTP+JSON spec:

{
  "resourceSpans": [{
    "resource":  { "attributes": [ ... ] },          // optional
    "scopeSpans": [{
      "scope":  { "name": "<your-instrumentation>", "version": "1.0.0" },
      "spans":  [ <span>, <span>, ... ]
    }]
  }]
}

Each <span> is an object whose required fields are traceId, spanId, name, kind, startTimeUnixNano, endTimeUnixNano, attributes, and (for non-root spans) parentSpanId. See Endpoints and Request body encoding for the encoding rules (string-encoded times, hex traceId / spanId, integer kind / status.code, all attribute values as stringValue).

The set of attributes to set on each span is defined in Message contracts. See Attribute reference for the full attribute list. Refer to the Agent run example for an end-to-end working sample with the Bearer token in the header and the body inline.

You can send all the spans of a run in a single POST body (preferred - one request, one trace) or across multiple POSTs. The server reconstructs the run from traceId + parentSpanId + gen_ai.conversation.id, so each span carries enough to be correlated either way.

Message contracts

This section defines what spans you can emit and which attributes go on each. For the full attribute-by-attribute spec, see the Attribute reference.

Operation types

Every span you send must carry gen_ai.operation.name set to one of these four values (case-insensitive). Any span with a missing or unrecognized value is silently dropped and counted in partialSuccess.rejectedSpans.

Span hierarchy and run grouping

Agent 365 reconstructs a run from the standard OTLP span graph (traceId, spanId, parentSpanId) plus the run-wide attributes from the Attribute reference.

Six rules:

1. Always set parentSpanId on every non-root span. Without it, the run's tree structure can't be reconstructed.
2. Reuse the same traceId across every span in a run.
3. Set gen_ai.conversation.id on every span with the same value. This is the primary join key for "all spans in this run". It is not propagated automatically.
4. Set microsoft.channel.name on every span with the same value. Tool spans missing the channel / conversation can inherit them from their parent invoke_agent only if the parent is in the same OTLP request, so set them on every span yourself.
5. Set microsoft.session.id on every span when you have a logical session.
6. For agent-to-agent calls where the child agent is in a separate request, reuse the same gen_ai.conversation.id and use the microsoft.a365.caller.agent.* attributes (see Attribute reference) to capture the caller-agent context.

The four-span tree in the Agent run example is the canonical shape.

Common run shapes

Onboarding checklist

Run through this checklist before going to production.

Next steps

• Attribute reference - Per-attribute spec and value-picking guidance.
• Troubleshooting - Verifying ingestion, common pitfalls, and error responses.