Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
The Azure Monitor pipeline extends the data collection capabilities of Azure Monitor to your local data center and multicloud environments. It enables at-scale collection, transformation, and routing of telemetry data before sending it to Azure Monitor in the cloud. The pipeline can cache data locally, sync with the cloud when connectivity is restored, and route telemetry to Azure Monitor in cases where clients can't send data directly to the cloud.
Use cases
Specific use cases for Azure Monitor pipeline include the following:
- Scalability. The pipeline can handle large volumes of data from monitored resources that other collection methods, such as Azure Monitor agent, might limit.
- Periodic connectivity. Some environments have unreliable connectivity to the cloud or long unexpected periods without connection. There might also be periods of planned maintenance or need to temporarily disconnect from internet for security reasons. The pipeline can cache data locally and sync with the cloud when connectivity is restored.
- Reduce network bandwidth. Transformations in Azure Monitor pipeline can filter and aggregate data before sending it to the cloud, reducing the amount of data transmitted over the network.
Implementation
The Azure Monitor pipeline is a containerized solution that you deploy on an Arc-enabled Kubernetes cluster located in your local data center or another cloud provider. It's built on open‑source technologies from the OpenTelemetry ecosystem optimized for performance and scale, while ensuring portability and interoperability. It consists of multiple components that work together to collect, process, and transmit telemetry data to Azure Monitor in the cloud. Configuration in the pipeline determines the data collected from the local clients and how to process it before sending it to Azure Monitor. Configuration in Azure Monitor understands the data delivered from the pipeline and how to process and store it in a Log Analytics workspace.
Supported configurations
| Supported distros | Supported locations |
|---|---|
| - Canonical - Cluster API Provider for Azure - K3 - Rancher Kubernetes Engine - VMware Tanzu Kubernetes Grid |
- Canada Central - East US - East US2 - Italy North - West US2 - West Europe |
For more information, see Product availability by region.
Sample architecture
The following diagram illustrates a sample architecture of the Azure Monitor pipeline. This diagram illustrates the following concepts.
- The pipeline is deployed in third-party clouds, and any physical locations with devices and applications to collect data from.
- The pipeline is deployed on an Arc-enabled Kubernetes cluster at each location and in each third-party cloud provider.
- Data sources include the following:
- Syslog being collected from sources such as network devices and agents running on local servers. Collected by default on TCP port 514.
- OpenTelemetry (OTLP) being collected from applications. Collected by default on TCP port 4317.
- The pipeline forwards data across local firewalls to Log Analytics workspaces in Azure Monitor.
- Once data is collected from the pipeline, it's available to any Azure Monitor features accessing that data.
Next steps
- Configure Azure Monitor pipeline on your Arc-enabled cluster.
- Modify data before it's sent to the cloud using pipeline transformations.