Add Apache Kafka as source in Fabric Real-Time hub (preview)

This article describes how to add Apache Kafka as an event source in Fabric Real-Time hub.

Prerequisites

• Access to the Fabric workspace with Contributor or above permissions.

• An Apache Kafka cluster running.

• Your Apache Kafka must be publicly accessible and not be behind a firewall or secured in a virtual network. If it resides in a protected network, connect to it by using Eventstream connector virtual network injection.

• If you plan to use TLS/mTLS settings, make sure the required certificates are available in an Azure Key Vault:

  • Import the required certificates into Azure Key Vault in .pem format.
  • The user who configures the source and previews data must have permission to access the certificates in the Key Vault (for example, Key Vault Certificate User or Key Vault Administrator).
  • If the current user doesn’t have the required permissions, data can’t be previewed from this source in Eventstream.

Data sources page

1. Sign in to Microsoft Fabric.

2. If you see Power BI at the bottom-left of the page, switch to the Fabric workload by selecting Power BI and then by selecting Fabric.

   

3. Select Real-Time on the left navigation bar.

   

4. The Streaming data page opens by default. Click on the Add data button to get to the Data sources page.

   

   You can also get to the Data sources page directly by selecting the Add data option in the left navigation bar.

   

Add Apache Kafka as a source

On the Add data page, type in the search bar and select Apache Kafka.

Configure Apache Kafka connector

1. On the Connect page, select New connection.

   

2. In the Connection settings section, for Bootstrap Server, enter one or more Kafka bootstrap server addresses. Separate multiple addresses with commas (,).

   

3. In the Connection credentials section, If you have an existing connection to the Apache Kafka cluster, select it from the dropdown list for Connection. Otherwise, follow these steps:

   1. For Connection name, enter a name for the connection.
   2. For Authentication kind, confirm that API Key is selected.
   3. For Key and Secret, enter API key and key Secret.

      Note

      If you only use mTLS to do the authentication, you can add any string in the Key section during connection creation.

4. Select Connect. 

5. Now, on the Connect page, follow these steps.

   1. For Topic, enter the Kafka topic.

   2. For Consumer group, enter the consumer group of your Apache Kafka cluster. This field provides you with a dedicated consumer group for getting events.

   3. Select Reset auto offset to specify where to start reading offsets if there's no commit.

   4. For Security protocol, select one of the following options:

      • SASL_SSL: Use this option when your Kafka cluster uses SASL-based authentication. By default, the Kafka broker’s server certificate must be signed by a Certificate Authority (CA) included in the trusted CA list. If your Kafka cluster uses a custom CA, you can configure it by using TLS/mTLS settings.
      • SSL (mTLS): Use this option when your Kafka cluster requires mTLS authentication, and you must configure both a custom server CA certificate and a client certificate in TLS/mTLS settings.

   5. The default SASL mechanism is typically PLAIN, unless configured otherwise. You can select the SCRAM-SHA-256 or SCRAM-SHA-512 mechanism that suits your security requirements.

   6. If your Kafka cluster uses a custom CA or requires mTLS, expand TLS/mTLS settings and configure the following options as needed:

      • Trust CA Certificate: Enable Trust CA Certificate configuration. Select your subscription, resource group, and key vault, and then provide the server ca name.

      • Client certificate and key: Enable Client certificate and key configuration. Select your subscription, resource group, and key vault, and then provide the client certificate name.

        If you don’t use mTLS but still use SASL_SSL with your custom CA cert, then you can skip this client certificate configuration.

      Note

      The TLS/mTLS settings in this section are currently in preview.

      For sources in a private network, ensure that the Azure Key Vault containing your certificates is connected to the Azure virtual network used by the streaming virtual network data gateway for Eventstream connector virtual network injection (for example, via a private endpoint).

   

Stream or source details

1. On the Connect page, follow one of these steps based on whether you're using Eventstream or Real-Time hub.

   • Eventstream:

     In the Source details pane to the right, follow these steps:

     1. For Source name, select the Pencil button to change the name.

     2. Notice that Eventstream name and Stream name are read-only.

   • Real-Time hub:

     In the Stream details section to the right, follow these steps:

     1. Select the Fabric workspace where you want to create the eventstream.

     2. For Eventstream name, select the Pencil button, and enter a name for the eventstream.

     3. The Stream name value is automatically generated for you by appending -stream to the name of the eventstream. This stream appears on the real-time hub's All data streams page when the wizard finishes.

2. Select Next at the bottom of the Configure page.

Review and connect

On the Review + connect screen, review the summary, and select Add (Eventstream) or Connect (Real-Time hub).

View data stream details

1. On the Review + connect page, if you select Open eventstream, the wizard opens the eventstream that it created for you with the selected Apache Kafka source. To close the wizard, select Close at the bottom of the page.
2. You should see the stream in the Recent streaming data section of the Real-Time hub home page. For detailed steps, see View details of data streams in Fabric Real-Time hub.

Related content

To learn about consuming data streams, see the following articles:

• Process data streams
• Analyze data streams
• Set alerts on data streams