Azure Logic Apps and Azure Event Hubs (Middleware Friday)

In this blog post, we’ll take a look at Azure Logic Apps and Azure Event Hubs. This topic was covered by Kent Weare in the 16th episode of Middleware Friday.

Scenario

To understand the topic better, let’s consider an IoT scenario where we have multiple IoT devices pumping telemetry data into Azure Event Hubs every 30 seconds. The scenario can be explained based on the below diagram. We will be focussing on Azure Logic Apps and using the EventHubs connector to ingest high-speed telemetry data, and publishing the data into Power BI real-time data set.

For this scenario, the first step will be to create Event Hubs and allow the devices to ingest telemetry data into Event Hubs. Next, we will use the Event Hub connector (which is currently in Preview) in Logic Apps to pull these Event Hubs messages and then use the Power BI connector to publish the telemetry data into a dashboard.

The main prerequisite for this blog post is that you need to have Event Hubs configured under your subscription.

Step 1 – Create Event Hubs under your Azure Subscription

Let’s get started with creating an Event Hub –

1. Open Azure Portal
2. Click New (+) -> Internet of Things -> Event Hubs
3. Click Create 
   1. Enter the name of the Event Hubs namespace
   2. Select the Pricing Tier
   3. Choose your Azure Subscription, Resource Group and Location
   4. Click Create to create the Event Hub
      
4. Once your namespace is created, click Event Hubs under Entities in the left hand navigation pane.
   
5. Click +Event Hub
   
   1. Enter the Event Hub name
   2. Select the partition count by moving the slider
   3. Select the message retention duration by moving the slider
   4. Choose the remaining (required) settings and click Create
      
6. The Event Hub will be created as shown below
   

Step 2 – Set the Shared Access Policies for Logic Apps

For a more seamless experience, we recommend you to add the permissions for Logic Apps to be able to retrieve data from your Event Hubs (through the connector). To achieve this, you can create a shared access policy for LogicAppsSubscribe with the Manage, Send, Listen permissions.

1. Under Settings in the left hand navigation menu, select Shared Access Policies
   
   
2. Click Add to add a new Shared Access Policy
   1. Enter the policy name
   2. Under claim, select the check box for Manage. This will automatically set the Send and Listen status for the policy.
   3. Click Create
      

If you do not perform this step, you will face the below error when you try to use the Event Hubs connector within Logic App.

Step 3 – Define the Logic App for the scenario

Let’s construct the Logic App for the scenario –

1. Log in to Azure Portal
2. Create a new Logic App. Let’s name it, in this case, MF-EventHub.
3. In the Logic App Designer, let’s start off with the Event Hub Connector
   
4. The best part about the Event Hubs connector is that you will see the list of configured Event Hubs under the same Azure subscription. This is a cool feature that you no longer have to go searching for your Event Hubs. Provide a connection name for the Event Hubs and click the Event Hub of your choice from the list.
   
5. In the next step, select the Shared Access Policy from the list. Select the policy that we created earlier in Step 2 and click Create.
   
6. With this, we have wired our Event Hub with the Event Hub Logic App connector. You will now be able to choose your Event Hub from the drop down and set the frequency for the trigger to execute. For this scenario, we will set the frequency to 30 seconds.
   
7. The next step is to create the Action. We will select the Data Operations – Parse JSON action. For this, we will create a sample message and pass the message (payload) to generate the schema of the message. In the Parse JSON action, click the Use sample payload to generate schema link to enter the message body as a JSON request and click Done.
   
8. You will notice that the message schema is generated that will allow us to use the type data in any downstream connectors (say, Power BI)
   
9. If you notice keenly, you will see that the message schema has an array that has, say, multiple request names with values etc., When we add the next step (action) to push the data into Power BI, Logic Apps has the capability to find out that the message schema has a batch of records and will automatically create a For Each loop for each record in the message schema.
10. Before you can configure the Power BI connector, you need to Sign In (in the connector) to create a connection to Power BI. And also, you need to have the Power BI graph ready in the Power BI console. Now, in the Power BI connector, once you have provided the authorization credentials, you will see the fields where you can select the Workspace name, Dataset, Table, and then define the attributes that can be published into Power BI.
    
11. Our Logic App is now ready!
    

Test – Trigger messages into the Event Hub

Now that the Logic App is ready, we will use Turbo360 to send events (messages) into the event hub. In Turbo360, the first step will be to associate a namespace (related to Event Hub). Once you have associated the namespace, you can click on Manage Namespace from the Home Dashboard. Select Activities from the left-hand navigation menu and click Create.

In the Activity Configuration Details section, provide a name for the message. Next, paste the message that will be sent into the Event Hubs and eventually into Power BI. You can make the additional settings and if you are happy with the changes, click Save. You have now created an activity to trigger messages into the Event Hub.

Click the Play button (Run Once) to trigger the message. By doing this, we are actually triggering the message into the Event Hub. The Logic App connector will pick up the message from the Event Hub and process the message further and put the message into Power BI dashboard. The Power BI dashboard will look as shown below.

Therefore, this is a very quick way you can provision a Power BI dashboard that consumes data from Azure Event Hub using Logic Apps.

As a part of the feature content, Kent also gave his opinion about when you need to use Azure Stream Analytics and Azure Logic Apps, especially in an Industrial IoT scenario. You can watch the video to know more about this – Watch video

Feedback

You can give your feedback about Middleware Friday episodes, any special topic of interest, or any guest speaker whom you would like to see in Middleware Friday. Simply tweet at @MiddlewareFri or drop an email to middlewarefriday@gmail.com.

You can watch the Middleware Friday sessions here.