Azure Event Grid
With Event Grid, we have the option to react to events, both for Azure entities as well as beyond Azure. As the team that works on Service Bus also built Event Grid, it is not surprising that these services work together very well. The announcement of integration between these two services was on February 26 and allows us to listen for messages without available listeners, as well as dead letter messages in the various Service Bus entities. This means we no longer need to do any polling on our Service Bus topics and queues. But instead can have an event-driven architecture.
Especially for scenarios where messages come in intermittent intervals, it allows us to react on these, without listeners continually checking for new messages. Instead, we can place Azure Event Grid in between, which triggers whenever new messages arrive, implementing a push model instead of the pull model that we usually use when listening on Service Bus entities.
Placed on the namespace level of Service Bus, Event Grid will look for messages on all queues and subscriptions within, and trigger an event for each of these entities where they are available. Do keep in mind that, currently, Azure Event Grid is only available for the premium tier of Service Bus. Furthermore, if the Service Bus entity did not have any messages, and a new message arrives, Event Grid will immediately trigger. However, once an event triggers, new messages coming in right after that can take up to two minutes to trigger another event, so this will be something we have to keep in mind while setting up our architecture.
Extending our messaging scenarios
Let us have a look at how we can use Azure Event Grid to enhance our messaging scenarios. For this, we will tackle a common use-case, where our partner places messages onto a Service Bus topic, and depending on the country property; we will route it to one of our subsidiaries.
In this case, we have a Logic App which polls on their Service Bus topic subscription for one of the subsidiaries, while the other uses a custom client to retrieve the messages. However, as our partner is not continuously sending messages, most of the polling actions will yield no results. In Azure Logic Apps, we pay per execution of operations, and thus we are spending money for nothing in many of these. In the custom client, we are wasting unnecessary CPU cycles and calls to Azure. Would it not be nice if instead of continually having to check if our partner placed a message on the topic, we would get a notification when a message is available? This is where Azure Event Grid comes in, as it allows us to send out an event whenever messages become available on Service Bus. This new implementation looks like the following.
The Logic App and custom client no longer poll on their subscriptions, but instead, an event triggers them. Once an event triggers one of the subsidiary systems, it will go out to their corresponding Service Bus topic subscription, and retrieve all available messages. This means that we no longer will be doing any polling, but instead, work with an event-driven way.
Implementation
Setting up Event Grid integration with Service Bus is simple, so let us have a look at how we can implement the use-case we discussed in the previous paragraph. We start by creating the various components needed for our scenario. The first one is a Service Bus namespace.
Note: Only premium namespaces allow for Event Grid integration. Thus we will need to use this here as well. When doing this implementation just for learning or demo purposes, we recommend deleting this when finished, as the premium namespace does come with some costs, which are excellent for production but probably are a bit much to keep running just for test or demo purposes.
In the Azure portal, create a new Service Bus premium namespace, making sure to set all the correct properties.
Next, we will create a topic in the namespace, which will receive the messages from our partner.
Now, add two subscriptions for routing to the subsidiaries. For now, we will not change the filters, so messages end up in both subscriptions; however, in a real case, these would use properties to route to one or the other.
We now have our prerequisites set up, so let us add an Event Grid subscription for the first subsidiary on it, which will be the custom client. To do this, we go to the namespace and select the Events blade.
As we will find, there are already several out of the box sample scenarios, which allow us to get started quickly with Event Grid. However, for our use-case, we will not use these samples but add an Event Grid subscription.
For the Event Types select ActiveMessagesAvailableWithNoListeners, as this will trigger when new messages are available on the subscriptions. For the endpoint, create a RequestBin endpoint by clicking Create a RequestBin and copying the Bin URL (change HTTP to HTTPS). On the Create Event subscription blade, set the Endpoint Type to WebHook, and set the endpoint to the RequestBin URL we just retrieved. Give the subscription a name, and switch over to the Additional Features tab.
As this is the subscription for Subsidiary A, we only want an event to trigger when a message is available in its corresponding Service Bus topic subscription. For this, on the Additional Features tab, set the Subject Ends With filter to the name of the topic subscription. This is also the place where to set up dead lettering for the Event Grid subscription, as well as configure the retry policy. Once we have set everything up, click the Create button to add the Event Grid subscription to the Service Bus namespace. The first time this will trigger an error, but this is to be expected, see the next part for this.
Be aware, that Event Grid needs to validate our endpoint before we can use it. To do this using RequestBin, after clicking the Create button, refresh the RequestBin, which will now show a message from Event Grid like the following.
[{
"id": "7cb7c0fb-e403-4e7c-b0c5-d16f396813b8",
"topic": "/subscriptions/fdf3a3a3-c8f5-472f-8367-6a9a4a6c11a9/resourceGroups/EnhanceMessagingWithEventGrid/providers/Microsoft.ServiceBus/namespaces/ServiceBusEldert",
"subject": "",
"data": {
"validationCode": "7A01AEA4-4B1A-4D13-8843-1C4A19E56689",
"validationUrl": "https://rp-westeurope.eventgrid.azure.net/eventsubscriptions/subsidiaryasubscription/validate?id=7A01AEA4-4B1A-4D13-8843-1C4A19E56689&t=2018-09-01T11:57:41.2214389Z&apiVersion=2018-05-01-preview&token=atNzU8gt8YQP4%2f%2f8aDzrHU9%2bEW1cPmT7enqP7v8LfkA%3d"
},
"eventType": "Microsoft.EventGrid.SubscriptionValidationEvent",
"eventTime": "2018-09-01T11:57:41.2214389Z",
"metadataVersion": "1",
"dataVersion": "2"
}]
The important part here is the validationCode and validationUrl fields. The validationCode field is what we would use if this were an implementation in an actual client. Which would then need to respond with this code to the request from Event Grid. However, as we are using RequestBin, this is not an option. Therefore, we use the validationUrl field instead. This we open in a new browser window. So Event Grid knows we own this endpoint. This will result in a message that Event Grid successfully validated the endpoint. If instead, we receive a message indicating an internal server error occurred, click on the Create button once more in the event subscription blade, and get the new validationUrl in RequestBin, as this will happen when the validation URL has timed out. Once we have validated the endpoint, our new Event Grid subscription will show up in the Events blade.
For the second subsidiary, we will create a new Logic App. This Logic App does the processing of the messages in the corresponding topic subscription.
Once the Logic App finished creation, open the designer and select the When an Event Grid event occurs trigger. This will allow us to listen for Event Grid events and start a Logic App each time these occur.
Sign in to the Azure Event Grid connector using your Azure account.
Set the parameters of the trigger to get events from the Service Bus namespace we created earlier. Make sure to include the Event Type and Suffix Filter here as well to only listen for available messages on the topic subscription for subsidiary B. This will make sure we receive an event once messages are available. So now we need to read the messages from the Service Bus topic subscription. To do this, we will add a Get messages from a topic subscription action, which retrieves all the messages available. Subsequently, we loop through these messages and do our processing of each message. Finally, remember to save the Logic App.
Now, when we switch back to the Service Bus namespace. Open the Events blade, we will find a new subscription for the Logic App as well.
Testing
To test the solution, we will use Turbo360’s capabilities to send messages into Service Bus. If you do not have a subscription yet, you can create one for free. To try out this great tool for monitoring and working with Azure’s serverless services. In Turbo360, add the namespace we created earlier, so we can manage the service bus namespace from here. To do this, go to the Namespaces blade, and choose to associate a namespace using a connection string. Grab the connection string on the Shared access policies blade in the Azure portal, and associate it with Turbo360.
Next, go to the Dashboard blade. Associate the topic we implemented earlier with the default composite application in Turbo360. Thus allowing us to send messages to this topic.
We now go to the Activities blade for Service Bus and create a new Send Messages activity. Configure the parameters, select the topic we just associated, and save the activity.
Finally, run the activity from Turbo360, which will place a message in the Service Bus topic. The topic then routes the message to the topic subscriptions for both subsidiaries, triggering the corresponding Azure Event Grid subscriptions. Consequently, Azure Event Grid sends an event to both RequestBin as well as the Logic App we created. In RequestBin, we will see the body of the message coming in, which will look something like the following.
[{
"topic": "/subscriptions/fdf3a3a3-c8f5-472f-8367-6a9a4a6c11a9/resourcegroups/EnhanceMessagingWithEventGrid/providers/Microsoft.ServiceBus/namespaces/servicebuseldert",
"subject": "topics/subsidiaries/subscriptions/subsidiarya",
"eventType": "Microsoft.ServiceBus.ActiveMessagesAvailableWithNoListeners",
"eventTime": "2018-09-01T12:36:53.4387835Z",
"id": "e707ac59-df41-4507-92da-2f1a12da8afc",
"data": {
"namespaceName": "servicebuseldert",
"requestUri": "https://servicebuseldert.servicebus.windows.net/subsidiaries/subscriptions/subsidiarya/messages/head",
"entityType": "subscriber",
"queueName": null,
"topicName": "subsidiaries",
"subscriptionName": "subsidiarya"
},
"dataVersion": "1",
"metadataVersion": "1"
}]
In Logic App run history, we will see that it received the event. And, then the messages from the Service Bus topic subscription.
Conclusion
In conclusion, we have seen today how easy it is to extend our Azure Service Bus entities with Azure Event Grid. By adding the capabilities that Azure Event Grid provides, we no longer need to worry about polling for new messages arriving on Service Bus. But can instead use events to notify us. This opens many options for us, both inside of Azure, but also well beyond. This allows for a truly open event-driven architecture.
