This blog is an extraction of the session “Building Event-Driven Integration Architectures” presented by Dan Toomey at Integrate 2020. Dan Toomey is a Microsoft Azure MVP and works as a Senior Integration Specialist at Deloitte. He is also a Pluralsight BizTalk Author, you can get to him at, Dan Toomey – BizTalk Author | Pluralsight.
Introduction to Events
Events generally are light-weight facts or notification that triggers a reaction.
Types of Events
Discrete Events
Discrete Events are significant and stand-alone events (i.e) they don’t rely on other events to make them actionable. An example of a discrete event is a notification on ‘a file is created in a container or a directory’, all that the subscriber needs to action this is available in this single event.
Series Events
Series events are time-based events they acquire significance when looked in the context. These events are plotted in the graph. Temperature variation in the room over a period is a good example of a series of events.
Event-Driven vs. Event Sourcing
Event Notifications are event-driven, it broadcasts announcements about various state changes, and it is completely decoupled.
Event sourcing is significant and needs to be persistent, it also has replaying capability. Auditing is a good example of event sourcing.
Understanding the difference here is critical to choose the right platform to handle the events.
Why Event-Driven Integration
We are being bombarded with data everywhere coming from mobile devices, IoT, etc., In real-time we build APIs to drive applications, capture those events, and react to those events.
Dan Tommy’s words regarding Data-Centric, “Five years ago, I came across the Copernican Shift, that shift was Data-Centric, and it is most suited for traditional applications. The highest priority here is to store data and to preserve the stored data. Later, the events gained a higher priority. Storing and persisting becomes secondary and reacting to those events is the highest priority.
Event-Driven integration helps us stay connected to the application by reacting to the events on-time. It is cheaper and more efficient than polling. The Publisher is decoupled from the receivers. If the publisher wants to enhance the applications, they can extend the subscribers without any impact. Therefore Event-Driven Integration is flexible and scalable.”
CNCF CloudEvents
Below is an instance of a cloud event notification to describe the standard defined under version 1.0:
- specversion indicates the instance
- type describes the kind. Here it is “pull”
- source indicates where the request comes from
- subject is a unique identifier to identify the information
- id unique identifier for notifications
- time to distinguish between the events if multiple events have same meta data
- datacontentype and data are payload
- comexampleextension1 and comexampleothervalue are custom attributes
Azure Event Grid
Azure Event Grids are Microsoft platform for Event Notification. It is asynchronous method for capturing events and sending them around in a decoupled fashion. It works on Publish Subscribe fashion and it supports advanced filtering and routing capabilities. It might appear like Service Bus but not as Event Grid uses the Push Push model. It is a very efficient model that actively pushes the data to the subscribers. And if an event needed to be captured you can use built-in Event sources and publishers without writing any code.
Here there are lots of event sources, choose custom events and here you can manage all your events in one place. In the Event Handlers choose Web Hooks, Since Web Hooks can push events outside the Azure services.
Check out this a comprehensive guide on ‘Azure Event Grid’ to brief all the debriefed queries related to Event Grid. It provides useful links to explore and exercise further.
Reliability and Performance
Event Grids can handle about 10 million events per second in every region. It can deliver the notifications to the subscribers and has exponential backoff for the events those are not delivered to the subscribers.
Azure Event Grid can handle up to 2.5 trillion requests per day and the success rate is 99.9998% and the failures are 5 million. The purpose of mentioning the count of failures here is to emphasize building the applications with failures in mind. Fortunately, Event Grid does help us with this.
Dead Letter Handling
Event Grid supports Dead Letter Handling, which can be turned on when the topic and the subscription are chosen. Provide a storage blob container to send events.
Please refer this blog on ‘Event Grid Delivery and Retry Policy’ to understand more.
Limitations
Here are a few limitations regarding Event Grid that was mentioned by Dan Toomey,
- The message size limit is 64 KB in GA but in preview, it supports up to 1MB. But it is recommended to keep the event notifications light.
- No guarantee of ordering
- No replaying of events, this means Event Grid is not a platform for Event Sourcing where Event Hubs can help
- Retry Capability is available
Creating Topics
Topic creation can be done in two methods, either from the event source, if the source is an Azure Service with support to events.
Topic can be created from the Event subscriber as well say in this case a Logic App.
Comparison
Microsoft Azure support three excelling messaging services, Service Bus, Event Hubs, and Event Grids. Each service has its own capability of dealing with messages and events. Understanding their significance mentioned below is necessary to choose the right.
Also check this blog on ‘Azure Service Bus, Event Hub and Event Grid, Which one to choose?’ to understand the significance of the different messaging services in Azure to choose them right in solving your problem.
Demo – VM Resize Notification
This was the most interesting segment of the session. To see the concepts in action. The scenario here is, a smart manager wants to keep the Azure spending in control. Hence wants to receive an alert when a VM size is increased beyond the allowed limit by a developer who has got contributor right on the resource. The application depicted below, built in an Event-driven architecture can indeed meet this requirement.
Let’s understand,
- Virtual machines are not yet supported as Event sources of Event Grids; hence we shall create Event Grid subscription on Resource Group where the Virtual machine lives
- Whenever the VM is resized the Event Grid would send an event notification which triggers a Logic App
- However, this trigger would not contain any details about Virtual Machine in fact any change in the Resource group not essentially the resize of the VM would trigger this Logic App. Hence, we need to add a condition to check the trigger subject contain Virtual Machine before proceeding further. Calling a function app can help us fetch the current size of the Virtual machine.
- Another Azure function is invoked to compare the current size against the allowed sizes mentioned in the Storage table
- In case of the allowed limit being breached an email, notification is sent to the smart Manager subsequently the developer can await a surprise 😊
Let’s implement,
- Building event-driven applications are made a cakewalk with the introduction of the ‘Events’ section on the supported event sources of Azure Event Grid. In this scenario, On the Resource Group is the event source by choosing Logic App as the event handler, the required Event Grid Topic and Subscriber gets implicitly created.
- In the Logic App trigger definition, specify the Subscription, Resource type, Resource group and there are set of events for the Resource group. Choose WriteSuccess event. The Logic App is now to set to get triggered on any change in the Resource Group.
Subsequent actions in the Logic App definition should achieve the flowing sequence
- Condition to validate of the trigger has been raised due to change in VM
- Invoking the Function App to gather the current size of the VM
- Invoking the Function App to compare the current size against the allowed limit mentioned in the Storage Table (pre-requisite the Storage Table should contain valid threshold values set) and returns a Boolean
- Sending a high- priority email notification through the Outlook connector when the VM size has been increased beyond the defined limit
It’s time to check if all works as expected. Go ahead and modify the VM size.
The change triggers the Logic App, whose successful run delivers a high priority email notification in the Managers’ inbox
Turbo360 can add value
Here is where I would like to emphasize how Turbo360 can complement the Azure portal in supporting the Serverless Application built above.
Holistic Application view
Manage and monitor the Serverless services with the context of the application rather than dealing with them as siloed resources in the Azure portal using Composite Applications in Turbo360. Get any failures in them brought into the spotlight on the application dashboard.
Let your operations and support team better understand how the application resources interact with each other along with their current status readily available in the Service Map.
View and Process Dead lettered events
Don’t you find your Event Grid as a black box? management capability in Turbo360 can throw light on this. View and process the dead lettered events in an Event Grid Subscription. Perform bulk resubmission or view event details to repair and resubmit. Achieve those that are not possible in the Azure portal.
Automated Task
- Resubmit the Logic App runs failed due to a specific error in the specific run action without any manual intervention
- Simulate a real-time environment using the configurable ‘Send Events’ activity for Event Grid Topics
Auto Correct Status
Map the Logic Apps and Function apps to the Threshold monitor in Turbo360 to not only to monitor their status and get alerts when not met but also to auto correct their state without any manual intervention. Ensure your resources are always on.
Comprehensive monitoring
- Explore 5 different types of monitors in Turbo360 to monitor the resources in various required perspectives
- Status monitor to get a consolidate report of the status of all the associated resources
- Threshold monitor to alert on violations in state and properties with the capability to auto correct status of Logic Apps and Function Apps
- Data Monitor to monitor Logic Apps, Event Grid Topics and Subscriptions, Function Apps on a range of metrics to measure their performance, efficiency, reliability, consumption and much more
- Watch Monitor detect the failure in Logic Apps and Functions Apps
- Achieve end- to- end tracking on the event flowing through the Logic App and Functions using Business Activity Monitoring
- Integrate with available notification channels
Advance Security
Define a granular user access policy on the Composite Application inviting users or groups from Azure Active Directory. Govern and Audit user actions on the Logic Apps, Event Grid and Function Apps.
Also refer this blog, ‘Azure Event Grid: Cab Availability Management System’ to understand how an Event driven application can be better managed and monitored using Turbo360
Q&A highlights
Below are a few questions raised by the participants for which Turbo360 blog content can provide more clarity on:
- I have been using Azure Service Bus Topics for discrete application events? Should I be using Event Grid instead and if so are Service Bus Topics now redundant?
Dan’s Response: Definitely not. SB is for messaging, EG is for event notifications. Different uses, different features supported
Please check this blog on ‘Azure Service Bus Topics vs Event Grid’ for more clarity on this topic.
- Will the event grid ever support ordering?
Dan’s Response: I don’t expect it would. I would use Event Hubs for that as it definitely supports ordered replay.
Please check this blog on ‘Azure Event Grid vs Event Hub’ for more clarity on this topic.
This session had a couple of challenges raised by participants for which Turbo360 has got solutions. Let’s check them below
- Is there something like the Service bus explorer to browse events?
Dan’s Response: Oh, I wish there were! Hopefully, someone will develop a tool for this! 🙂
Turbo360 has already got the capability to let you browse through the dead lettered events in an Azure Event Grid Subscription. It is also possible to view event details, delete the event, resubmit all events at a go or repair an event and resubmit the same to the Topic. Explore more on this feature here.
- Is it possible to get an alert when an unmatched topic comes in?
Turbo360 offers monitoring solutions to detect arrival if unmatched events in the Event Grid Topic. In fact, not only detecting the unmatched events, it is possible to detect failed events in the Topics. Alerts can be generated on the arrival of Dead lettered or dropped events in the subscriptions as well. Explore all possible monitoring for Event Grid Topics and subscriptions in Turbo360 here.
With integrations to 7 different notification channels in Turbo360, it is indeed convenient to keep the stakeholders informed on the event status. This same can be highlighted on a service map for a better understanding of the operations or support team too.
Here is a quick overview of the features in Turbo360 for the management and monitoring of Event Grid Topics and Subscriptions.
Wrap up
In this blog post, we discussed the Event-Driven Integration design which are real-time responsive applications and that are cost-effective. Event Grid is a Microsoft Azure service that sends Event Notifications and it is the easiest solution for building Event-Driven Applications. Also, explore Turbo360 to support the Event-Driven applications with ensured operational efficiency. Thanks to Dan Toomey for such a lively and informative session.
