Azure Event Grid domain is a fully managed event routing service that simplifies the development of event-based applications by providing a centralized event routing service for cloud-based applications and services.
Delivery of events
Event Grid domains are known for their reliable delivery of events. The domains ensure that the events are delivered to the configured destination endpoint at least once by receiving an acknowledgment from the destination endpoint.
Sometimes the events might not get any acknowledgment sent to the Event Grid domain from the destination endpoint. In such cases, the Event Grid domain decides if it should try to resubmit, drop, or dead letter the event based on the error acknowledgment received. If the error received contains 400 (Bad request), 413 (Request is too large), or 401 (Unauthorised) errors, the Event Grid Domain either drops the event or dead letters the event if the dead letter is configured. If the error in the acknowledgment belongs to none of the above, then the Event Grid domain tries to resend the event based on the retry policy configured.
Dead letter events
If the event is not delivered to the destination endpoint, it is stored as a dead letter event. We can configure how to handle such dead letter events in Azure Event Grid. The dead letter events are usually stored in the configured Storage Account as blobs. In a business-to-business scenario, the event must be delivered to another part of the system to continue the process flow. In case of any failures or missing events, the dead lettering will help restore the missed transactions by ensuring no data is lost.
Why is it essential to monitor the Event Grid Domain?
Consider the above scenario where a Logic App emits events to the Event Grid domain. The Event Grid domain has two direct subscriptions configured with Logic Apps. These events are later sent from the domain subscriptions to the following systems, salesforce, and SAP. Sometimes the events might not be able to reach the destination endpoints due to any one of the reasons explained earlier.
In this case, the Logic App connected to the SAP system has been disabled due to maintenance. When the domain receives an event, the event will be forwarded to both the Logic Apps, and the event at the SAP system Logic App will fail. The Event Grid domain will try to trigger the event based on the retry policy configured. After reaching the maximum number of retries, the event grid will push the event to the configured storage account as a dead letter.
Azure Event Grid Domain Monitoring in Azure
Azure provides various metrics related to the status of the events in an Event Grid domain like Published events, and delivery Failed events, Dropped events, and Dead letter events, which can be visualized as charts.
In addition to visualization, there are few options available for monitoring Azure Event Grid domains,
Diagnostic settings – A diagnostic setting can be configured to send the logs and metrics of the Event Grid domain to different destinations. The diagnostic setting logs can be configured for either logs or metrics. The logs can be further categorized as all logs or audit logs. A log analytic workspace, storage account, or event hub can be configured as the destination endpoint for sending the logs and metrics.
Log alerts – These alerts are triggered based on a log search query. An Alert can be configured by defining alert conditions based on the query results.
Metric alerts – These alerts work based on the metrics. The alerts will be sent to the configured destinations when the metric value meets the threshold value. For example, we can configure an alert and be notified when the dead letter counts for the Event Grid domain is greater than 10.
Azure Event Grid Domain Monitoring using Turbo360?
Turbo360 is a viable product that comes in handy to monitor the Event Grid domains and ensure the delivery of events. To tackle various business needs, Turbo360 offers a solution called Business Application. A business application is a logical group where we can associate various resources from multiple subscriptions and resource groups under a single roof and monitor the same based on our preferences. In terms of Event Grid domains, we can associate a set of Azure Event Grid domains, Topics, and Subscriptions to a Business Application and monitor them.
The Azure resources in a Business Application can be monitored using a monitor profile. A profile is a collection of monitor rules defined for different Azure Resource types. A profile can be created with rules to monitor the Event Grid domain resources and applied to a Business Application, which will initiate monitoring all its resources.
Consider the above scenario, where an event might get dead-lettered for various reasons. We want to be notified immediately whenever an event is dead-lettered.
To achieve this, we should associate the Event Grid domain, Event Grid domain Subscriptions, Storage account blob, and Logic Apps with a business application.
A monitoring rule can be defined for the Event Grid domain with the following metric “dead letter events count,” which will initiate the monitoring in the background.
In case of any dead letter events in the domain, Turbo360 will create an alert. Alerts can be configured with different notification channels like Slack, Microsoft Team, etc. In this case, we can configure an email notification.
If multiple Event Grid domains need to be monitored, we can use a monitor profile, which is a set of rules defined for Azure resource types and can be mapped to Business Applications. All the resources in a Business Application will get monitored based on the rules configured in the profiles and send a consolidated status notification.
Managing Dead letter events in Turbo360
In the Azure portal, the dead letter events can be viewed only by accessing the configured storage account and downloading the blob. But Turbo360 automatically finds the Storage account configured for dead-lettering, allowing us to view the dead-letter events in the domain subscription context.
Resubmitting events
In addition to accessing the dead-letter events, Turbo360 allows resubmitting the events to the source.
Resubmit to a domain: This will resubmit the events directly to the domain, forwarding the events to all the domain subscriptions. If we take the above scenario, the dead letter event will be resubmitted to both Logic Apps despite successfully sending it to the Salesforce system Logic App.
To API endpoint: This method will resubmit the events to an API endpoint. To ensure the processing of duplicate event processing in other subscriptions, we can configure the endpoint pointed to the required domain subscription. Consider the same example mentioned in the dead letter section. In this scenario, it is enough to resubmit the dead letter event to the SAP system Logic App alone.
As mentioned in the scenario, if the failure was due to a maintenance window in the Logic App, resubmitting the event to the Logic App endpoint once it is restored is sufficient to make sure the event is in place and processed by the SAP system.
Repairing and resubmitting events
If in case the resubmission of the event fails because of an error in the data that is being sent, then the event data can be manipulated and resubmitted to the required destination using the “Repair and Resubmit” option.
To conclude, Azure provides basic tooling to monitor the Event Grid domains. Still, when it comes to a business with many integrations in using components like Event Grid domains, Logic Apps, etc., Turbo360 has the best tool set for not just monitoring but to act upon and resolving the failures efficiently.
When you are looking for a cost-effective, ever-evolving solution to manage your Azure resources, Turbo360 might be the right product for you. You can have an obligation-free trial, but you could also discuss your challenges regarding managing your Azure resources with our team over a demo.
