Here is a blog about Azure Event Hubs monitoring and how Turbo360 helps you do it.
Azure Event Hub is an event collection service and big data streaming platform. It is highly scalable and can handle millions of events per second. Azure Event Hubs are simple, secure real-time data and instantly connect millions of devices across platforms.
An overview of Azure Event Hub monitoring, logging, and alerting concepts
Let’s understand the following important concepts.
Event publisher
A client sending events to the Event Hub is an event publisher. Events are sent via HTTPS, AMQP 1.0, or Kafka protocols. Events are published either individually or in batches. Event publications exceeding the limit of 1MB are rejected.
Partitions
The Event Hub organizes and stores the events a publisher sends in one or more partitions. A partition key helps process events in an orderly manner. Events sharing the same partitioning key are stored and delivered in order of arrival.
Event consumer
A client receiving events from an Event Hub is an event consumer. Consumers receive the events via the AMQP 1.0 session when they are available.
Sending Events to Event Hubs
Visual Studio 2022 sends events to Event Hub in this blog. Create a .NET Core console application. Install the Azure.Messaging.EventHubs NuGet package. Paste the following code in the Program.cs file.
- Initialize an EventHubProducerClient object with the primary connection string of the Event Hub namespace and the Event Hub name.
- Create a batch of events by creating a list of EventData and adding events.
- Assign PartitionId to the batch of events using SendEventOptions.
- Finally, send the events using EventHubProducerClient.SendAsync method.
using Azure.Messaging.EventHubs;
using Azure.Messaging.EventHubs.Producer;
EventHubProducerClient eventHubProducerClient = new EventHubProducerClient("PrimaryConnectionString", "EventHubName");
List eventList = new List();
for (int iteration = 0; iteration < 10; iteration++)
{
eventList.Add(new EventData(new BinaryData(Encoding.UTF8.GetBytes("Event Content"))));
}
try
{
SendEventOptions sendEventOptions = new SendEventOptions() { PartitionId = "PartitionId" };
await eventHubProducerClient.SendAsync(eventList, sendEventOptions);
}
finally
{
eventHubProducerClient.DisposeAsync();
}
Build the project and run the program. Events will be sent to the Event Hub.
Send Events to Event Hub from Turbo360
A more innovative alternative to the solution above is the Send Event Automated task for Event Hubs in Turbo360. It is a cloud-based platform that lets users manage and monitor the app's performance on the Microsoft Azure cloud platform.
One can simulate a real-time scenario by triggering many events in an Event Hub without writing a single line of code. The Turbo360 encapsulates the underlying complexity and hence improves operational efficiency.
Azure Event Hub logging
Azure Event Hubs logging provides information about the operations done under the Event Hub namespace.
Diagnostic Log in Event Hubs
Event Hubs generate diagnostic logs to provide insights into the health and performance of the Event Hubs and the associated components. These logs contain information about events such as data ingress and egress, errors, and other operational metrics.
Azure Event Hub diagnostic setting defines which platform logs and metrics are collected from a resource and where to stream them. Diagnostic configurations are limited to up to 5 to send different logs and metrics to independent targets.
Enabling diagnostic log
- Click on Diagnostic settings under the Monitoring section of the Event Hub Namespace.
- Click on Add diagnostic setting.
- From the category details, select the log category.
- From the destination details, select the destination.
Event Hubs logs storing destinations
Once configured, Event Hubs will start collecting and storing logs based on the diagnostic settings configuration. Logs can be stored in the following destinations.
Log Analytics workspace
The metrics are stored here in log format. The stored logs can be searched via Log Analytics. It can also be integrated into queries and alerts and visualized with existing log data.
The following query retrieves the logs from the AzureDiagnostics table.
AzureDiagnostics
| where TimeGenerated > ago(1h)
| where ResourceProvider == "MICROSOFT.EVENTHUB"
| where Category == "RuntimeAuditLogs"
Azure Storage Account
Storing the logs and metrics in an Azure Storage account helps analyze, restore, etc. It is cheaper than Azure Monitor logs and Log Analytics workspaces, and logs can be kept here indefinitely.
Azure Event Hubs
Once the data is in Event Hub, it can be streamed to other third-party Security information and event management systems or external analytics solutions.
Azure partner solutions
Integrations can be made between Azure Monitor and other non-Microsoft monitoring platforms. It is useful when already a partner solution is used.
How to monitor Event Hubs in the Azure portal?
By defining alert rules, monitoring can be set up in the Azure portal. The Azure portal's Event Hub, Namespace Monitoring section, is where alert rules can be specified. Azure offers many metrics corresponding to the resource's essential aspects. With the alert rules, metrics and their corresponding thresholds can be specified.
When monitoring the resource individually, Azure Monitor comes in helpful. When resources that comprise a business flow must be monitored as an application, monitoring might be challenging. The alerts obtained using Azure Monitor are resource-level warnings. It isn't easy to follow the notifications when several Event Hubs are being monitored.
How to monitor Azure Event Hubs with Turbo360?
The Business Application module in Turbo360 has an Azure Event Hub Monitoring Tool for seamless Event Hub monitoring. Business Applications in Turbo360 can monitor the Azure Event Hubs, which act as a container that groups resources of a specific business flow.
A Business Application can be built by adding the required Azure Event Hubs. Monitoring can be performed based on metrics like Server Errors, Incoming Requests, etc. Additionally, the state of the Event Hubs can be monitored, ensuring that the resources are always enabled. An automated configuration can also be enabled to auto-update the resource's state to the expected state.
Users can set monitoring rules using monitoring profiles for multiple resources of the same type or other types. It is possible to define and apply a monitoring profile to the Business Application to monitor the Event Hubs rather than specifying the rules at the resource level.
Instead of applying the profile each time an Event Hub is added to the Business Application, an option to automatically apply the profile can be enabled, reducing the overhead of manually applying the profiles.
The resources are monitored, and the status of the resources is automatically updated once the resources are added to the Business Application to which the monitoring profile is applied.
Sometimes, it's necessary to do more than monitor the Event Hub's metrics and state. Through Log Analytics, Turbo360 allows monitoring of the Event Hub using the logs set up in the diagnostic configuration.
Log Analytics: querying and monitoring
Turbo360 enables querying and monitoring logs by adding the Log Analytics resource to the Business Application.
Queries can be executed on the Log Analytics resource, and the results can be visualized in different forms, like tables and charts.
Besides executing the queries and visualizing the logs, monitoring can also be performed on them. Query monitoring supports both scalar and non-scalar queries.
How to configure Event Hubs alerting in Turbo360?
Alerts are configured through the escalation policies in Turbo360. It is a set of rules to propagate the alerts to configured notification channels after an interval.
When the resources violate the configured thresholds, an alert incident is created. It can either be closed or acknowledged based on the business requirements. These incidents can be escalated up to 5 times. The escalation stops once it is acknowledged or closed.
Conclusion
Monitoring Azure Event Hubs is essential to keep business running smoothly. Azure delivers Event Hubs monitoring through Azure Monitor by setting up alert rules. The alert rules aid in the early detection of errors and performance snags. However, monitoring several Event Hubs necessitates numerous alert setups, making it difficult to keep track of the notifications.
By combining Event Hubs into a Business Application and using monitoring profiles, Turbo360 extends its lead over Microsoft Azure. It helps to proactively identify and fix problems, reduce downtime, and increase productivity.
Azure provides basic monitoring capabilities and works well with a few event hubs. Turbo360 is the perfect choice for businesses managing multiple resources.
