Microsoft Azure Event Grid

Concepts

What is an Event?

Events are the data or information traveling through Azure Event Grid that describes the actions that took place in a system. All events have common properties like a source of an event, the time at which the event occurred, and a unique identifier. Every event has detailed information that is relevant to that event. Each event is independent with a size up to 64KB covered by General Availability Service Level Agreement.

In general, an event is sent to indicate something that has happened or changed. However, the actual object that was changed is not part of the event data. Instead, a URL or identifier is often passed to reference the changed object.

What is a Publisher?

Publishers are users or a third party that chooses to send events to Azure Event Grid. Microsoft publishes events for various Azure services and from the user’s application also.

What are Event sources?

The event source is the origin for an event to take place and it is responsible for transmitting events to Azure Event Grid. Every individual event source is linked to one or more event types. Azure Event Hub has the concept named event publisher which people often tend to confuse with event sources. Publisher in Event Hub is the user or organization that permits to send events to Azure Event Grid.

What is an Event Grid Topic?

Event topics classify events into groups and provide an endpoint where the sources transmit events. Azure Event Grid Topic is been created by the publisher and finalizes whether an event source needs one topic or more than that. Topics are classified into two categories depending upon the needs.

How does Azure Event Grid work?

Azure Event Grid can connect to any application that you create, and the Events generated by the application can be pulled and published to different other destinations. And the fun part is that all of these can be achieved with minimum coding efforts or even with no coding.

System Topic

System topics are the in-built topics available under Azure Services. These topics are not visible on Azure subscription because it is owned by the publisher, but the user can subscribe to it. In order to subscribe, the user must provide information about the resource from which they want the events to be received. Events can be subscribed if the user has access to the resources. These are the predefined set of Azure Services that can generate events. When the event source is a System Event Grid Topic, the Event generation and Event schema are handled by default. Some of the System Event Grid Topics are

• Azure Subscriptions (management operations)
• Container Registry
• Event Hubs
• IoT Hub
• Media Services
• Resource Groups (management operations)
• Service Bus
• Storage Blob
• Azure Maps

Custom Topic

Custom topics are application and third-party topics. Unlike system, topic users can view the custom topic in the Azure subscription once it has been created or if the user has access to it. Custom topics can be used to manage larger solutions where a custom topic can be created for each category of related events.

Azure Event Grid subscriptions

Event subscriptions outline the event residing under a topic which the event handler wants to receive. Events can be filtered based on their type or subject so that the user can ensure that the event handler receives only relevant events. Regarding the event subscription expiration, the subscription automatically expires after the expiration time period that has been set by the user. The Event Subscriptions can be created for both the custom and system Event Grid Topics. The Event Subscriptions also have filters to select the required events even before they are received by the endpoints. The destination endpoints those can be configured with the Event Subscriptions are

• Azure Automation
• Azure Functions
• Event Hubs
• Hybrid Connections
• Logic Apps
• Microsoft Flow
• Queue Storage
• Service Bus
• Webhooks

Event Handlers

An event handler is an application or resource that receives events from Azure Event Grid, in simple words, it is the place where an event is sent. The handler proceeds with certain actions to process the event. Subscribers can decide which event they prefer to handle, and Event Grid notifies every subscriber on the availability of a new event. Users can make use of any Azure service or their custom webhook as a handler. Depending upon the handler used, Azure Event Grid exercises different mechanisms to ensure the delivery of the event.

Security

Security is one of the important parameters in any resource. Azure Event Grid provides security for subscribing to topics, and publishing topics. When subscribing, you must have adequate permissions on the resource or event grid topic. When publishing, you must have a SAS token or key authentication for the topic. It facilitates security on the three concerns,

• Authentication: Authentication is the process of validating and verifying the identity of a user or process. The user authentication ensures that the individual is recognized by the Azure platform. Azure Event Grids provide few options for authorizing access through,
  • Securing webhook endpoint with Azure Active Directory
  • Authenticating clients using Shared Access Signature or key
• Authorization: Every request to a secure resource must be authorized to ensure that the client has required permissions to access the data. Authorizing a user can be done based on two roles,
  • Built-in roles
  • Custom roles
• Network Security: The security features available under network security are, service tags for egress, IP Firewall rules for ingress, and private endpoints for ingress.

Messaging Services

Azure offers two categories based on the communication forms, message-based delivery, and event-based communication.  The message-based delivery includes,

• Azure Service Bus

Event-based communication technologies are,

• Event Grid
• Event Hub

The fundamental of any business application is the communication between its processes. As a business application has different processes involved, it is necessary to choose the right medium of communication between them. The foremost action is to analyze whether the communication is via messages or events.

Azure Event Grid Vs Service Bus

With adequate understanding of Azure Event Grid in the sections above let us understand now Service Bus to compare them both. Azure Service Bus is a messaging service on cloud used to connect any applications, devices, and services running in the cloud to any other applications or services. As a result, it acts as a messaging backbone for applications available in the cloud or across any devices.

Queues

Messages are sent to and received from queues. Queues enable you to store messages until the receiving application is available to receive and process them.

Messages in queues are ordered and timestamped on arrival. Once accepted, the message is held safely in redundant storage. Messages are delivered in pull mode, which delivers messages on request.

Topics

While a queue is often used for point-to-point communication, topics are useful in publish/subscribe scenarios.

Topics can have multiple, independent subscriptions. A subscriber to a topic can receive a copy of each message sent to that topic. Subscriptions are named entities, which are durably created but can optionally expire or auto-delete. In some scenarios, you may not want individual subscriptions to receive all messages sent to a topic. If so, you can use rules and filters to define conditions that trigger optional actions, filter specified messages and set or modify message properties.

A business application might opt for events for certain processes and move to a message for other processes. In order to set the required communication, it is necessary to analyze the application’s architecture and its use cases.

Messages are raw data that toggles between the sender and receiver processes in the application. It contains the data itself, not just the reference. The sender process expects the receiver process to process the message content in a specific way.

Events are less complex than messages and frequently used for broadcast communications. It is a notification that indicates any occurrence or action that takes place in the process.

Use Azure Service Bus to deal with messages. When the business needs to deal with Events it becomes necessary to make an informed choice between Event Hubs and Event Grids. The following section can help with this.

Azure Event Grid Vs Event Hub

Azure Event Grid and Event Hub are services in Azure that can handle Events. However, they are significant in the type of events they deal with. With the understanding of Azure Event Grid in the previous sections let us now focus on Event Hubs to choose appropriately.

Event Hubs

Azure Event hubs – Event Ingesting service

Event Hub is a data ingestion service that streams a huge count of messages from any source to provide an immediate response to business challenges. The event Hub comes into play when handling of events along with data is required. Event hubs perform certain additional tasks apart from just being an event broadcaster.

Namespace

The namespace provides a dedicated scoping container, where you can frame multiple event hubs.

Event Publisher

Any resource that sends data to an event hub is a publisher. Events are published by using HTTPs or AMQP protocol.

Capture

Event Hubs Capture permits automatic capturing of the data published by the Event Hubs and conserve it either in Blob Storage account or in an Azure Data Lake Service account.

Partitions

Event Hubs splits up the streaming data into partitions in which the consumer or receiver will be restricted only to a specific subset or partition of the data.

SAS Tokens

Shared Access Signature in a namespace is considered as an access token to grant limited access to the namespace. The access is purely based on the authorization rules, which are configured either on a namespace or an entity.

Event Consumers

Any application that reads the data from the event hub is a consumer. Multiple consumers can be allocated for the same event hub, they can read the same partition data at their own tempo.

Scenarios such as telemetry require the ability to capture events that arrive at high volumes. That’s where Event Hubs can help. This is a service that is specifically built to deal with event streams.

The world we live in is reactive. Events take place, and we respond to those events. The reactive nature of Azure services is no exception to this paradigm. A newly updated blob to Storage service needs to be processed to import data into CosmosDB. A provisioned resource needs to be tagged. Ability to notify subscribers scaled-out between multiple regions with non-ubiquitous receiving mechanisms (webhooks, queues, streams, etc). Reacting to the events taking place on a cloud-scale is not what the Service Bus or Event Hub is built for. That’s what Azure Event Grid was created for.

Azure Event Grid’s whole purpose is to allow publishing and handling events crossing datacenter boundaries cheap and straightforward, helping with building reactive architecture.

Read more: Azure Event Grid vs Event Hub Comparison

Why Azure Event Grid 

Azure Event Grid is a simple but multi-purpose event distribution system. It can be used to deliver discrete events to subscribes, where those events will be received reliably and at lesser time.  If the user needs a simple event publish subscribe infrastructure, with trusted publishers, then Azure Event Grid will be the go-to option.

Various Applications of Azure Event Grid

Let us have a look at different applications where we could leverage the capabilities of Azure Event Grid. Events are a major component in many architectures. Here are some Azure Event Grid applications that can really help us better understand and implement these in an easy and effective way.

Serverless architectures

Being a Serverless service, Azure Event Grid easily fits in with many serverless architectures. It allows us to trigger various event sources and can fan out the events to several event handlers. By using serverless event handlers like Azure Functions and Logic Apps, we can do highly scalable event processing, leveraging the true value of a serverless architecture.

For example, let’s say we have a system that places an invoice into Blob Storage. We can then use Azure Event Grid to trigger off this and have a Function that is subscribed to this type of event. In the function, we encrypt the invoice using PGP (Pretty Good Privacy), which is an encryption method used for encrypting and decrypting data. The encrypted file is then placed onto another blob storage account. Another Event Grid topic gets triggered by this and has a subscription with a Logic App, which sends the encrypted invoice out via an email.

Enterprise integration

Azure Event Grid can be a great asset in an Enterprise Integration scenario. Azure Event Grid provides reliable messaging with retries and dead lettering, so users can be sure that no messages are lost. With its highly scalable throughput, the user is assured that they can have systems creating events at an extremely high pace, and the integrated solution will keep up. For this scenario, let’s take a Dynamics instance which creates messages and places them in a Service Bus queue. Then use Azure Event Grid to trigger this queue and have a subscription that starts a Logic App. The Logic App will then retrieve all messages from the queue and update the corresponding entities in SAP.

DevOps

We can also integrate many details about our Azure services through Event Grid. For example, when resources are being created or modified. This allows our DevOps team to automate their work, for example by calling a workbook whenever a virtual machine is created, automatically adding it to a virtual network and set up security.

IoT

Azure Event Grid also integrates with Azure IoT Hub, allowing us to be notified whenever devices have been created or deleted. This allows us to, for example, dispatch an engineer when a new device has come online. The engineer can then finish setting up the device and make sure all configuration has been completed.

Cloud Events

Azure Event Grid can also be integrated with resources outside of Azure. Custom topics can be used as event sources, which allow us to send events in the Event Grid schema to Event Grid. We can also send out events with subscriptions to webhooks, allowing us to send events to any system supporting these. However, it does not stop there, as Event Grid is one of the first services to support the Cloud Events standards. Cloud Events is a standard that allows working with events across platforms and gives us a specification for describing event data in a common way. This way we can have any platform or application working with events, to implement a common format, allowing easy integration and interoperability. With names like Azure, AWS, Google, IBM, and many more backing this standard. We can expect this to become the de facto standard for exchanging events across providers.

Delivery and Retry Policy

Event Grid delivery

Azure Event Grid ensures reliable delivery of events to the destination Endpoints. It makes sure that the events are delivered at least once to the destination endpoint. Whenever the event is not received by the configured destination endpoint, the acknowledgment is not sent by the destination endpoint. This mechanism is utilized by the Azure Event grid in handling the retries. Events are sent to the event subscribers i.e. the configured destination endpoints immediately. This makes sure the timely delivery of events from the Event Sources to the Destination endpoint with minimum Latency.

The Azure Event Grid delivers each event individually to the subscribers. The destination endpoints receive the events as an Array with a single event in them.

Event Grid Retry Policy

The Azure Event Grid waits 30 seconds for an acknowledgment from the destination endpoint. If the acknowledgment is not received, the Event Grid queues the event for retry. Azure Event Grid applies an exponential retry policy to ensure event delivery.

Azure Event Grid retries delivery on the following schedule on the best effort basis:10 seconds

• 30 seconds
• 1 minute
• 5 minutes
• 10 minutes
• 30 minutes
• 1 hour
• Hourly for up to 24 hours

If the destination endpoint responds within 3 minutes, the event grid will try to remove the message from the retry queue on a best effort basis, but this doesn’t guarantee that no duplicate events.

Azure Event Grid adds small randomization to all retries and may even skip some retries if the destination endpoint is unhealthy or unavailable for a long time.

For deterministic behavior of Event Grid retry configure the event time to live and maximum retries or delivery attempts in the Subscription Retry Policy.

Geo disaster recovery

The downfall is a common pitch faced by any cloud provider. Similarly, Azure regions or datacentres encounters a downfall if no availability zones are used. Due to the fall in datacentres, data processing gets affected, and switching the processing into a completely different region or datacentre seems critical. To overcome the disrupt Geo-disaster recovery and Geo-replication comes into action, which are important features for any enterprise. Azure Event Grid owns an automatic geo disaster recovery (GeoDR) of meta-data for new and existing domains, topics, and event subscriptions. If an entire Azure region faces a downfall, Event Grid will already have all the event-related infrastructure metadata synced to a paired region. The new events will begin to flow again any need for manual intervention by the user.

The Disaster recovery is measured by two metrics,

• Recovery Point Objective (RPO)
• Recovery Time Objective (RTO)

Recovery Point Objective

The Recovery Point Objective refers to the minutes or hours of data that might be lost. There two categories under RPO,

Metadata RPO: The RPO for the event is zero minutes. Whenever a resource is created in Azure Event Grid, it is rapidly replicated over regions. If any failover occurs, no metadata is lost

Data RPO: If a healthy system is caught up on existing traffic during regional failover, the RPO events stay up to 5 minutes.

Recovery Time Objective

The Recovery Time Objective is the minutes or hours the service downtime occurs. There are two categories under RTO,

Metadata RTO: It occurs within 60 minutes, where the Event Grid will start to accept create, update or delete calls for topics and subscriptions

Data RTO: Like metadata, it occurs within 60 minutes. Even Grid starts receiving new traffic followed by a regional failover.

Key Features of Azure Event Grid

Azure Event Grid is a lightweight notification and alerting solution. It has the following characteristics

• Dynamically scalable
• Messaging features like
  • At least once delivery
  • Dead Lettering
  • Includes retry mechanism
• Advanced message routing and filtering capabilities
• Uses push mechanism, which means there is no need for long polling.
• Provides out of the box integration with various Azure services, both as event sources and event handlers.
• Lower cost
• Serverless

How do I create an Event Grid in Azure?

Below steps can be followed to create an event grid

• Login to your Azure portal and select the respective Resource Group.
• Select an existing resource group or select Create new and enter a name for the resource group.
• Provide a unique name for the custom topic.
• Select a location for the event grid topic.
• Select Review + create at the bottom of the page.

Manage Azure Event Grid in Serverless360

Managing the activities in a service like Azure Event Grid, which deals with different events requires a tool with outstanding management and monitoring capabilities. Serverless360 comes into the picture to fulfill all the complex management needs required by the Azure services. Know more about Serverless360 to deal with Event Grid in real-time.

The different scenarios exhibit how powerful the Event Grids are, but the real challenge is to manage it in the Azure portal. There are certain challenges faced by Azure users to manage and monitor event hubs in the Azure portal.

• Lack of deeper/integrated tooling
• No consolidated monitoring
• No dead-letter event processing in Event Grid
• No Auditing

To overcome all these challenges Serverless360 has come up with an extensive set of capabilities.

Event Processing

In certain scenarios, there are chances that the events would become dead-lettered when the Relay attains the maximum number of listener count. Those events may be business-critical events and they need to be processed. To solve this business-critical problem. Serverless360 comes with the solution called Dead-letter Event Processing with which users can repair & resubmit the event to the Event Grid Topic. We can also view the event details

Automated Tasks

When there is a need to test our business orchestration, we need to send some events to Event Grids so that event will send to the endpoint through the subscription. But in the Azure portal, it is not possible to test the orchestration. To solve this challenge, Serverless360 brought in the capabilities called Automated Tasks with which users will be able to send events to both Event Hubs and Event Grid Topic. It is also possible to schedule these Activities too.

Monitor Azure Event Grid in Serverless360

Azure provides the entity level monitoring on their metrics, but the actual need would be Consolidated monitoring at the application level. User can monitor Azure Event Grid effectively using the metric-based monitoring available under Serverless360 Azure Event Grid Monitoring Tool.

Data Monitoring

Event grid topics can be monitored on the following metrics in Serverless360

Failed Events: Event sent to the Topic but rejected with an error code.

Published Events: Event successfully sent to the Topic and processed with a 2xx response.

Unmatched Events: Event successfully published to the Topic, but not matched to an Event Subscription. The event was dropped.

Publish-Subscribe Latency: Time is taken by the Topic to publish the event to the Event Subscription.

Event grid subscriptions can be monitored on the following metrics in Serverless360

Delivered Events: Event successfully delivered to the Subscription’s endpoint and received a 2xx response.

Matched Events: Event in the Topic was matched by the Event Subscription.

Dropped Events: The event was not delivered, and all retry attempts were sent. The event was dropped.

Delivery Failed Events: Event sent to Subscription’s endpoint but received a 4xx or 5xx response.

Dead Lettered Events: Events, that can’t be delivered to an endpoint, can be sent to a Storage Account defined as dead letter location

Destination Processing Duration: Time is taken to process the event from Event Subscription to the destination endpoint.

Operational Capabilities at Serverless360 for Azure Event Grid

Apart from the Managing and Monitoring capabilities, there are certain operational capabilities offered by Serverless360. These competencies aids in the enhancement of the Event Grid processing and in turn reduces the time consumption.

Sending Events to Event grid Topics

Consider a scenario where there are multiple event grid subscriptions created for a single event grid topic in a business process. There may be a need to verify whether the events sent to the event grid topic is received by all the event grid subscriptions. Serverless360 provides this capability of sending the events to event grid topics. Events can be sent to the event grid topics associated with the composite application of Serverless360.

The following event properties can be configured for the Event to be sent to the Event Grid Topic.

• Subject
• Data Type
• Event Type

Processing the dead-letter events of Event subscription

Consider a scenario where the endpoint configured to an event subscription was down for an hour and the events sent to it from the event grid topic have been dead-lettered. Now when the endpoint is available to receive the events there may be a need to reprocess the dead-lettered events of the event grid subscription.

This can be achieved using Serverless360. The dead-lettered events can be either resubmitted or repaired and resubmitted to the event grid topic.

Resubmit dead-lettered events

Resubmitting the dead-lettered event to an event grid topic preserves all the properties of the event grid event like Id, Subject, Event Type, Event Time, Data Version, Metadata Version, and Event Data.

Repair and Resubmit dead-lettered events

Repair and resubmitting of dead-lettered events can be used when there is a need to modify the event properties. For example, Event time of the dead-lettered event can be modified before resubmitting it to the event grid topic.

Delete dead-lettered events

Once the dead-letter events are processed manually either through resubmission or repair and resubmission, the dead-letter events can be deleted from the configured dead-letter destination.

Uses Cases/FAQ

Azure Event Grid can be used when your application deals with discrete events. Predominantly, when there is a need for your application to work in a publisher/subscriber model and to handle event but not the data, unlike Event Hubs.

Let us consider, a real-time e-commerce scenario When a user is shipping and moving stuff, where they are reacting to things like the item has been pulled from the shelf, item was bought to the postal area, item has been shipped, or this shipment was rejected and so on, in this case instead of doing a central control workflow user would do it in an Event-based model running active workflows for each product item.  Azure Event Grid is ideal for these kinds of reactive scenarios. 

Conclusion

From this pillar page, we can understand that Azure Event Grid facilitates building event-driven serverless apps that can effectively solve a real-time business problem with a focus on the core logic rather than the infrastructure. Event Grid is designed for high availability, consistent performance, and dynamic scale. Event Grid can simplify event-based apps, as this serves as a single service to manage routing of all events from any source to any destination. Experience better enhancement on the Event Grid monitoring, management, and operational activities using Serverless360.