Microsoft Azure offers various services for messaging and events. Today, there are four options for messaging and events in Azure: Service Bus, Storage Queues, Event Hubs, and Event Grid. Two deal specifically with events – Event Hubs and Event Grid, and two for Azure messaging – Service Bus and Storage queues. Before we discuss these Azure Services – let’s describe first what are events and messages.
Azure Messaging versus Eventing Services
Messages and events are not the same. Messages have a clear intent – they are sent for a specific action or response. When a message is sent to a warehouse system requesting an item to pick – this message has an intent or a command that requires an action – the client or system sending the message is expecting a response in this case. An event, on the other hand, reflects a fact – an occurrence in the past. Let’s say the requested item leaves the warehouse and the system sent out a notification message there could be many systems or devices interested in it. This notification, i.e., event, however, doesn’t define intent as the publisher is not aware of these systems and devices nor doesn’t have to be.
Now that difference is apparent between events and messages we will discuss the Azure Services associated with them.
Azure Service Bus
The Azure Service Bus is one of the oldest services of the Azure platform – it is a highly reliable, brokered cloud messaging system. This service is aimed for enterprise messaging scenarios and offers middleware technologies like message queueing and publish/subscribe messaging. Furthermore, the service bus provides decoupling between services and applications. The critical capabilities of the service bus are:
- Queues are offering an asynchronous messaging capability with first-in-first-out (FIFO) message delivery. Only one consumer receives each message. Messages are persisted on the queue, and the consumer does not have to pick up the message straight away, i.e., the message publisher and consumer do not have to be connected to queue at the same time. Hence, queues are suitable for store-and-forward of messages in a particular scenario.
- Topics and subscriptions offer a similar capability as queues. However, there can be more consumers for messages that are sent to a topic. A topic has one or more subscriptions. Furthermore, subscriptions can have filters, and thus various messages to a topic can end up in different subscriptions belonging to the topic.
- Relay provides a gateway to connect on-premise services to Azure, without having to open a firewall connection to the network. The Relay has two features – Hybrid Connections and WCF-Relays. Both enable a secure connection to on-premise services. The difference is that hybrid connection uses the open standard web socket, while WCF-Relay uses the legacy Windows Communication Foundation (WCF) to enable remote procedure calls.
Azure Service Bus is available in three tiers:
- Basic – queues and scheduled messages, and message size up to 256 KB.
- Standard – on top of basic with Topics and subscriptions, transactions, sessions, and de-duplication.
- Premium – on top of the standard, and message size up to 1 MB.
Azure Event Grid
Azure Event Grid is one of latest service added to the Azure Platform. This service enables event-driven, reactive programming through a publish-subscribe model – in the Building reactive solution with Azure Event Grid, you can read more about it.
With Event Grid, there is a concept of publishers of events, which do not expect how the events are handled, and subscribers who decide how to handle the events. Publishers emit events to Event Grid – centrally manages events by routing them to subscribers of them. Furthermore, the routing can be done on the event type, or by a pre- and or postfix.
Event Grid has the following characteristics:
- dynamically scalable
- low cost
- serverless
Both Azure Service Bus with Topics and Subscriptions and Event Grid offer a pub-sub model. However, they are different in behavior as shown in the picture below.
Azure Event Hubs
You can view Azure Event Hubs as a significant data pipeline. This service is designed to ingest large data streams generated by devices and services. Moreover, Event Hubs facilitates the capture, retention, and replay of telemetry. In case your solution requires a high throughput of data ingestion then Event Hubs is the right service.
With Event Hubs, you can define so-called ‘consumer groups’, which allows you to read the stream of events. In case you have one receiver read the stream of events, then you can use the default consumer group. However, when you have multiple receivers for the stream concurrently, and each wants to read the stream at its rate – then you will need to set up numerous consumer groups. Furthermore, each receiver will manage an index (or offset) – a pointer to where in the stream of events it will start reading. The receiver can begin at the beginning of the stream and read to the end and subsequently wait for new events – or it can start reading part way through the stream.
Event Hubs support the following protocols: HTTPS, AMQP, and AMQP over WebSockets. Furthermore, it supports Shared Access Policies for Security, and .NET/C for language support for building solutions. It is available in three tiers:
- Basic – offers a maximum of 20 throughput units with 1 MB/s ingress and 2MB/s egress per unit. The message size is up to 256 KB, and retention of 1 day, one consumer group, and 100 brokered connections.
- Standard – on top of basic it offers, lower costs for throughput units and messages. Furthermore, you get 20 consumer groups, and a 1000 brokered connections.
- Dedicated – offers a dedicated environment with a maximum message size of 1 MB, 50 throughput units, a retention period of seven days, and 25000 brokered connections.
Storage Queues
Like Service Bus Queues, Azure Storage queue offer asynchronous messaging, and the publisher and consumer do not have to be connected at the same time. The message can be stored and picked up later by the consumer. A storage queue is suitable as a task queue, and messages can remain on the queue for a maximum of seven days.
Azure Storage Queues differ from Service Bus Queues – each serves a different purpose (see picture below).
See also Storage queues and Service Bus queues – compared and contrasted document on Microsoft docs for comparison of these cloud queue technologies.
When to choose what?
In the previous sections, we discussed the four Azure messaging services that are available. Now we will talk when to choose which Azure messaging service by using sample reference cases. Note that each of these Azure messaging services can be part of your solution depending on your requirements. Note that these services are not competing with each other!
Stream data into Azure
When devices and services generate large volumes of telemetry data then Event Hubs is a candidate service for you when you want to ingest that data. Subsequently, multiple applications can be based on a publish-subscribe mechanism consumes that data.
Telemetry data can be for instance log data (website visitors, click behavior) from your website. Furthermore, you want to process the data real-time with Stream Analytics and push specific page click behavior to Data Lake – and store visitor data in a Cosmos DB collection. To support this scenario, you create two consumer groups, one for a function that will store visitor events (for instance log in) to the Cosmos DB collection (user logins) and one for a Stream Analytics job – the results of individual queries are stored in Data Lake.
For pricing details of Event Hubs see the pricing page.
Handle events with Event Grid
The Event Grid service on the Azure Platform is suitable to route events coming from Azure resources like Event Hub, Azure Storage, and third party (custom) to any given event handler or WebHook. You can, for instance, create an Event Grid Topic, and send your application events to Event Grid to leverage its reliable delivery, advanced routing, and direct integration with Azure. Furthermore, you can make use of Event Grid and Logic Apps to process data without writing code.
Let’s assume you ingest public data, i.e. weather data using a Logic App with a schedule trigger mechanism. The Logic App processes an array of data and stores every part of the array as an individual json file in blob storage. The blob storage is configured with Event Grid, and one or more Logic Apps handles each blob created event.
For pricing details of Event Grid see pricing page.
Azure Messaging Service for Enterprise
With the Azure messaging in Service Bus, there are several scenarios you can think of where this service brings value. Service Bus offers, topics and subscriptions (pub-sub), and message queues. When you require a message broker in Azure, then Service Bus is fit for your solution.
Also, refer Service Bus message broker blog for good insights.
Assume you want to process flat-file batch files (1,2) in Azure through multiple micro-services – each service has one responsibility. The microservices (3) shown in the diagram below are activated by a message in a service bus queue. The queue is an intermediary between services and decouples them from each other. The microservices are packaged into a container and are hosted in a managed Azure Kubernetes Cluster (AKS). The microservices can scale, and this also accounts for the queues, i.e. they can handle the massive throughput of messages that can activate the microservices to process the EDI files (parse, translate, store) (4).
There are some samples available for the Azure Service Bus through the following link. For pricing details of the Azure Service Bus see pricing page.
Task Queue
Storage Queues are a part of the Azure Storage Account service. Furthermore, they feature a simple REST-based GET/PUT/PEEK interface, providing reliable, persistent messaging within and between services. You can consider storage queues:
- when you must store a significant amount of message, i.e. over 80 GB,
- time-to-live for your messages is less than seven days,
- your application needs to track progress for processing a message inside of the queue,
- or require server side logs of all of the transactions executed against your queues.
When for instance, you want to process a workload asynchronous or pass messages from one web role to a worker role or function, you can use a simple queue like Storage Queue. Similar to the sample in the Enterprise Messaging scenario you can use a Storage queue, however, if it needs to be in an ordered manner, then Service Bus queues are a better fit because of the strict ordering (First-in-first-out).
Assume you need to upload a large number of high-resolution images and each need to be resized and analyzed asynchronously – then images are stored first, and then task messages are sent to storage queues, i.e., task messages for resizing on one storage queue, and task messages for analysis on another.
The diagram above shows how you could use simple queues for a particular scenario. For pricing details of Storage Queues see the pricing page.
Summary
Four Azure messaging services each serving a purpose and have a fit in your architecture or solution. Below you will find a summary of these services, how they fit in a segment, characteristics, and what type of concept (Serverless, Big Data, Enterprise).
Wrap up
In this blog, we described the Azure messaging and event services. Each has its characteristics and can be the right service to adhere to your requirements. These services do not compete with each other and are in some use cases complementary to each other. Hopefully, this blog post has given you some insights in these Azure Messaging services and provide guidance when to choose what.
Another good read for determining what service is best for you is the blog post from Clemens Vasters – Events, Data Points, and Messages – Choosing the right Azure messaging service for your data.
Also, read Steef Jan-Wiggers blog on “How To Build A Reactive Solution With Azure Event Grid“.
