Introduction
The integral part of any Enterprise application is communication between the micro-processes and the sub-applications involved in the orchestration designed to provide a solution to the business problem. This inter-process communication can be achieved through message passing. Microsoft Azure facilitates the message passing service through Service Bus Queues and Topics.
Partition Enabled Queues and Topics
In conventional Service Bus Queues and Topics, all the messages are stored in a single message store and all the requests to that Queue or Topic are handled by a single message broker. This will directly impact the performance of the Queue or Topic.
In partitioned Queues and Topic Subscriptions, the messages will be stored in multiple fragments (Partitions) and the requests are handled by the dedicated message broker of that partition.
This improves the performance of the Service Bus Queue or Topic as it eliminates the bottleneck of the single message store and message broker serving multiple requests.
Working of Partitions
There are multiple partitions with the partitioned Service Bus Queue and Topic. Each partition will be stored in a separate message store and handled by its own message handler. Messages are sent to the partitions based on the partition key or in a random manner.
Whenever messages are requested in a partitioned Service Bus Queue or Topic, the messages are queried across multiple partitions or message stores. The first message recieved will be returned to the receiver.
All the remaining messages will be cached in the Queue and returned if there is any further request for the messages. The message reception of receiving client is the same for both the conventional Queues and the partitioned Queues.
Partition Key Selection Algorithm
Based on the message properties sent to the partition Queue, the partition key is chosen to direct the message to different partitions.
Session-Id
If a message sent to the Service Bus Queue has the session-id property set, then the provided session-id will be used as the partition key. The messages with the same partition key will be stored in the same partition. This ensures that the messages that belong to the same session are stored in the same message store and are being handled by the same message broker. The session enabled Queues can be used when ordering in messages is mandatory.
Partition Key
The partition key property will be used to identify the partition within the Queue where the message must be stored when the session-id property of the message is not set. When both the session-id and partition key of the message is used, it is important to make sure that both the session-id and partition key are the same. On failing the condition, the message will be treated as invalid and an exception will be encountered at the message transmission end.
Message-Id
If a Service Bus Queue or Topic is a duplicate detection enabled entity, then the message-id of the message will be treated as the partition key. This helps the Service Bus in removing the messages with duplicate message-id. The main idea behind this concept is the messages with the same message-id are handled by the message broker, making sure that only messages with unique message-id within the specified time window exists in the Queue or Topic. This applies only when the Queue or Topic is enabled with Duplicate Detection property.
Random Algorithm
If none of the above scenarios are met, the messages are stored in multiple partitions based on a round-robin manner. A message is stored in any one of the partitions if a partition chosen from the algorithm is not available. This assures the messages are successfully stored in any of the message stores even if a message store is not available at the message transmission.
The major disadvantage of this message handling is that the message ordering due to the partition key provided is not guaranteed.
The Operation Timeout parameter set while sending the message decides the appropriate assignment of messages to the partitions within the queue. The minimum value for the operation time out should be at least 15 seconds. It is recommended to maintain the default value of Operation Timeout (60 seconds).
Using Transactions with Partitioned Queues and Topic Subscriptions
When messages must be sent to a partitioned Queues and Topic Subscriptions within a transaction, it is mandatory to specify the partition key to the message. The partition key can be either a session-id, partition key or message-id. As mentioned above, the messages sent to session enabled entities must have the session-id property set.
using Microsoft.ServiceBus.Messaging;
using System;
using System.Transactions;
namespace PartitionEnabledQueues
{
class PartitionEnabledQueues
{
static void Main(string[] args)
{
MessagingFactory messagingFactory = MessagingFactory.CreateFromConnectionString("connection string");
var queueClient = messagingFactory.CreateMessageSender("EntityPath");
CommittableTransaction committableTransaction = new CommittableTransaction();
using (TransactionScope transactionScope = new TransactionScope(committableTransaction))
{
BrokeredMessage brokeredMessage = new BrokeredMessage("This message is from Turbo360");
brokeredMessage.PartitionKey = "Partition Key";
queueClient.SendAsync(brokeredMessage).GetAwaiter().GetResult();
transactionScope.Complete();
}
committableTransaction.Commit();
Console.ReadKey();
}
}
}
Scenario to Use Partition Enabled Queues and Topics
Consider an e-commerce scenario, where customers from different geographical regions place their orders. All the orders are served by multiple order processing applications each responsible for a geographical region.
The orders are maintained in Service Bus Queues as messages. If conventional Service Bus Queue is used to serve the orders to the order processing application, the performance will be degraded by the performance of the message store and message broker as all the requests are handled by the same message broker.
This degradation in performance of the order processing application can be eliminated by using the partition enabled Service Bus Queue. The orders from each geographical region can be sent to the Service Bus with geographical region name as the partition key of the message. The messages with the same partition key will be stored in the same partition within the Queue. This will reduce the overhead of the Service Bus in serving the requests of the order processing application.
Drawbacks with Partition Enabled Queue and Topic
- Partition enabled Queues and Topics can be created only in Standard Namespaces and not in Premium Namespaces.
- It is possible to create only up to a maximum of 100 partitions enabled Service Bus Queues or Topics within a Service Bus Namespace.
- It is possible to send only the messages belonging to the same session within a single transaction.
Partition Entity Handling in Turbo360
Turbo360 allows us to create Service Bus Queues and Topics with partitions.
Messages can be sent to the partition enabled entities with a partition key and session-id values set using the automated activities of Turbo360.
It is possible to view the partition key of the messages sent to the Service Bus Queues and Topic Subscriptions.
Strengthen your Azure Service Bus monitoring, and get powerful toolsets and actionable insights to troubleshoot messaging issues with the help of Turbo360.
Conclusion
Hope this article meets the purpose of understanding the real-time use case of partition enabled Service Bus Queues and Topics. Also, it describes the clear advantages over conventional Azure Service Bus.
