When working and dealing with asynchronous messaging patterns – in this case, using Azure Service Bus – depending on the requirements, you will find many scenarios when you need to reprocess messages. Sometimes, a message failed because a system was offline for a certain period, there was a bug with the service, and we needed to resend specific messages and many other reasons. Sometimes, we need to design our processes for re-entry or have the availability to resubmit messages without requiring them from the original systems.
Of course, to accomplish this, we have many options:
- From designing our processes to be robust and reentrant, for example, implementing try-catch mechanisms inside Logic Apps, and if there is a failure, we re-insert the message back to the Service Bus.
- Using Azure Event Hubs to store the messages that arrive through the Service Bus for possibly later being re-processed.
- Using the Dead-letter queue.
- Using Turbo360 to resubmit & delete messages in Azure Service Bus.
- And probably many other options.
It is also essential to understand what you consider to replay or resubmit messages that were processed by Service Bus:
- Is it about resubmitting all messages on the dead-letter queue?
- Is it about resubmitting a specific message on the dead-letter queue?
- Is it about resubmitting a specific message that, from an integration point of view, failed to integrate?
- Is it about resubmitting a specific message that, from an integration point of view, was successfully integrated but, for some reason, failed internally on the external system, and they need the message to be resent?
Depending on the scenario and what you consider to be your understanding of resubmitting/replaying Service Bus messages, each has different types of implementations.
In this two-part POC Tutorial, we discuss how we can replay messages from Azure Service Bus. In this first part, we explore how we can resend messages that have expired or, for some reason, entered the dead letter queue, and later, in PART 2, we explore how we can resend a message of our choice.
When dealing with Azure Service Bus, if you are not careful, sometimes some messages can get lost for various reasons. Still, the most common is that the Filter Properties do not comply with the Subscription Filters, and in those cases, they will not be consumed by the subscribers. In other cases, the messages spend too much time without being consumed or fail to be delivered to the subscriber. Both situations led to the messages being sent to a dead letter queue (the dead letter queue can be configured to exist when we are creating a new topic or queue or after the creation).
Sometimes, when this happens, we would like to be able to replay (or resubmit) the messages that have been sent into that dead letter queue and resend them into the topic/queue again to be consumed, so how can we achieve that?
A way to replay dead-letter queue messages?
Of course, there are multiple ways you can “force” to reprocess messages that exist on the dead-letter queue:
- By using Service Bus Explorer.
- By using Turbo360.
- By using Functions.
- And many more…
- or by using a Logic App!
But first and foremost, you need to create an Azure Service Bus namespace and to do that, you need:
- On the Portal Search bar, search for Azure Service Bus
- Click on it, and next click on + Create
- Next, you will have some fields to populate:
- Subscription
- Resource Group
- Namespace name
- Location
- Pricing tier: Regarding the pricing tier and for the sake of this document, we are using the Standard pricing tier to access the Topics.
- Next, click on Review and Create and create the Azure Service Bus Namespace.
After creating the namespace, you will see that you can create Queues or Topics. Let’s go with the last one.
- So click on +Topic
- Give a name to the Topic you want to create and click on Create.
- Next, click on the Topics on the lateral menu to access the topic we created
- Click on the name of the Topic, and this way, you enter the Topic.
As previously explained, within the topics, you can establish subscriptions. A topic subscription is a named virtual queue that acts as a message destination for subscribers. Each subscription is associated with a specific topic and can have its own set of rules and filters for receiving messages. Clients who have subscribed to this particular topic will subsequently receive messages that pertain to this specific subscription.
Next, proceed to:
- Inside the previously created topic, click on the + Subscription button to initiate the creation of a new subscription.
For the purposes of this Proof of Concept (POC), we will intentionally set a brief lifespan for the messages. This aligns with the testing scenario where messages may end up in the dead letter queue either due to message expiration or filter-related issues.
Here are the steps to follow:
- Provide a name for the subscription.
- Set a short time to live for the messages. (15 seconds)
- Specify a brief lock duration. (20 seconds)
- Enable dead lettering upon message expiration.
- Enable the Move messages that cause trigger filter evaluation exceptions to the dead letter subqueue.
- Now, if you click on your subscriptions inside the topic on the left menu, you will see that you have your subscription created, so click on it.
- Now, inside the subscription is where you will insert the filter that you want to be associated with your messages so they can be consumed by those who are filtering for receiving these kinds of messages.
- As you can see, you just need to click on +Add filter and write the SQL filter you want to be associated with your message; these are like key pair, Key, and value.
user.DemoFilter = 'Demo'
Now, you will be ready to send a message to this topic, and to do so, you need to know that, like for HTTP requests, we typically use Postman. In cases like these, we will use Service Bus Explorer:
- You can download Service Bus Explorer from this page.
- After you download and extract the files, click on the one we show you below:
- This will open the Service Bus Explorer.
What you will have is a blank canvas, as you can see in the image, and to change that, you need to create a connection to your Azure Service Bus Topic. To do that, click on File – Connect. A new Panel will open, and in the dropdown menu, choose Enter Connection String. But where is this connection string, and how to get it?
- Back in Azure Service Bus in the Azure Portal, access your topic, and click on Shared Access Policies on the left panel.
- Next, click on Add and give the SAS Policy a name. By default, the RootManagedSharedAccessKey policy is basically the owner or master user. You will see that the Manage option is selected, which means automatically, the Send and the Listen will be enabled also.
- Now, clicking on the Policy you just created, you will see the Primary Connection String.
- Copy that Primary Connection String and paste it on the Panel we were working on Service Bus Explorer.
- Click OK on the Connect to a Service Bus Namespace panel inside Service Bus Explorer, and you should now be connected to your Topic.
- Next, right-click on the topic and choose Send Messages.
- That will open a new window where you should be able to define the message payload and properties to be submitted to your topic.
- To submit a message or several messages (also possible), you just need to click Start.
- After that, you will see that one or more messages have arrived and are active in your subscription, but if you keep refreshing about 20 seconds later (what we have defined previously), the message will be moved to the dead-letter queue.
From here, we know that those messages will stay there and can no longer be consumed, so how can we achieve the goal of replaying those messages into the queue again, containing the same information?
Well, once again, there are multiple ways you can accomplish that, for example:
- By using Service Bus Explorer.
- By using Turbo360.
- By using Functions.
- And many more…
Today, we will describe how you can automate this process using a Logic App! To do that:
- Access the Azure Portal and search for the Logic Apps on the search bar. And then click on it.
- We will create a Logic App consumption, so give it a name relevant to you, your subscription, a resource group, and the location, and click on review and create, and create.
- This is a simple POC, so we are going to keep our business logic straightforward. In real cases, we can implement some additional logic to decide what to reprocess or not. This will be the look of our Logic App.
- What we have here is a Trigger from Azure Service Bus for When a message is received in a topic Subscription (auto-complete), you can configure the connection with Azure Active Directory and provide the details for connection or by using, for example, Logic App Managed Identity or a Connection String.
- After the connection is established with the Namespace of your Azure Service Bus, you can now configure the Trigger.
- So, we have the Topic name and the subscription name, and as for the subscription type, we want to deal with the messages from the dead letter queue.
- Then, you can define how often you want to pull the messages from the dead letter.
- Now, if we send a message through Service Bus Explorer to our topic, after 20 seconds, the message will be in the Dead-letter queue, and it will be picked by our Logic App. The outputs of the Service bus Trigger will contain all information about that message. You will see all of this information and sometimes more:
{
"DemoFilter": "Demo",
"DeadLetterReason": "TTLExpiredException",
"DeadLetterErrorDescription": "The message expired and was dead lettered.",
"DeliveryCount": "1",
"EnqueuedSequenceNumber": "75",
"EnqueuedTimeUtc": "2023-08-29T14:37:27Z",
"ExpiresAtUtc": "2023-08-29T14:37:42Z",
"Label": "Service Bus Explorer",
"LockedUntilUtc": "2023-08-29T14:40:10Z",
"LockToken": "6619cac9-ea3f-4843-bebf-e344b1af426a",
"MessageId": "5fcde388-9bf8-4aa4-835e-83d763f40e53",
"ScheduledEnqueueTimeUtc": "0001-01-01T00:00:00Z",
"SequenceNumber": "1",
"Size": "319",
"State": "Active",
"TimeToLive": "150000000"
"Content": "This is a demo! in a base64 string"
}
- Knowing this, we can now use another shape on our Logic App process to send the messages into the topic or queue once again.
- So, we just added a new action from the Azure Service Bus, this time a Send Message. And we configured it like this:
- You can add more parameters according to your needs, but to make this work, you will definitely need the Properties. This is because it was there that the Key and Value are passed, and the filter in the subscription can recognize that as a valid message.
- Now, if you save the Logic App, wait 30 seconds, and wait again for 30 seconds, what you will see is that the Logic App runs after these 30 Seconds, getting the messages that are entered in the dead letter queue and replaying them into the topic main queue, and because we have settled a short time to live for the messages in that queue (15 seconds) and a short time to lock the messages (20 seconds), in a way this could become an infinite loop – Be aware of that! This is just a POC.
- As you can see in these runs, the four messages entered the main queue – after 20 seconds, were sent into the dead letter queue – and the Logic App processed those messages again into the topic main queue, and that repeated again.
Once again, despite being simple to automate, the reprocessing of messages stuck in the dead-letter queue is important to consider implementing additional logic to control how many times we are going to resubmit the messages back to the main topic or queue to avoid infinite loops of the messages. At some point, we need to give up that message or store it in an external system in order to be analyzed.
One thing that it is virtually impossible to accomplish using default capabilities inside Service Bus is to reprocess a specific message based on some identifiers; let’s say I want to reprocess the order message X from client Y that is in failure.
In the next part, we will explore how we can replay a single message of our choice.
Hope you have enjoyed this POC, and we will see you in the next one.
