We have come across many business scenarios involving message resubmission to an Azure Service Bus entity with ‘Requires Duplicate Detection’ property enabled, and the requirement is to prevent message loss during resubmission. Turbo360 features are designed after a careful investigation into real word integration solution challenges and feedback from our existing customers. With a lot of attention to detail, we have built this capability to avoid message loss. Especially, when you try to resubmit messages to Azure Service Bus entities with ‘Requires Duplicate Detection’ property enabled. To understand this article, it is a pre-requisite to have knowledge of the ‘Requires Duplicate Detection’ property and ‘Duplicate Detection Time History Window’ property on an Azure Service Bus entity.
Requires Duplicate Detection
When set to true, the ‘Requires Duplicate Detection’ property will ensure that all enqueued messages with a duplicate value of the Message-Id property within the ‘Duplicate Detection History Time Window’ will be ignored.
Duplicate Detection Time History Window
If duplicate detection is enabled for a queue the ‘Duplicate Detection History Time Window’ value will specify how long the Message-Id values for the received messages will be retained to check for duplicate messages. The property is a Timespan with a default value of 10 minutes and a maximum allowed value of 7 days.
Now, consider a business scenario, where an integration solution orchestration has an application upstream, that pushes messages into a queue with the following properties:
- Requires Duplicate Detection: True
- Duplicate Detection History Time Window: 10 minutes
- Time To Live: 1min
NOTE: Once the entity is created with the ‘Requires Duplicate Detection’ property enabled, it can never be modified.
The application downstream is expected to process the active messages from the queue defined above.
Turbo360 can facilitate Queue property definition at the time of creation itself which is not possible in Azure Portal.
In the image below you can see that Turbo360 is displaying the properties of the created queue. By turning on ‘Enable Dead-Lettering on Message Expiration’ and sets ‘Default message Time to Live’ to 1 minute. With this configuration, as the application upstream enqueue messages to the created queue, after 1 minute the messages will automatically be dead lettered due to ‘TTL Expired’ exception.
Turbo360 has the capability to send messages to queue for testing purpose, to mock the application upstream. Send 10 messages to the queue and after 1 minute, the active messages will be dead-lettered.
The challenge with this scenario
In the above-explained scenario, the message in the queue get dead -lettered before the downstream application could process them. Even if the ‘Default message Time to Live’ is redefined to a higher value, the requirement is to resubmit the already dead lettered messages. As the ‘Duplicate Detection History Time Window‘ is 10 mins, while resubmitting the dead lettered messages, the message will be lost being detected as the duplicate message.
Solution with Turbo360
Turbo360 provides a solution to overcome the above challenge and achieve the above requirement. By enabling the ‘Requires Duplicate Detection’ property while resubmitting message, the system would provide the following options.
- Regenerate the selected single or multiple messages id (new GUID will be created and assigned to every message id)
- Edit the message id to meet your business requirement, like invoice/202#/####.
As the new message ids are regenerated, these messages will not be lost, when resubmitted to the queue with ‘Requires Duplicate Detection’ property enabled. Also, Turbo360 helps you to automate these tasks by creating Activities. For example, while creating a ‘Dead- Letter Activity’ on a ‘Requires Duplicate Detection’ property enabled ServiceBus Messaging Entity. Enabling ‘Regenerate Message Ids’ option will take care of resubmitting messages with the newly generated GUID as message Id.
Read more documentation on Turbo360 capabilities – Documentation link
