What is Auto-forwarding?
Azure Service Bus is a feature-rich broker. Among those multiple features is auto-forwarding feature. Auto-forwarding allows chaining queues and topics in the same namespace to move messages from one entity to another. To provide an idea of how this feature operates, let me tell you about my aunt Jolly.
Aunt Jolly is amazing. On everyone’s birthday, she sends a personal birthday card handwritten with the names of her three cats. No matter how many times we tried to “upgrade” her to “21st century”. She remained loyal to the old-fashioned way; sending cards using mail service to deliver to your mailbox rather than sending email greeting. She keeps all addresses, phone numbers are written down in her contacts book, she’s using to phone on a happy day to wish years of happiness and good health. You can imagine how devastated she’d be if would ever discover that her personalized card did not make it. And yet I’ve managed to save this stress from her despite my frequent address changes when moved around. All thanks to the mail forwarding service.
In case you’ve never used the service, here’s how it works. When you move and change your address, a postal forwarding is set up associated with your name and old address. This redirects everything to the new address if you’ve activated the service. The service guarantees that nothing will go to the old destination and always will arrive at the new destination.
Strengthen your Azure Service Bus monitoring, and get powerful toolsets and actionable insights to troubleshoot messaging issues with the help of Turbo360.
Auto-forwarding with Azure Service Bus is similar to mail forwarding service. The feature enables chaining entities such as queues and subscriptions to other entities in the same namespace. When enabled, messages arrive at a forwarding entity, the service automatically removes those messages and places in the new destination specified by the forwarding rule. It’s a lossless operation with exactly-once delivery semantics.
When do I need this feature?
Ironically, the feature is never appealing when designing a new system. But comes handy when an existing system is growing and you modify. For example, systems that made certain assumptions relevant for the time when built, but no longer holding after a while. An example of how auto-forwarding can become of value would be a classical story of inter-departmental development. Department A has develops its system and a queue for reports to dispatch via email. The queue is “email-reports”.
Meanwhile, another department, let’s call it department B, implements a similar service. But the queue to queue up reports for mail out is “reports-out.” At some point, both departments realize that running two different services to mail out reports was less than optimal and agree to consolidate those. Both departments had their systems in production, running and generating reports. Stopping those systems just for email consolidation would be far from optimal. Not to mention that there’s no gained value in redeploying the systems just to modify email service queue name. So how to consolidate email sending?
This is where auto-forwarding comes into the picture. To build, test and deploy a consolidate email delivery service. You use “email-service” as an input queue for every new development. But not for the already existing systems departments A and B. For those departments, their specific mail services were deprecated, and the appropriate queues re-configured to have auto-forwarding enabled to point to the new “email-service” queue. From then and on, any report sent from A’s system to “email-reports” or from B’s system to “reports-out” queue will auto-forward to the “email-service” queue to process by the consolidate email service.
Good things come in small packages
Auto-forwarding is not magic. It computes on the broker side that has to move messages behind the scenes. And as such, it has a cost associated with it. Now imagine making a mistake and setting up a circular reference as described in the following diagram:
Not only this would cause an excessive compute utilization, but it will also get the system in a faulty state by moving messages infinitely throughout circular referencing queues. That’s the reason why Azure Service Bus has a cap on a number of times a single message can be auto-forwarded. Each time you move a message from one entity to another is “hop”. Service Bus permits up to 4 hops. At the moment a message exceeds the maximum number of hops, the message will move to dead-letter with a reason. Stating that the message has exceeds the maximum number of hops. When planning to utilize auto-forwarding, keep this simple and hard rule of no more than four consecutive hops.
System hops vs. User hops
Every developer knows that if everything were simple in life, it would be extremely boring. Auto-forwarding feature is no exempt from this life wisdom. Users can configure auto-forwarding explicitly as well as implement implicit by other feature such as send-via. Send-via is a Service Bus mechanism to enable transactional message processing by utilizing a transfer queue. During the messaging transaction, the outgoing messages rather than been sent to the destination are first “staged” in the transfer queue of the incoming message queue. Once the incoming message is successfully complete. The transaction is marked as complete, the staged messages are forwarded to their destination(s).
When the staged message is moved from the transfer queue to the destination queue, that’s a hop. User-specified auto-redirects are counted as “user hops.” Send-via auto-redirect is a “system hop” and counts towards the maximum number of consecutive hops a message can have undergone.
Danger, Will Robinson!
A messaging topology describes the physical and logical layout of a networked messaging system. A topology represents the way you arrange the entities in the namespace and how they communicate with one another. When building more sophisticated topologies, it’s common to utilize auto-forwarding. Keep in mind the number of hops messages will have to undergo, including transactional processing scenarios i.e. if a message will be auto-forwarded several times and at its last hop processed in a transactional manner, the number of user hops cannot exceed 3 (maximum number of allowed hops minus the transactional processing hop). Failure to do so will cause headaches later.
Auto-forwarding use cases
Fanning messages in
As mentioned earlier, entities capable of auto-forwarding are queues and topic subscriptions. In scenarios where you use multiple entities to receive and process messages, processing can be consolidate by auto-forwarding all messages to a single queue for processing. For example, a service can create multiple subscriptions. Where you use a single SQL filter per subscriptions for simplicity of management. Rather than processing messages off every subscription, auto-forwarding can redirect all filter out messages to deliver to a single subscription or queue for processing. This would allow monitoring and scale out/in the service without trying to do this across multiple subscriptions the service creates throughout its lifetime. This pattern also significantly simplifies dead-lettered queues monitoring, which is the next use case.
Centralized dead-letter queue
Each queue and subscription have a dedicated dead-letter queue to store poison messages. Production systems tend to have a significant number of those. With each one having its dead-letter queue, trying to monitor all dead-letter queues can quickly become a source of pain. Centralized dead-lettering offers a convenient way to monitor a single entity for poison messages thanks to auto-forwarding. To implementing this pattern, queues or subscriptions must be configured to auto-forward dead-lettered messages to the centralized dead-letter entity which is just an arbitrary entity. Thanks to Azure Service Bus service, auto-forwarded dead-lettered messages will be enriched with headers to provide the reason message has been dead-lettered, as well as the original queue where messages became poisonous.
Check out the blog on Azure Service Bus delete message to understand how to resubmit and delete messages in Azure Service Bus Messaging.
Temporary message redirection for reprocessing
In some situations, there’s a need to move messages from one entity to another for further investigation and reprocessing. The simplest and easiest way to achieve that is to utilize auto-subscriptions for a short period, removing it shortly after.
De-duplication
Next use case is a case where a system that has not been originally designed to handle duplicate messages can encounter those as a result of system modifications and is required to perform de-duplication.
Below is as an example of a system that has implemented fan-in pattern to consolidate messages. Unfortunately, it has also introduced an edge case where duplicates get created and processed. De-duplication was not turned on for “entity-3” and changing that is no longer an option without deleting and recreating it. In production, removing and re-adding “entity-3” is not possible. For de-duplicated messages coming to “entity-3”, an additional entity is set up, “dedup” with RequiresDuplicateDetection property set to true and the desired de-duplication time windows specified via DuplicateDetectionHistoryTimeWindow property. “Entity-3” will auto-forward all fan in messages and send to “dedup”. Where duplicates get eliminate, ensuring system gets each message only once.
It’s essential to consider a number of hops messages that will undergo to ensure they do not exceed the maximum consecutive number of hops permitted.
Building complex topologies
Moreover, last, but not the least use case for using auto-forwarding is building complex topologies. Systems that require sophisticated topologies usually tend to leverage auto-forwarding one or another way. This either flow messages to specific entities or to follow some kind of convention. These topologies would employ all the patterns described above to achieve what’s required. A rule of thumb for building a successful topology is to take advantage of the native features Azure Service Bus provides. But keeping in mind that any advanced feature (such as auto-forwarding) is an additional compute that comes at a cost.
To learn more about auto-forwarding feature, visit Microsoft documentation.
