Welcome again to another BizTalk Server to Azure Integration Services blog post. In my previous blog post, I discussed how you can migrate a BizTalk request-response routing solution with LOB Adapters. Today, we are to address simple one-way routing solutions inside the BizTalk Server platform.
In the realm of enterprise application integration, the concept of routing plays a pivotal role. Specifically, in Microsoft’s BizTalk Server, message routing and content-based routing (CBR) emerge as vital features, enabling businesses to efficiently manage and route messages based on their content and data.
It is quite common to see BizTalk Server clients with a File Transfer application on their systems, where, for example, a system creates a flat file on a local folder, and BizTalk Server picks the file and sends it to an FTP or a Secure FTP. You may be thinking, this is old school.
We don’t do this kind of integration anymore! Yes, it can be old school, but it works like a charm, and don’t be fooled if you think that organizations will want to invest in changing these processes. They don’t! So, that means that we will still need to implement these scenarios in Azure.
What are content-based routing and basic routing?
If you are unfamiliar with the term content-based routing, Content-Based Routing is a messaging pattern and feature of Microsoft BizTalk Server that allows you to route messages based on their content, properties, or context.
With content-based routing, you can define routing rules that determine where a message should be sent (or subscribed) within your integration solution based on the data within the message or its associated properties – in BizTalk Server context, it will typically be a Send Port (or send Port Group), but in rare implementations, it can also be an Orchestration.
This helps direct messages to their appropriate destinations, a fundamental aspect of business process automation and integration. At its core, content-based routing in BizTalk Server revolves around the idea of inspecting a message’s content and making routing decisions based on this analysis.
Unlike traditional routing, which typically depends on fixed endpoints, CBR allows for the dynamic determination of destinations based on the data within the message itself. This capability is particularly beneficial in complex business scenarios where messages need to be dispatched to different systems or workflows based on specific criteria.
Where do promotions happen inside the BizTalk Server?
When we speak about promotions inside BizTalk Server, we usually associate it with the process of making certain parts of a message easily accessible and identifiable to the BizTalk engine. This is typically done for routing and processing purposes. There are two types of promotions in BizTalk:
- Property Promotion: This involves promoting specific fields from a message to the message context. This makes these fields easily accessible for various purposes like routing, correlation, or tracking. Property promotion is configured in the schema of the message type within BizTalk. > This usually happens on the Receive Pipeline.
- Distinguished Fields: These are similar to promoted properties but are used differently. Distinguished fields are also selected from the message and made accessible in the message context, but they are used for orchestration business decision purposes. They cannot be used for routing or correlation.
However, there are several stages and components that add meta-information to the context of a message through the use of message context properties (some of them are promoted that allow routing). The message context is a collection of metadata that travels with the message as it moves through the BizTalk Server. This context includes both system properties, which BizTalk predefines, and custom context properties, which users can define. Here’s a brief overview of how and where this meta-information is added:
- Promoting Properties: When you promote a property in a message schema, this property becomes part of the message context. This is done in the BizTalk Schema Editor, where you define your message schemas. Promoted properties are used for routing and correlation purposes.
- Pipeline Components: When a message is processed through receive and send pipelines, various components within these pipelines can add, modify, or remove properties in the message context. For instance, the XML disassembler component in a receive pipeline can promote properties from the message to the context.
- Orchestrations: Within a BizTalk orchestration, you can manipulate the message context. This includes adding custom context properties or reading existing ones. This is done using expression shapes and message assignment shapes within the orchestration.
- Custom Pipeline Components: You can create custom pipeline components to perform specific operations on the message context. These custom components can be integrated into either receive or send pipelines to modify the message context as needed.
- Adapters: Some adapters in BizTalk Server can also add specific context properties to messages. For example, the HTTP adapter can add properties related to the HTTP transport, like HTTP URL, method, etc. Or the FILE adapter will promote the file name.
Here is a diagram of the BizTalk Server runtime architecture and where message context is added to the message:
BizTalk Server Publish and Subscribe Architecture
To understand the BizTalk Server solution and how we can design a similar solution in Azure, we also have to understand the pub/sub architecture of BizTalk Server. Everything, and I mean everything, in BizTalk Server is pub/sub!
BizTalk Server uses a publish and subscribe (or pub/sub) messaging model. The pub/sub model is highly scalable at both the database and processing levels. This pub/sub-routing mechanism can manage large volumes of messages and large individual messages and can interact with a wide variety of back-end systems. This design also makes adding and changing subscribing components on the fly extremely easy.
- Publication
- Publication is the process by which messages are placed in the MessageBox database. The principal publishing component in BizTalk Server is the receive port, which receives and processes messages into the database. Orchestrations also publish messages as they are sent from the orchestration to the MessageBox database for further routing.
- Subscriptions
- Subscriptions are criteria on which message routing is determined. Orchestrations, send ports, and send port groups can each subscribe to messages. Each subscription allows the subscriber to initiate or continue the processing of a message. Subscriptions are managed by the MessageBox database.
- When a message meets the specifications of a subscription, the message is passed from the MessageBox database to the subscriber. If multiple subscribers exist for a given message, each gets a copy of the message.
Within the Azure Integration Services offer, we only see these pub/sub capabilities in Azure Service Bus that can easily replace the BizTalk Server MessageBox, but we have to build the surrounding pieces. With that, I mean that we don’t have the concept of receiving/sending ports and pipelines on Azure, so we need to replace them.
Common BizTalk Server Routing Scenarios and Design Approaches with Azure Integration Services
Simple File Transfer
As I mentioned before, it is pretty standard to see BizTalk Server clients with a File Transfer application on their systems, where, for example, a system creates a flat file on a local folder, and BizTalk Server picks the file and sends it to an FTP or a Secure FTP.
BizTalk Server was considered many years ago as a no-code, low-code platform, something that Microsoft tries not to associate anymore and bind that term to Azure Integration Services, but in fact, BizTalk Server is and still is in many cases a no-code, low-code platform! We don’t need a single line of code or a Visual Studio solution to accomplish this. What we need is:
- To create a receive port with a receive location, specify the FILE adapter to read the file type from a shared network or a local folder. It is important here to specify the out-of-the-box PassThru Receive Pipeline. This pipeline contains no component and is used mainly for basic message routing.
- Then, create a send port, one this time with a PassThru Send Pipeline, and add a filter to subscribe to all messages from the receive port.
The goal now is to understand how we can accomplish this with Azure Integration Services (AIS).
In this simple scenario, in my point of view, the most straightforward and most practised approach is to:
- Create a Logic App that reads the files from the FILE System (or any other place) and sends it to FTP.
Simple File Transfer based on the file name structure
This is a similar scenario: a system creates a flat file on a local folder, and BizTalk Server picks the file and sends it to an FTP or a Secure FTP. However, the file name structure includes the partner for each we want to send and the type of the document:
- File naming convention: <partner>–<msg-type>-<doc-id>.csv
So that means the BizTalk Server needs to slip the name to extract these values and promote them to the context of the message to route them to different partners.
Once again, this is relatively easy to accomplish in BizTalk Server. For that, we need to:
- Create a Visual Studio Solution and
- Add a new property schema with two fields: client name and message type
- Create a simple custom receive pipeline component that reads the file name and promotes the client name and message type into the context of the message.
- Finally, create a custom receive pipeline and add the above custom receive pipeline component into the first stage of the pipeline.
- To create a receive port with a receive location, specify the FILE adapter to read the file type from a shared network or a local folder. It is important here to specify the custom Receive Pipeline we created earlier.
- Then, create one or more send ports, configure the channel (FTP, SFTP, …), set the PassThru Send Pipeline, and add a filter to subscribe to the expected client name and message type.
How can we accomplish this with Azure Integration Services (AIS)?
Of course, we could do it only with a single Logic App like the previous one, but that means if we have another partner tomorrow, we need to change and redeploy the entire solution and also suffer a minor offline. Something we want to avoid. We want to create a more decoupled solution.
To accomplish that, we can implement the following architecture:
- Create a Logic App that reads the files from the FILE System (or any other place), parses the file name, and sends it to a Service Bus Topic. Here, the Logic Apps also promote into the context of the message, the client name and the message type.
- On the Service Bus Topic, we will create one or more subscriptions for each partner and message type.
- Finally, we will create a Logic App for each partner and message type that subscribes from the topic subscription and sends it to the expected destination.
A very simple and elegant solution. However, we have to take into consideration the size of the messages. If the messages are greater than 256 KB (standard tier), then this approach will not work, and we need to arrange a different approach.
A different approach with identical behaviour is replacing the Service Bus with:
- An Azure Function that reads configuration from App Configurations.
- An App Configuration to store the URL of the partner/message type Logic Apps.
Then, the Logic App reads the files from the FILE System (or any other place) and parses the file name. After that, we will invoke an Azure Function to read the configuration from App Configuration – we need an Azure Function because it doesn’t exist the App Configuration connector for Logic Apps. Finally, by using the HTTP Connector, the Logic App will redirect the file to the expected “child” process for that specific partner and message type.
If there is a need to create a new process for a new partner or message type, we just need to create that “child” process, a.k .a. Logic App, and store the URL trigger of that LA in the App Configuration. Simple as that.
Now, if the message size is bigger or equal to 100MB, then the use of Logic Apps is out of the equation, and we need to arrange a different solution.
Content-based routing
Finally, if we are working with content-based routing, that normally means that we will extract data from the message, like the partner name, and add it to the context of the message in order to be routed in the system.
Content-based routing is very easy to accomplish in BizTalk Server and is quite the same approach as the previous one but even simpler. In this case, we need to:
- Create a Visual Studio Solution and
- Create or import the schema of the message and promote the desired element. This automatically creates a property schema with all the promoted fields: client name.
- To create a receive port with a receive location, specify the FILE adapter to read the file type from a shared network or a local folder. It is important here to specify the out-of-the-box XML Receive pipeline.
- Then, create one or more send ports, configure the channel (FTP, SFTP, …), set the PassThru Send Pipeline, and add a filter to subscribe to the expected client name.
How can we accomplish this with Azure Integration Services (AIS)?
The solution on AIS will be precisely the same as the previous one:
- A Logic App reads or receives an XML, parses the XML, extracts the values we want to promote, and sends it to a Service Bus Topic. Here, the Logic Apps also promote into the context of the message and the client name.
- On the Service Bus Topic, we will create one or more subscriptions for each partner.
- Finally, we will create a Logic App for each partner subscribing to the topic subscription and sending it to the expected destination.
Migrating BizTalk Platform one-way routing solutions
Of course, we can take this opportunity, if possible, to modernize our solution. For example, if the partners now expect a JSON and not an XML file, the Logic App that is receiving the messages can translate them to JSON and send it to the Service Bus. Once again, this works depending on the size of the messages.
I hope you find these architecture samples useful and stay tuned for more BizTalk Server to Azure Integration Services.
