Aggregation mapping pattern in BizTalk to Azure Integration Services migration

Let’s embark on a new journey as we begin a series of blog posts dedicated to the migration of BizTalk Server to Azure Integration Services. I’d like to highlight that when I mention the migration to Azure Integration Services (AIS), I’m making a clear distinction from Logic Apps.

This differentiation is important because, contrary to what some consultants and salespeople may suggest, migrating BizTalk Server entirely to Logic Apps is not a viable path! If we do a straightforward comparison with BizTalk Server, Logic Apps is the equivalent of an orchestration, and BizTalk is much more than an orchestrator.

Role of aggregation pattern in BizTalk to AIS migration

It is pretty common to find BizTalk Server solutions/processes that receive a request and, based on some information from that request, obtain more information from another internal or external system. Then, we need to combine the two messages and map certain data from both messages to generate a message or request to be sent to a third system.
Afterwards, it becomes necessary to combine the two messages and extract specific information from both, culminating in the generation of a message or request to be dispatched to a third system. This is a well-known mapping pattern: Aggregation Pattern.

Basically, the Aggregation Pattern could also be another example or a subset of the content enricher pattern (you can know more about this here: BizTalk mapping patterns & best practices), but sometimes when we exchange messages between different systems, we will need to gather information from multiple external sources, this is also known as scatter-gather pattern, and once a complete set of messages has been received, we need to process them as a whole and combine or merge parts of information from each correlated message to create the expected message by the target system.

Difference between Aggregation pattern and content enricher pattern

So the main difference between the Aggregation Pattern and the Content Enricher one is that, in the Content Enricher, we’re usually talking about mapping one-to-one messages, and in the Aggregator pattern, we are dealing with multiple inbound messages that were collected from the original request and need to be mapped and aggregated into a single outbound request. For example, we want to bill the client’s order after all items have been pulled from the financial system or warehouse. Also, Content Enricher can happen inside the Aggregation Pattern along with other types of patterns.

Reference to this pattern– Enterprise Integration Patterns: Aggregator

In BizTalk Server, this is a pretty simple operation since BizTalk Server orchestrations allow the creation of maps with multiple inputs and outputs through the use of a Transform shape. In order to maximize the value and enjoyment of this exercise, why don’t we begin by crafting a BizTalk Server solution from the ground up and then migrate it?

The purpose of this solution is as follows:

• Receive a request with an individual’s information.
• Retrieve additional details from the person using their ID (as we lack an alternative system, we will simulate this interaction by generating a static response message).
• Merge the two messages and provide the combined output as the result.

Building the BizTalk Server solution

To create our BizTalk Server project to implement these requirements, we need:

• Open Visual Studio and Create an Empty BizTalk Server Project, giving a name to that project and a location, then click OK

• Looking at your Solution Explorer, this is what you will get:

• Next, right-mouse-click on the project and click on Add > New Item > Schema
  • Give a name that makes sense to you to the schema and click on Add (we named it Person)

• By default, that schema comes with a root named Root, but you can and should change it. In this case, we also called it Person.
• Next, right-click on the Person root and click on Insert Schema None > Child Field Element, and we added four child field elements:
  • Id
  • FirstName
  • LastName
  • Address

• Next, we added another Schema, this time we named it: PersonDetail.
  • Add the following Child Field Elements:
    • Id
    • Birthday
    • Gender
    • Phone
    • MaritalStatus
    • FamilyHousehold

• Next, we added a third schema. At this time, we named it ExternalSystem – This will be the output schema message.
  • And add the following Elements:
    • FamilyHouseHold
    • FullName
    • Gender
    • MaritalStatus
    • Birthday
    • Address

• Now, we need to create an Orchestration to create the business flow and create the map that is going to support this aggregation transformation. Now, the goal here is not to explain in detail how you create a BizTalk Server orchestration but for you to have an overview, these are the main steps:
  • We have created three messages on the orchestration viewer each of these messages is defined with the schemas we created above.
  • And one XmlDocument variable.
  • In this case, we will receive a Person message that will trigger the orchestration – Point 1.
  • In a real case scenario, we will transform this incoming request into a request to a web service for example and we will invoke that expecting a response – message PersonDetail – Point 2.
  • In this exercise, you will not see point 2 implemented and instead, we will emulate this by setting up a static response with a PersonDetail information- Point 3.
  • Then we created a map using the Transform shape to apply the requirement transform between the two messages and the ExternalSystem schema to produce our response – Point 4 – see more details down.
  • Finally in Point 5, we send the response back to the caller with the result of the aggregation.

• The correct orchestration flow will be like this:

• Next, to create the correct map, we have to use the Transform shape we add in the orchestration. For that, double-click on the Transform shape and set the following configuration:
  • Select the option New Map.
  • Apply a decent name to your map, for example, mapPersonDetailToExternalSystem.
  • On the Transform table, set the following source schemas:
    • Person
    • PersonDetails
  • And still on the Transform table, set the following destination schemas:
    • ExternalSystem
  • Click OK.

• The Transform shape will allow us to create a new map through the Transform shape configuration window, and from there, we can define one or several source schemas and one or several destination schemas. In this case, we are going to define two source schemas and one destination schema. Once we click OK, being the scenes, BizTalk Server will create an internal schema with the following:
  • InputMessagePart_0 containing the schema of Users
  • and InputMessagePart_1 containing the schema of Addresses

• From there, you can apply all the transformations needed to create the destination message according to the requirements.

If we deploy this solution, configure it, and test the solution, what happens is that, giving, the following input message:

<ns1:Person xmlns:ns1="http://BTS.AggregationPattern.Person">
   <Id>1</Id>
   <FirstName>Sandro</FirstName>
   <LastName>Pereira</LastName>
   <Address>Carvalhos</Address>
</ns1:Person>

And emulating the WebService result response with:

<ns2:PersonDetail xmlns:ns2="http://BTS.AggregationPattern.PersonDetails">
   <Id>Id_1</Id>
   <Birthday>04-04-1978</Birthday>
   <Gender>M</Gender>
   <Phone>916688115</Phone>
   <MaritalStatus>M</MaritalStatus>
   <FamilyHousehold>3</FamilyHousehold>
</ns2:PersonDetail>

We would get the following result after the transformation:

<ns0:ExternalSystem xmlns:ns0="http://BTS.AggregationPattern.ExternalSystem">
   <FamilyHouseHold>3</FamilyHouseHold>
   <FullName>Sandro Pereira</FullName>
   <Gender>M</Gender>
   <MaritalStatus>M</MaritalStatus>
   <Birthday>04-04-1978</Birthday>
   <Address>Carvalhos</Address>
</ns0:ExternalSystem>

Quite simple, don’t you think?

The challenge

Now, the main goal here, and what we really pretend, is to migrate this BizTalk solution to Azure Integration Services. This is a straightforward integration that, at first glimpse, is quite simple to migrate; however, there is one big challenge by default: the Transform XML action inside Logic Apps does not allow us to provide multiple inputs like the BizTalk Server Transform shap! So the biggest question is: How can we implement this Aggregation Mapping Pattern inside Logic Apps?

To migrate this solution using Azure Integration Services, we need to use one of these two stack options:

• Logic App Consumption + Integration Account
• or Logic App Standard

For this proof-of-concept, we decided to use the first option:

• Logic App Consumption + Integration Account

Building the Logic App solution

To migrate this BizTalk Server solution, maintaining the exact requirements, we need to:

• First, create a Logic App Consumption – we are using a Logic App Consumption, but you can apply the same in Logic App Standard – give it a subscription, a resource group, and a location, and click on Create.
• And this will be the final look of our Logic App will be like this.

• First, start by adding the Request > When a HTTP request is received trigger, and then you can save the Logic App. This will generate a link that you can copy to be used to trigger the Logic App.

• Next, add a Variables > Initialize variable action. Set the name to be Person and type String, and on the value, we will place the sample of PersonDetail; this will emulate a response from the web service:

<ns2:PersonDetail xmlns:ns2="http://BTS.AggregationPattern.PersonDetail">
   <Id>1</Id>
   <Birthday>04-04-1978</Birthday>
   <Gender>M</Gender>
   <Phone>919600000</Phone>
   <MaritalStatus>M</MaritalStatus>
   <FamilyHousehold>3</FamilyHousehold>
</ns2:PersonDetail>

• The next step is to use a Transform XML action to apply an XSLT map as we did in BizTalk Server in order to combine the Person and PersonDetail messages into an ExternalSystem message. Because we are using a Logic App Consumption, we need to use an Integration Account to store the map and invoke it through the XML > Transform XML action.
  • The good thing is that you don’t need to recreate the transformation; you can simply extract the XSLT code from the BizTalk Server map and add it to the Integration Account in order to use it inside the Logic App. To do you need to:
    • On the BizTalk Server Visual Studio solution, right-click on your map and select Validate Map.

• • • On the Output window, apply CTRL + left-mouse-click on the first link, and this will show you the XSLT code behind the BizTalk Mapper:

• • • Access that file on the file system and upload it to your Integration Account by:
      • On the Azure Portal, search for your Integration Account; if you don’t have it, you need to create one.
      • On the left menu three, under Settings, select Maps and click + Add.

• • • • On the Add Map panel, browse for the map in your machine and click OK.

Note: before you can see the maps that exist in the Integration Account, you need to access the Workflow settings option of the Logic App and associate the Integration Account you want to use

• Now that we have uploaded the map to our Integration Account, we can go back to our Logic App and add an XML > Transform XML action and add the following configuration:
  • On the Map field, select mapPersonDetailToExternalSysten.
  • On the Content field, set the following payload:

<ns0:Root xmlns:ns0="http://schemas.microsoft.com/BizTalk/2003/aggschema">
   <InputMessagePart_0>
      @{triggerBody()}
   </InputMessagePart_0>
   <InputMessagePart_1>
      @{variables('Person')}
   </InputMessagePart_1>
</ns0:Root>

Here is where things can get confusing or weird if you don’t have knowledge of BizTalk Server, and you may ask yourself:

• Where did I get this structure for the Content field?
• There is a BizTalk reference in the namespace. Can I still use it, or should I change it?

Well, both questions are simple to answer. First, this is a BizTalk Server migration using as much as we can that already exists in BizTalk Server. Yes, there is a BizTalk reference in that namespace, but that is just a namespace that is a string. It does not reference any assembly from BizTalk Server. So, you can and you should keep using it. You could change that if you want, but then you need to make small changes to the XSLT code also, and that is not the intention here.

Getting that correct structure is also relatively easy if you understand and know a little about BizTalk Server. If you access the BizTalk Server Visual Studio solution, right-click on our map and select Test Map.

This will provide on the Output windows a dummy input structure to our map in order to test it inside Visual Studio:

If you open it by pressing CTRL + left-mouse-click, you will see the global structure that you need to use:

Amazing how simple is to use the same principles of BizTalk Server in our Logic App!

• To finalize our Logic App, we just need to add a Request > Response action and set it as:
  • On the Status Code property, put 200.
  • On the Headers property, set Content-Type: text/xml
  • And on the Body property, set the result of the Transform XML action.

• Save your Logic App.

Now let’s test our Logic App using Postman:

• Open Postman and copy the link from the Request > When a HTTP request is received trigger.
• Set the HTTP Method to be a POST and add the following header:
  • Content-Type: text/xml; charset=utf-8

• On the Body tab, select raw and set the following payload and click Send

<ns1:Person xmlns:ns1="http://BTS.AggregationPattern.Person">
      <Id>1</Id>
      <FirstName>Sandro</FirstName>
      <LastName>Pereira</LastName>
      <Address>Carvalhos</Address>
</ns1:Person>

• This will execute the Logic App, and if everything executes properly we will get the following response:

<?xml version="1.0" encoding="utf-8"?>

<ns0:ExternalSystem xmlns:ns0="http://BTS.AggregationPattern.ExternalSystem">
    <FamilyHouseHold>3</FamilyHouseHold>
    <FullName>Sandro Pereira</FullName>
    <Gender>M</Gender>
    <MaritalStatus>M</MaritalStatus>
    <Birthday>04-04-1978</Birthday>
    <Address>Carvalhos</Address>
</ns0:ExternalSystem>

I hope you find this content useful, and stay tuned for more BizTalk Server to Azure Integration Services.

Related reading

• A complete guide on Azure Integration Services
• An ultimate guide on BizTalk to Azure migration
• BizTalk to Azure migration resources