This blog is an extraction of the webinar “Azure Integration in Production with Logic Apps and more” presented by Alex Murov and Kelvin van Geene in “Integration Monday” series.
The following features of Azure App Services and much more are discussed in this blog;
- API Apps
- BizTalk Services
- Service Bus (Queues)
- Logic App
- Cloud Service
- AS2 on Azure BizTalk Services vs Logic App Connector
How it all Started
About two and half years ago, while making this webinar, a client approached a team who was working on a proof of concept (PoC) for CRM online. Integration was required between local systems and CRM online. So, PoC had to be done on CRM online but not necessary for local systems as they exist already.
Assumptions that have to be considered
- No custom development on CRM side
- SAP-PI will be used as an on-premise middleware system
- The AS2 connection must be used to integrate with SAP-PI
- Cost for development and operations must be kept to the bare minimum
Considering the above assumptions, the selection fell on Azure BizTalk Services for the realization of PoC.
This is the simple representation on the model side to integrate local systems and also CRM online. There is an SAP-PI, out of the box, which acts as a Middleware for on-premise components. Next t,o it, there is an Azure Cloud service and finally, CRM online environment. In addition, there is a CRM Organization Service which is used to send messages to the CRM online environment.
BizTalk Environment
In Azure Cloud service, there is an AS2 agreement at the very beginning, which is based on BizTalk Services.
The above picture shows the basic UI of the BizTalk environment. There are four Partners, three Agreements, and nine Bridges. The first six bridges that you see represent AS2 connections for developing quality assurance and production to the three-tier architecture. The rest of the bridges represent a one-way pipeline where the messages will be received from the client and routed to their destinations based on certain properties.
In the next phase of the Azure Cloud service, Service Bus Queue is present. It is used as a substitute for message box in BizTalk server which will preserve the messages. The Service Bus Queue will be helpful in a scenario where the system might undergo some faults in Message routing, Message processing, and also Dead-letter queues.
The above picture shows the production environment of BizTalk Services. There is a source DXI queue (1) which will route the messages from the SAP PI to Service Bus Queue (2) from there, those messages will be picked up and routed to the particular destinations. There are multiple destinations and the messages will be routed based on the organization name.
Queues
Once the messages are routed to their respective destinations, they will be saved in the Queues as listed in the above picture.
Now, the messages will end up in the Service Bus Queue and those messages have to be uploaded to CRM online environment. For this purpose, Cloud Services come into action, acting as a substitute for adapters that will be connected to the CRM system online via organization service and upload available messages from Service Bus Queues.
But, the messages cannot be routed if it is Dead-letter. SendGrid is used to notify the operation team about the reason for Dead-letter message and also the specific message will be displayed.
The above picture represents the Cloud Service environment. There are two instances of Run history. The first Run history represents the message which is sent to the CRM environment and also the other represents the message which is fetched from the CRM environment.
At a certain point, the development team faced a lot of exceptions from BizTalk Services. To address this issue, they decided to move on to a different solution. At the time, a new tool (Logic Apps) was introduced by Microsoft to play with these exceptions and also BizTalk Service was replaced by Logic App.
What is Logic App?
Basically, it allows the developer to design simple but effective workflows in the cloud. It gets triggered based on some Actions.
In the above example, the Logic App gets triggered by Recurrence after which it searches for tweets and also based on the condition, certain actions will take place. Finally, the result will be stored in SQL Azure. It is a high-level design of Logic App. There are a lot of components out there that can be used to build different Logic Apps.
Why do you need to start with Logic App?
The following points justify the question
- Simplicity/ Flexibility
- Most common adapters are already available.
- Azure BizTalk Service deprecation.
Advantages over Cloud Services
Cost efficient: It is expensive to work on Virtual Machines when you use the cloud.
Quick to deploy: It is easy and also quick to deploy the services towards Azure.
Start and Stop almost instantly: The services can start and also stop instantaneously whereas in the cloud, to start spinning up and shutting down the services will almost take 20 minutes.
Operations Monitoring in Portal UI: The monitoring operations become easy in the Azure portal UI.
Reusability of components: The components of the Logic Apps can be used in other similar scenarios to solve different business problems.
Disadvantages of Logic Apps
Information retrieval: It is not easy to retrieve information if something goes wrong.
Formatting of data: Formatting of data is done only through a web viewer.
Visual Studio integration: Logic App provides only minimal integration with Visual Studio.
Settings / Configuration management: It is hard to configure Logic Apps.
Beta version of Logic Apps: At the time of making this webinar, Logic Apps were in beta and also changes were made since then.
Specific Implementations
The BizTalk services were replaced by Azure Logic Apps. The Logic App connects the Azure Service Bus Queue and next to it, the AS2 connector is used. It checks every minute for the availability of the message in the queue and once it is available, the messages will be moved to AS2 connector. The AS2 connector will send the message to SAP-PI environment. The Run history can also be checked in the Azure portal after the execution of the Logic App.
Scenario – Big Data (Volume data)
There is a Data Warehouse with a certain view on the volume of data, which is generated by the client. Assume there are 15 million flat records which are being uploaded from one site to the SQL database. Those records are moved to the Web Job from which the messages will be processed and also notified to the Service Bus Queue. From the Service Bus Queue, the data will be sent to the Blob storage and also another Web Job will read those notifications. Based on the notification, the web Job will create new records in Document DB as JSON files. The 15 million records can be displayed through Windows App using Web App.
Logic App – Sales order
The Recurrence trigger will check for a request from the hosted API every five minutes. Based on the request, it will get orders from the CRM environment which will be consolidated into a single message. Then, the message will be passed to the SAP-PI environment through the AS2 connector.
This is the functional view of the integration process that defines the process behind the curtain. At one side, there is a windows application which will sink to the back-end system of the CRM online where the messages will be picked up and sent as events to Service Bus Queue. Then, the Logic App gets triggered and also receives the message from the Queue. Based on the information in those messages, the Logic App consolidates them.
In this phase, we can either send messages to the distributors directly through Send Grid or to the SAP-PI On-premise which in turn will route the messages to their respective destinations based on the Headers.
Scenario – Message validation
Since this solution is going to be implemented across different countries in the future, the above Logic App validates the message against the JSON schema that has been defined for different countries. The messages will be validated when they are sent from CRM online to SAP-PI system. The JSON schemas will be stored in Azure SQL Database.
There are three flows in the picture, the first flow is to create a new account, the second flow is to monitor the message and finally, the third flow is to capture the orders. But these flows can be implemented in the current architecture only in the future because the message validation uses the V2 version of the Logic Apps where the AS2 connector is not available. Once the Logic Apps team introduces the AS2 connector, this model will be implemented.
Plans for integration
Different types of integrations will be implemented in the existing architecture. Multiple development and operations teams will work together to develop the functionalities. Since there are different contracts, a common subscription will be provided for all the messages to connect with the SAP-PI environment.
In the near future, the whole architecture will be implemented using Azure Logic Apps. The above picture shows the implementation model.
You can watch the webinar “Azure Integration in Production with Logic Apps and more” in Integration Monday series.
