Tracking IDocs for Integration Scenarios with Turbo360 BAM

Suppose you are a user of the Microsoft integration stack, and your organization also uses SAP. In that case, you will likely have use cases where an IDOC triggers integration processes in SAP being published.

One of the good things about Logic Apps on Azure is that a connector for SAP allows you to register to receive IDOCS published by SAP, and you can then use them in your integration processes.

We implemented a pattern where we had a generic Logic App registered to receive IDOCS and would then use the Azure Service Bus to implement a pub/sub pattern to allow multiple different interfaces to act upon the receipt of an IDOC into the integration platform.

Below is a representation of this architecture.

It is quite a handy pattern as it allows you to add new IDOCS and new interfaces being triggered via Service Bus in a plug-and-play style.

The Challenge

Because we have several interfaces triggered by IDOCS being published, it is challenging to track which Logic App run processed each IDOC received by the integration platform and which downstream interface processed each IDOC.

The net result of this challenge is that the interfaces are harder to support, lots of time is spent tracking down issues, and the business users and SAP team feel like they have no visibility into the processes that are important to the effective running of their applications.

Overcome the challenges in attaining the root cause of critical issues with end-to-end tracking. Book a demo!

The Value of BAM in this Scenario

We felt like BAM could help us a lot here. It is beneficial to track the IDOCS being published and received by the integration platform, tracking their participation in interfaces, and then making this accessible. It would help us reduce those support costs and give a better-quality service to the users who depend on our integration platform.

In BAM, we could achieve distributed tracing by creating business processes and transactions that would represent an excellent, more straightforward view of our interfaces. Then, we could create searchable properties that our integrations could use to make it easy to track and trace the information we needed.

BAM Global Properties

In BAM, we have global properties where you declare properties that will be searchable. Interfaces will use common business values, and users may wish to find information using this property. For example, a BAM user may want to search for a transaction via an Order Number.

We started by declaring the properties which are common business properties such as:

• CUSTOMER-NO
• ORDER-NO

These have business meaning, but in different interfaces, they may be in other messages and message formats. For example, in one interface, they may be in a JSON message, and in another, they could be in XML. Having your interfaces connected to BAM allows you to map properties to the implementation at both design and runtime to ensure you can extract the needed information.

SAP specific properties

Next up, we added some SAP-specific properties. In an IDOC, the EDI_DC40 segment is common to all the IDOCS we receive, and we felt that promoting some properties from this segment would be an excellent way to make helpful information searchable in BAM.

The IDOC properties in the EDI_DC40 segment we are interested in including the following:

• DOCNUM is the IDOC number
• STATUS is the status of the IDOC
• IDOCTYP is the type of IDOC
• MESTYP is the message type for the IDOC
• CREDAT is the date the IDOC was created
• CRETIM is the time the IDOC was created

We create the following list of properties in Turbo360 BAM:

Logic App-Specific Properties

We also need some Logic App properties in BAM so that when we escalate to the IT support team, they can easily see which Logic App they need to check and if they need to study more detail on the technical implementation of the interface.

To do this, we add the below two properties to the BAM Global Properties too:

• LOGICAPP-RUN-ID
• LOGICAPP-NAME

At this point, we now have all the properties we would want for tracking our IDOCS, and we would begin to create our transaction and Business Process.

BAM Business Process Setup

We chose a dedicated business process for our SAP-focused interfaces so they could easily be managed together. In the business process, we had a transaction called IDoc Published to the EAI Platform, as shown below.

Within the transaction, we created a simple activity diagram, as shown below, representing a simple view of the Logic App. It receives, parses, processes the IDOC, and then publishes it to Service Bus. The Logic App might be around 20 – 30 actions. Still, the BAM user will likely be either an IT support user, SAP application user, or business user. They are not so interested in the implementation detail of the Logic App they want to know about the key milestones, which are:

• • The IDOC was received
  • It was successfully published to Service Bus
  • There was an error

We can configure the properties on the IDOC Received from SAP action, allowing us to map the technical properties from the Logic App to the BAM properties. The below picture shows the properties we configured. The first properties we extract at the start of the transaction.

We promote some additional properties from the body of the IDOC after we have parsed it.

There are two types of properties we are using here:

• Message context properties which the Logic App provides at design time
• Message Body properties which we are extracting from the message the Logic App sends to BAM at runtime

In this case, the message header properties were the Logic App name and ID and the SAPActionUri from the SAP connector.

The message body properties use XPath against the message body. The Logic App sends the IDOC XML to BAM and allows to extract data from it to show in BAM.

Note:

Sending a message body is optional; if you choose to do it, you also have the option to archive the message body so that the BAM user can access it later if needed. If you do not archive it, it will be discarded when the message body is processed.

When configuring message properties from the SAP IDOC, you would use an XPath like the one below, extracting the DOCNUM, which is the IDOC number.

Receive/idocData/EDI_DC40/DOCNUM/text()

Adding BAM to the Logic App

Most people will probably use the Turbo360 connector JSON with a custom connector in Logic Apps to set up a connector for BAM. Several other ways to set up your Logic Apps to call BAM, such as via API Management.

In the Logic App, you will add an action to start your transaction and then for the checkpoints you want to send to BAM.

Below is the Start Transaction shape I have configured in my Logic App.

It provides information that maps to my BAM transaction and the first step within it. I am also passing the IDOC message as a message body to extract some properties.

Next, in my Logic App, I would do some work to process the IDOC and send a notification that it is received or the message to Service Bus. I would then add a checkpoint shape for BAM, as shown below. It is completing my transaction.

Runtime Tracking

At runtime, I can use the tracking features of BAM to query for the instances of my transactions that have received an IDOC and then publish them into the integration platform. I can create a search like the one below, which uses the friendly columns from my BAM properties to allow me to see the IDOC type, the IDOC number easily and which logic app processed it, as you can see below.

One of the most common support queries will be, “what happened with IDOC [add idoc number here].”

With this query, we can check how we received it and see if it was successful. I can also open up the transaction by clicking and seeing what happened, as shown below.

What if there is an issue?

The BAM tracking view allows your BAM users to have self-service-friendly ideas for complex interfaces. If there was an obvious problem and they needed to escalate it, then they could use the assign feature shown below, where they can link the transaction to another user in Turbo360. It might allow the BAM user to escalate to the Integration support person to take a deeper look at what is happening. It will enable them to find the transaction quickly.

There are also options to add a transaction as a favorite to you can easily find the one you are working with later if you need to take some time to investigate a problem.

Reprocessing

Suppose you wanted to allow the BAM users to reprocess a failed message without needing an Integration Expert to step in. Then it is possible to configure a reprocessing step for the transaction. You might set up a way for the BAM user can push the IDOC back to the Logic App if you need to get it processed again. There is more information about reprocessing in this article.

Performance Dashboards

In BAM, I can also create dashboards to give me an easy-to-consume view of how my processes run. Below is an example of one of the widgets I use where I can see how many IDOCS we receive over time.

It is interesting to see we get a few bursts of load in these scenarios.

Downstream Interfaces

Once we have received our IDOCS and published them to Service Bus for other interfaces, we can also add BAM to those interfaces, and we will be able to track and monitor them with similar features, as shown here.

One of the key benefits is mapping properties such as the IDOC number where it is used in other interfaces so we can easily trace the flow of an IDOC to other systems within our Business Process.

Try BAM

If you want to explore how BAM can help you implement interfaces that integrate with SAP or other systems using the Microsoft Integration Services, go to the Turbo360 BAM for more info and a trial.