Distributed Tracing Guide (2024)

What is Distributed Tracing?

Distributed Tracing is a method to trace requests or messages flowing through distributed systems. Distributed Tracing brings connectivity or visibility of messages flowing through various services using a unique identifier. This identifier is passed to different services to correlate them as a single flow. Distributed Tracing helps you trace ids or trace data in a system. It also adds value to business by,

Having access to all necessary data from a single place – When we build applications in the cloud, it is not just a single service that does the whole job. We need to integrate multiple services to complete a business application. Since services are spread across platforms, users don’t get holistic application visibility. Lack of visibility results in performance issues and degrades users’ efficiency as they spend much time investigating how the application is built and where the failure has happened.
Reducing overall complexity – Since we have all the transaction monitoring, we can quickly identify where a failure has happened in a business process.
Focus on innovation – With the reduction in the overall complexity, support is offloaded to Business users, allowing experts to focus on business innovation.

How does Distributed Tracing work?

Let us take the booking scenario of a Cab Booking System backed with the Azure Serverless services and see how distributed tracing technique will help detect the exceptions and track the custom properties of the messages flowing through the business process.

The above is the Cab booking application built with the Azure Serverless servvices. In this scenario, the User submits a new booking request through the exposed web application. The booking request gets submitted as a message to the Service Bus Queue. Then, a Logic App listens to the Service Bus Queue to process the incoming bookings. Once all the processing gets done, the message is pushed into Service Bus Topic with the label “confirmed “or “rejected.” These bookings move to a Function App that accesses the telemetry data to decide on booking confirmation and revert to the Logic App after confirmation/ rejection. In such modern applications, there will be various challenges faced by different stakeholders of the organization.

Here, it is possible to track data flowing through multiple components of the business solution through distributed tracing.

Why Does Your Business Need Distributed Tracing?

Today’s integration scenarios span across boundaries and are distributed in nature, spanning across on-premise and cloud. This brings its unique challenges; one of them is getting the end-to-end visibility of your transactions.

Most organizations soon realize the need for an end-to-end traceability/ monitoring or distributed tracing solution that business users can use. The simplest and best example is “Where is my purchase order acknowledgment? “. Typically, teams spend about 20-30% of their time addressing this challenge by building custom solutions like logging components and web dashboards. Most often, the solution will be premature as building an enterprise-grade distributed tracing/monitoring solution requires enormous time and effort.

This is the exact problem that needs to be solved. It is required to monitor the business flows or transactions using the hybrid business application to ensure business continuity. The time spent on each step of a business transaction, the success or failure of each stage, and the flow of business data through the application are variables that can be measured with Distributed Tracing. Any of the following factors could lead to the failure of business transactions:

Invalid or incorrect data
A lack of resources to process the data
Business logical mistakes

The transactions failing due to the above reasons can be easily found if tracked and monitored simultaneously. Distributed tracking can diagnose the cause of failure and fix them.

When Do You Use Distributed Tracing?

Consider a business user who needs message tracking through every stage in the above business activity. The User should get informed of any exception in the business transaction and the reason behind the failure. A Business development manager would need to have analytic information on the booking trends at various locations.

Also, in the above scenario, all the Azure services might be independent and decoupled. This challenge will make it very hard to track all the data in a single place. Apart from tracking, business users would also need to:

Get notified on failed transactions
Reprocess the failed transactions
Track generic properties
Perform better analysis with tracked data

Organizations will attempt to build a custom logging solution with web dashboards to address this challenge. However, these solutions often turn out to be premature as it requires enormous time and effort of skilled resources to build enterprise-grade tracking/monitoring solutions. Turbo360 understands the pain points and developed BAM to help users perform end-to-end distributed tracing on our Serverless Applications.

What are the benefits of distributed tracing?

Distributed tracing’s core benefit can be attained from Turbo360 BAM. Consider the Cab Booking application, and there would be a need to track the Driver Id, User Id, and Location. Configuring the necessary properties while creating the business process can be tracked in the BAM section.

Users can even archive the message and their header flowing through each stage. Thus, it is also possible to be aware of every stage’s status and the critical properties of messages flowing through the stage.

From the picture above, even a support person can easily understand and track the business data flowing through the Azure services.

Querying

In Turbo360, BAM users can experience advanced filtering with the help of Querying. However, when multiple business processes are getting tracked in the BAM, there would be a need for some efficient filtering mechanism to stay focused. With Search queries in BAM, Users can easily filter the Business processes from various perspectives. In addition to query filtering, it is also possible to filter based on the time interval.

Users can also save and use the query for Monitoring and Analytics. Tracked Transactions can be queried for specific results based on the transaction status and tracked properties in a user-friendly manner. The queries also support the LIKE operator, and we can also group multiple search query clauses using AND – OR Operators, which helps us retrieve more focused records. For example, if I want to retrieve only the employees from the development branch, I can use the Query Branch = “Development,” which returns only the required records.

Reprocessing

In this scenario, a user booked a cab, and a driver was assigned. Soon after the booking, the driver went offline, and this transaction would fail at the driver validation stage of the transaction. Another available driver should be assigned to restore this transaction and complete the booking.

With the help of Reprocessing, the Support person himself can change the required data and reprocess the message to the stage configured with reprocessing. Turbo360 provides four types of reprocessing endpoints. They are as follows,

HTTP Endpoints
Service Bus Queue
Service Bus Topic
Event Grid Topic

The message body archived at the selected stage will be sent to the configured reprocess endpoints which also helps reprocess the failed instances by investigating the failure later.

Exception Handling

If there is any discrepancy in the custom tracking, it would be hard for people with less experience to fix/handle it. But Turbo360 BAM comes in very handy and provides a similar experience to a native connector with much more exception details that can be captured.

The exceptions could pop up anytime due to invalid data, system, or user errors. For example, consider the Cab Booking Application; there would be a need to track the Driver Id, User Id, and Location, which could cause exceptions. These parameters can get tracked in the BAM section by configuring the necessary properties while creating the business process.

User can even archive the whole message and their header at each stage. It is also possible to know the status of each stage and the transaction.

Exceptions can be tracked on Azure resources like Azure logic apps and Functions Apps. To track exceptions in Azure Logic Apps, the User needs to deploy the BAM custom connector into the user location where the logic app is available. Once the connector gets deployed, the User can handle the exceptions in the Logic App by providing the necessary information to the BAM connector. When the User wants to track the exceptions in resources like Azure Functions, Worker role, and Web job, they can use the exposed BAM APIs. By sending the exception details to the APIs secured with API Key, the User can track the exceptions shown in the above image. Use the try-catch and the checkpoint library to track the exception in the code. Monitoring

We can also set up Monitors to monitor the transactions occurring in a flow. There are two types of monitors available for BAM. They are,

Query Monitors
Exception Monitors

Query Monitoring

Query monitors monitor the transactions by periodically querying the transactions with the configured query and validating the results against the configured threshold error and warning values. The User will have to enable the Rules Evaluation Frequency and Maximum alerts per hour to receive the query alerts. The rules evaluation frequency will calculate the next run time considering the configured start time and selected frequency. For example, if the schedule runs every 15 minutes, the rules evaluation frequency will fetch the number of records from the Turbo360 tracked data based on the configured query monitor.

Exception Monitoring

BAM allows users to log exceptions along with exception codes in the stages of the business transaction. The logged exceptions will be displayed to the users when selecting the transaction instance. Users will also need alerts immediately whenever an exception gets logged in any of the stages of the transactions. The exception monitor in Turbo360 comes to serve this purpose.

Exception alerts from BAM can be filtered to prevent spamming notifications in our inboxes. There are a variety of filtering options available in the Business process monitor that are,

Max Alerts
Frequency
Per Exception Code
Per Transaction

Users will be able to see the list of alerts generated for Query Alert and Exception Alert under Alert History

Analytics

Today, most business processes happen in real-time environments. Most business users would require an aggregate of information, like how exactly is business process X doing? Is there any anomaly in the number of transactions? Etc. This information can only be based on data trends. Any data trends need to be plotted as a chart to be better understood.

We can set up dashboards to visualize the trend of our business process. We can create multiple widgets with different queries. It also helps us understand our business by viewing the transactions grouped by tracked properties. This kind of business dashboard can be made in Turbo360 BAM.

Analytics Dashboard will get automatically created when a business process gets created newly. This Auto-Generated Dashboard will give insights on the Number of failed Transactions, Successful transactions, and more. Users can also customize this dashboard as per the requirement.

Lower the TCO

BAM in Turbo360 solves enormous challenges in end-to-end message tracking through the business process and the need for analytics. It reduces the Total Cost of Ownership and minimizes the effort spent building a solution to do end-to-end trackability by 20%.

How is distributed tracing used in microservices?

The need for observability has increased with the rise of microservices-based cloud applications and their corresponding complexities. Distributed Tracing is the key to the much-needed observability across services. It facilitates maximum visibility within the infrastructure and supports the incident management team in maintaining the reliability of the architecture. The microservices architecture has introduced an all-new way to scale an application (cloud) with several independent services, thus facilitating high resiliency, scalability, productivity, and efficiency. Implementing distributed tracing helps achieve active monitoring of transaction flows, leading to early detection of business failures and effective restoration of business with minimal impact on the customer.

Distributed tracing solutions enable users to track requests through a software system distributed across multiple applications, services, databases, and intermediaries like proxies. It allows for a deeper understanding of what is happening within the application. These solutions produce graphical representations that show the time the request took on each stage and listed each known stage.

How to implement distributed tracing in microservices?

To achieve end-to-end real-time distributed tracing of the message and metadata flowing through the microservices, Turbo360 Business Activity Monitoring can be the best fit. It can track fundamental properties and allow users to locate a transaction by querying for the property value. It also enables dynamic monitoring of transaction exceptions and any violations in the threshold limits set. It is much easier to correlate data flow within the User’s system.

One of the main advantages of the Turbo360 BAM is the ability to visualize business transactions in graphical form and detailed data.

Implement distributed tracing in microservices

In a microservice architecture, there would be many services integrated. This graphical view will help us quickly identify any failures or exceptions in any stages and Action them accordingly.

Users can set up a business process and add transactions with multiple stages. Each of the stages may execute in any of your business components.

BAM now supports automatic resource identification while tracking transaction instances, revealing the source from which the transaction instance is executed. The source of transaction instance execution is automatically identified, and the respective resource type icon is visible in the Message flow while tracking transaction instances when a transaction instance is triggered using Logic App, Power Automate, or Custom APIs.

An added advantage in BAM is dynamic reprocessing, which is useful when the User wants to reprocess the tracked property values as a header. It also offers a solution by allowing the instances to regenerate the message-id, ensuring that no transaction instances are lost concerning their corresponding endpoints. Multiple transaction instances can also renew their message ids simultaneously by selecting the required transaction instances and reprocessing.

What is the difference between distributed tracing and logging?

Distributed Tracing reveals what occurred between services/components and their relationships, whereas logs show what happened within the service. It is particularly crucial for microservices because poor component integration often results in problems.

Logs are utilized for any level of activity, be it a high-level action or a particular low-level detail. Traces, on the other hand, give the User an entire understanding of the architecture. Distributing Tracing enables request tracking across autonomous services and modules to provide observability.

Apart from everything mentioned, users would also seek the same or added benefits from a business perspective. Turbo360 caught hold of this point and curated the BAM feature at a business level. Below are some expectations raised from our customers,

Notify my stakeholders of failed transactions
Reprocess the failed transactions
Track based on details like ID, Name, or Location
Perform better analysis with tracked data

Turbo360 came up with BAM, a feature set that would ease work for all organization members considering all these points.

Distributed Tracing tools

We track data from different services in distributed tracing, but how do we visualize them? Visualization is a tedious task. To achieve this is where the distributed tracing tools come into play. Distributed tracing tools give you an end-to-end visualization of how user requests perform in different services.

With these distributed tracing tools, you can eliminate traveling across services to identify where and why the failure has happened. You have access to these details in a single window. Here’s the list of the top tools available for distributed tracing in the market:

Azure Application Insights
Turbo360 BAM
SigNoz
App Dynamics
Dynatrace
Nodinite

Distributed Tracing with Turbo360 BAM

Turbo360 Azure Distributed Tracing Tool is a way to provide a simplified business-focused view on what might be a complex underlying system or group of system interactions that execute to fulfill the business transaction. If you have a BAM solution, you can significantly simplify the ability to support these business transactions.

As we observe the integration scenarios today, they span boundaries that help us build complex business applications quickly and invite their challenges. The critical challenge is the lack of end-to-end visibility of the message flowing through the business transactions. When a transaction fails, it is hard for business users to understand where the failure occurred and its reason. The simplest example could be “Where is the booking confirmation?” in the above Cab Booking Application.

Which is the best tool for distributed tracing?

Turbo360 BAM is an end-to-end distributed tracing solution that can address this challenge by reducing the effort to instrument the business application to provide end-to-end visibility on the message flowing through various business application services involving Azure entities.

Turbo360 BAM Architecture

One of the critical design decisions in the BAM architecture based on customer feedback was the importance of owning your data and having access to it. As a result, you can configure your own Azure SQL Database and Storage account, where your BAM transactions and messages get pushed.

You will then instrument your components to push BAM events to our BAM service, where you will have a dedicated endpoint that will process your BAM events. We will then move your data to your data components. The Turbo360 portal will access your BAM data at runtime when you use our BAM searches and dashboards. The below picture represents this architecture.

Tracking Hybrid Solution with Turbo360 BAM

This section will explore the core concepts and ideology of how BAM works to track a hybrid solution.

BAM Concepts

There are three main terms to understand before exploring the core concepts of BAM. They are,

Business Process – A collection of one or more related business transactions.
Transaction – An operation of the business which represents a set of business activities
Stage – An activity within the transaction. A transaction is a combination of 1 or more stages.

Business Process

Nowadays, we see different integration solutions like on-premises, Azure only, and Hybrid integration solutions. Irrespective of the integration solutions or technology, they serve as a vital tool for a business process for an organization. Consider an example of a logistics company, Ship Anywhere, which will have many integrations with the carriers. These interactions can be Booking Requests, Confirmation, Shipping Instructions, Milestone Status, etc. All these transactions come together and form a Logistics Business Process for Ship Anywhere.

Turbo360’s business process can be modeled as depicted in the picture above. Thus, the simplest way to think about a Business Process is that it is a group of Business Transactions.

Transactions

When there is a business process, it involves several Transactions. A Transaction can be a series of actions or stages.

So, in Turbo360 BAM, the list of transactions involved in this business process can be taken as Business Transactions: Booking Request, Booking Confirmation, Shipping Instructions, Invoice, and Milestone Status. Aggregating the transaction and stages are up to the customers, and there is no restriction to doing it precisely. In Turbo360, we will implement the transaction like an activity diagram which will be a familiar view to users’ operators and business users.

Implementing BAM

The first step in configuring the business process is identifying the logical grouping of transactions. For example, you may decide to keep all transactions specific to a business unit such as Finance and Supply Chain in their respective business processes or at a more granular level. Once you arrive at a strategy to group the transactions, you will define the business process, related transactions, and stages in the Turbo360 portal.

Let us consider the Order Processing scenario for the implementation. In this scenario, the Order Processing business process has a new order, fulfilling and shipping charges as different transactions.

The transaction is designed into a flow with stages, as shown in the above picture.

Before configuring the Business process, the User should associate SQL Database and Storage Account to store the BAM-related data. To associate both the resources to Turbo360 BAM, the User must provide the credentials mentioned below:

SQL Database: Connection string required
Storage Account: Either Connection string or Service principal required

Followed by Storage Account configuration, the Business process creation, and transaction configuration can be done.

The added advantages during stage configuration are custom properties. Users can use these properties to track the transaction based on a specific property like Order Id or Shipment Id in the above scenario. There are two different kinds of properties BAM supports,

Global propertiesare on a transaction level. They will become visible in the grid; you can update them during a single business process instance and use them across different business processes. Any changes in the global properties will be reflected in all the stages where these properties get mapped.
Stage propertiesare defined on a stage and are more used in a specific scenario in a particular transaction/stage. They will become visible in the Message flow details; they will be stored on every different stage again when used. For example, if the message is an XML message, the User can use the Xpath as Property Source and JSON Path for a JSON message.

The next step is to instrument the technical solution to tell BAM when checkpoints occur in the transactions. Turbo360 BAM provides the following components, which you can plug into your business flow to track the transactions happening.

Custom connectors for Logic Apps and Microsoft Flow
.NET Library for Azure Functions and OnPrem APIs

Connectors

If users are familiar with Azure Logic Apps, they must have used the designer’s view to create their Orchestrations. To create it, the User will use an Action—an action otherwise known as a connector, e.g., Facebook, Outlook, Twitter, etc. We provide this connector to enable users to track data from their Business Logic into Turbo360 BAM.

Suppose users are using Logic Apps or Power Automate. The User might want to use the custom connector and add the custom connector using the JSON definition for the connector.

Use cases of distributed tracing

We will explore a real-time use case we developed for our customers to have a practical understanding of how Turbo360 BAM, a distributed tracing solution, will benefit from a business viewpoint. This use case will help you comprehend how BAM helps the real-time company.

Problems with the Current System

If done manually for each new employee, giving access to various systems, applications, or tools is challenging for the HR team. As a result, businesses are attempting to automate this process by utilizing different cloud services. However, this method results in a lack of message visibility, making it challenging for the HR staff to locate failures and reprocess data. Even for the errors that have been found, we should start the business process from the beginning because doing otherwise results in a duplicate employee ID.

Employee data is processed concurrently with the migration to a new system. The biggest obstacle here is reducing the record to the information of a single person. Due to the high volume of transactions, any unsuccessful transaction could go unreported.

Advantages of instrumenting BAM

With BAM, you can instrument your business process to track the business-critical data flowing in the business process and search for any transaction using the tracked properties.

The simplified view of your business process allows the HR team to review issues and progress. Even without touching the technical components HR team can make some quick fixes.

With Action required tabs in BAM, concentrate only on the failed transaction without missing out on any. Utilizing the BAM query, the User can filter records based on the transaction property like Emp_Name, Emp_Id, etc.

With a better GUI, get to know in which stage of the business transaction the error lies. Even a non-technical person can understand what is happening in their business process.

Reprocess a failed transaction to any stage of your business process right from the Turbo360 portal. It reduces the user’s effort in switching between applications/portal.

The actionable dashboard enables a user to perform better analytics. Users need not spend much time understanding business health. Turbo360’s BAM dashboard helps users to make business-critical decisions effectively.

What are the uses of BAM? – A quick summary

Here is a quick summary of the benefits of BAM discussed in this Pillar Page:

Business Users can view tracked data from a business point of view rather than a technical point of view
It is possible to track the properties at different stages in a Microsoft flow
It is possible to filter or search the transaction instances based on the configured properties in addition to the status of the runs or transaction instances
It is possible to get alerted whenever there is an exception logged in the stages of the Microsoft flow
It is possible to set up a custom user access policy so that users having only the required privileges can access the message content tracked in the Microsoft flow
We can set up a customized dashboard to monitor the transactions happening in a Microsoft flow
It is easier to find the exact stage where the transaction instance has failed and fix it with minimal cost
It is also possible to track the stages that constitute the same business flow but is present in a completely different infrastructure using the Checkpoint with Correlation action of BAM

Webinars and Documents

This Webinar covers in detail the capabilities of Turbo360 to facilitate managing and monitoring a business flow with BAM. All concepts are dealt with real-world business use cases for better understanding.

Need a better Azure Management platform?

Turbo360 helps to streamline Azure monitoring, distributed tracing, documentation and optimize cost.

Distributed Tracing