This blog is the continuation of the blog “Get the best out of Azure Data Factory – Part 1“. It explains how Turbo360 addresses the pain points and enhances the usage of Azure Data Factory. Turbo360 is a single platform solution that enables Operations and Support Teams to manage and monitor Azure Serverless services more efficiently.
Let us have a quick recap of part 1 before looking into new features that Turbo360 provides for Azure Data Factory.
- Monitor the Data Factory, Pipelines, and Integration Runtimes based on metrics.
- Trigger a pipeline execution, stop the execution of the pipeline.
- Search for a pipeline run based on its execution status, start time, and end time.
- Visualize the trigger runs and Integration Runtimes related to a Data Factory.
- Visualize Integration Runtimes and the Nodes, Linked Services, and Pipelines connected to it.
Listed above are the capabilities of the Azure Data Factory provided by Turbo360 discussed in part 1. Now let us discuss a few other features offered by Turbo360 for Data Factory.
Visualizing Data Factory Pipeline runs
Consider a scenario where there are multiple failed runs in a pipeline at a specific time interval, and there is a need to investigate the cause of the failures. In such cases, Turbo360 can retrieve the failed runs based on run status in the specified time window.
Once the failed runs in a specific time interval are retrieved, the next step is to find the activities which caused the failure. Turbo360 allows the user to view all the activities belonging to the pipeline run by clicking on the name of the run. These activities can also be retrieved based on their execution status.
Once we have found the activity which caused the failure of the pipeline run, the next step is to investigate the input and output bindings of the activity which caused the failure. Turbo360 provides an option to visualize the input and output bindings of the activities by clicking on its name.
The next step would be to run the pipeline with the correct values. Turbo360 extends its capability to trigger a Data Factory Pipeline by dynamically passing values to the already configured pipeline parameters without having to switch to the Azure portal. Each time, the pipeline can be triggered with a distinct set of parameter values. When a parameterized pipeline run is re-run, the parameters of the original run are carried forward to its execution.
This scenario explains how Turbo360 can efficiently analyze the failures caused in the Data Factory Pipeline and rectify the same with the least effort.
Monitoring Integration Runtime
Consider a scenario in the Integration Runtime where the number of available nodes in the cluster goes down for some reason, or there is a sudden spike in the CPU utilization of the Integration Runtime, indicating some failure during non-business hours. Now here arises a need for a user intervention to set things correctly.
To mitigate such scenarios, Turbo360 allows the users to configure monitoring on such critical properties and notifies the users when any violation is observed on the set threshold values. Additionally, periodic notifications can also be delivered even when there are no violations to know the status of the critical metrics.
Granular user access policy
Any number of users can be added to the Turbo360 account to manage and monitor the resources associated with it. However, not all users can be granted the authorization to perform all the operations. Turbo360 allows fine-grained access control definition to the users with personalized functional capabilities to restrict unauthorized access.
Consider a scenario where a user should be given access only to perform monitoring on the Data Factory, or a user should be restricted from performing certain operations. All these requirements can be achieved by enforcing the user access policy provided by Turbo360.
Roadmap for Azure Data Factory in Turbo360
Turbo360 constantly adds new features to its portfolio. The roadmap for Azure Data Factory includes the below list of items
- Support to Integration Runtime Node that includes metric monitoring and dashboards capabilities
- Property monitoring on Data Factory and Integration Runtime
- Stopping a pipeline trigger
Conclusion
This blog briefs on some pain points faced by the users of Data Factory and how Turbo360’s Azure Data Factory Monitoring capabilities can address those scenarios. Stay connected to learn more about Turbo360’s features in managing the Azure Serverless services.
