Challenges you may face in implementing cloud-native integration solutions

“Decisions are made, you are going full stop into Azure for your native cloud integrations.”

Cloud-native means you leverage the PaaS services Azure offers such as the Service Bus, Logic Apps, API Management, Azure Functions, and Event Grid. Furthermore, you design solutions with these services and deliver value to your business. While doing so, you face different challenges than with developing integration solutions on-premise.

With Azure you face new dynamics – you do not control any servers, you only manage the services. The infrastructure is abstracted away from you, and you pay for what you use – it is consumption based. It scales to certain thresholds when consuming those services. Moreover, what you now need to worry about aspects such as dependencies between the services in your solution, monitoring, resiliency, and scale – the more you consume resources (services), the more you pay. These are some of the challenges you face when implementing cloud-native integration solutions.

Introduction

In the on-premise world, you are used to the traditional N-tier approach for your applications where you have a client-facing tier, a processing (business) tier, and the data storage tier. The processing or storage tier you can find integration solutions such as BizTalk, processing message – transforming and routing to different subsystems. However, in the cloud, you can have different ways as you built solutions from a collection of services. With Cloud-native integration, you design and build a solution with Azure platform services such as:

• Service Bus for brokering and decouple dependencies between services,
• Logic Apps to build workflows to run a process,
• API Management to create API gateways,
• Azure Functions running small pieces of code,
• and Event Grid supporting event-driven solutions.

The solution you design can be event-driven, where you use a pub-sub model. When something emits an event (publish) to Event Grid, the service sends the event to each subscriber. The subscribers handle the events; some of the event handlers are Functions and Logic Apps. Alternatively, when designing a messaging solution you can use Service Bus, and leverage the pub-sub model with topics and subscriptions. A message arrives on a topic and subscribers can receive the message. Here the subscribes can also be Functions and Logic Apps. Both are examples of cloud-native integration solutions.

What does cloud-native integration solution mean?

Cloud-native concerning integration means an approach to design, building and running integration solutions in Azure. The goal with the approach is to quickly build an integration solution, valuable for the business, scales and is cost-effective. Furthermore, the solutions run on platform services; deployments are automated and resilient to failure. In a nut-shell, cloud-native is about speed, agility and reducing operational costs.

Challenges

With design and building cloud-native integrations, you face some challenges with implementation, reducing costs, performance, resiliency, security, and monitoring. Each challenge we discuss in the following paragraphs is in our view key to having a sustainable cloud-native integration solution.

Implementation

Your IT-department or engineering team is now responsible for building and maintaining new integration solutions in Azure. As part of a new cloud strategy moving forward, you and your colleagues are responsible for those integration solutions. The immediate challenges in this context are the skill set of you and your team, designing, building, deploying, and maintaining the solution in Azure.

Skillset and implementation challenges

With building integration solution in Azure, the skill set differs from what you and your team members are familiar with on-premise. First of all, you have to learn new technologies, consider other implementation patterns, and apply different deployment strategies. The new technologies vary from messaging to business flows. With the Service Bus, you have the means to create queues and topics to decouple services from each other. API management provides a gateway into your API’s or Logic App endpoints. Nevertheless, you can learn these technologies easily through the massive amount of online content, courses, and available videos ranging from channel9 to Middleware Friday. Furthermore, Microsoft just recently launched a site for Azure learning for free. Other resources you can examine or subscribe to are:

• eDx online courses
• Pluralsight – on-demand learning technology
• Opsgility

When implementing integration solutions in Microsoft Azure you gather requirements first, select the appropriate services in Azure, and create the design. With the implementation design, you can look at, for instance, the available patterns for messaging and resiliency. These patterns include pipes and filters, queue-based load-levelling, and competing consumers. Moreover, the patterns can help you create a solid integration solution design. Generally, Microsoft provides you with a rich set of content in their Azure Architecture Center.

Costs

An essential aspect of building an integration solution in Azure are costs.  You can run some of the services under a consumption plan, others on fixed costs – you reserve some resources, which you pay per hour or day. Below you can review a table with the costs of each service that you could incur for your integration solution.

Azure Price Calculator

Each service has its costs, and it can be difficult to predict the costs of your integration solution to the exact dollar/euro cent. However, with these prices in mind, you can create estimations of the costs using the Azure price calculator.

The pricing pages and calculator can be of great value when assessing the costs of operating your integration solutions in Azure. However, not included in the costs are lead-time for the production of your solution, nor the time you or your team members spend on operations. In a nutshell, the costs in Azure are operationally driven (OPEX) and not through the acquisition of infrastructure and licenses (CAPEX).

To reduce costs generally with cloud-native integration solutions is to pick the right service, determine the workload, and deployment strategy. With a deployment strategy in the context of the cloud, is that you require a setup with a development, test, and production environments. You can scale out and down your test and if necessary your acceptance environment. You only provide the services through ARM-templates when you do testing and decommission them afterward. This approach helps reduce costs. Furthermore, in your test or acceptance environments, you can select the lower SKUs for API Management or Service Bus. Again this approach leads to cost-effectiveness.

Performance

With the costs of your integration solutions, you also have to consider the workload aspect. What is the workload that the integration solution will handle? The workload ultimately drives the consumption and thus the costs. With that in mind, you have to think about what service fits best for your workloads. For instance, handling messages with Functions can be a better option than with Logic Apps. In the When to use Logic Apps and Azure Functions blog post we further dive into that particular option.

With BizTalk on-premise, you might have an environment set up, which is optimized for throughput. To handle similar workloads in Azure, you need to consider if you can meet the same throughput requirements. An option can be to leverage IaaS and lift-and-shift your environment into VMs in Azure to control costs with regards to spinning up a large number of services. Certain services are more cost-effective than others. Generally, you can look at the Azure architecture center for guidance and examine the performance and scalability patterns, and the scalability checklist.

Resiliency

Even in Microsoft Azure, your solution can fail if you misdesign your solution, or not made resilient by applying for instance error handling. Furthermore, services could become unavailable when a data center is offline or having issues. Therefore, you need to consider these factors. You need to think about what potentially can go wrong in your integration solutions, what the impact could be, and how to recover. Furthermore, it is imperative that you test your solution and automate the deployment to the various environments.

Within Logic Apps, and Functions you can think about implementing error- and exception handling to enhance robustness. You can leverage ARM templates for deployment of your services, and Continuous integration and deployment pipelines for your artifacts. Furthermore, you can develop unit- and integration tests for your integration solution wherever necessary. Also, you can spread your services over various Azure regions and create them redundant if necessary to enhance the availability of your solution. Note that Microsoft provides guidance for resiliency through a checklist, and also on how to create a solid design with resiliency in mind.

Monitoring

With your integration solution operational, the next challenge is monitoring. You do not need to manage infrastructure or anything – Microsoft as a cloud provider does that for you. You do need to keep an eye out for the health of your solution is leveraging the monitoring capabilities Microsoft Azure provides in Application Insights, Azure Monitor, OMS, etc – a few blog posts on the Turbo360 blog about monitoring options are available for you:

• Different options available for Monitoring Azure Serverless Components
• Monitoring a Composite Cloud-Native Solution using Turbo360

In any case, you need to set up monitoring using Microsoft’s services and potentially a third-party SaaS solution. Turbo360 is built as a complement to Azure portal capabilities like Application Insights and Azure Monitor.

Besides monitoring your solution you also need to keep an eye out for running costs of your integration solutions and environments. Fortunately, Microsoft offers cost management and billing through the Azure Portal. With cost management, you can analyze and optimize cloud costs. Moreover, Microsoft offers guidance to leverage those capabilities.

To conclude, there are several services and solutions available to keep the costs in check and keep your cloud-native integrations healthy. Furthermore, based on the size and complexity of your solutions you need to set up the right size of monitoring.

Security

Security in cloud-native integration solution is different towards on-premise integration solutions. Logic Apps, for instance, use connectors and these have their security. Furthermore, with Functions, each particular binding comes with its security. Both provide you with an extra burden to manage them. Each Logic App connection towards a connector results in a managed API connection, and with Function, you maintain the connections in the integration part of the function itself. These are just examples of security aspects of just Logic Apps and Functions. With API Management you control access to your API endpoints, which can include Logic App and Function endpoints. Access to Service Bus is through policies you manage in that service, and Event Grid has security with Authentication Keys and SAS-tokens.

Here is a list of a few resources for security and authentication of the various integration services:

• Secure access in Azure Logic Apps
• Authenticate with managed identities
• Service Bus authentication and authorization
• API Management Authentication policies
• Event Grid Event Grid security and authentication
• Security in Azure App Service and Azure Functions

The list is to illustrate that security is a crucial aspect of any cloud solution including integration ones, but also a significant complex aspect. Moreover, you have to think about security with your integrations concerning the data flowing through them, and the businesses it supports.

Wrap up

When you read through this blog post, there are various aspects around the implementation of cloud integration solutions. Aspects that you should not ignore and take lightly. Integration in Azure is being just as complex and challenging as it is in the on-premise world.

In this blog post, we explained the meaning of cloud-native in the context of integration, and the challenges you may face when implementing cloud-native integration solutions. The intention is to give you an understanding of what you face when implementing cloud-native integration solutions. Hopefully, we succeeded with that in this blog post.