One of the fundamental promises of the cloud, when organizations embrace it, is significant cost savings compared to its on-premises costs.
However, organizations to realize savings is required to proactively plan and monitor the application’s cost at a granular level.
Azure cost anomaly detection involves promptly identifying, rectifying, and analysing unexpected Azure cost events to minimize their impact on the business.
According to the FinOps foundation, cloud cost anomaly is as defined:
“Anomalies in the context of FinOps are unpredicted variations (resulting in increases) in cloud spending that are larger than would be expected given historical spending patterns.”
What is unpredicted variation in cost anomaly detection?
Unpredicted variation in the context of anomaly detection refers to the unexpected fluctuations in cost within a system.
It involves identifying predicted or baseline cost patterns for a period, comparing them with the actual accumulated cost so far in the day/month, and deciding whether to panic.
The cost anomaly detection platform’s prediction mechanism or algorithm is crucial in determining the actual cost anomaly and eliminating false positives.
Why should you care about Azure cost anomaly detection?
Spotting an anomaly is like a red flag waving—it could mean a glitch in the infrastructure, a sneaky software bug, a potential cyber-attack, or some other issue causing a steady rise in unexpected cloud costs.
Managing anomalies is a way of playing defence to keep costs in check or at least soften the blow when these unexpected events occur. Without a solid anomaly management strategy, you’re crossing your fingers and hoping to stumble upon unusual spikes in spending—and that’s not a position any organization wants to find itself in.
For instance, the following image shows that there is a sudden spike in the cost at the start of the month and eventual drop from a temporary or short-lived resource in a subscription. You may not want to action this at the end of the month rather soon after the unusual spend event occurred.
Cost anomalies can be categorized into four types based on the granularity
- Anomalous spike in Azure Service costs: It refers to the cost spikes of a particular Azure service like VM, App Service and Storage for the past days compared to normal.
- Anomalous spikes in Cost per Usage: It determines the cost fluctuation per unit of usage. For instance, if there is a spike in cost per hour of compute, this may indicate an increase in pay-as-you-go vs discount plans or shift towards expensive services.
- Anomalous spikes in business unit cost: It goes to the granular approach of identifying cost spikes in certain business units like cost per team or environment allowing the engineers to take ownership of their cost. It supports to implement one of the core principles of FinOps “Everyone takes ownership for their cloud usage and optimize spending”
- Anomalous drop on unit economics: This signifies drop in the ratio between revenue derived from a business unit (e.g.: product feature) built using certain Azure services to the cost of running those services is an indication of loss of efficiency in the way resources are used. It supports to implement another core principles of FinOps “Decisions are driven by business value of cloud”
Different tooling options available for Azure cost anomaly detection
When it comes to detecting anomalies in Azure costs, several tooling options are available to help businesses identify irregularities and optimize spending. When the company has very minimal spending with just a couple of subscriptions, they can probably manage with the native tools and the once with complex environment might require an advanced tooling that goes beyond the traditional methods.
Native Azure Cost Management
The cost anomaly detection feature provided in the Azure portal needs to be matured and provide cost anomaly insights at the business unit level.
When writing this blog post, you can only create an anomaly alert within limited scopes like subscription, resources group, and resources.
These alerts often go unactionable as they only tell you that the cost has spiked somewhere without clarity on who owns it and its impact on the business.
Build vs Buy
Often, teams encounter a situation between building their own solution or buying a pre-existing one, like a SaaS platform.
Building cost anomaly detection software from scratch offers the advantage of customization. However, this approach typically involves significant time and effort in development, testing, and deployment. Buying a pre-built, most likely SaaS-based, cost anomaly detection solution reduces time to market.
Turbo360 – A SaaS based cost anomaly detection solution
Turbo360’s Azure Cost Management tool goes beyond traditional budget monitoring setups by introducing automated anomaly detection and alert mechanisms. The platform leverages historical cost data to autonomously identify and notify relevant stakeholders of any unexpected cost changes.
Unlike conventional approaches that require users to set thresholds manually, Turbo360 dynamically assesses average cost differences daily and monthly. Also, the anomalies can be detected at any desired business unit level, such as per product or team.
How does Turbo360 detect cost anomalies?
The platform leverages a statistical method that provides a more sophisticated approach in identifying cost anomalies than a threshold-based approach.
The anomaly detection algorithm leverages the historical data up to 90 days to predict the cost patterns and determine the cost anomalies.
The system offers a two fold approach to anomaly detection:
Daily Variation Analysis: By comparing the average cost differences between successive days within a month, the system can promptly identify any unusual fluctuations. This real-time monitoring ensures that stakeholders receive immediate alerts when deviations occur, enabling swift response and proactive cost management.
Long-term Trend Analysis: To provide a comprehensive perspective, the system also analyses monthly cost differences up to the last 90 days. This allows for the detection of broader trends and patterns, enabling stakeholders to identify and address potential cost anomalies that may have developed over a more extended period.
How is anomaly detected in time series?
In Turbo360, the anomaly detection process consists of several steps. First, it involves gathering historical data and establishing a baseline for normal behaviour. Subsequently, a statistical test is applied to current daily costs to decide whether they are normal or abnormal.
This comparison helps assess whether the observed variation is significant or falls within the expected ranges.
To determine significance, Turbo360 typically utilizes a prediction confidence interval. If the data point falls outside this confidence interval, it is considered an anomalous event.
Cost anomaly detection with unit economics
Turbo360 offers significant value by enabling users to identify cost anomalies at a granular unit cost level, such as per team, environment, or product. Organizations can enhance the accuracy and relevance of identifying cost anomalies by incorporating unit economics into anomaly detection.
This approach goes beyond relying solely on historical data and forecasts, addressing the limitation of lacking granularity and reducing the occurrence of false positives.
To illustrate, consider the scenario of notifying about a cost alert tailored to an engineering team responsible for respective features or resources. This targeted approach ensures that relevant alerts are delivered, minimizing the chance of overlooking anomalies and promoting a sense of ownership within the team to investigate and address the reasons behind that cost spike.
With this, you can achieve one of the core principles of FinOps, which is empowering the engineers to take ownership of their cloud usage.
The below picture represents the cost anomaly feature in Turbo360. Once you navigate the Cost Analyzer module, you will notice an option to configure the organization tree hierarchical view, allowing you to create unit costs for the desired segments. The system automatically assigns relevant costs to this cost group.
Further, to see the automatically detected anomalies in the unit cost context, you need to navigate to the monitoring tab, under which a line graph will be shown with the usual cost spend pattern and indicate anomalous events, if any, with a red dot.
Also, the platform allows you to view the historical incidents detected so far. The alert typically contains the usual cost per day or month and the current cost with potential root cause indicating the subscription and resource type where the cost has spiked.
In the Manage section, you can select the daily or monthly frequency you want to scan for cost anomalies. Users can configure notification channels like Teams, Slack, or a Service Now ticket.
How FinOps and Finance teams can mitigate bill shocks?
With timely information on the cost anomalies, the FinOps team can ensure the extra expenses are allocated to those accountable by implementing the show-back policy.
In this process, anomalies are accounted for and reported to the finance team, aiding them in forecasting cloud expenses for the remaining period. To illustrate, if the budget is maintained in a CSV file, tracking anomaly changes might appear as follows. This method ensures that the Finance team can proactively forecast the budget and effectively manage unforeseen shocks.
Conclusion
In conclusion, embracing Azure cost anomaly detection is not just about cost savings; it’s about fortifying your organization against unforeseen expenses through cost surveillance with immediate view on cost spike, optimizing cloud utilization, and driving continuous improvement in your cloud strategy.
With the right tools and strategies in place, businesses can navigate the complexities of Azure cost management with confidence.
While the tools in native Azure portal is not matured enough to help you identify anomalies and reduce Azure cost, there is an alternative option to choose Turo360 which comes as an out of the box SaaS solution. Start your free trial to explore the platform.
