Slash your cloud bill by moving data workloads to cost-effective compute
These are tough times for technology
There is a tech recession going on, with layoffs across the industry and big tech companies losing most of their market value.
It’s a cause for reflection and rationalisation of all activities in the organisation. It’s also a moment of truth for your data workloads. Which ones are worth it and which ones cost more than they bring?
Cloud costs are the silent killers of data platforms
We talked about this before, here: https://blog.dataminded.com/why-rising-cloud-costs-are-the-silent-killers-of-data-platforms-52a98b371f28. Cloud costs, while small at the start of the project, can become the biggest pain point of your data journey. Yes, engineers are expensive. But it’s not a recurring cost that you have to swallow every month for every use case ever built. It’s a linear cost, per use case. Your cloud bill, on the other hand, is not linear, it’s compounded. And the bill is due every month. It reminds me of this quote by Albert Einstein:
After years of migrating data workloads to the cloud, companies are starting to notice the compounding effect on their monthly bills and are wondering what to do about it.
Where to go from here?
So what do I propose? Go back on-premise and ditch the cloud all together? For most companies, this is not realistic either. It takes a big upfront investment and a long-term commitment to go back on-premise.
Yet, you can face the problem head-on, by following this 4-step proces:
1. Reduce the license costs
These days, license costs can make up more than 50% of your cloud bill. Example data services that can become very expensive: Databricks, Snowflake, AWS EMR (Serverless), AWS Glue, AWS Redshift, Azure Data Factory, … There are more cost-effective services available.
“But then we have to run it ourselves”. Not always, but often, yes. So, don’t do it to save $500 on your monthly bill. But $5K or $50K? Becomes tempting.
Also, there are low-cost (managed) alternatives on the market. They don’t work for every use case, but if you are serious about reducing your cloud bill, you should put them on the table: Aiven Kafka, Airbyte, DuckDB, Ahana Cloud, Conveyor, …
2. Go for SPOT instances
When available and smartly configured on your target compute platform, SPOT instances are a great way to reduce cloud costs. SPOT instances are typically 60–80% cheaper than on-demand instances. Yes, you need to build redundancy and resiliency into your data platform, but shouldn’t you do that anyway?
A typical way on achieving this is by running on Kubernetes clusters. You get some extra overhead of running a Kubernetes cluster, but on the plus side, you get to reduce your instance cost by 60 to 80%
3. Right-size your resources
It’s bonkers that we still need to discuss this in 2022. Wasting resources can take many forms:
- You don’t need clusters of 3 nodes, with m5.2xlarge instances, to basically run a shell script or basic python script. This is often configured because that’s the minimum size cluster you can schedule
- No, not every job needs 300GB of RAM and 128 cores. Observe the actual usage of your resources and finetune your jobs accordingly
- Do you really need those long-running clusters that can’t dynamically scale up and down? We have auto-scaling groups now, and there are plenty of services, like KEDA, that can help you auto-scale based on your specific needs.
- Idle notebooks. yes those costs do add up
These measures can easily save another 50% of your cloud compute bill
4. Monitor, analyse, iterate
Having the ability to attribute cloud costs to individual projects, or even individual jobs and pipelines, is crucial. You can have a much more focused discussion on where you’re spending most, where to act, and whether those actions actually have an impact.
There is a cut-off point, beyond which further refactoring will cost more than it will bring. Often you can get 80% of the savings with 20% of the effort.
A note on the percentages above: With each of the steps, I claimed to save 50% of cloud costs, or more. Of course, these don’t add up to 150%. AWS won’t be sending you money any time soon. They add up by only having to pay 50% o 50% of 50% or 12.5%. Those are insane savings. And you don’t get there for every job, and for every pipeline. Also, cloud compute cost is only part of your overall cloud bill. Still, with the right focus, you are able to make a dent in the universe.
Where to get started saving on cloud?
On your own
We have published a great blog and github repository on how to move workloads to containers, which you can then deploy on low cost infrastructure. You can get started on your own, here: https://blog.dataminded.com/from-notebook-hell-to-container-heaven-20cbe05100a1
Managed solution with Conveyor
If you are looking for a more managed solution, we have packaged all our learnings and practices in Conveyor, a managed runtime to run all your compute workloads at very low cost.
Free consultation on how to reduce cloud costs
You can also book a meeting directly with me here to discuss your specific challenges related to cloud costs. We will give you advice on what to focus on and how to tackle the problem
