{"id":631,"date":"2024-10-05T09:00:00","date_gmt":"2024-10-05T09:00:00","guid":{"rendered":"https:\/\/datablog.roman-halliday.com\/?p=631"},"modified":"2024-10-04T17:07:40","modified_gmt":"2024-10-04T17:07:40","slug":"this-one-little-trick-will-save-you-up-to-79-on-your-cloud-refresh-costs","status":"publish","type":"post","link":"https:\/\/datablog.roman-halliday.com\/index.php\/2024\/10\/05\/this-one-little-trick-will-save-you-up-to-79-on-your-cloud-refresh-costs\/","title":{"rendered":"This one little trick will save you up to 79% on your cloud refresh costs"},"content":{"rendered":"\n

Yes, the title is clickbait, but the savings are real! The approach can be applied to save money on any cloud data infrastructure. The saving in question applies to regular refreshes, not other data processes (more on that in the future).<\/p>\n\n\n\n

The Problem<\/h2>\n\n\n\n

It’s not the first time I’ve looked at the numbers thinking this, and it won’t be the last. I glanced at the cloud admin dashboard and thought “Hmmm, those costs have gone up. It’s probably less cost in engineer resource than the amount we could save by making changes”. The place to start is the famous “low hanging fruit”, easy changes with significant impact.<\/p>\n\n\n\n

The lowest hanging fruit<\/h3>\n\n\n\n

Most of us (especially those with experience in ‘bare metal’, in-house infrastructure) will set our regular data refresh to be hourly and then get on with adding value for our consumers. Over time the data volumes will increase, the business will ask for more and one day someone will look at the bill for cloud costs and say “Wow, that’s got bigger”. If you can get that number down (especially before someone notices it), you are winning with your stakeholders.<\/p>\n\n\n\n

If you are running a data refresh (or any process) hourly, seven days a week, that’s 168 executions in a week. Whatever your cost per refresh, 168 multiplied by any base number will probably add up to a significant increase on it.<\/p>\n\n\n\n

24 * 7 = 168<\/code><\/pre>\n\n\n\n
The Solution<\/h2>\n\n\n\n
You probably already guessed it… But in my own wise words:<\/p>\n\n\n\n
Don’t run your refresh every hour of every day if you don’t need to!<\/p>David Roman-Halliday<\/cite><\/blockquote><\/figure>\n\n\n\n
The best case<\/h3>\n\n\n\n
If your data consumers are only working during office hours in one country, you probably only need to refresh the data for while they are working. If that’s 9 till 5, Monday to Friday then the hourly refresh only needs to run 09:00 – 16:00.<\/p>\n\n\n\n
That’s seven times a day, five days a week. A total of 35 executions per week (a saving of 133 executions on our starting point of hourly), which is a difference of 79.17%<\/p>\n\n\n\n
7 * 5 = 35 -- Mon -> Fri, 09:00 -> 16:00<\/pre>\n\n\n\n
A more realistic case<\/h3>\n\n\n\n
Realistically, some people start earlier (or want the numbers ready for their 9am meeting). While some may work a little later than finishing bang on 5pm. There may be reports and updates which need to go out including weekends.<\/p>\n\n\n\n
This still clocks in at 54 refreshes (far fewer than 168) a week, to refresh from 8am until (and including) 6pm Monday through Friday and twice a day (for example the start of the morning working hours, and the start of the afternoon).<\/p>\n\n\n\n
  10 * 5 = 50 -- Monday  -> Friday : 08:00 -> 18:00
+  2 * 2 =  4 -- Saturday & Sunday : 08:00  & 13:00
           54<\/pre>\n\n\n\n
That’s still a serious reduction of 67% in volume of executions, which is a significant saving on regular refreshes.<\/p>\n\n\n\n
Coding it in<\/h2>\n\n\n\n
As most orchestration and related tools work with cron style definitions for the schedule, we can save time (and make it easier to read\/check) using crontab.guru<\/a>. Some examples are:<\/p>\n\n\n\n