Repository navigation
Feature: fallback pool (prefer on prem, fallback to another cloud) #442
Description
Activity
Hi @brandonkal !
You already can! 😁 Well, at least for the case where one pool is at capacity.
You can switch your entity pool balancer type from
roundrobintopackby running:garm-cli repo update --pool-balancer-type pack YOUR_REPO_ID
Then you can set priorities to your pools:
garm-cli pool update YOUR_POOL_ID --priority 200 garm-cli pool update YOUR_BACKUP_POOL_ID --priority 100
Pools are sorted by priority in descending order. So if the priority is set to
pack, GARM will try to fill up the highest priority pool before moving on to the next one.And on that note, I did a terrible job at documenting things. I just noticed I have just one mention of priorities in the docs, but absolutely no example of how to use the feature 😓.
Give it a shot and let me know if it works.
to view pool priorities when listing them, you can run:
garm-cli pool ls --all --long
Thank you this is good to know. My hypervisor host is without internet today so I have to set up an AWS fallback (which has been on my list but now is a must). Ultimately I want to:
- run arm jobs only on AWS
- run amd64 jobs on premises (with some queuing even)
- fallback amd64 jobs to AWS if the hypervisor is down or there are 10+ amd64 jobs waiting in the queue.
I suppose for now I can set my primary pool to have 0 capacity.
You can use the priority option and just swap priorities when you notice an outage. Set a priority of
0for the pool that has an outage and anything higher than0for the other pool(s). You might need to force remove erred runners.You can define as many pools as you like. You can also have multiple pools with the same labels that can handle the same jobs. I recommend you use a single, unique label for pools (eg:
linux-arm64-large). Define multiple pools with the same label but different priorities, and you're good to go.If you only define a single pool for arm64 on aws, then that is the only place where arm64 will spin up.
You can also enable/disable pools. So if you notice that a provider is down, you can disable pools for that provider.
- locked and limited conversation to collaborators
on Jul 1, 2025
It would be nice to give pools priority. So that you could either: