Repository navigation
Load Balance HealthCheck Fail when Making Changes to HA Proxy Setting #9582
Description
Activity
Each time when update lb, the haproxy.cfg is regenerated by ACS mgmt or agent.
in that case, all your haproxy changes are gone.I do not think your changes on timeout settings caused the issue.
but the other settings, I am not sure@weizhouapache I think there are two issues here:
- Why would the VR suddenly have health check fail related to LB when we performed the above action?
- How can we change the HA Proxy Default Setting to support the desired settings above? (Timeout Client/Server 50s -> 30Min)
@weizhouapache I think there are two issues here:
- Why would the VR suddenly have health check fail related to LB when we performed the above action?
It might be related to your changes. Have you faced the similar issue before ?
- How can we change the HA Proxy Default Setting to support the desired settings above? (Timeout Client/Server 50s -> 30Min)
AFAIK, no
You can find supported customized lb config by searchinghaproxyin global settingsWe went through HA Proxy in Global Settings but couldnt find anything that helps us overcome this issue.
Where is haproxy.cfg stored? Is it in the systemVM template? If so, would customizing the systemVM template and changing our VPC Network Offering work?
For the health check fail, no we havent encountered this issue before. This is new to us.
We went through HA Proxy in Global Settings but couldnt find anything that helps us overcome this issue.
Where is haproxy.cfg stored? Is it in the systemVM template? If so, would customizing the systemVM template and changing our VPC Network Offering work?
For the health check fail, no we havent encountered this issue before. This is new to us.
the /etc/haproxy/haproxy.cfg in VR is generated by ACS class
https://github.com/apache/cloudstack/blob/main/core/src/main/java/com/cloud/network/HAProxyConfigurator.javathe timeout settings are hard-coded. you can
- change the values in the code
- build
coresub-project - replace cloud-core-XXX.jar on kvm hosts and restart cloudstack-agent
I see @weizhouapache , i think we have no choice.
How do we build this component? We've never done this before. Is there any guide we can refer to?
I see @weizhouapache , i think we have no choice.
How do we build this component? We've never done this before. Is there any guide we can refer to?
You can refer to https://github.com/shapeblue/hackerbook/blob/main/2-dev.md#building-cloudstack
- linked a pull request that will close this issue[router] make a distinction between fatal errors, warnings and unknown as healthcheck result #10710
on Aug 22, 2025 - added this to Apache CloudStack 4.22.0 and removed this from Apache CloudStack 4.21.0
on Aug 29, 2025 - moved this from Todo to Done in Apache CloudStack BugFest - Issues
on Sep 22, 2025 - moved this from In Progress to Done in Apache CloudStack 4.22.0
on Sep 22, 2025
ISSUE TYPE
COMPONENT NAME
CLOUDSTACK VERSION
CONFIGURATION
OS / ENVIRONMENT
SUMMARY
We have 50 Autoscale Groups within the same Virtual Router.
The Default Setting for VR Load Balancer is where:
We have a use case where lots of our customers have very long lived TCP Connections. So we went into the VR, and made the following changes:
After that, we tried accessing services using Autoscale Group Load Balancer. It working for 3-5 Minutes and then we were suddenly we got cut off. We couldnt PING or Telnet the VMs under Autoscale anymore.
We checked and saw VR started throwing HealthCheck issues where: 'Missing Load Balancer For XXXXXX'
To resolve the issue temporarily, we added a new LB Rule and removed it, to force HA Proxy to get the new info.
As a result, the HA Proxy Config was reloaded and the settings all got overrided.
But strangely enough, once we repeat the steps above, the load balance issue no longer there. It seems to be intermittent.
STEPS TO REPRODUCE
EXPECTED RESULTS
ACTUAL RESULTS