UI extremely slow when db.ha.enabled=true after MySQL source failure (4.22.1.0) #14172

Activity

1. nabassa1323

   added

   bug

   on Sep 15, 2026
2. DaanHoogland

   linked a pull request that will close this issueallow loadbalanced DB #14279on Oct 1, 2026
3. DaanHoogland commented on Oct 1, 2026

   

   DaanHoogland

   on Oct 1, 2026

   Contributor

   More actions

   @nabassa1323 I had claudecode create a PR; #14279. Can you see if this meets your requirements and test it?

   I think this was never implemented seriously and I think we would need to classifyt it as a new feature, though some claims had been made.
4. DaanHoogland

   removed

   bug

   on Oct 1, 2026
5. DaanHoogland

   added this to the 4.22.2 milestone on Oct 1, 2026
6. github-actions

   added

   component:database

   component:management-server

   component:mysql

   status:Has a PR

   type:new-feature

   on Oct 1, 2026
7. github-actions commented on Oct 1, 2026

   

   github-actionsbot

   on Oct 1, 2026 – with GitHub Actions

   More actions

   🎯 Triage report

   Reporter observes that with db.ha.enabled=true on MySQL 8.x, after the configured source becomes unreachable, the management server UI becomes extremely slow because the connector keeps retrying the dead source on (nearly) every request. Manually promoting the replica and pointing db.cloud.host/db.usage.host at it (per the install guide's Failover procedure) restores normal performance. A maintainer has already commented that this DB HA mechanism was likely never fully implemented/tested, and a candidate PR (#14279) has been opened to address it.

   📊 Assessment

   Dimension	Value	Reasoning
   Type	type:new-feature	Maintainer comment indicates the existing db.ha.enabled client-side failover was never seriously implemented and should be treated as a new feature rather than a regression/bug fix.
   Component	component:database, component:management-server, component:mysql	Issue concerns MySQL connector failover behavior within the management server's DB HA configuration.
   Severity	n/a	Not labeled as a bug; classified as new-feature work per maintainer.
   Labels	type:new-feature, component:database, component:management-server, component:mysql, status:Has a PR	See reasoning above; status:Has a PR added since #14279 is linked.
   Coding agent	Suitable	Well-documented repro steps, config, versions, and a maintainer-linked candidate PR (#14279) already exist; mainly needs validation/testing by the reporter against that PR.

   🔗 Similar issues

   No clearly duplicate or related open issues were found via search for "db.ha.enabled MySQL failover replica".

   💡 Notes and suggestions

   • PR allow loadbalanced DB #14279 was opened in response to this issue — worth asking @nabassa1323 to validate it against their reproduction steps (2 MS nodes, GTID replication, source outage simulation).
   • Consider updating the admin guide (currently references MySQL 5.1/5.5 testing) once the fix/feature lands, to reflect supported MySQL 8.x behavior and clarify whether manual promotion remains the documented production path.
   • Relevant connector settings called out by reporter: secondsBeforeRetrySource, queriesBeforeRetrySource, initialTimeout — may need tuned defaults or documentation if the new implementation still relies on MySQL Connector/J failover semantics.

   Generated by Daily Issue Triage · sonnet50 77.8K · ◷

   Add this agentic workflows to your repo

   To install this agentic workflow, run

   gh aw add githubnext/agentics/workflows/daily-issue-triage.md@d7c1dc4b72b00607a67caaffdcc216cb64379cf9
   
8. nabassa1323 commented on Oct 6, 2026

   

   nabassa1323

   on Oct 6, 2026

   Author

   More actions

   @nabassa1323 I had claudecode create a PR; #14279. Can you see if this meets your requirements and test it?

   I think this was never implemented seriously and I think we would need to classifyt it as a new feature, though some claims had been made.

   Hi @DaanHoogland,

   Thanks for pointing to #14279 — we tested a backport of the change on CloudStack 4.22.1.0 (injected TransactionLegacy into the management fat JAR on both MS).

   Setup

   2 management servers
   MariaDB GTID, primary + replica
   db.ha.enabled=true with StaticStrategy
   replica listed in db.cloud.replicas / db.usage.replicas
   PR timeouts: connectTimeout=3000, socketTimeout=30000, loadBalanceBlocklistTimeout=30000
   Test Stopped the primary MySQL and the management server on the primary site, leaving the other MS (and the replica DB) up.

   Result

   The new path is active (com.mysql.cj.jdbc.ha.MultiHostConnectionProxy in the stack).
   No more UI / API slowness compared to before — the previous hang is gone.
   We also did not see HikariPool / createTimeoutException in this run.
   The remaining MS talked to the replica, but write paths failed with: The MySQL server is running with the --read-only option so it cannot execute this statement e.g. SELECT … FROM op_ha_work … FOR UPDATE from HighAvailabilityDaoImpl.take.
   So the latency/hang issue looks fixed by the PR. After losing the primary MS + primary DB, without promoting the replica to read/write, the surviving MS stays responsive but write operations fail on read-only.

   Is the expected model for db.ha + loadbalance that the replica must be promoted (or already writable) for CloudStack to remain fully usable after primary loss?

   Happy to retest with any config tweak you recommend.
9. DaanHoogland commented on Oct 6, 2026

   

   DaanHoogland

   on Oct 6, 2026

   Contributor

   More actions

   I think it is expected, yes. In a primary / replica scenario failover will have to be manual. I think we can further improve in the future. I also know there are people using galdera in the wild. For that you need at least three instances of the DB. If you have a scenario for failover that works with this change I think we should merge.

   cc @sureshanaparti @weizhouapache can you add your thoughts?
10. nabassa1323 commented on Oct 14, 2026

    

    nabassa1323

    on Oct 14, 2026

    Last edited by nabassa1323

    Author

    More actions

    @DaanHoogland,
    Yes, we have a failover scenario that works with this change:
    Primary MySQL (and primary-site MS) down → with #14279, the remaining MS stays responsive (no UI hang / no Hikari timeout). It uses the replica.
    Manual promote of the replica (read_only=OFF, stop replication as needed) → writes succeed again and CloudStack is fully usable.
    So: automatic writable failover is still not there (expected for primary/replica), but manual failover works and the hang is fixed. On that basis we think #14279 should be merged.
11. weizhouapache commented on Oct 14, 2026

    

    weizhouapache

    on Oct 14, 2026

    Member

    More actions

    11. @DaanHoogland,
    Yes, we have a failover scenario that works with this change:
    Primary MySQL (and primary-site MS) down → with #14279, the remaining MS stays responsive (no UI hang / no Hikari timeout). It uses the replica.
    Manual promote of the replica (read_only=OFF, stop replication as needed) → writes succeed again and CloudStack is fully usable.
    So: automatic writable failover is still not there (expected for primary/replica), but manual failover works and the hang is fixed. On that basis we think #14279 should be merged.

    thanks @nabassa1323
    looks nice