Skip to content

[ISSUE #12657]✨Build the release Broker with profile-guided optimization - #12658

Merged
mxsm merged 1 commit into
mainfrom
perf-release-broker-pgo
Nov 15, 2026
Merged

mxsm merged 1 commit into
mainfrom
perf-release-broker-pgo

Conversation

@mxsm

@mxsm mxsm commented Nov 15, 2026

Copy link
Copy Markdown
Owner

Which Issue(s) This PR Fixes(Closes)

Brief Description

The release service images now ship a Broker built with profile-guided optimization (PGO).

  • docker/Dockerfile.base
    • builder-base installs the rustup llvm-tools component, which provides llvm-profdata for the pinned toolchain.
    • service-builder gets a build argument ROCKETMQ_BROKER_PGO (default true). With true, the Broker is built three times over: an instrumented build with the release features, a training run on loopback, and the release build with the merged profile. Both Broker builds use their own target directory because their RUSTFLAGS differ. false keeps the previous plain build for faster local images. NameServer, Controller and Proxy are unchanged.
  • scripts/pgo-train-broker.sh starts a NameServer and the instrumented Broker on loopback and drives the client-production-benchmark example: 20,000 synchronous sends, 200,000 asynchronous sends with 256 in flight, 200,000 LitePull messages, 100,000 asynchronous sends with 16 in flight, and four concurrent sessions of 50,000 sends. It stops the Broker with SIGTERM so the profile is written, fails the build when a step fails or no profile appears, and keeps client and server output in log files that it prints on failure. Training takes about 25 seconds. The profile is generated in the build, never committed, so it always matches the code.
  • Cargo.lock: cc 1.4.6 to 1.11.2 (and find-msvc-tools 0.1.12 to 0.1.15). cc 1.4.6 forwards -Cprofile-generate and -Cprofile-use to clang, so the builder image's clang 14 instrumented the C dependencies against an older profiling runtime than rustc's, and the instrumented Broker crashed at startup (even --help exited with SIGSEGV; the same build with gcc did not). cc 1.10 and later no longer forward these flags (cc-rs issue 1986).
  • Docs: the container deployment page (English and zh-CN) and scripts/README.md describe the build argument and the training script.

Measured gain (Linux Docker, 16 CPUs, Rust benchmark client, 1 KiB messages, Broker built with the release features otlp-metrics,otlp-traces,otlp-logs,tls,allocator-mimalloc, PGO against the same features without PGO, interleaved pairs, pairs with either side below 85% of the batch median throughput dropped, 0 failures):

Mode Cycles per message Instructions per message Throughput
async, 16 workers 259.9k to 194.4k (-25.2%, 3 of 3 pairs lower) -22.1% +17.2%
async, 4 workers 219.1k to 172.8k (-21.1%, 3 of 3) -21.7% +19.1%
sync 537.4k to 393.0k (-26.9%, 4 of 4) -16.8% +12.9%

The Broker copied out of the broker image built from this Dockerfile.base (clang for the C dependencies), against the same plain build (built with gcc in the benchmark container), 3 interleaved pairs each, 0 failures: async with 16 workers -26.1% cycles (3 of 3 lower), -21.9% instructions, +19.7% throughput; sync -29.6% cycles (3 of 3), -18.0% instructions, +16.4% throughput.

Build cost: in the service-builder stage on that machine, the instrumented Broker took 7 min 52 s, the benchmark client 3 min 22 s, training 24 s and the PGO Broker 11 min 55 s; the whole stage took 40 min. The image publish workflow allows 240 minutes.

How Did You Test This Change?

  • docker buildx build --file docker/Dockerfile.base --target service-builder and --target broker at HEAD: built, trained (one raw profile, merged without warnings) and installed the PGO Broker. Before the cc upgrade the same build failed at the training step with the SIGSEGV described above.
  • The measurements above: the same commands run in the benchmark container, plus the Broker copied out of the built image.
  • bash -n scripts/pgo-train-broker.sh; the training script was also run on its own against an instrumented Broker.
  • cargo check -p rocketmq-broker --features allocator-mimalloc with the upgraded cc.
  • npm run build in rocketmq-website: both locales generated.

🤖 Generated with Claude Code

@mxsm
mxsm merged commit bbcebc2 into main Nov 15, 2026
19 of 23 checks passed
@rocketmq-rust-bot

Copy link
Copy Markdown
Collaborator

🔊@mxsm 🚀Thanks for your contribution🎉!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Enhancement✨] Build the release Broker image with profile-guided optimization

3 participants