Repository navigation
Add a JVMTI agent for causal profiling of Java - #290
Merged
Merged
Conversation
The Rust, Go and Swift bindings are thin: they hand progress points to libcoz,
which samples the PC, maps it to a source line through DWARF, and delays native
threads. None of that works on the JVM -- JIT-compiled methods live in anonymous
memory with no DWARF, and the threads to delay are Java threads libcoz has never
seen. So Java gets a real agent, which reimplements the causal profiling loop in
JVM terms and emits coz's own JSON profile so `coz plot` reads it unchanged.
Sampling uses GetAllStackTraces + GetLineNumberTable. JCoz instead used
AsyncGetCallTrace, a private undocumented HotSpot entry point; everything here is
documented JVMTI 1.2, needs no tools.jar (gone since JDK 9) and loads statically
with -agentpath, so JEP 451 does not apply.
Virtual speedup is real: for each sample on the selected line the agent owes
every other thread a pause of speedup * sample_period, paid with
SuspendThreadList/ResumeThreadList and batched so the VM reaches a safepoint once
per batch rather than once per sample.
No lock is ever shared with a Java thread. The agent suspends threads at
arbitrary points, so if a suspended thread held a lock the agent then wanted, it
would deadlock the VM. The progress-point registry is therefore an append-only
lock-free list; everything else is touched by the agent thread alone.
Three things this took to get right:
- coz.Coz keeps its "is the agent attached" flag in a holder class. The agent
finds the class with JNI FindClass, which *initializes* it -- before
RegisterNatives has bound the natives. A static field on Coz would latch
"not profiling" for the life of the VM.
- Idle paths must keep sampling. `recent` (the pool the next line is drawn
from) is cleared once a line is drawn, so sleeping without sampling left the
profiler unable to ever select a line again.
- The experiment's duration is wall time *minus the delay inserted*, matching
libcoz (profiler.cpp:362). Pausing other threads makes the run longer in real
seconds while making the selected line relatively faster; that paused time is
not progress. Reporting raw wall clock gave every line a negative slope.
Verified on macOS arm64 (Temurin 17) and Linux aarch64 (Temurin/OpenJDK 21):
18-19 experiments, slowWork's loop at slope 0.37-0.50 (R^2 up to 1.00, max
+41%), fastWork's loop flat. Sampling is 1.7:1 across the two loops, matching
their 2:1 work ratio.
Documented caveat: the agent samples at safepoints, and GraalVM's JIT leaves hot
counted loops without a safepoint poll, pushing mean time-to-safepoint from
~100us to ~100ms. Use a HotSpot JDK or -XX:-UseJVMCICompiler; `verbose=1` prints
the number so you can check.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Measured on example/Toy.java under HotSpot C2, the loop header (line 47) takes 1666 samples and the line that does the work (line 48) takes zero: every sample lands on the strip-mined back-edge safepoint poll. The README said the bias was "close enough to attribute the loop", which is true, but the surrounding text still read as line-level attribution. It is loop/method-level. Also records why AsyncGetCallTrace is not the default: it is exported from libjvm on both HotSpot and GraalVM, but it is undocumented, in no specification, absent from non-HotSpot VMs, and needs per-thread CPU timers that macOS lacks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
# Conflicts: # .github/workflows/ci.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Java is the one language here that cannot be a thin binding.
Rust, Go and Swift hand their progress points to
libcoz, which samples the PC, maps it to a source line through DWARF, and inserts virtual delays by pausing native threads. On the JVM none of that holds: JIT-compiled methods live in anonymous memory with no DWARF, so a sampled PC maps to nothing, and the threads to delay are Java threadslibcozhas never heard of.So Java gets a real JVMTI agent that reimplements the causal profiling loop in JVM terms, and emits coz's own JSON Lines profile so
coz plotreads it unchanged.Design
GetAllStackTraces(documented JVMTI)AsyncGetCallTrace(private, undocumented HotSpot)coz.Coz.progress()in sourcetools.jar(removed in JDK 9)-agentpath(JEP 451 N/A)Virtual speedup is real, not simulated: for each sample on the selected line the agent owes every other thread a pause of
speedup × sample_period, paid withSuspendThreadList/ResumeThreadListand batched so the VM reaches a safepoint once per batch rather than once per sample.No lock is ever shared with a Java thread. The agent suspends threads at arbitrary points; if a suspended thread held a lock the agent then wanted, it would deadlock the VM. The progress-point registry is an append-only lock-free list, and everything else is touched by the agent thread alone.
Three bugs worth calling out
coz.Cozkeeps its "agent attached" flag in a holder class. The agent locates the class with JNIFindClass, which initializes it — beforeRegisterNativeshas bound the natives. A plainstatic final boolean ENABLED = probe()onCoztherefore ran first, caughtUnsatisfiedLinkError, and latched "not profiling" for the life of the VM. Symptom:coz agent attached: falsewith the agent demonstrably loaded.Idle paths must keep sampling.
recent— the pool the next line is drawn from — is cleared once a line has been drawn. Any path that slept without sampling left it empty, soselect_line()returned nothing forever and the profiler silently stopped sampling.Experiment duration is wall time minus the inserted delay. This is what
libcozdoes atprofiler.cpp:362(duration = elapsed - experiment_delay). Pausing the other threads makes the run longer in real seconds while making the selected line relatively faster; the paused time is not progress. Reporting raw wall clock madeperiod = duration/deltagrow with the speedup and gave every line a negative slope (−0.514 on the toy).Verification
java/example/Toy.java: two threads per round,slowWorkdoes twice the work offastWork.macOS arm64 (Temurin 17) — 18 experiments, stable over 3 runs:
Linux aarch64 (OpenJDK/Temurin 21) — 19 experiments, both runs:
Sampling lands 1666 vs 944 on the two loops — a 1.77:1 ratio against their 2:1 work ratio.
The GraalVM caveat (documented, and CI pins around it)
The agent samples at safepoints, so its sample rate is bounded by time-to-safepoint. Measured on the same toy:
GetAllStackTracesGraal left no safepoint poll in the hot counted loop, so the agent sampled ~10×/second and could attribute nothing. Use a HotSpot JDK, or
-XX:-UseJVMCICompileron GraalVM.verbose=1prints the number so users can check;java/README.mdsays "if that number is in the milliseconds, your profile is not trustworthy."This is also why the CI job pins
distribution: temurin.Contents
java/agent/coz_agent.cpp— the agentjava/src/coz/Coz.java—progress/begin/end/scope/isAvailable; all no-ops without the agentjava/example/Toy.java,java/README.md,java/CMakeLists.txt-DBUILD_JAVA_AGENT=ON(off by default; needs a JDK)lang-javaon Linux + macOS, asserting experiments are attributed toToy.java🤖 Generated with Claude Code