Skip to content

Add a JVMTI agent for causal profiling of Java - #290

Merged
emeryberger merged 3 commits into
masterfrom
java-jvmti-agent
Jul 10, 2026
Merged

emeryberger merged 3 commits into
masterfrom
java-jvmti-agent

Conversation

@emeryberger

Copy link
Copy Markdown
Member

Java is the one language here that cannot be a thin binding.

Rust, Go and Swift hand their progress points to libcoz, which samples the PC, maps it to a source line through DWARF, and inserts virtual delays by pausing native threads. On the JVM none of that holds: JIT-compiled methods live in anonymous memory with no DWARF, so a sampled PC maps to nothing, and the threads to delay are Java threads libcoz has never heard of.

So Java gets a real JVMTI agent that reimplements the causal profiling loop in JVM terms, and emits coz's own JSON Lines profile so coz plot reads it unchanged.

Design

this agent JCoz
stack sampling GetAllStackTraces (documented JVMTI) AsyncGetCallTrace (private, undocumented HotSpot)
progress points coz.Coz.progress() in source JVMTI breakpoints set over JMX/RMI
client none separate RMI/CLI process, needs tools.jar (removed in JDK 9)
agent load static -agentpath (JEP 451 N/A) static, but control path uses dynamic attach
macOS yes Linux/x86 only

Virtual speedup is real, not simulated: for each sample on the selected line the agent owes every other thread a pause of speedup × sample_period, paid with SuspendThreadList/ResumeThreadList and batched so the VM reaches a safepoint once per batch rather than once per sample.

No lock is ever shared with a Java thread. The agent suspends threads at arbitrary points; if a suspended thread held a lock the agent then wanted, it would deadlock the VM. The progress-point registry is an append-only lock-free list, and everything else is touched by the agent thread alone.

Three bugs worth calling out

  1. coz.Coz keeps its "agent attached" flag in a holder class. The agent locates the class with JNI FindClass, which initializes it — before RegisterNatives has bound the natives. A plain static final boolean ENABLED = probe() on Coz therefore ran first, caught UnsatisfiedLinkError, and latched "not profiling" for the life of the VM. Symptom: coz agent attached: false with the agent demonstrably loaded.

  2. Idle paths must keep sampling. recent — the pool the next line is drawn from — is cleared once a line has been drawn. Any path that slept without sampling left it empty, so select_line() returned nothing forever and the profiler silently stopped sampling.

  3. Experiment duration is wall time minus the inserted delay. This is what libcoz does at profiler.cpp:362 (duration = elapsed - experiment_delay). Pausing the other threads makes the run longer in real seconds while making the selected line relatively faster; the paused time is not progress. Reporting raw wall clock made period = duration/delta grow with the speedup and gave every line a negative slope (−0.514 on the toy).

Verification

java/example/Toy.java: two threads per round, slowWork does twice the work of fastWork.

macOS arm64 (Temurin 17) — 18 experiments, stable over 3 runs:

example/Toy.java:47 |   0.496 |  1.00 | +     41.3%    <- slowWork's loop
example/Toy.java:55 |   0.090 |  0.21 | +     14.1%    <- fastWork's loop

Linux aarch64 (OpenJDK/Temurin 21) — 19 experiments, both runs:

example/Toy.java:47 |   0.396 |  0.90 | +     40.6%
example/Toy.java:55 |   0.007 |  0.02 | +      2.3%

Sampling lands 1666 vs 944 on the two loops — a 1.77:1 ratio against their 2:1 work ratio.

The GraalVM caveat (documented, and CI pins around it)

The agent samples at safepoints, so its sample rate is bounded by time-to-safepoint. Measured on the same toy:

JIT mean GetAllStackTraces sample ticks
HotSpot C2 147 µs 1671
GraalVM JIT 103,862 µs 105

Graal left no safepoint poll in the hot counted loop, so the agent sampled ~10×/second and could attribute nothing. Use a HotSpot JDK, or -XX:-UseJVMCICompiler on GraalVM. verbose=1 prints the number so users can check; java/README.md says "if that number is in the milliseconds, your profile is not trustworthy."

This is also why the CI job pins distribution: temurin.

Contents

  • java/agent/coz_agent.cpp — the agent
  • java/src/coz/Coz.java — progress / begin / end / scope / isAvailable; all no-ops without the agent
  • java/example/Toy.java, java/README.md, java/CMakeLists.txt
  • -DBUILD_JAVA_AGENT=ON (off by default; needs a JDK)
  • CI job lang-java on Linux + macOS, asserting experiments are attributed to Toy.java

🤖 Generated with Claude Code

emeryberger and others added 3 commits July 9, 2026 08:31
The Rust, Go and Swift bindings are thin: they hand progress points to libcoz,
which samples the PC, maps it to a source line through DWARF, and delays native
threads. None of that works on the JVM -- JIT-compiled methods live in anonymous
memory with no DWARF, and the threads to delay are Java threads libcoz has never
seen. So Java gets a real agent, which reimplements the causal profiling loop in
JVM terms and emits coz's own JSON profile so `coz plot` reads it unchanged.

Sampling uses GetAllStackTraces + GetLineNumberTable. JCoz instead used
AsyncGetCallTrace, a private undocumented HotSpot entry point; everything here is
documented JVMTI 1.2, needs no tools.jar (gone since JDK 9) and loads statically
with -agentpath, so JEP 451 does not apply.

Virtual speedup is real: for each sample on the selected line the agent owes
every other thread a pause of speedup * sample_period, paid with
SuspendThreadList/ResumeThreadList and batched so the VM reaches a safepoint once
per batch rather than once per sample.

No lock is ever shared with a Java thread. The agent suspends threads at
arbitrary points, so if a suspended thread held a lock the agent then wanted, it
would deadlock the VM. The progress-point registry is therefore an append-only
lock-free list; everything else is touched by the agent thread alone.

Three things this took to get right:

  - coz.Coz keeps its "is the agent attached" flag in a holder class. The agent
    finds the class with JNI FindClass, which *initializes* it -- before
    RegisterNatives has bound the natives. A static field on Coz would latch
    "not profiling" for the life of the VM.

  - Idle paths must keep sampling. `recent` (the pool the next line is drawn
    from) is cleared once a line is drawn, so sleeping without sampling left the
    profiler unable to ever select a line again.

  - The experiment's duration is wall time *minus the delay inserted*, matching
    libcoz (profiler.cpp:362). Pausing other threads makes the run longer in real
    seconds while making the selected line relatively faster; that paused time is
    not progress. Reporting raw wall clock gave every line a negative slope.

Verified on macOS arm64 (Temurin 17) and Linux aarch64 (Temurin/OpenJDK 21):
18-19 experiments, slowWork's loop at slope 0.37-0.50 (R^2 up to 1.00, max
+41%), fastWork's loop flat. Sampling is 1.7:1 across the two loops, matching
their 2:1 work ratio.

Documented caveat: the agent samples at safepoints, and GraalVM's JIT leaves hot
counted loops without a safepoint poll, pushing mean time-to-safepoint from
~100us to ~100ms. Use a HotSpot JDK or -XX:-UseJVMCICompiler; `verbose=1` prints
the number so you can check.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Measured on example/Toy.java under HotSpot C2, the loop header (line 47) takes
1666 samples and the line that does the work (line 48) takes zero: every sample
lands on the strip-mined back-edge safepoint poll. The README said the bias was
"close enough to attribute the loop", which is true, but the surrounding text
still read as line-level attribution. It is loop/method-level.

Also records why AsyncGetCallTrace is not the default: it is exported from
libjvm on both HotSpot and GraalVM, but it is undocumented, in no specification,
absent from non-HotSpot VMs, and needs per-thread CPU timers that macOS lacks.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@emeryberger
emeryberger merged commit 56ac187 into master Jul 10, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant