BTrace 3.0: Same Old Trick, New Bag
- A bit of history
- Why a release candidate
- New address
- One jar
- Stop writing files for one-line questions
- Extensions, and what they are allowed to do
- One jar, part two: the fat agent
- Attaching to production without sweating
- Leaving without a trace
- The ASM ceiling
- Wire protocol
- Agentic investigations
- What breaks
- Try it
- PS: how this got done
A bit of history
BTrace is old. When I wrote about unattended execution in 2020, I was already calling its origins “more than 10 years ago”. The trick has not changed since. You write a small, restricted Java class and point it at a running JVM. The agent weaves your probe into the bytecode of the methods you care about. No restart, no recompilation, and a verifier that keeps your probe from taking the JVM down with it.
The trick still works. Everything around it had aged.
The last 2.x release, 2.2.6, went out in November 2024. By then my list of grievances was longer than any patch release could fix:
- Three jars (
btrace-agent.jar,btrace-boot.jar,btrace-client.jar) that had to sit next to each other on disk. Fine for a tarball, hopeless for jbang or Maven. - A one-line question (“which method is slow?”) cost a whole class, with imports, annotations and
BTraceUtils.on every call. - A package name,
org.openjdk.btrace, pointing to a place BTrace has not lived in for years. - GPLv2 with the Classpath Exception. A fine license, and a reliable way to start a conversation with someone’s legal department.
- Every new JDK meant waiting for an ASM release, bumping it, and hoping the generated bytecode was still right.
- The
libs=profiles, which appended jars to the global classpath of the application you were observing. - An agent started with
-javaagentlistening on port 2020 for anyone who cared to deploy a probe.
So 3.0.0 is a major release in the proper sense of the word, and the first one in years that I use because I want to, not because I have to. What follows is what changed, roughly in the order you will bump into it.
Why a release candidate
You can install 3.0.0-RC1 today. The short version lives on the project site, btrace.io/btrace; this post is the long one. I would normally ship the final version and fix whatever turns up in a patch release. This time that does not work, because of how much changed at once.
Between 2.2.6 and the RC1 tag there are 165 commits touching 1,869 files, about 197,000 lines added and 31,000 removed. Roughly 640 of those files are main source, 590 are tests, and 240 are documentation and demos. The package namespace, the license, the jar layout, the wire protocol, the instrumentation backend, the extension framework and the attach security model all changed in the same release. A patch release can fix a bug. It cannot take back a Maven coordinate, a package name or a protocol default once people depend on it.
So the candidate comes first. The release process is built for exactly this: Maven Central artifacts are permanent, a broken candidate is superseded by RC2 instead of being deleted, and when a candidate is good, 3.0.0 is released from the same commit. Whatever you try in RC1 is what 3.0.0 will be, unless you find something wrong. If you do, file an issue.
An RC is deliberately kept out of every channel that tracks “the latest release”. It is on Maven Central, it is a GitHub pre-release with archives, deb and rpm packages, and it has version-tagged container images. It is not on SDKMAN, it is not behind the btrace@btraceio jbang alias, and it is not on the Gradle Plugin Portal. The images have version tags only, so there is no latest yet: a plain docker pull ghcr.io/btraceio/btrace fails until 3.0.0 creates the tag. And sdk install btrace still gives you 2.2.6. Everything below that you can run against RC1 says 3.0.0-RC1.
New address
BTrace now lives in io.btrace: the sources, the Maven coordinates (io.btrace:btrace), the runtime API, all of it. The license is now Apache 2.0.
For 2.x scripts kept as source, the change is a mechanical rename, and a helper script does it in bulk with .bak backups.
For compiled 2.x probes sitting in a deployment, you do nothing. When the agent loads a persisted probe it scans it for org/openjdk/btrace/ references. Only if it finds any does it rewrite every class, method, field, descriptor and annotation reference to io/btrace/, in memory. The file on disk stays as it is. The same path also upgrades truly ancient probes compiled against com/sun/btrace, because apparently those still exist.
One jar
The three-jar layout is gone. btrace.jar is the agent, the client and the launcher at once. Its manifest declares io.btrace.boot.Loader as Premain-Class, Agent-Class and Main-Class.
The obvious way to get there would be to dump everything into one jar and put the whole jar on the boot class path. That would push about two thousand classes (ASM, the compiler, the instrumentation engine, the client) into the bootstrap classloader of every application BTrace touches. Namespace pollution, ASM version conflicts with the application, memory spent for nothing.
Instead, the jar uses classdata masking, a trick I borrowed from the dd-trace-java agent, which has shipped one jar with hidden class data for years. Only the small API surface (about 200 classes) is stored as regular .class files and is visible to the bootstrap loader. Everything else (about 1900 classes) is stored under META-INF/btrace/ with a .classdata extension. The JVM does not consider those to be classes at all, so the application never sees them. A tiny MaskedClassLoader loads them on demand when the agent or the client needs them. Same 3.8 MB jar, two very different views of its contents. The details, including why this was the thing that made jbang work at all, are in the Masked JAR Architecture document.
The practical upshot: jbang io.btrace:btrace:3.0.0-RC1 <pid> script.java works, and docker pull ghcr.io/btraceio/btrace:3.0.0-RC1 gets you an image (Debian, -alpine and -distroless variants, all on JDK 21) you can copy the distribution out of as a layer. Pin the version tag in anything you build more than once. The latest tags do not exist yet. They first appear with 3.0.0, and from then on they move with every release, including the next major one.
Stop writing files for one-line questions
The feature I use most, by a wide margin. BTrace 3.0 has oneliners, shamelessly inspired by DTrace:
btrace -n 'OrderService::* @return if duration>200ms { print method, time }' <PID>
The grammar is small. A class pattern, a method pattern, a location (@entry, @return, @error), an optional filter, and a list of actions (print, count, time, stack). Patterns take wildcards or /regex/. The filter can look at duration or at args[N].
# who is constructing exceptions, and from where
btrace -n 'java.lang.Exception::<init> @entry { print self, stack(5) }' <PID>
# how often is this cache hit
btrace -n 'com.myapp.Cache::get @entry { count }' <PID>
# the swallowed exception, with the call path
btrace -n 'OrderService::validateOrder @error { print method, stack }' <PID>
The <PID> does not have to be a number any more. The exact main class name works, as jps shows it, and so does '~<regex>' matched against the JVM’s command line, as in btrace '~OrderService' .... If the pattern matches more than one JVM, the client lists them and asks.
Underneath, a oneliner is an ordinary probe. The client turns it into a @BTrace class, compiles it, and deploys it like a file you would have written yourself. You can look at the generated source. The property has to reach the btrace client, not the application, and the launcher passes anything you append to the command line straight to the client’s argument parser, so set it through JAVA_TOOL_OPTIONS:
JAVA_TOOL_OPTIONS="-Dbtrace.oneliner.dump=true" btrace -n 'OrderService::processOrder @return { print method, time }' <PID>
A oneliner is a real probe with the training wheels on. It goes through the same verifier as everything else.
When you outgrow a oneliner, the next step is a script file, and that got shorter too. The flat DSL drops the BTraceUtils. prefix. The compiler injects import static io.btrace.BTrace.* and import io.btrace.core.annotations.* into any script that does not import the DSL itself. The minimal script is now literally this:
@BTrace
public class OrderTiming {
@OnMethod(clazz = "OrderService", method = "processOrder", location = @Location(Kind.RETURN))
public static void onReturn(@ProbeMethodName String method, @Duration long duration) {
println(method + " took " + str(duration / 1_000_000) + "ms");
}
}
No imports. Compare that to a 2.x script and you will understand why I kept putting off writing probes for small questions.
Extensions, and what they are allowed to do
The 2.x Aggregations API is gone, and the thing that replaced it is a good place to meet extensions. Here is a probe tracking latency percentiles with btrace-metrics, HdrHistogram under the hood:
@BTrace
public class LatencyHistogram {
@Injected(optional = true)
private static MetricsService metrics;
private static HistogramMetric chargeCardLatency;
@OnMethod(clazz = "OrderService", method = "chargeCard", location = @Location(Kind.RETURN))
public static void onChargeCardReturn(@Duration long durationNanos) {
if (chargeCardLatency == null) {
chargeCardLatency = metrics.histogramMillis("chargeCard.latency");
}
chargeCardLatency.record(durationNanos / 1_000_000);
}
@OnTimer(5000)
public static void report() {
HistogramSnapshot h = chargeCardLatency.snapshot();
println("chargeCard p50=" + h.p50() + "ms p95=" + h.p95() + "ms p99=" + h.p99() + "ms");
}
}
@Injected is the whole API surface from the script’s point of view: ask for a service, get a service. Deploy it and every five seconds you get a line of percentiles for the method you care about, computed by a real histogram library rather than by whatever you could squeeze through the verifier.
Now the uncomfortable part. Look at what that extension needs in order to work: HdrHistogram, a third-party library, loaded and running inside the JVM you are observing, plus the reflection and threading that come with it. In 2.x there was exactly one way to get code like that into a probe: the libs= profiles, which appended jars to the global classpath of the application and hoped nothing conflicted. Calling into such a jar also required unsafe mode, euphemistically called “trusted”, so the verifier was off too. A “safe” tracing tool mutating the application classpath and switching off its own safety net so that it can compute a percentile. I could never write that sentence with a straight face. The profiles are gone, and I do not miss them.
The replacement is the extension framework. An extension is a pair of jars. A small API jar (the MetricsService interface and its histogram types, nothing else) lands on the bootstrap classpath so probes can link against it. The implementation, HdrHistogram included, is loaded in isolation and bridged to the probe through an invokedynamic call site. The application never sees the implementation, and the implementation never sees the application’s classes.
Then permissions. Every extension declares what it needs, and the BTrace Gradle plugin computes most of that by statically scanning the implementation for JDK APIs that imply a permission. An extension cannot quietly forget to mention that it uses reflection or starts threads. btrace-metrics ends up declaring THREADS, REFLECTION and CLASSLOADER, all three privileged. A privileged extension does not link unless the operator of the target JVM allows it, as an agent argument or in a policy file:
java -javaagent:btrace.jar=allowExtensions=btrace-metrics MyApp
There is deliberately no client-side flag. The person deploying a probe does not get to grant themselves reflection or threads on a JVM they do not own. When an extension is blocked, the agent says so, and says why:
$ btrace -le <PID>
Failed Extensions:
1. btrace-metrics: Blocked privileged extension. Required=[THREADS, REFLECTION, CLASSLOADER]
This is where the optional = true in the probe above earns its keep. When the operator has not allowed the extension, the probe still deploys, and the field is bound to a stub that throws on every call. The failure is visible and local, not fatal. If you would rather have the calls do nothing, ask for a no-op shim instead: per field with @Injected(optional = true, mode = SHIM), or for every optional injection in the target JVM with -Dbtrace.extension.shimMode=shim. Without optional = true there is no fallback at all, and the deployment fails fast. The recorded demo of exactly this, the same probe deployed twice with only the agent’s policy changed in between, is tutorial 04.
Metrics is one of eight extensions that ship with 3.0. The others:
- StatsD export, which predates 3.0 and remains the proven way to push numbers at a real backend.
- An OpenTelemetry exporter, covered in a moment.
- Runtime contracts: latency budgets, call-rate limits and assertions checked on live calls.
- A handful of utilities.
- Three aimed at AI workloads: LLM call tracing (tokens, latency, estimated cost per model), RAG quality (vector store query latency and similarity scores), and a GPU bridge for ONNX Runtime, DJL and TensorFlow inference timing.
The AI ones exist because “which method is slow” has, for a lot of applications, turned into “which model call is slow and how many tokens did it burn”. The answer is produced the same way as everywhere else in this post: a probe on the client method, nothing added to the application, nothing left behind when the probe goes. A btracex CLI, with a small TUI, lists and inspects what is installed and edits the policy. Third-party extensions have an extension registry.
How small is an extension? The OTel exporter
To get a feel for how little an extension is, here is the one that started as a weekend experiment and ended up in the bundled set: btrace-otel. I wanted to know whether BTrace could speak OpenTelemetry. The API is one interface, OTel, with a handful of methods. The implementation is roughly six hundred lines. It keeps spans in a lock-free bounded queue, aggregates counters and gauges per name, and posts both to <endpoint>/v1/traces and <endpoint>/v1/metrics from a background thread once per flush interval.
@BTrace
public class CheckoutSpans {
@Injected
private static OTel otel;
@OnMethod(clazz = "com.example.OrderService", method = "checkout", location = @Location(Kind.RETURN))
public static void onCheckout(@Duration long durationNanos) {
otel.span("OrderService.checkout", durationNanos);
otel.counter("orders.checkout", 1);
}
}
Notice there is no start and no end call. BTrace already knows when the method returned and how long it took, so the span is built after the fact: the end timestamp is taken when span() is called, the start is end minus duration. On its own, each such span is the root of a fresh trace. To hang it off the application’s trace, spanInTrace(traceId, parentSpanId, ...) takes the ids, which a script can carry in @TLS fields after picking them off a traceparent header at the request boundary. newTraceId() and newSpanId() let one script build a small tree of its own. Counters go out as monotonic sums with delta temporality, the kind that adds up correctly on the receiving side. Gauges export the last value seen.
The extension runs inside the traced JVM, so it is configured there: script arguments first, then btrace.otel.* system properties on the application’s command line. Because it opens sockets, starts a thread and reads those properties, it declares NETWORK, THREADS and SYSTEM_PROPS, and goes through the same operator gate as everything above:
java -javaagent:btrace.jar=allowExtensions=btrace-otel \
-Dbtrace.otel.endpoint=http://otel-collector:4318 \
-Dbtrace.otel.service.name=order-service \
MyApp
What it does not do matters as much as what it does. There is no retry with backoff and no Retry-After handling on the transport. There are no compression or authentication headers, no OTEL_* environment variables, and no histograms or attribute sets on metrics. It does not read the application’s active OpenTelemetry context; if you want correlation, your script carries the ids. Every payload it produces is decoded in the test suite by the official OTLP protobuf parser, which rejects unknown fields and wrong types, but it has not yet been run against a real collector and a real backend. The honest summary is “BTrace can export spans and counters to an OTLP endpoint”, with a short, public list of what a production integration would still need. What the exercise did show is that the framework makes the boring part free. Isolation and permissions came with the first commit. A weekend extension gets exactly the same gate as the bundled ones, and once it earned its keep, the same tests.
One jar, part two: the fat agent
Extensions on disk are fine on a laptop. They are not fine on a Spark executor, a Hadoop node, or a distroless container. There is no $BTRACE_HOME there, no extensions/ directory, no shared filesystem, and nobody who wants to copy anything anywhere. Those environments can cope with exactly one thing: a -javaagent jar.
So 3.0 ships a Gradle plugin, io.btrace.fat-agent, that builds that one jar:
plugins {
id 'io.btrace.fat-agent' version '3.0.0'
}
btraceFatAgent {
baseName = 'demo-btrace-agent'
btraceVersion = '3.0.0'
embedExtensions {
file('btrace-metrics.zip') // an extension package
project(':my-custom-extension') // or a subproject in the same build
maven('com.acme:acme-ext:1.2') // or published Maven coordinates
}
}
./gradlew fatAgentJar stages the masked BTrace engine, then every extension you listed, and writes a jar whose manifest still says Premain-Class: io.btrace.boot.Loader. It attaches like plain btrace.jar.
One caveat for the RC. The io.btrace.fat-agent and io.btrace.extension plugin ids are published to the Gradle Plugin Portal only with the final release, so the build above resolves once 3.0.0 is out. With RC1 you can already use the result: the io.btrace:btrace:3.0.0-RC1 artifact on Maven Central is itself a fat agent, as described below.
How the extensions go in is the classdata masking trick again, applied one level up. An extension’s API classes, the interfaces a probe links against, are copied into the jar as regular .class files, so the bootstrap loader sees them. The implementation classes are renamed to .classdata and filed under the extension’s own directory, META-INF/btrace-extensions/btrace-metrics/io/btrace/metrics/MetricsServiceImpl.classdata for example. The JVM ignores them. Each extension gets its own small classloader that resolves class names only below that prefix.
That per-extension prefix is not decoration. It is what lets two extensions each ship a com.example.Helper without one silently running the other’s bytecode. Each embedded extension also gets a META-INF/btrace-extensions/<id>/extension.properties, and the manifest carries a BTrace-Embedded-Extensions list that the agent reads at startup instead of scanning a directory. Same isolation as an installed extension, with the directory folded into the jar.
Two details I like. First, the published io.btrace:btrace artifact is a fat agent. It is built by this same plugin and already embeds the eight bundled extensions, which is why jbang users get MetricsService without installing anything. Your own embedExtensions entries are deduplicated by id against that list, so the plugin is really for adding what the engine does not carry: your own extension, or a third-party one.
Second, embedding does not launder permissions. The plugin reads BTrace-Extension-Permissions from the extension’s API jar manifest, refuses to build if the attribute is missing rather than defaulting to “no permissions”, and copies it into the embedded metadata. At runtime the embedded extension goes through the same privileged gate as a filesystem one, so the deploy line still has to say what is allowed:
java -javaagent:demo-btrace-agent.jar=allowExtensions=btrace-metrics MyApp
You can also bake probes in. A bundledProbes { } block points at a directory of compiled probe classes (the output of btracec, in its package tree) and names the ones to embed by their binary name:
btraceFatAgent {
bundledProbes {
from layout.buildDirectory.dir('compiled-probes').get().asFile
include 'com.example.OrderStartupProbe'
}
}
The class ends up at META-INF/btrace-probes/com/example/OrderStartupProbe.class. An agent argument selects which embedded probe starts with the JVM:
java -javaagent:demo-btrace-agent.jar=probes=com.example.OrderStartupProbe,output=stdout MyApp
Probe names are exact binary names. A name that is not in the jar fails startup loudly rather than being ignored, which also keeps probes= from reaching outside the META-INF/btrace-probes/ namespace. For a Spark job that means one jar in spark.executor.extraJavaOptions, with the extension and the startup probe inside it, and nothing else to distribute. The full walkthrough, including what the staged jar looks like when you unzip -l it, is tutorial 08.
Attaching to production without sweating
The 2.x situation was this: a JVM started with -javaagent:btrace-agent.jar opened port 2020 and accepted a probe from anyone who could reach it. Dynamic attach was gated by the operating system (same user, same host). The prepared endpoint was gated by nothing.
In 3.0, prepared mode is an agent started at launch with no script, waiting for a client:
java -javaagent:/path/to/btrace.jar=port=0 MyApp
Three things happen. The agent binds a loopback address only. It generates an owner-protected token file. And it publishes the address, the port and the token file path as JVM properties, which a 3.0 client reads through the Attach API before it authenticates. The token never shows up as a process argument or a system property. port=0 lets the OS pick a free port, so there is nothing to collide with.
There is a bindAddress= option, but it exists to pick the loopback family, 127.0.0.1 or ::1. Give it a wildcard or a non-loopback address and the agent rejects it, logs one line naming the option, and starts without a command server while the application runs on. A 2.x client cannot talk to a 3.0 prepared endpoint. That is the point.
Ask the doctor first
The other half of this is btrace doctor:
$ btrace doctor <PID>
BTrace doctor for PID 89156
Status: READY (exit 0)
Target JDK: 17.0.12 (Oracle Corporation)
Attach API: available
Target attach: accessible
Dynamic agent loading: not tested (doctor never loads an agent)
BTrace endpoint: ready at 127.0.0.1:58360
Authentication: required; credential file readable
Actions: read target system properties and detached; no agent load or BTrace command connection was attempted.
Doctor reads the published properties through the Attach API and detaches. It never loads an agent and never opens the command socket, which makes it safe to run against anything, including the JVM you are not supposed to touch. The exit codes are stable (0 ready, 2 preparation required, 3 inaccessible), so a deployment script can gate on the result, and btrace doctor <PID> --json prints the same report as JSON for anything that would rather parse than grep. When the answer is “not ready”, doctor says what to change on the next deployment rather than leaving you to guess.
Budgets the client cannot raise
Authentication decides who may attach. It says nothing about how much a probe may cost once it is in. For that, 3.0 adds probe budgets, set by whoever launches the JVM:
java -javaagent:/path/to/btrace.jar=port=0,probeMaxRate=10000,probeTtl=600 MyApp
probeMaxRate caps how many times per second a probe’s handlers on application code may fire. probeTtl caps how long a probe stays installed. Both can also be set as system properties (-Dio.btrace.probeMaxRate, -Dio.btrace.probeTtl), and the smaller value wins. A client that attaches later cannot set or raise them. The agent checks once a second, and a probe over budget is switched off and removed. The client exits with code 5 and the reason:
BTrace: probe HotPathProbe removed: 23478274 hits/s exceeded probeMaxRate=10000
The same line lands in the target JVM’s log, which is the only record when the probe was running detached. One consequence worth spelling out: while a budget is set, trusted mode is refused. An unverified probe is arbitrary code, and arbitrary code is exactly what a budget cannot contain. The hands-on version of all of this, prepared mode, doctor, attach with no flags, and a probe getting evicted by its budget, is tutorial 12.
Leaving without a trace
Attaching is only half of the production story. The other half is what happens when you leave. Anyone who has run a Java agent in production has met the variant that cannot be removed. The bytecode stays patched, a stub keeps getting called on every invocation, and the only way back to the original performance is a restart, which is exactly what you attached an agent to avoid. For a tool meant to be attached for ten minutes and then forgotten, that is disqualifying.
Here is the whole cycle on a deliberately unfair target: a single thread calling one cheap method in a tight loop, reporting its own throughput every two seconds. Watch the number drop when the probe lands, and come back after the client exits.
Two mechanisms make the recovery work. Probe handlers in 3.0 are dispatched through invokedynamic call sites, and when the client exits those call sites become no-ops. At the same time the agent retransforms the instrumented classes back to their original bytecode. The first covers the window in which the old bytecode is still executing. The second removes it. The drop itself is the worst case on purpose: the probe handler is empty, so everything you see is call-site dispatch on a method invoked tens of millions of times per second. A real production method is called orders of magnitude less often. An agent you cannot remove is a liability. This one leaves.
The ASM ceiling
This one is for the people who, like me, have had to explain to someone why their shiny new JDK broke the tracing tool.
BTrace’s instrumentation engine is built on ASM. ASM only parses class files up to the major version it knows about and throws on anything newer. Every JDK release therefore meant: wait for ASM, bump ASM, re-validate the generated bytecode, release. Six months later, repeat.
BTrace 3.0 keeps ASM as the default backend, with a validated ceiling of class file version 71 (Java 27). That number is raised deliberately, after the instrumentation output has been checked on a new class file version, not automatically with every ASM dependency update. Anything above the ceiling is routed to a second backend built on the JDK’s own ClassFile API (java.lang.classfile, final since JDK 24). The JDK always understands the class file format it ships with, so this backend is forward-compatible by construction. Today that means JDK 28 early-access builds (class file version 72) get instrumented without BTrace knowing anything about them in advance.
The selection has three cases. Class file version 71 or below goes to ASM. Anything newer goes to the ClassFile API backend, which is compiled against Java 24 and loaded reflectively and lazily, the first time such a class arrives, so the rest of the agent stays Java 8 compatible and the jar is not even a multi-release jar. And if that load fails, the class stays uninstrumented and the attached client is told so, as [BTRACE WARN] Could not instrument <class> with the real cause. Handing the class to ASM instead would only have produced ASM’s “unsupported class file version” and hidden the reason.
All probe kinds and handler parameters are supported on the new backend, with one documented gap: @Duration on SYNC_ENTRY/SYNC_EXIT handlers. The backend document has the rest.
Wire protocol
Nobody will notice this one, which is the point. The client and the agent used to talk Java serialization. 3.0 defaults to a binary V2 protocol: 3-6x faster command transmission and 2-5x smaller payloads, per the protocol benchmarks.
Every connection starts with a negotiation. The receiving side looks for the BTR2 magic prefix and falls back to V1 if it is missing, so a 2.x client talks to a 3.0 agent and vice versa on dynamic attach, with no configuration. V1 itself is no longer the V1 you remember. Its payloads are encoded type by type instead of through ObjectInputStream, which closes the pre-authentication deserialization hole an unauthenticated peer on port 2020 used to have, and -Dbtrace.comm.allowV1=false turns the fallback off entirely where every peer is 3.0. The one wrinkle: a 3.0 client connecting to a 2.x agent waits for a V2 reply that never comes before falling back, which costs about 5 seconds on the first connect. If you know the agent is 2.x, -Dbtrace.comm.protocol=v1 on the client skips the wait.
V2 sessions also notice when the other side goes quiet. The client sends a heartbeat once the deploy is confirmed, and an agent that misses enough of them closes the connection. The probe keeps running and becomes detached, exactly as if the client had closed its socket, so btrace -lp finds it and btrace -r resumes it. TCP keepalive backs this up for V1 sessions, where no heartbeat exists.
Agentic investigations
I have written about jfr-mcp before, so the shape of this will not surprise anyone. BTrace has an MCP server, and it lives in the btraceio/agent-plugins repository as the btrace-observability plugin:
/plugin marketplace add btraceio/agent-plugins
/plugin install btrace-observability@btraceio-agent-plugins
The server runs through jbang and downloads the single io.btrace:btrace jar on first use, the masked jar earning its keep again. It exposes a deliberately short list of tools: list JVMs, deploy a oneliner or a script, read what a probe has printed, send an event, list probes, and detach, exit or stop a probe. Codex and Pi are covered by the same repository.
What makes this more than “an LLM with a shell” is the skill suite that ships with the server. There are eleven skills. One coordinates and the rest are specialists: endpoint diagnostics, latency analysis, failure analysis, oneliner triage, probe lifecycle, data safety, runtime access (SSH, Docker, Kubernetes, sidecars), startup and packaging (fat agents, distroless images), MCP operations, and a migration skill for the retired libs profiles. You do not pick them. You ask an incident question in operational language and the coordinator composes them.
A session, end to end
The repository ships this as demo 06. Like the other demos it has a run.sh, which starts the target and leaves it running for you to point your own AI host at. The recording below is different from the others in this post, though. It is not a scripted replay but an unedited Claude Code session, rendered from its transcript, which is committed next to the demo. A record-session.sh re-records one on your machine if you want to see what your model does with the same prompt.
The target is a checkout service. Four worker threads call checkout, which screens each order for fraud, reserves inventory and authorizes the payment. An order the screen flags goes to a manual review queue, which is a normal business outcome, and the service code is not wrong anywhere. The fraud screen and the payment client live in a vendor SDK that is shipped as a binary only, on purpose. The service source tells the assistant what checkout does. It cannot tell it what happens inside FraudScreen or PaymentClient. For that it has to probe the live JVM.
The session is one prompt, in the words an on-call engineer would use:
Customers report that checkout is sometimes slow, and a lot of orders are ending up in manual review, yet there is no fraud spike and nothing in the error logs. The checkout service is running on this machine and we cannot restart it. Its source is in this directory; payments and fraud screening come from a vendor SDK we only have as lib/acme-sdk.jar. Find the root cause of both problems, down to what happens inside the SDK, and back every claim with numbers from the running service. Keep probes narrow, and leave nothing attached when you are done.
The assistant lists the JVMs, reads the service source and the permissions file, and tries the shell, which is denied because the session has none. Its first deploy_script fails to compile: it wrote the probe against the 2.x package name. It fixes the imports and then deploys four probes, each removed before the next. First a timing probe on FraudScreen.screen and PaymentClient.authorize. Then a short entry trace of all com.acme.* methods, to discover the SDK’s internals. Then timing on those internals. Last, a probe on RiskServiceException construction together with refreshSession starts and slow authorize calls, with thread names.
Its conclusion, with the numbers the probes printed rather than numbers it imagined: refreshSession takes 311 to 453 ms about once a second while holding a lock, and the authorize calls of all four workers stall together, finishing within about 10 ms of each other, one per checkout-0 to checkout-3. On the other problem, deepScore throws RiskServiceException and screen turns it into a hold verdict. In one window, 14 throws matched 14 holds one to one, and the throws were as fast as the successes (10 to 32 ms against 10 to 33 ms), so the “timeout” in the exception message is not a long call. It states what it inferred rather than measured (the shared lock, from the pattern; it had no lock-wait stacks), reports every probe removed, and confirms with list_probes. Twenty turns, about a minute of session time, both planted defects found inside a library with no source.
The wording, the probe shapes and the number of tool calls differ from run to run. That is why the demo ships with a rehearsed prompt (PROMPTS.md) and a “do not record if” list. That list is the interesting document. It is a description of what a bad investigation looks like: widening the scope to *::* when a class was enough, leaving a probe attached at the end, detaching where it should exit, printing argument values, stating a cause as fact that no probe showed, or answering from general knowledge without ever calling a tool.
Guardrails
The part I consider important is that the assistant gets fewer privileges than you have, not more. The MCP tools take a PID, a probe and a port, and nothing else. There is no parameter that turns the verifier off, so the assistant cannot put a JVM into trusted mode even if it wanted to. Privileged extensions still need the operator’s policy on the agent side, and probe budgets, if the operator set any, apply to the assistant’s probes like anyone else’s. The data-safety skill defaults to method names, durations and exception types rather than request bodies, tokens or customer data. The scope rules are explicit: an exact method in production, a bounded class or package for discovery, and a broad match only as a time-bounded escalation with a stated event rate and a rollback plan. Every deployment plan the plugin produces names a target, an output destination, an observation window and a stop condition.
Read the tool calls, not just the answer. The one thing worth saying no to is an assistant that suggests you run btrace -u by hand.
Pairing it with JFR
The division of labor is the one I described in the jfr-mcp post. JFR is history: it tells you which resource, which interval and which candidate hotspot. BTrace is the present: it confirms, narrowly and live, whether that candidate is still doing what the recording suggested. When the jfr-analyzer plugin from the same repository is installed, a jfr-btrace-interop skill carries the target identity, the timestamps and the hypothesis from one workflow to the other. The rule at the end is the same as everywhere else in this post: the probe comes out when the observation window ends.
What breaks
A major version is allowed to break things, and this one does. The complete list is in the migration guide; the short version:
| What you have | What happens |
|---|---|
| Compiled 2.x probes | Nothing. Rewritten in memory at load time. |
| 2.x script sources | Rename org.openjdk.btrace to io.btrace, or delete the imports entirely. |
btrace-agent.jar / -boot / -client in launch scripts |
Point them at the single btrace.jar. |
libs=<profile> |
Removed. The agent ignores the argument like any other unknown one and loads nothing. Migrate to an extension. |
Aggregations API |
Removed. Use btrace-metrics for histograms and stats, plain maps and counters for the rest. |
| 2.x client against a 3.0 prepared agent | Rejected. Prepared mode is authenticated now; upgrade the client. |
| Targets on Java 8-16 | Still work for all of 3.x, with a one-time deprecation warning. Gone in 4.0. |
That last row deserves a sentence. BTrace 3.0 runs against Java 8 through 27 and beyond, but I am not going to pretend that testing every probe kind against everything from Java 8, which shipped in 2014, up to Java 16 is a good use of anyone’s time in 2026. If the warning bothers you, -Dbtrace.suppressJavaDeprecationWarning=true turns it off.
Try it
# jbang, no installation (straight from Maven Central)
jbang io.btrace:btrace:3.0.0-RC1 -n 'com.myapp.*::* @return if duration>100ms { print method, duration }' '~myapp'
# or any of these
docker pull ghcr.io/btraceio/btrace:3.0.0-RC1
curl -LO https://github.com/btraceio/btrace/releases/download/v3.0.0-RC1/btrace-v3.0.0-RC1-bin.tar.gz
The usual shortcuts, the btrace@btraceio jbang alias and sdk install btrace, switch to 3.0.0 when it is released. Until then they stay on 2.x, which is why the lines above spell out the version.
The repository ships a set of replayable demos, each with a run.sh, and a tutorial series that walks the same ground in ten-minute pieces. The recordings in this post come straight from those demos.
BTrace has been asking the same question for well over fifteen years: what is this JVM doing, right now, without me restarting it? 3.0 does not change the question. It just stops making you work so hard to ask it.
PS: how this got done
A word on how a one-person, evenings-and-weekends project shipped this much in one release. Most of what is above started as an idea I would normally have parked for lack of time: a second instrumentation backend, a masked jar, an authenticated prepared mode, a DSL that compiles to probes. Working with coding agents changed the economics. I could try a crazy idea in an afternoon, throw it away if it did not hold up, and when it did, have the agent write the tests, the golden files, the migration script, the tutorials with their replayable demos, and the misuse corpus that now runs on every push. That last part is the one that matters. Those are the checks I would never have had the evenings to write by hand, and 3.0 has far more of them than any release before it. The ideas are still mine, and so are the mistakes. The verification is what I could not have afforded before.
BTrace is developed at btraceio/btrace under the Apache 2.0 license, and has a project site at btrace.io/btrace. The 3.0.0-RC1 release notes are here.