One Recording, Many Silos

A JFR recording from a busy service is absurdly rich. CPU samples, allocation samples, GC pauses, exceptions, thread states, endpoint spans, lock contention events. All timestamped, all in the same file, all covering the same two minutes of life.

And yet, almost everyone analyzes them one event type at a time.

You look at CPU samples. You see hot methods. You look at allocations. You see big objects. You look at exceptions. You see error counts. Each view is correct, but each is also incomplete. The interesting questions are the ones that span two views at once. Which endpoint is responsible for GC pressure? Are the exceptions actually correlated with CPU waste, or are they harmless? Do threads allocate memory while they are parked and waiting?

These are not exotic questions. They are the first things a performance engineer wants to know when triaging a production incident. The data to answer them is right there in the recording. The problem has always been connecting it.

The Hard Way

Suppose you have a recording with 44,000 CPU samples and 438,000 endpoint span events. You want to know how much CPU each endpoint consumes.

Both event types carry a localRootSpanId field that identifies the distributed trace they belong to. In theory, you just join them on that field. In practice, that means:

1. Export all execution samples with their span IDs
2. Export all endpoint events with their span IDs and endpoint names
3. Load both into a script or a spreadsheet
4. Join on span ID
5. Group by endpoint name
6. Count

It works. It’s also tedious enough that nobody does it during an incident at 3am. You end up eyeballing thread names in the flamegraph and hoping they correlate with something useful.

If you want to cross allocations with exceptions on the same trace, that’s another export, another join, another script. Each question is a small project. So in practice, people don’t ask these questions. They stick to single-event-type queries and miss the connections.

decorateBy

jfr-shell has an operator called decorateBy that does this join inline, as part of a query pipeline. You tell it which event type to use as context, which field to join on, and which fields to pull from it. The result is the original events enriched with fields from the second type, accessible as $decorator.* in downstream operators like groupBy or select.

There are two flavors. decorateByTime matches events that overlap in time on the same thread, useful for things like “which lock was this thread waiting on when this sample was taken.” decorateByKey matches events that share a value in a specified field, useful for any kind of ID-based correlation: trace IDs, span IDs, request IDs, GC cycle IDs.

The query for “CPU budget per endpoint” looks like this:

events/datadog.ExecutionSample
  | decorateByKey(datadog.Endpoint,
      key=localRootSpanId, decoratorKey=localRootSpanId,
      fields=endpoint)
  | groupBy($decorator.endpoint)

One query. No exports, no scripts, no spreadsheets. Takes a few seconds to run. And because it’s just another pipeline operator, you can chain it with everything else: filters, aggregations, top-N, sorting.

A Recording to Play With

I had a recording from a gRPC-based profile analysis service. Two minutes, 16 megabytes. Here’s what was in it:

• 44,454 CPU execution samples (Datadog async profiler)
• 13,492 method samples with trace correlation
• 4,186 exception samples
• 820 allocation samples
• 438,041 endpoint span events
• 116 GC pauses, including one full GC at 3.6 seconds
• 8 cpu-intensive worker threads, gRPC/Netty I/O threads, a Kafka consumer

The Datadog profiler puts a localRootSpanId on most of its events. This is the distributed trace root. When an execution sample, an allocation sample, and an endpoint event share the same localRootSpanId, they all belong to the same request. That is the join key for everything that follows.

Five Queries

1. Where does the CPU actually go?

events/datadog.ExecutionSample
  | decorateByKey(datadog.Endpoint,
      key=localRootSpanId, decoratorKey=localRootSpanId,
      fields=endpoint)
  | groupBy($decorator.endpoint)

| endpoint                                 | count |
+------------------------------------------+-------+
| (untraced)                               | 23578 |
| com.datadoghq.profiling.analyzer.Analyz… | 19270 |
| processProfileRecord                     |  1418 |
| libstreamingFetch                        |   153 |
| SubmitSeries                             |    35 |

53% of CPU has no trace context at all. That’s GC threads, Kafka housekeeping, JIT compilation, internal bookkeeping. The main gRPC Analyze endpoint takes 43%. And there’s a processProfileRecord step eating 3.2% that you’d never notice in a flat flamegraph because it gets mixed in with everything else.

Before this query, you have 44,000 undifferentiated samples. After it, you have an endpoint-level CPU budget.

2. Which endpoint is driving GC?

events/datadog.ObjectSample
  | decorateByKey(datadog.Endpoint,
      key=localRootSpanId, decoratorKey=localRootSpanId,
      fields=endpoint)
  | groupBy($decorator.endpoint, agg=sum, value=weight)

| endpoint                                 | allocation weight |
+------------------------------------------+-------------------|
| com.datadoghq.profiling.analyzer.Analyz… |      1,237,346 KB |
| (untraced)                               |        161,972 KB |
| processProfileRecord                     |         94,296 KB |
| libstreamingFetch                        |          2,396 KB |

Same idea, different signal. The gRPC Analyze endpoint is responsible for 83% of all allocation weight: about 1.2 GB in two minutes. That’s almost certainly what’s feeding the 116 GC pauses and the 3.6-second full GC.

When you see 116 GC pauses in a summary, the natural question is “caused by what?” Allocation profiling alone tells you which object types are hot, but not which business operation produced them. This one query connects the two.

3. Wall-clock profile by trace operation

events/datadog.MethodSample
  | decorateByKey(datadog.Endpoint,
      key=localRootSpanId, decoratorKey=localRootSpanId,
      fields=operation)
  | groupBy($decorator.operation)

| operation            | count |
+----------------------+-------+
| grpc.server          | 10686 |
| processProfileRecord |  1017 |
| (unmatched)          |  1697 |
| libstreamingFetch    |    82 |
| SubmitSeries         |    10 |

MethodSample events are wall-clock samples. They capture what threads are doing regardless of whether they are on CPU or waiting. That makes this a different lens than query #1: it includes time threads spend parked, blocked, or sleeping. Decorating them with the endpoint’s operation field shows that grpc.server dominates at 79% of traced wall-clock time. That’s surprising, because the heavy lifting in this service is processProfileRecord, the step that actually analyzes incoming profiles. It only accounts for 7.5% here. So the question becomes: what’s the gRPC layer doing with all that time? Is it serialization overhead, TLS, frame handling? The wall-clock view raises questions that a CPU-only profile would not.

4. How much CPU is spent on failing traces?

This one is my favorite.

events/datadog.ExecutionSample
  | decorateByKey(datadog.ExceptionSample,
      key=localRootSpanId, decoratorKey=localRootSpanId,
      fields=sampled)
  | groupBy($decorator.sampled)

| error-tainted? | count |
+----------------+-------+
| true           | 22529 |
| (clean/none)   | 21925 |

This doesn’t correlate CPU with endpoints. It correlates CPU with exceptions. Every CPU sample whose trace also threw at least one exception gets tagged. Every sample from a clean trace, or from background work with no trace at all, falls into the other bucket.

The result: 50.7% of all CPU is spent on traces that also threw at least one exception.

Now, a caveat. The profiler tracks all exceptions, caught and uncaught alike. A try/catch around a socket timeout still produces an exception sample. So “error-tainted” doesn’t necessarily mean “failing.” Some of those traces might be perfectly healthy code paths that just happen to use exceptions for control flow (looking at you, Java I/O).

Still, 50% is a striking number. It tells you that half your compute shares traces with exception activity. Whether that is harmless or a real problem, you now know where to look. Filter those traces by exception type and you will quickly separate the expected EOFException noise from the real errors.

5. Heap allocations by wall-clock thread state

events/datadog.ObjectSample
  | decorateByKey(datadog.MethodSample,
      key=localRootSpanId, decoratorKey=localRootSpanId,
      fields=state)
  | groupBy($decorator.state, agg=sum, value=weight)

| thread state | allocation weight |
+--------------+-------------------|
| RUNNABLE     |      1,046,390 KB |
| (unmatched)  |        261,461 KB |
| PARKED       |        188,159 KB |

This joins two completely different profiling signals: heap allocation samples and wall-clock thread state samples, correlated by trace. For each allocation, it grabs a wall-clock sample from the same trace and pulls the thread state at that moment.

A word of caution: this is a loose correlation. decorateByKey picks one wall-clock sample from the matching trace. It doesn’t tell you what the thread was doing at the exact instant of the allocation. What it does tell you is whether a given trace was predominantly active or idle during the recording window.

With that in mind: 1 GB of heap allocations belong to traces that also show RUNNABLE wall-clock activity. No surprise there. But 188 MB belong to traces where the sampled state was PARKED. That doesn’t prove the thread was parked when it allocated, but it flags those traces as worth a closer look. Why is a trace that spends time parked also producing meaningful heap pressure?

Why This Matters

Every one of these queries takes two things that JFR already records and asks a question that neither can answer alone. CPU samples don’t know which endpoint they serve. Allocation samples don’t know if their trace threw an exception. Exception counts don’t know how much CPU the failing traces consumed.

The data was always there. The recording has always contained all of it. What was missing was a way to connect the signals without leaving the query language and reaching for external scripts.

decorateBy is that connection. And it’s not limited to Datadog-specific events. Standard JDK events work the same way. Decorate execution samples with jdk.JavaMonitorEnter by time overlap to find CPU under lock contention. Decorate allocation samples with GC phase events to find which allocations happen during collection. Decorate file reads with endpoint spans to attribute I/O to business operations. Any two event types in the same recording can be joined, as long as they share a time range or a key field.

The recording is always richer than a single-event-type query can show. Most of the interesting performance stories live in the space between two signals.

Sidebar: the bug that almost ate this post. When I first ran these queries, every single one returned zero matches. It turned out there was a path-navigation bug in DecoratedEventMap: the $decorator.field path was being split into two steps ("$decorator" then "field"), but the map only recognized the full string "$decorator.field" as a single key. A four-line fix made the bare "$decorator" lookup return the decorator map directly, and everything started working. Lesson: when your feature silently returns empty results for every input, it’s probably not a data problem.

jfr-shell is part of JAFAR and available via JBang and Maven Central.

The Flamegraph Lie

Flamegraphs are the standard answer to “where is the CPU going?” You get a beautiful tree of call stacks, widths proportional to sample counts, and you can visually spot the methods that dominate. It is a deservedly popular visualization.

But a flamegraph aggregates everything. Two minutes of recording become one static picture. A method that burned 10% of CPU for two straight minutes looks exactly the same as a method that burned 100% of CPU for twelve seconds and was idle for the rest. Both show the same width in the graph. Both report the same sample count.

This matters more than you might think. The first case is a steady load, probably a core loop doing its job. The second is a burst, maybe a periodic batch, a cache rebuild, a sudden queue drain. The diagnosis is different, the fix is different, and the urgency is different. But in a flamegraph, they are the same rectangle.

What You Actually Want

When you find a hot method in a profile, the next questions are almost always temporal:

• Was it hot the entire time, or only during a spike?
• Is it one thread doing all the work, or is it spread across many?
• Is this a steady-state cost I should optimize, or a transient burst I should investigate?

None of these questions can be answered by a tool that throws away the time axis.

stackprofile

jfr-shell now has a stackprofile pipeline operator that keeps the time axis. Instead of collapsing all samples into a single tree, it builds a weighted call tree where every node carries two extra pieces of information: a time-bucket distribution showing how samples spread over the recording duration, and a per-thread breakdown showing which threads contributed.

The syntax is what you would expect from a pipeline operator:

events/jdk.ExecutionSample | stackprofile()

It accepts a few optional parameters:

A bottom-up view with finer time resolution:

events/jdk.ExecutionSample | stackprofile(direction=bottom-up, buckets=20, minPct=0.5)

Reading the Output

Each row in the result is a frame in the call tree, annotated with:

• total: samples in this frame or any of its children
• self: samples where this frame is the leaf (the actual work)
• time buckets: a sparkline showing sample distribution across the recording
• threads: per-thread sample counts with percentages

The operator also classifies each frame into a category:

That last category is the interesting one. A steady hotspot is a method that consumes significant CPU at a constant rate for the entire recording. It is not reacting to a spike, it is not processing a batch. It is always there, always burning. These are classic N+1 query candidates, polling loops, or inefficient serialization paths. They are the kind of thing that flamegraphs make look “normal” because they blend into the background.

Bursty vs Steady: A Concrete Difference

Consider two methods that each account for 5% of total CPU in a two-minute recording:

Method A — steady hotspot:

time buckets: ▃▃▃▃▃▃▃▃▃▃  (uniform across all 10 buckets)
threads:      worker-1 (40%), worker-2 (35%), worker-3 (25%)
category:     ◆◆ steady-hotspot

This method is always running, spread across multiple worker threads. It is a baseline cost. If you want to reduce CPU, this is a method worth optimizing because the savings compound across the entire recording.

Method B — bursty hotspot:

time buckets: ▁▁▁▁▁▁▁█▁▁  (concentrated in bucket 8)
threads:      batch-processor-1 (100%)
category:     ◆ hotspot

Same percentage, completely different story. This method fired once, in a single burst, on a single thread. It is probably a scheduled task, a cache refresh, a one-time computation. Optimizing it would save nothing during normal operation. The right response might be to move it off the hot path or run it at a less contended time.

A flamegraph cannot tell these apart. stackprofile can, because it kept the time axis.

Flamegraph vs stackprofile

The two operators complement each other. Neither replaces the other.

Use flamegraph when you need to understand call paths and caller-callee relationships. Use stackprofile when you need to understand temporal behavior and thread affinity.

Prior Art and How stackprofile Differs

The idea of adding a time axis to profiling data is not new. Several tools have tried different approaches:

• Flame Charts (Chrome DevTools, Polar Signals) preserve temporal ordering by not merging stacks at all. Every individual call is shown in time sequence. This is faithful to the recording, but for a two-minute JFR with tens of thousands of samples, the result is too noisy to navigate. You get time, but you lose the ability to spot patterns.

• Differential Flame Graphs (Brendan Gregg) compare two profiles, A and B, coloring frames red or blue based on the delta. Great for before/after comparisons, but they only answer “did this method get hotter or colder between these two snapshots?” They cannot show a continuous temporal distribution within a single recording.

• dotTrace Timeline Profiling (JetBrains) gives a full thread-level timeline with call stacks at each point. Powerful, but .NET-only, and it is a GUI tool rather than a composable pipeline operator.

• binjr can browse JFR method profiling events as time series and show histogram density distributions. It gives you the temporal view, but it works at the individual event level rather than aggregating into a call tree.

stackprofile takes a different path. It merges stacks like a flamegraph (keeping the output compact and navigable) but annotates every node in the tree with a bucketed time distribution and per-thread breakdown. You get the structural clarity of a merged call tree plus the temporal insight of a flame chart, without the noise of either extreme.

For AI Agents Too

stackprofile is also exposed as an MCP tool (jfr_stackprofile) through jfr-mcp. The MCP version returns structured JSON with numeric fields for every frame: totalPct, selfPct, pattern, category, timeBuckets array, and threadCounts map. This makes it straightforward for an AI agent to programmatically identify bursty hotspots, detect N+1 patterns, or compare thread affinity across frames without parsing sparklines.

claude mcp add jafar -- jbang jfr-mcp@btraceio --stdio

Then ask Claude to investigate temporal CPU patterns, and it will use jfr_stackprofile alongside jfr_flamegraph and jfr_hotmethods to build a more complete picture.

Where It Shines

stackprofile works in the plain CLI, but its full potential shows in two places: the TUI and MCP.

In TUI mode, the results are color-coded by category. Normal frames stay neutral, hotspots get highlighted, and steady-hotspots are immediately visible. You can jump directly to the next hotspot without scrolling through hundreds of normal frames. When a recording has thousands of frames, that navigation alone saves real time.

Through MCP, an AI agent gets structured JSON with numeric fields it can reason over programmatically. It can scan for steady-hotspots, compare thread affinity across frames, and detect N+1 patterns without parsing visual output. The combination of jfr_stackprofile with jfr_flamegraph and jfr_hotmethods gives the agent three complementary views of the same CPU data.

The plain CLI gives you the data. The TUI makes it navigable. MCP makes it queryable by machines.

Try It

# Install via JBang, open a recording
jbang jfr-shell@btraceio -f recording.jfr --tui

# Top-down call tree with temporal bucketing
jfr> events/jdk.ExecutionSample | stackprofile()

# Bottom-up hot methods with 20 time buckets
jfr> events/jdk.ExecutionSample | stackprofile(direction=bottom-up, buckets=20)

# Only frames above 2% of total
jfr> events/jdk.ExecutionSample | stackprofile(minPct=2.0)

The next time you find a hot method in a flamegraph, ask yourself: was it hot the entire time, or just for a moment? stackprofile will tell you.

jfr-shell is part of JAFAR and available via JBang and Maven Central.

The Pitch You Didn’t Ask For

A JFR recording is one of the richest artifacts a JVM can produce. CPU samples, GC pauses, memory allocations, thread states, lock contention, I/O latency, class loading, JIT compilations, exceptions — all timestamped, all in one file. The problem was never the data. The problem is that a 200MB recording from a production JVM that melted down at 3am contains all of it at once, and the human staring at it has to know which questions to ask, in what order, and how to connect the answers.

Or you could let an AI do that part.

Not “let an AI hallucinate a diagnosis from the filename.” Actually let it open the recording, query events, correlate thread states with lock contention, extract resource utilization metrics, run systematic performance methodologies, and explain what it found — with evidence.

That’s what jfr-mcp does. It’s a Model Context Protocol server that exposes the full depth of JFR analysis as a set of tools that any MCP-capable AI agent (Claude, for instance) can call directly. No copy-pasting terminal output. No narrowing down to a single event type and hoping it’s the right one. The AI gets the whole recording, the whole query language, and established performance analysis frameworks to work with.

What MCP Actually Is (30-Second Version)

MCP is a protocol that lets AI agents call external tools in a structured way. The agent sees a catalog of available tools, each with a name, description, and parameter schema. When the agent decides it needs to, say, open a JFR file, it emits a tool call with the right parameters, the MCP server executes it, and the result flows back as structured data.

Think of it as giving the AI a CLI it can actually use, except it’s JSON over stdio instead of bash over a terminal.

JBang: Zero-Install Java Distribution

Before we get into the MCP server itself, a word about how it’s distributed.

jfr-mcp is packaged as a fat JAR — single file, all dependencies baked in. But “download a JAR, find the right Java, set the classpath, run it” is the kind of ceremony that kills adoption. So it’s published to a JBang catalog instead.

JBang is a launcher that resolves, caches, and runs Java applications with zero setup. It downloads the right JDK if you don’t have one, fetches the artifact from Maven Central, caches it locally, and runs it. First invocation takes 10–30 seconds (it’s pulling ~70MB of JDK plus the JAR). Every subsequent run starts in under 2 seconds from cache.

If you don’t have JBang yet:

# macOS
brew install jbangdev/tap/jbang

# Linux / macOS (without Homebrew)
curl -Ls https://sh.jbang.dev | bash -s - app setup

# Windows
scoop install jbang
# or: choco install jbang
# or: iex "& { $(iwr https://ps.jbang.dev) } app setup"

# SDKMAN (any platform)
sdk install jbang

Or skip all that — the one-liner install script handles JBang installation too:

curl -Ls https://raw.githubusercontent.com/btraceio/jafar/main/jfr-mcp/install.sh | bash

This installs JBang if needed, then installs the jfr-mcp command. One script, no prerequisites.

The catalog alias is jfr-mcp@btraceio for stable releases and jfr-mcp-dev@btraceio for development snapshots. You can run directly without installing:

jbang jfr-mcp@btraceio          # run without installing
jbang app install jfr-mcp@btraceio   # install as a command, then:
jfr-mcp                              # run by name

To update, force a reinstall:

jbang app install --force jfr-mcp@btraceio

For dev snapshots where Maven Central caching may serve stale versions:

jbang --fresh jfr-mcp-dev@btraceio

Two Transports: STDIO and HTTP

The MCP server speaks two protocols. Which one you want depends on who’s talking to it.

STDIO Mode

jfr-mcp --stdio

In STDIO mode, the server reads JSON-RPC messages from stdin and writes responses to stdout. No network port, no HTTP, no SSE. The server runs as a subprocess of whatever launched it — which is exactly how Claude Desktop and Claude Code expect to talk to MCP servers.

This is the mode you want for AI integration. The client (Claude) spawns the server process, sends tool calls over stdin, reads results from stdout, and kills the process when the conversation ends. Each invocation is a fresh instance. No port conflicts, no “is the server already running” questions.

HTTP/SSE Mode

jfr-mcp                          # default: port 3000
jfr-mcp -Dmcp.port=8080          # custom port

In HTTP mode, the server starts a Jetty instance exposing two endpoints:

• /mcp/sse — Server-Sent Events stream (establishes a session)
• /mcp/message — JSON-RPC message endpoint

This is for web-based MCP clients, custom integrations, or manual testing with curl. The server auto-detects if the port is already in use and exits silently — so you won’t accidentally start two instances fighting over port 3000.

For most people reading this post, you want --stdio.

Wiring It to Claude

Claude Code

One command:

claude mcp add jafar -- jbang jfr-mcp@btraceio --stdio

That registers an MCP server named jafar that Claude Code will start using JBang in STDIO mode. Next time you start a conversation, Claude has 13 new JFR analysis tools available. No restart needed.

Claude Desktop

Edit ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "jafar": {
      "command": "jbang",
      "args": ["jfr-mcp@btraceio", "--stdio"]
    }
  }
}

Restart the app. The hammer icon in the input area should now list the jfr tools.

For development snapshots, swap the alias:

{
  "mcpServers": {
    "jafar": {
      "command": "jbang",
      "args": ["--fresh", "jfr-mcp-dev@btraceio", "--stdio"]
    }
  }
}

The --fresh flag tells JBang to bypass its cache and fetch the latest snapshot — useful when you’re testing changes that were published minutes ago.

Building from Source

If you prefer to skip JBang entirely:

cd jafar
./gradlew :jfr-mcp:shadowJar
java -jar jfr-mcp/build/libs/jfr-mcp-*-all.jar --stdio

Requires Java 25+. The Claude Desktop config for a manual JAR looks like:

{
  "mcpServers": {
    "jafar": {
      "command": "java",
      "args": ["-jar", "/path/to/jfr-mcp-0.15.0-all.jar", "--stdio"]
    }
  }
}

Why JFR Deserves Better Tooling

Most people who touch JFR recordings end up tunneling in on one event type. Open the file, look at CPU samples, maybe check GC pauses, close the file. That’s like reading the first chapter of a detective novel and declaring the butler did it.

A JFR recording is a correlated, timestamped snapshot of everything the JVM was doing. CPU and native method samples. GC phases and heap summaries. Thread lifecycle events. Monitor contention. File and socket I/O. Object allocations. Class loading. JIT compilation phases. Exception throws. And the power of JFR is that all of these events share a timeline — you can correlate a GC pause with the thread states that were blocked during it, or trace an allocation spike to the exact call path that triggered it.

The MCP server is designed to treat the recording holistically. It doesn’t just expose “query this event type.” It exposes systematic analysis frameworks that cross-cut multiple event types and produce a coherent picture.

The Toolbox

The server exposes 13 tools organized in two layers.

Core Tools: Opening, Querying, Learning

jfr_query accepts the same JfrPath expressions that jfr-shell uses:

events/jdk.ExecutionSample | groupBy(eventThread.javaName) | top(10)
events/jdk.GCPhasePause | count()
events/jdk.FileRead[bytes>1000] | groupBy(path)

The AI doesn’t need to know JfrPath in advance. It calls jfr_help first, reads the syntax reference, and constructs queries from there. This is one of those details that sounds minor but matters a lot in practice — the agent is self-teaching, not hardcoded.

Analysis Tools: Methodologies, Not Just Queries

This is where the holistic approach lives. These tools don’t just run a single query — they correlate across event types and apply structured analysis frameworks.

jfr_diagnose is the entry point for the “I don’t know what’s wrong” case. The server inspects the recording — is there high exception volume? GC pressure? CPU saturation? Thread contention? — and runs the relevant analyses automatically, composing jfr_use, jfr_tsa, and other tools as needed. It’s the tool an AI reaches for first when you say “something is slow and I don’t know why.”

What a Conversation Looks Like

Here’s a realistic exchange. You drop a JFR file in front of Claude and ask what’s wrong:

You: Open /tmp/checkout-service.jfr and tell me why latency spiked.

Behind the scenes, Claude works through the recording methodically:

1. jfr_open — opens the recording, gets session ID and timestamp range
2. jfr_use — runs USE Method analysis. CPU utilization is 92%, thread saturation is high (long monitor waits), memory shows frequent GC pauses. Now it knows which resources are stressed.
3. jfr_tsa — runs Thread State Analysis. 40% of checkout-handler threads are BLOCKED, not RUNNABLE. The bottleneck isn’t just CPU — threads are waiting on something. TSA correlates this with jdk.JavaMonitorEnter events and identifies InventoryCache.lock as the contention point.
4. jfr_hotmethods — of the threads that are running, com.example.cart.PriceCalculator.recomputeTotals dominates CPU samples
5. jfr_query — targeted follow-up: events/jdk.JavaMonitorEnter[monitorClass='InventoryCache'] | top(5, by=duration) confirms the lock is held for 200ms+ during cache refresh

Claude then synthesizes: USE analysis shows CPU saturation and thread contention. TSA identifies that checkout threads spend 40% of their time BLOCKED on InventoryCache.lock. The lock holder is PriceCalculator.recomputeTotals, which is also the CPU hotspot. The latency spike correlates with cache refresh intervals. Recommendation: decouple the cache refresh from the read path.

Notice the structure: USE first to identify which resources are stressed, TSA to understand how threads are affected, then targeted queries to confirm the specific mechanism. Each finding cross-references a different slice of the same recording. That’s the holistic approach — not “look at CPU samples” but “systematically check every resource class, then drill into the ones that hurt.”

The Self-Teaching Trick

When you first connect the MCP server, the AI has never seen JfrPath. It doesn’t need to have. The conversation typically goes:

1. AI calls jfr_help with topic overview
2. Reads the syntax reference (event paths, filters, pipeline operators, aggregation functions)
3. Starts constructing queries

This is possible because jfr_help returns comprehensive documentation broken into topics: overview, filters, pipeline, functions, examples, event_types. The AI reads what it needs, when it needs it.

It won’t always write a perfect query on the first try. But the error messages from jfr_query are informative enough that the AI can self-correct — adjust a field name, fix a filter syntax, try a different aggregation. It’s the same loop a human goes through, just compressed into a few hundred milliseconds of API calls.

USE and TSA: Standing on the Shoulders of Giants

The most important design decision in jfr-mcp is not which JFR events it can parse. It’s that the analysis tools implement real performance engineering methodologies rather than ad-hoc “let me grep for big numbers.”

The USE Method

Brendan Gregg’s USE Method is a systematic approach to resource analysis: for every resource in the system, check three things — Utilization, Saturation, and Errors. It’s simple, exhaustive, and it prevents the common trap of fixating on one metric while ignoring the actual bottleneck.

jfr_use applies this to four resource classes extracted from the JFR recording:

A single jfr_use call cross-references all of these event types and returns a structured report with actionable insights. When an AI runs this as a first step, it immediately knows which resource classes are healthy and which need investigation — before looking at a single stack trace.

Thread State Analysis

Gregg’s TSA Method asks a different question: instead of “which resource is stressed,” it asks “what are threads actually doing with their time?” A thread is either running on-CPU, waiting for I/O, waiting for a lock, sleeping, or parked. The distribution across these states tells you whether your application is CPU-bound, I/O-bound, lock-bound, or just idle.

jfr_tsa groups execution samples by thread state — RUNNABLE, WAITING, TIMED_WAITING, BLOCKED — and breaks down the distribution globally and per-thread. When it finds threads spending significant time in BLOCKED state, it correlates with jdk.JavaMonitorEnter and jdk.JavaMonitorWait events to identify which locks are responsible and who is holding them.

This is the kind of analysis that, done manually, requires querying three or four different event types and cross-referencing timestamps. The MCP server does it in one call and returns structured results the AI can reason about.

Why This Matters for AI-Driven Analysis

An AI without methodology is a pattern matcher looking for big numbers. An AI with methodology has a playbook: run USE to classify resource bottlenecks, run TSA to understand thread behavior, then drill into the specific event types that USE and TSA flagged. jfr_diagnose orchestrates exactly this — it inspects the recording characteristics, runs the appropriate methodologies, and chains into targeted follow-ups based on what it finds.

The AI doesn’t need to be an expert in JFR event types. It needs to follow a framework that an expert designed. The frameworks are baked into the tools.

What It Doesn’t Do

A few things to be upfront about:

• No live attach. The MCP server works with recorded .jfr files, not live JVMs. You still need to capture the recording first (jcmd, JDK Mission Control, or continuous profiling infrastructure).
• No visualization. Results come back as structured data and text, not rendered charts. The AI can read and reason about them; if you want a flamegraph SVG, pipe the folded output from jfr_flamegraph into your favorite renderer.
• The AI can be wrong. It’s working with real data and real methodologies, but its interpretations are still AI-generated. Treat them as a knowledgeable colleague’s first take, not as gospel. The difference is that every claim is backed by a query you can re-run yourself.

Why Not Just Use jfr-shell Directly?

Good question. If you know JfrPath well and have a hypothesis to test, jfr-shell (especially with the TUI) is probably faster. You know what you’re looking for, you write the query, you get the answer.

The MCP server shines when:

• You don’t know where to start. The AI can run jfr_diagnose, form hypotheses, and chase leads without you directing every step.
• You want a summary for someone else. Ask the AI to write up findings. It’ll produce a narrative backed by USE/TSA data, which is more useful in a Slack thread than a raw query dump.
• You’re reviewing multiple recordings. Open three recordings from different services, ask the AI to compare thread behavior across them. It’ll manage the sessions and cross-reference results.
• You’re tired. It’s 3am, the pager went off, and you need to triage fast. Let the AI do the first pass while you get coffee.

jfr-shell gives you precision. The MCP server gives you leverage. They’re not competing — jfr-mcp uses the same parser, the same query engine, and the same analysis primitives under the hood.

Try It

# One-liner: installs JBang (if needed) + jfr-mcp
curl -Ls https://raw.githubusercontent.com/btraceio/jafar/main/jfr-mcp/install.sh | bash

# Or manually via JBang
jbang app install jfr-mcp@btraceio

# Wire it to Claude Code
claude mcp add jafar -- jbang jfr-mcp@btraceio --stdio

# Then in any conversation:
# "Open /path/to/recording.jfr and diagnose the performance issue"

jfr-mcp is part of JAFAR and available via JBang and Maven Central. Source on GitHub.

The Problem with Scrolling

If you have used jfr-shell for any non-trivial analysis, you know the drill. You run a query, get a wall of text, scroll up to see the header, lose your place, scroll back down, squint at a column that wrapped awkwardly, then run the query again with --limit 10 because your terminal buffer just swallowed the first thousand rows.

It works. It’s fine. It’s also the terminal equivalent of reading a spreadsheet by printing it on a receipt roll.

Starting with version 0.14.0, jfr-shell ships with a full-screen terminal UI mode built on TamboUI, a ratatui-inspired widget framework for Java. Launch it with --tui and the scrolling stops. Everything stays put. You navigate, filter, drill down, and export - all without losing context.

Getting In

jfr-shell --tui
jfr> open recording.jfr

That’s it. Launch the TUI, open a recording. Same queries, same JfrPath, same everything. The difference is that the output no longer scrolls off into the void.

The Layout

The screen is divided into a few fixed zones, top to bottom:

• Status bar - shows the active session and recording name
• Results pane - your query output, rendered as a table or tree
• Command input - where you type, with the familiar jfr> prompt
• Tips & hints - rotating usage tips and context-aware keyboard shortcuts

The results pane is the main attraction. It holds a navigable table with row selection, column sorting, and horizontal scrolling. No more piping into less and pretending that’s ergonomic.

Tabs

By default each command is rendered into a scratch-tab — it will get replaced by the next command. Pin the tab with Ctrl+P to prevent it from being overwritten. Pinned tabs stick around and you can switch between them with { and }.

Search and Filter

Press Ctrl+F and start typing. The table filters in real time - only rows containing your search term are shown, with matches highlighted in yellow. The status line shows the hit count. Press Enter to lock the filter, Esc to cancel.

This works the same way you would expect from any reasonable tool: case-insensitive, substring match, instant feedback.

Sorting

< and > cycle through sort columns. Alt+r reverses the sort direction. The currently sorted column is indicated in the header. Combined with filtering, this lets you answer questions like “which thread had the most samples” without writing a top() aggregation.

The Detail Pane

Select a row and press Enter. The results pane splits: the table stays on the left (60%), and a detail view appears on the right (40%). The detail view shows the full structure of the selected row, rendered as a tree, with nested fields, annotations, and type information all expanded.

This is where the TUI earns its keep. In the old REPL, inspecting a complex event meant running show metadata in a separate query, cross-referencing field names, and hoping your mental model held. Now you just arrow down to a row and hit Enter.

Detail subtabs (switch with [ and ]) let you inspect different aspects of the selected row. Shift+Tab moves focus between the results table and the detail pane. The detail pane has its own scrolling, its own search (Ctrl+F while focused there), and its own cursor navigation.

Event Browser

Type events to see every event type present in the recording, listed as a navigable table with event counts. This is the starting point for exploring an unfamiliar recording - you see what’s there before writing any queries.

Constant Pool Browser

The constant pool browser is one of those features that sounds niche until you need it - and then it’s the only thing that matters.

Type constants to see all constant pool types. Select a type and press Enter to browse its entries, or type constants jdk.types.Symbol to jump straight to a specific type. Navigate with arrow keys, press Ctrl+F to filter the type list.

For large constant pools, entries are paginated automatically so the UI stays responsive even when a recording has hundreds of thousands of entries.

Metadata

metadata lists every event, type and annotation defined in the recording as a navigable table. Field names, types, dimensions - all browsable without leaving the TUI.

Session Switching

Working with multiple recordings is common when doing comparative analysis - “before” and “after” a change, or multiple services in a distributed trace. Alt+s opens a session picker overlay. Arrow to the session you want, hit Enter. The results pane updates to show results from the new active session.

Completion

Tab in the command input opens a completion popup. It’s context-aware: event type names after events/, field paths in filters, function names in aggregations, file paths after open. Type to narrow the list, arrow keys to select, Enter to accept.

Export

Ctrl+E exports the active tab’s table data to a CSV file. A popup opens with a default path under ~/.jfr-shell/exports/; edit it or just press Enter. Done. No more --format csv > output.csv and rebuilding the query from memory.

History

Ctrl+R opens reverse-i-search, exactly like bash. Start typing and it finds the most recent matching command. Ctrl+R again jumps to the next match. Enter accepts, Esc cancels. Command history persists across sessions in ~/.jfr-shell/history.

Combined with Shift+Up/Down for simple history scrolling, you never have to retype a query.

Cell Picker

Press @ to open a popup listing every field and value of the currently selected row. Arrow to the one you want, press Enter — the value is inserted into the command input and copied to the clipboard. Useful for grabbing a thread name, stack trace hash, or constant pool ID without retyping it.

The Keyboard Cheat Sheet

The hints bar at the bottom of the screen adapts to the current focus. When you’re in the results pane, it shows shortcuts like ↑↓:row <>:sort col Ctrl+F:search Ctrl+P:pin. When you’re in the command input, it shows Enter:run Tab:complete Ctrl+R:search history. It’s always telling you what’s available without you having to memorize anything.

Here’s the full set:

Under the Hood

The TUI is built on TamboUI, a Java terminal UI framework inspired by Rust’s ratatui. TamboUI provides the widget set (tables, trees, text inputs, tabs, scrollbars, blocks with borders), the constraint-based layout system, and the styling primitives. jfr-shell composes these into the full-screen application.

The rendering loop is simple: draw the frame, wait for input with a 100ms timeout, handle the keystroke, repeat. Commands execute asynchronously in a background thread, with a braille spinner animation in the status area while they run. Results stream into the active tab as they arrive.

The terminal backend is JLine-based, using raw mode for direct keystroke capture and the alternate screen buffer so your shell history stays clean when you exit.

No external GUI dependencies. No Electron. No web browser. Just ANSI escape codes and a well-organized widget tree.

Try It

# JBang (easiest)
jbang jafar-shell@btraceio --tui
jfr> open recording.jfr

# Or build from source
./gradlew :jfr-shell:shadowJar
java -jar jfr-shell/build/libs/jfr-shell-*-all.jar --tui

If you have been using jfr-shell in REPL mode and getting by just fine, the TUI won’t change what you can do. It changes how it feels to do it. Queries that used to involve scrolling, re-running, and squinting now involve pointing and pressing Enter.

The REPL is still there, unchanged, for scripts and non-interactive use. --tui is strictly additive.

jfr-shell 0.14.2 is available on Maven Central and via JBang. Source on GitHub.

The short version:
our OTLP profiles were perfectly valid according to protoc, but profcheck insisted our attribute_indices were out of range.
The long version is below. It involves:

• An evening that felt like November at 16:30
• Two different proto schemas with the same message name
• Field numbers quietly rearranged between commits
• A cat that refused to care

1. The Symptom: Profcheck vs. Reality

We were adding sample attributes support to an OTLP profiles converter. The flow was:

1. Convert internal data → OTLP ProfilesData
2. Serialize to protobuf
3. Validate using:
   • protoc (canonical protobuf implementation)
   • profcheck (OpenTelemetry profile validator)

The results:

• protoc: ✅ all good
• profcheck: ❌ screaming about attribute_indices

The errors looked like:

sample[0]: attribute_indices: [0]: index 2 is out of range [0..2)
sample[1]: attribute_indices: [0]: index 3 is out of range [0..2)
sample[2]: attribute_indices: [0]: index 4 is out of range [0..2)
...
sample[99]: attribute_indices: [0]: index 101 is out of range [0..2)

We expected every sample to reference a single attribute at index 1. Instead, we got a nice ascending staircase: 2, 3, 4, …, 101.

On a good day, that pattern would be annoying. On a cold, wet evening with terminal light reflecting off the window and the cat side-eyeing the radiator, it was downright offensive.

2. Context: What We Thought We Were Encoding

We had a simple model:

• AttributeTable
  • Index 0: sentinel
  • Index 1: "sample.type: cpu"
• Each sample:
  • attribute_indices = [1]

Quick debug logging:

System.out.println("AttributeTable size: " + attributeTable.size());
System.out.println("Sample attributeIndices: " + Arrays.toString(sample.attributeIndices));

Output:

AttributeTable size: 2  // [0]=sentinel, [1]=sample.type:cpu
Sample attributeIndices: [1]

So in memory:

• Table size is correct
• Index is correct
• Everything looks boringly sane

This is the point where you start suspecting the wire format.

3. Wire Format Autopsy

3.1. First Hex Dump (The Red Herring)

We dumped the file:

hexdump -C /tmp/debug_cpu.pb | head -50

We saw patterns like:

08 01 10 01 18 02 22 01 01 2a 08 ...
08 01 10 01 18 03 22 01 01 2a 08 ...
08 01 10 01 18 04 22 01 01 2a 08 ...

Decoding:

• 08 = (1 << 3) | 0 → field 1, wire type 0 (varint) → stack_index

• 01 = value 1

• 10 = (2 << 3) | 0 → field 2, wire type 0 (varint)

• 01 = value 1

• 18 = (3 << 3) | 0 → field 3, wire type 0 (varint)

• 02/03/04 = incrementing values (looked like link_index)

• 22 = (4 << 3) | 2 → field 4, wire type 2 (length-delimited)
• 01 01 = packed length 1, value [1]

At first glance, this suggested:

• Field 2 was being emitted as a single varint, not a packed repeated field

• That clashed with our encoder call:

  encoder.writePackedVarintField(
      OtlpProtoFields.Sample.ATTRIBUTE_INDICES, sample.attributeIndices);
  

So either:

1. The encoder was misbehaving, or
2. We were looking at the wrong file

The cat, being more experienced with humans than protobuf, silently voted for (2).

3.2. The Stale Artifact

That hex dump was from an old debug file generated before recent refactoring.

After:

rm /tmp/debug_cpu.pb
# rebuild + rerun generator
hexdump -C /tmp/debug_cpu.pb | head -50

…the wire format now matched the expected packed encoding, and protoc decoding aligned perfectly with our structures.

So:

• Fresh .pb → ✔
• Our own decoder → ✔
• protoc → ✔
• profcheck → still ❌

Verdict: the problem was not “we wrote garbage.” It was “someone else is reading it differently.”

Outside, the rain kept going. Inside, the cat fell asleep. We moved on.

4. Calling in Protoc as Referee

We wired in a canonical decode using the trunk OTLP profiles proto:

protoc --decode=opentelemetry.proto.profiles.v1development.ProfilesData \
    --proto_path=/proto/opentelemetry-proto \
    opentelemetry/proto/profiles/v1development/profiles.proto \
    < profile.pb

Result:

• protoc decoded without complaint
• The decoded ProfilesData matched the trunk proto layout
• attribute_indices were [1] everywhere

So for the schema we pointed protoc at:

• Our payload was 100% spec-compliant
• Our data matched expectations

At this point, there were only two realistic options:

1. profcheck is buggy
2. profcheck is using a different schema than the one we’re validating against

Option (2) is more boring and much more likely.

5. Profcheck’s Reality: The Go Module

The real turning point came from looking at the Go module docs:

https://pkg.go.dev/go.opentelemetry.io/proto/otlp/profiles/v1development#Sample

The generated Go struct:

type Sample struct {
    StackIndex         int32    `protobuf:"varint,1,opt,name=stack_index"`
    Values             []int64  `protobuf:"varint,2,rep,packed,name=values"`
    AttributeIndices   []int32  `protobuf:"varint,3,rep,packed,name=attribute_indices"`
    LinkIndex          int32    `protobuf:"varint,4,opt,name=link_index"`
    TimestampsUnixNano []uint64 `protobuf:"fixed64,5,rep,packed,name=timestamps_unix_nano"`
}

Field numbers according to the Go module:

1. stack_index = 1
2. values = 2
3. attribute_indices = 3
4. link_index = 4
5. timestamps_unix_nano = 5

Now compare that to the proto from GitHub trunk:

message Sample {
  int32 stack_index = 1;
  repeated int32 attribute_indices = 2;
  int32 link_index = 3;
  repeated int64 values = 4;
  repeated fixed64 timestamps_unix_nano = 5;
}

Field numbers in trunk:

1. stack_index = 1
2. attribute_indices = 2
3. link_index = 3
4. values = 4
5. timestamps_unix_nano = 5

Let’s put that into a table.

5.1. Field Number Mismatch

So:

• In trunk, attribute_indices = 2
• In the Go module, values = 2, attribute_indices = 3

Someone reshuffled the field numbers between commits.
The Go module was pegged to an older layout, while trunk had a newer one.

We had implemented against trunk.
profcheck was compiled against the Go module.

Result:

• We wrote:

  • Field 2 → attribute_indices
  • Field 3 → link_index
  • Field 4 → values

• profcheck decoded:

  • Field 2 → values
  • Field 3 → attribute_indices
  • Field 4 → link_index

So from profcheck’s point of view:

• Our link_index staircase (2, 3, 4, …, 101) appeared in its attribute_indices field
• It checked those against an attribute table of size 2
• And fairly yelled: “index 101 is out of range [0..2).`

Both sides were internally consistent.
They just didn’t agree on the meaning of field number 2+.

6. The Real Root Cause

The core issue was:

The proto definition in the Go module used by profcheck did not match the trunk version in the GitHub repo.
The Go module was pinned to an older commit where the field numbering was different.

Thus:

• Our encoding:
  • Correct for trunk schema
  • Verified by protoc using that schema
• Profcheck’s decoding:
  • Based on an older schema
  • Interpreted our tags with its own field map
  • Misread link_index as attribute_indices

No exotic protobuf edge case.
No subtle encoder bug.
Just plain schema drift masquerading as “validation errors.”

7. The Fix: Align with Profcheck’s Schema

From a practical standpoint, we had two options:

1. Fight profcheck and enforce trunk schema everywhere
2. Align our field numbering with the schema that the ecosystem is actually using right now (the Go module)

We chose option 2. The code change was almost embarrassingly simple:

// Sample fields
public static final class Sample {
  public static final int STACK_INDEX = 1;
  public static final int VALUES = 2;               // Was 4
  public static final int ATTRIBUTE_INDICES = 3;    // Was 2
  public static final int LINK_INDEX = 4;           // Was 3
  public static final int TIMESTAMPS_UNIX_NANO = 5; // Unchanged

  private Sample() {}
}

After that:

• Our encoder wrote field numbers matching the Go module layout
• profcheck and protoc both interpreted the payload consistently

7.1. Validation After the Change

We re-ran our checks:

# Canonical validation
protoc --decode=opentelemetry.proto.profiles.v1development.ProfilesData \
    --proto_path=. \
    opentelemetry/proto/profiles/v1development/profiles.proto \
    < profile.pb

# Ecosystem validation
profcheck profile.pb

Results:

• protoc → ✅
• profcheck → ✅

The only remaining profcheck warnings were about timestamp ranges in synthetic test data, i.e. not protocol issues.

The mystery staircase of attribute_indices was gone. The logs looked calmer. Outside was still damp and miserable, but at least the protobuf wasn’t.

8. Dual Validation Setup (With Docker)

To avoid “works on my machine” in the future, we containerized the validation environment.

FROM golang:1.23-alpine AS builder
# Build profcheck from OpenTelemetry sig-profiling repo
RUN git clone https://github.com/open-telemetry/sig-profiling.git
WORKDIR /build/sig-profiling/tools/profcheck
RUN go build -o /profcheck .

FROM alpine:latest
RUN apk add --no-cache protobuf protobuf-dev git

WORKDIR /proto
RUN git clone --depth=1 https://github.com/open-telemetry/opentelemetry-proto.git

COPY --from=builder /profcheck /usr/local/bin/profcheck

RUN cat > /usr/local/bin/validate-profile << 'EOF'
#!/bin/sh
set -e

PROFILE_FILE="$1"

if [ -z "$PROFILE_FILE" ]; then
  echo "Usage: validate-profile <profile.pb>" >&2
  exit 1
fi

echo "=== protoc decode ==="
protoc --decode=opentelemetry.proto.profiles.v1development.ProfilesData \
    --proto_path=/proto/opentelemetry-proto \
    opentelemetry/proto/profiles/v1development/profiles.proto \
    < "$PROFILE_FILE" > /tmp/decoded.txt

echo "Decoded profile written to /tmp/decoded.txt"

echo
echo "=== profcheck ==="
profcheck "$PROFILE_FILE"
EOF

RUN chmod +x /usr/local/bin/validate-profile

With this image:

• Everyone in the team validates profiles with the same:
  • profcheck build
  • protoc version
  • OTLP proto checkout

CI uses it, local dev can use it, and nobody has to guess which version of which proto they’re really talking to.

9. Practical Takeaways

9.1. “Spec-Compliant” Needs a Commit Hash

It’s not enough to say:

“We follow the OTLP profiles proto.”

You must also know:

• Which commit of that proto you follow
• Which commit your tools (Go modules, profcheck, agents, exporters) are generated from

Two schemas with the same package and message names but different field numbers are a silent disaster.

9.2. Protoc Is Necessary, Not Sufficient

protoc tells you:

“This payload is valid for the proto you gave me.”

It does not guarantee:

• That this proto matches what profcheck was generated from
• That your Go/Java/Python modules are in sync with your .proto checkout

Think of protoc as the local judge, not the whole court.

9.3. Hex Dumps Still Matter

Hex dumps and manual decoding are tedious, but they:

• Prove which field numbers are actually on the wire
• Show whether you’re emitting packed vs. non-packed fields correctly
• Help you spot “incrementing values in the wrong field” patterns

When you’re stuck on a cold night, it’s basically looking for footprints in a snowstorm.

9.4. Stale Everything: Files and Schemas

Two equally annoying forms of “you’re staring at the wrong thing”:

• Old .pb artifacts from earlier builds
• Old proto versions baked into dependencies and tools

You have to invalidate both before you trust any conclusion.

9.5. Incrementing Values in a “Constant” Field = Schema Mismatch Alarm

If you expect:

• attribute_indices = [1] for all samples

but you see:

• 2, 3, 4, …, 101

assume:

• You are probably decoding the wrong field under the wrong schema, not just “off by one.”

10. Debugging Checklist for Protobuf Weirdness

If you find yourself debugging protobuf in a winter mood, use this checklist:

1. Verify internal data
   • Log tables, sizes, indices, actual arrays
2. Validate with protoc
   • Against the .proto you think you’re implementing
3. Inspect the wire
   • Hex dump
   • Decode tags: tag = (field_number << 3) | wire_type
4. Compare schemas
   • GitHub trunk .proto
   • Vendored .proto in your repo
   • Generated code (Go/Java/etc.)
5. Check tool versions
   • Which commit is profcheck (or other validators) built from?
6. Look at value patterns
   • Incrementing sequences
   • Constant offsets
   • Suspicious repetition
7. Regenerate everything
   • Delete old .pb files
   • Clean & rebuild
8. Assume version skew first
   • Before blaming protobuf
   • Before blaming the cat
   • Before rewriting your encoder twice

11. Postmortem: What Actually Happened

• The OTLP profiles proto evolved; field numbers were rearranged in Sample
• The Go module used by profcheck was locked to an older commit
• We implemented using the trunk proto layout
• Our encoder wrote tags according to trunk
• profcheck decoded tags according to the older Go module layout
• Our link_index became attribute_indices in profcheck’s view
• profcheck legitimately complained about “index 101 out of range [0..2)”

The fix:

• Update four constants in OtlpProtoFields.Sample to match the Go module’s field numbering

The lesson:

• When two validators disagree on a cold, wet night, suspect schema version skew first
• And always verify which reality your tools are compiled against

The cat, for the record, was right to stay on the radiator the entire time.

Or: How I Spent Hours Building a Sophisticated Solution to Automate Clicking a Button (Spoiler: It Still Doesn’t Work)

The Innocent Beginning

It started with such a simple request: “Help me auto-sync my fork with its upstream.” How hard could it be? Just fetch upstream changes, create a branch, push it, and open a PR. Five minutes, tops.

Narrator: It was not five minutes.

Act I: The Schema Validation Betrayal

I confidently started with what seemed like reasonable permissions:

permissions:
  contents: write      # push the sync branch
  pull-requests: write # open, approve & merge the PR
  workflows: write     # enable auto-merge

GitHub’s schema validator immediately slapped me with:

Schema validation: Property 'workflows' is not allowed

“Ah,” I thought, “clearly workflows isn’t a valid permission. Let me just remove that invalid line.”

Famous last words.

Act II: The Runtime Reality Check

With the “invalid” permission removed, the workflow passed validation but failed spectacularly at runtime:

! [remote rejected] upstream-sync -> upstream-sync 
(refusing to allow a GitHub App to create or update workflow 
`.github/workflows/main.yml` without `workflows` permission)

Wait, WHAT? The error message is literally asking for the workflows permission that the schema validator just told me doesn’t exist.

This is like being told you need a driver’s license to drive, but also being told that driver’s licenses are a myth.

Act III: The Documentation Rabbit Hole

Surely GitHub’s documentation would clear this up. I dove into their official docs, searching for the definitive list of valid permissions. What I found was:

• 🔍 Multiple documentation pages with different information
• 📝 Some pages mentioning actions: write for workflow-related operations
• 📋 Other pages showing examples with permissions that don’t match the schema
• 🤷 Zero consistency between schema validation and runtime behavior

I tried actions: write instead. Same error. The runtime system was still demanding the mythical workflows permission like a bouncer asking for a membership card to a club that doesn’t exist.

Act IV: The Catch-22

At this point, I was trapped in a perfect catch-22:

• Schema validation: “You cannot use workflows: write - it doesn’t exist!”
• Runtime system: “You must use workflows permission to modify workflow files!”
• Documentation: “¯\(ツ)/¯”

I tried adding workflows: write back, thinking maybe the schema validator was wrong. Result: Back to the original schema validation error.

It’s like GitHub’s left hand doesn’t know what its right hand is doing, and both hands are actively fighting each other while flipping you off.

Act V: The CLI Comedy of Errors

“Fine,” I said, “I’ll use GitHub CLI instead of those problematic actions.”

First attempt:

gh pr create --json number

unknown flag: --json

The GitHub Actions runner had an older version of gh that didn’t support modern flags. Because of course it did.

Second attempt (simplified):

gh pr create --head upstream-sync --base master

pull request create failed: GraphQL: Resource not accessible by integration (createPullRequest)

Apparently, the GITHUB_TOKEN doesn’t have GraphQL permissions for createPullRequest. Because why would the token designed for GitHub Actions be able to… create pull requests in GitHub Actions?

Act VI: The API Expedition

“Screw it,” I declared, “I’ll use the REST API directly!”

curl -X POST "https://api.github.com/repos/user/repo/pulls" \
  -d '{"title": "Automated upstream merge", "head": "upstream-sync", "base": "master"}'

Response:

{
  "message": "GitHub Actions is not permitted to create or approve pull requests.",
  "status": "403"
}

Ah, the plot thickens! There’s a repository-level setting called “Allow GitHub Actions to create and approve pull requests” that was disabled. This is a security setting that prevents automated PR creation.

After enabling this setting, the API finally worked! 🎉

Act VII: The Self-Approval Scandal

Success! The PR was created automatically. Now for the auto-approval…

{
  "message": "Unprocessable Entity",
  "errors": ["Can not approve your own pull request"],
  "status": "422"
}

Of course. GitHub doesn’t allow you to approve your own PRs. This is actually good security practice, but it means true automation is impossible.

Act VIII: The Branch Protection Plot Twist

“Fine,” I said, “I’ll just merge the changes directly to master and skip the PR entirely!”

git push origin master

! [remote rejected] master -> master (protected branch hook declined)

Organization branch protection rules prevent direct pushes to master. Because we’re in an enterprise environment where security policies actually matter.

The Bitter End: A Sophisticated Solution to Nothing

After hours of troubleshooting through GitHub’s permission maze, I had built a technically perfect workflow that:

✅ Successfully syncs upstream changes while preserving local workflow files
✅ Automatically creates well-formatted PRs with clear descriptions
✅ Enables auto-merge so PRs merge immediately after approval
✅ Handles edge cases and provides comprehensive error handling
❌ Still requires manual approval due to organizational branch protection rules
❌ Provides zero practical benefit over clicking “Sync fork” in the GitHub UI

The Lessons Learned

1. GitHub’s tooling is inconsistent: Schema validation and runtime behavior can directly contradict each other.

2. Error messages lie: When GitHub says you need “workflows permission,” it might mean something that doesn’t actually exist in the schema.

3. Documentation is unreliable: Multiple official sources can give conflicting information about the same feature.

4. Enterprise constraints trump clever solutions: No amount of technical sophistication can bypass organizational security policies.

5. Sometimes the simple solution is the right solution: Clicking a button in the UI might actually be more efficient than building a complex automation that doesn’t automate the painful parts.

6. Policy problems require policy solutions: The real blocker wasn’t technical - it was organizational rules that prevent true automation.

The Final Irony

The original problem was having to manually sync multiple forks daily by clicking through the GitHub UI. After hours of sophisticated engineering, I built a system that… still requires manually clicking through GitHub to approve PRs.

In the end, I automated everything except the part that actually needed automation.

The Real Takeaway

This experience perfectly illustrates the difference between technical possibility and practical value. Just because you can build something doesn’t mean you should. Sometimes the boring, manual solution is actually the right one - especially when organizational constraints make true automation impossible.

The GitHub “Sync fork” button exists for a reason. It’s simple, reliable, and achieves the same end result with the same amount of manual intervention as our sophisticated workflow.

Sometimes the real automation is the clicking I did along the way. 🤷‍♂️

P.S. - If you’re facing similar GitHub Actions permission issues, save yourself some time: check your repository settings first, accept that organizational policies exist for a reason, and don’t be afraid to embrace the simple solution. Your sanity will thank you.

Technical Appendix

For those brave souls who want to see the final working (but practically useless) workflow:

name: Upstream-sync → protected master
on:
  schedule:            # run every night
    - cron:  '7 2 * * *'
  workflow_dispatch:   # (optional) manual trigger

permissions:           # minimum perms the job needs
  contents: write      # push the sync branch
  pull-requests: write # open, approve & merge the PR

concurrency:           # never let two syncs race
  group: $-$
  cancel-in-progress: true

jobs:
  sync:
    runs-on: ubuntu-latest

    steps:
      # 1. full clone so we always have the latest tip
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
          token: $

      # 2. fetch upstream & copy it to a side branch
      - name: Update upstream-sync branch
        env:
          GITHUB_TOKEN: $
        run: |
          # Configure git identity
          git config --global user.email "action@github.com"
          git config --global user.name "GitHub Action"
          
          git remote add upstream https://github.com/openjdk/jdk.git
          git fetch upstream master
          
          # Create sync branch from current master to preserve workflows
          git checkout -B upstream-sync origin/master
          
          # Simple merge approach - let's see what happens
          if git merge upstream/master --no-edit --allow-unrelated-histories; then
            echo "=== Merge successful ==="
            git log --oneline -5
          else
            echo "=== Merge failed, trying alternative approach ==="
            git merge --abort || true
            git reset --hard upstream/master
            # Restore our workflow files after taking upstream
            git checkout origin/master -- .github/workflows/
            git add .github/workflows/
            git commit -m "Preserve local workflow files during upstream sync"
            echo "=== Alternative approach completed ==="
            git log --oneline -5
          fi
          
          git push -f origin upstream-sync

      # 3. Create PR and attempt auto-merge (constrained by org branch protection)
      - name: Create PR for upstream sync
        env:
          GITHUB_TOKEN: $
        run: |
          # Check if PR already exists
          PR_EXISTS=$(curl -s -H "Authorization: token $GITHUB_TOKEN" \
            "https://api.github.com/repos/$/pulls?head=$:upstream-sync&base=master" \
            | jq -r '.[0].number // empty')
          
          if [ -n "$PR_EXISTS" ]; then
            echo "PR #$PR_EXISTS already exists - updating it"
            curl -s -X PATCH -H "Authorization: token $GITHUB_TOKEN" \
              -H "Accept: application/vnd.github.v3+json" \
              "https://api.github.com/repos/$/pulls/$PR_EXISTS" \
              -d '{
                "title": "🤖 Automated upstream sync",
                "body": "**Automated nightly sync of upstream changes**\n\n📊 **What this includes:**\n- Latest commits from `openjdk/jdk17u-dev:master`\n- Preserves local workflow configurations\n- Ready for immediate merge\n\n🔄 **Auto-generated by:** `.github/workflows/dd-sync.yml`\n\n⚡ **Action required:** Just click **Approve** to merge automatically!"
              }'
            echo "✅ Updated existing PR #$PR_EXISTS"
          else
            echo "Creating new PR for upstream sync"
            PR_RESPONSE=$(curl -s -X POST -H "Authorization: token $GITHUB_TOKEN" \
              -H "Accept: application/vnd.github.v3+json" \
              "https://api.github.com/repos/$/pulls" \
              -d '{
                "title": "🤖 Automated upstream sync",
                "body": "**Automated nightly sync of upstream changes**\n\n📊 **What this includes:**\n- Latest commits from `openjdk/jdk17u-dev:master`\n- Preserves local workflow configurations\n- Ready for immediate merge\n\n🔄 **Auto-generated by:** `.github/workflows/dd-sync.yml`\n\n⚡ **Action required:** Just click **Approve** to merge automatically!",
                "head": "upstream-sync",
                "base": "master"
              }')
            PR_NUMBER=$(echo "$PR_RESPONSE" | jq -r '.number')
            if [ "$PR_NUMBER" != "null" ] && [ -n "$PR_NUMBER" ]; then
              echo "✅ Created PR #$PR_NUMBER"
              
              # Enable auto-merge immediately
              curl -s -X PUT -H "Authorization: token $GITHUB_TOKEN" \
                -H "Accept: application/vnd.github+json" \
                "https://api.github.com/repos/$/pulls/$PR_NUMBER/merge" \
                -d '{"merge_method": "merge"}' && echo "🚀 Auto-merge enabled" || echo "⚠️ Auto-merge requires approval"
            else
              echo "❌ Failed to create PR: $PR_RESPONSE"
            fi
          fi
          
          echo ""
          echo "🎯 **SUMMARY:** Upstream sync PR is ready!"
          echo "📝 **Next step:** Approve the PR and it will merge automatically"
          echo "🔗 **Link:** https://github.com/$/pulls"

This workflow technically works perfectly. It just doesn’t solve the actual problem. 🎭

Oh, not again …

Yes, you remember correctly. I did write a quite lengthy series of blog posts about my proposal for JFR context implementation. However, it turned out that the proposed changes were too intrusive and would never be accepted.

Things looked bleak for a while, because without the context in JFR, we would still need to rely on a separate JVMTI agent to do the cool things we can do, thanks to the context.

The plot twist

During my trip to Stockholm where my colleague Richard Startin and I talked about future-proofing the profiling support in the JVM, we also managed to secure a meeting with my former colleagues and, I dare say friends, Markus Grönlund and Erik Gahlin. They are the main force behind JFR and it made a lot of sense to meet them.

We ended up talking for many more hours than originally planned, and it was a very fruitful discussion. When trying to figure out what to do with the context in JFR, Erik suddenly asked: “Why don’t we just have special events to convey the context?”.

The idea

The idea was simple: we would introduce a new event meta type, @Contextual, which would be used to convey the context.

This idea is also pretty old and was originally dismissed because, in order to capture rapidly changing context, as would be the case for async or reactive applications, we would need to generate a lot of events. And that would be a problem - the recording size would blow up, and the transfer and processing costs would skyrocket.

Here, Markus and Erik stopped and asked - “Hm, but what if we emit the contextual event only when it applies to at least one other event?”.

The word “applies” here translates to an event being committed between calls to begin() and end() methods of the contextual event.

And sure, with this little tweak we would be able to attach an arbitrary context to virtually any JFR event. The number and types of the context fields would be limited only by what the user will be willing to pay in terms of memory and storage costs.

Juicing the idea

What if we don’t stop there? If we have contextual events we could also use the presence of the context as an alternative to the event threshold which is in use today. While thresholding allows focusing on the outliers, it is many times the ‘death by a hundred cuts’ that is the real problem. But in that case, thresholding would actually mask the real problem, as the short-lived events would never cross the threshold and be reported.

But if, instead of checking the event duration to cross the threshold, we could check if there is a contextual event that applies to the event, this would create a ‘magnifying glass’ effect, where all the fine details would be preserved as long as there is a context. And, considering that the context is present for operations of special interest to the user, it would be a perfect match.

The proposal

Putting all of this together I set out creating an early prototype of the contextual events. The prototype is still in the very PoC stage, but it already shows a lot of promise. I have patched the Datadog Java tracer to turn the existing TimelineEvent type to a contextual event and ran a bunch of applications to see how it behaves.

And it does what is expected - the context is easily attributable to various events, like CPU, Wallclock, or Allocation samples, but not only that. It is also possible to associate the context with built-in events like MonitorWait, etc.

In order to test the thresholding alternative, I set the threshold for ThreadPark and JavaMonitorWait events to 0ms (no threshold, and something you really don’t want to do in production). The results were astonishing - the recording was not bloated, the processing was not overwhelmed, and the events were still there, providing the fine-grained details of the application’s behavior.

What’s next

Currently, we are collecting feedback from the community and early adopters. So far, there hasn’t been much of the feedback, but I am hoping that this blog post will change that.

If you have any thoughts, ideas, or concerns, please feel free to use the PR to comment directly there. Or, if you prefer, you can leave a comment here.

(This is a copy of the detailed description of the proposal from the PR - just for the record if the PR would change in future)

Gnarly details

Design

Contextual annotation

A contextual event will be demarked by @Contextual annotation. This annotation wil be a simple indication that this particular event type is supposed to provide context to other events and tooling can handle it as such.

All custom fields of such annotated event type will then constitute the context.

Context driven behaviour

Although having the @Contextual annotation will allow the tooling to associate the context with other JFR events, there are more ways they can be utilized.

Conditionally emit events

The contextual events can be used to guard annotations of events which are too costly to emit unconditionally and using the durational thresholds would introduce too strong bias. An example would be JavaMonitorWait event.

If left unchecked, the emission rate of JavaMonitorEvent can overwhelm the recording. What’s worse is that the majority of the recorded events will provide very little additional information. Turning on the durational threshold will improve the situation, but will introduce bias where the JFR will not be able to point out too much time spent waiting on a lock, if each wait is shorter than the threshold. In addition to that, this event type might be frequently emitted from thread pools where threads are just waiting for work.

If the emission is bound to the presence of a context (contextual event) which will be activated only when an important work (what is important work will usually be defined by the user) is being done, providing laser focus on fine-grained details of the application’s behaviour.

Record only activated context

We are talking about an activated context (contextual event) when there is at least one other event committed on the same thread between calling begin() and end() of the contextual thread. We can also think about the context being ‘triggered’ by the regular events.

The concept of ‘active’ context is beneficial in lowering the overhead related to recording the context - eg. for the distributed tracers with context propagation it is possible to generated millions of contextual events per minute for certain frameworks (async and reactive ones are pretty notorious). This creates a huge pressure both when the recording is written and also when it needs to be processed. And most of these events will be literally useless because there would be no events the context could be applied to.

Controlling the behaviour via settings

The proposal is to use the standard JFR event settings mechanism to affect the behaviour of both contextual and regular events.

There will be a new setting called select and the following permitted values:

• if-context - the regular event will be emitted only if a context is present
• if-triggered - the contextual event will be emitted only if the context is triggered
• all - no context related restrictions are applied

The if-context option is valid only for non-contextual events. The if-triggered option is valid only for contextual events. The all option is valid for any event.

If an invalid option is provided, JFR will log a warning and the setting will be set to all.

The select setting is to be used in conjunction with other filtering mechanisms, like threshold.

Implementation

@Contextual annotation

The annotation implementation is pretty straightforward and there is nothing special going on there.

Activated context

In order to support selective emission of the contextual events only when they are activated the event class must be instrumented and a synthetic field named ^ctxOffset must be inserted there.

The field is used to track the number of events written while this context is open. The actual number does not matter, we just need to make sure we can tell there is at least one written event.

This information is then used in the shouldCommit() method of the contextual event type which needs to be changed to consult ^ctxOffset field and return false if that field is 0. That is, if the event’s settings contains select=if-triggered. Otherwise, the behaviour of shouldCommit() is not affected.

The ^ctxOffset field is updated from EventWriter, incrementing it on a new event commit.

It starts with a customer report

Recently, I received a report from a customer that they noticed RSS slowly growing over time in their application when they enable Datadog Java Profiler. There is no unexpected growth in heap usage, meaning that it is the native memory usage that is growing. Oh well, I should brace for a fun debugging session.

Reproducer

Quite expectedly, the customer was not able to share their application with me, so I have to come up with a reproducer. From the problem description there was nothing extraordinary with the application, yet our continuous reliability tests did not catch the issue. Usually, when I have no idea where to start, I reach out to the Renaissance Benchmark which provides a nice collection of diverse load generating cases. As luck wanted it, the Akka-uct benchmark was the first one I tried, and it exhibited the same behavior as the customer’s application. The RSS was growing quite visible with the interpolated rate of 60-70MiB per day but the Java heap usage was rather stable.

The investigation

Native Memory Tracking

The first thing I did was to enable the native memory tracking (NMT) and check the JVM native memory statistics. Unfortunately, the NMT did not show anything suspicious. The usage of the native memory managed by JVM stayed stable, which was not the case with the RSS.

ASAN

Ruling out the JVM managed native memory issues I refocused on our profiler library which is a native JVMTI agent and therefore can cause unexpected memory leaks at the native side. The AddressSanitizer (ASAN) is a great tool to catch such issues. It is touted as the ‘go-to’ tool for the native memory issues debugging. I compiled and linked the agent with ASAN enabled and ran the reproducer. Boom! It turns out that ASAN is very sensitive to SIGSEG (among other signals) and aborts analysis as soon as it discovers one. The profiler library is a JVM agent and is running inside the JVM process. And the JVM is using SIGSEG for internal signalling. To make things more interesting, the profiler library is relying on being able to handle SIGSEG when attempting to read potentially invalid memory - we are effectively peeking in the JVM internals, and we need to be very defensive there as it was not designed with this level of observability in mind.

There is a config option telling ASAN to ignore SIGSEGs, but it did not work for the LeakSanitizer (LSAN) subsystem.

Bottom line, ASAN is not an option for this case.

Valgrind

After the fiasco with ASAN, I turned to Valgrind. Valgrind is a bit more intrusive than ASAN, but it is also more flexible. And you don’t need to modify the build process to use it. One can just take the already existing debug build and run it under Valgrind using the memcheck tool.

Ok, let’s try this out. Fingers crossed!

Hm, it’s slow. I mean, really, really slow … but it seems to be working. After a long while I had my first Valgrind report. Unfortunately, it was totally swamped by warnings caused by the GC managing memory in ways Valgrind does not like. It is possible to configure the checks with the help of suppressions, but it is a tedious process. After a few more rounds of running the reproducer, waiting for the report, and tweaking the suppressions, I gave up. The amount of noise was just too high to be able to see the signal. Even with most of the nonsensical warnings suppressed, it gives literally thousands of leak suspects which are most probably not leaks at all, coming from GC or runtime itself.

==315313== LEAK SUMMARY:
==315313==    definitely lost: 20,029 bytes in 354 blocks
==315313==    indirectly lost: 114,141 bytes in 2,215 blocks
==315313==      possibly lost: 60,961,787 bytes in 244,974 blocks
==315313==    still reachable: 29,620,688 bytes in 3,188 blocks
==315313==        suppressed: 0 bytes in 0 blocks

I took L for Valgring as well. But wait, there is this tool in the Valgrind package, ‘massif’ which can be used to profile the heap usage. But no, false alarm. Running the rerpoducer with ‘massif’ will just take down the JVM with SIGSEG within a minute.

DDPROF

When the standard solutions failed, it is time to try some dog-food. The DDPROF (Datadog Profiler for native code) contains a new beta feature called ‘live heap profiler’. The live heap profiler samples OS level memory allocations (eg. via malloc or new) and tracks when the allocated chunks are released. Subjectively, it is much faster than Valgrind (can’t compary to ASAN as it did not work for me) and should be able to capture sufficiently prominent memory leaks.

The live heap profiling feature is currently available only in version 0.17.0 of DDPROF and once that version is downloaded and extracted it can be used like this:

ddprof -S akka-uct-jdk11-notrace-native -e sALLOC,mode=l --inlined_functions true -worker_period 300000 \
  java -Ddd.env=${WORKSPACE} -Ddd.service=akka-uct-jdk11-notrace \
  -Ddd.profiling.enabled=true -Ddd.profiling.ddprof.enabled=true \
  -Ddd.profiling.ddprof.liveheap.enabled=true -Ddd.profiling.ddprof.alloc.enabled=true -Ddd.profiling.upload.period=10 \
  -Ddd.profiling.timeline.events.enabled=true -Ddd.integration.renaissance.enabled=true -Ddd.env=memleak-test \
  -Ddd.trace.enabled=false \
  -Xmx512m -jar renaissance-mit-0.15.0.jar akka-uct -r 1000000

The important part is ddprof -S akka-uct-jdk11-notrace-native -e sALLOC,mode=l --inlined_functions true -worker_period 300000 which configures the DDPROF to track allocation with liveness information (-e sALLOC,mode=l), resolve inlined function (--inline_functions true) and to effectively not evacuate the liveness tracking table(-worker_period 300000). If the -worker_period is not specified the liveness tracking table will be regularly evacuated in order to prevent secondary memory leaks in the DDPROF itself (if an allocation is not released in the profiled application, the tracking entry in DDPROF would live forever without the evacuation). The rest of the command is just a standard configuration of both the native and Java specific profiler.

Having finished the setup in no time, I run the reproducer and keep it running for at least a few hours. The results are interesting. As I already mentioned, this is a great example of dog-fooding. Using the Datadog DDPROF tool to capture profiles that can then be analyzed in the Datadog platform. Once all the data is in, I can use the comparison feature of the Datadog Continuous Profiler to compare the sample of live allocations at the beginning of the run with the sample at the end of the run. Here is the result:

Fig1: Live heap profile comparison

The comparison is very intuitive, at least in my opinion. The red bars represent the increase of the retained allocations from that particular allocation site. The green bars represent the decrease of the retained allocations. The blue bars are showing the allocations that were not present in the first sample but are present in the second sample.

Using this information, it becomes obvious that something is off in the part of the code responsible for updating the thread information. There are internal maps for converting the native thread ID to the Java thread ID and its name. Since threads come and go, these maps are regularly cleaned up to avoid memory leaks. And here lies the problem.

// Reset thread names and IDs
MutexLocker ml(_thread_names_lock);
if (thread_ids.empty()) {
    // take the fast path
    _thread_names.clear();
    _thread_ids.clear();
} else {
    // we need to honor the thread referenced from the liveness tracker
    std::map<int, std::string>::iterator name_itr = _thread_names.begin();
    while (name_itr != _thread_names.end()) {
        if (thread_ids.find(name_itr->first) != thread_ids.end()) {
            name_itr = _thread_names.erase(name_itr);
        } else {
            ++name_itr;
        }
    }
    std::map<int, jlong>::iterator id_itr = _thread_ids.begin();
    while (id_itr != _thread_ids.end()) {
        if (thread_ids.find(name_itr->first) != thread_ids.end()) {
            id_itr = _thread_ids.erase(id_itr);
        } else {
            ++id_itr;
        }
    }
}

Can you spot it? Yes, it’s the second loop, iterating over id_itr but checking the thread_ids for the name_itr->first. To add insult to injury, the condition is wrong in both loops. The intention was to check if the thread_ids set, which is the set of the thread ids being referenced from the liveness tracker, contains a particular thread id and if not, remove it.

However, the condition does the exact opposite!! No wonder the memory was leaking. The thread names and ids were almost never cleaned up, even though they were not needed anymore.

And they lived happily ever after

One would think so, but no. Once the library caused memory leak had been fixed I re-ran the reproducer. Surprisingly, the RSS was still growing, although at much slower pace. The memory leak was fixed, but there was still something else causing the memory to grow. The DDPROF live heap profiler was pointing to some internal JFR (JDK Flight Recorder) structures, but that sounding like a red herring I decided to run the reproducer with a specific configuration where only JFR was enabled, to rule out any interference from the profiler library. And the memory was still growing.

Fig2: RSS size with JFR only

The DDPROF live heap profiler is pointing to the JfrStackTraceRepository class which seems to be growing continuously.

Fig3: Comparison of live heap profiles of JFR only reproducer

But, by all means, this should not be the case - re-checking the code that is responsible for the repository management, the repository is cleared regularly. I also peppered the code with additional, very verbose logging to see if things are as they seem and I could see that the repository was evicted regularly, all reference counts were going to zero after the eviction and there were really no dangling pieces of memory out there.

After a lot of head scratching and soul-searching, I mentioned the issue to a colleague who pointed out that they had similarly inexplicable memory leak in Ruby. And, surprisingly, the culprit was the internal fragmentation of the glibc malloc implementation.

The issue is well described by my colleague Brice Dutheil in his blog post

The remedy is to use an alternative allocator, like jemalloc or tcmalloc.

Once I siwtched to tcmalloc, the memory growth stopped and the RSS stabilized.

Fig4: RSS size with JFR and tcmalloc

Conclusion

The memory leak was fixed in no time and the fix will be deployed to our customers’ environments in the near future. It is a bit embarrassing that such a simple bug could have escaped our attention. Obviously, if the standard tools, like ASAN, were working for JVM we could have caught it immediately before ever being shipped. But if ifs and buts …

On the other hand, I was pleasantly surprised by the DDPROF live heap profiler. The setup was easy, the overhead almost undetectable, and the results were very intuitive - despite me being quite sceptical

The next steps will involve adding a specific configuration to our reliability tests to run the akka-uct benchmark with the DDPROF live heap profiler and use the results to assess the situation of possible memory leaks regularly.

The JFR memory leak was a bit more tricky. The DDPROF live heap profiler was pointing to the wrong direction, but with a bit of luck, the real culprit was discovered. Probably, the allocation patterns in JFR could be optimized to avoid the internal fragmentation, but until then one would need to use an alternative allocator to avoid the issue.

It is very interesting that the malloc fragmentation is affecting the DDPROF live heap profile. There is no other reason for the profile to be what it is but the fragmentation. Yet, the reference counting should still work there. There are certainly some interesting things to investigate and improve the live heap profiler.

Integrating with existing tracers

I’ve already shown the basic concepts of profiling context and talked about how the proposed JFR API would look like.

Although having a shiny new API to deal with the profiling context is nice, we cannot forget about the existing code already dealing with the context - the distributed tracers. Whether we talk aboutOpenTelemetry, OpenTracing, OpenCensus, or any of the proprietary implementations, the proposed JFR API must allow for an easy, cheap, and non-intrusive integration.

Searching for integration points

A brief look at OpenTelemetry and DD Tracer confirmed that they are already dealing with the context problem in a broader way than what the proposed JFR API is supposed to tackle. Namely, the distributed tracers are taking care of maintaining the context stack and propagating the context around via extensive instrumentation. And the propagation is not restricted to the same JVM but, as the ‘distributed’ part suggests, they are able to transfer the context to other processes, hosts, and networks just as easily.

The JFR API proposal was never meant to deal with the distributed context in all of its nuances - rather it is supposed to be a public and supported way any tracer implementation could use to make the JFR recordings aware of the context they are working with.

The fact that the tracers are already maintaining their own context rules out using the custom ContextType directly as that would require either a parallel machinery to maintain the context stack and perform the context propagation or setting up mirroring parts of the tracer context to the custom ContextType, potentially resulting in much increased allocation rates and memory copies. As we all can see, this approach would not meet any of the ‘easy’, ‘cheap’, or ‘non-intrusive’ requirements.

Exposing ContextAccess

Instead of relying on the stateful context representation via a custom ContextType, I propose providing a ContextAccess which can be generated for any type (POJO) with the only requirement of the class being annotated by @Name and at least one field or method returning a value being annotated by @Name as well.

Once such ContextAccess instance is obtained, it can be used to set or unset the context from a specific instance of the given type. The access implementation will use the information about which fields/methods are annotated by @Name to properly fill the context slots by the values extracted from those fields and methods.

A note about picking the annotation type

The @Name annotation was picked as something non-intrusive and is guaranteed to exist in all supported versions of OpenJDK, as long as it was compiled with JFR enabled.

An alternative would be to use specific annotations for context type declaration, but that would require an ASM-based annotation detection as opposed to the currently used reflection-based one. This is because the annotations coming from a newer JDK would not be resolvable in older JDKs.

Supporting primitive types for context values

In order to facilitate zero-conversion integrations, it is necessary to increase the set of supported context attribute types. If we stick to supporting only string, the amount of conversions between, e.g., long values and strings would definitely become rather costly, as it would have to be done every single time the context is manipulated.

Instead, the definition of a context type is relaxed to allow attributes of any primitive type as well as string/charsequence. This comes at no extra cost since JFR already supports all the primitive types anyway.

DDSpanContext example

Modify DDSpanContext class

// expose only `rootspanid', `spanid` and `operation_name` attributes.
@Name("dd-context") // add this annotation such that type can be accepted for context 
public class DDSpanContext
  implements AgentSpan.Context, RequestContext, TraceSegment, ProfilerContext {
  // bunch of fields defining the context - all of them private

  @Name("rootspanid") // context attribute getter method
  public long getRootSpanID() {
    return rootSpanID;
  }

  @Name("spanid")
  public long getSpanID() {
    return spanID;
  }

  @Name("operation_name")
  public CharSequence getOperationName() {
     return operationName;
  }

  // more context stuff
}

Obtain and use ContextAccess instance

DDSPanContext context = new DDSpanContext(...);
ContextAccess<DDPspanContext> access = ContextAccess.forType(DDSpanContext.class);
...
// context is all set up, just activate it
access.set(context);
...
access.unset(); // deactivation is not instance specific

If you want to see the actual working integration, you can find this draft PR quite interesting. The PR description contains the guide for building the patched DD tracer agent to run against the patched OpenJDK 21.

The original JFR context prototype had to be backported from OpenJDK 22 because the DD tracer cannot be built and used on OpenJDK 22 due to the lack of support in ASM. This is because ASM can support only released JDK versions.

The captured JFR would look something like this in JMC

Wrap up

In this blog series, I have attempted to build a case for introducing the notion of context to JFR events. To support my case, I also proposed the API form and provided a prototype implementation for OpenJDK, as well as a PoC for integration with the Datadog tracer agent.

My impression is that the current API form suits the purpose of building context-aware applications and libraries, and it also integrates well with existing context-tracking solutions.

The implementation itself is prototypical, covering mostly only the happy paths. However, it should be sufficient to gather initial feedback about the feasibility of this approach and to serve as the foundation for the JEP preparation phase.

As always, if you wish to try out the API or consider adding an integration for your favorite tracer, please reach out to me on X via @BachorikJ.

What is jmethodID and why do I care

Let me start with a short introduction to the concept of jmethodID.

When working in the realm of JNI (Java Native Interface), we can obtain references to Java methods which can later be used to, for example, invoke the method. The reference to the method is represented by a jmethodID type, which is an opaque pointer to a structure containing the method’s metadata.

Why do we need an opaque pointer instead of just using the method reference in the form of jmethod directly? The answer is simple - the method reference is not stable. With the ability to retransform classes, the method reference can change at any time, invalidating any previously obtained references. Therefore, JNI uses the jmethodID type to represent the method reference, which is stable and can be used to invoke the method even after retransformations. The runtime will ensure the jmethodID is updated to point to the correct method metadata after retransformation.

However, there is one caveat: the jmethodID is only valid as long as there is a strong reference to the class containing the method that the jmethodID is pointing to. Once the class is unloaded, the jmethodID becomes invalid, and when it is used to resolve to a jmethod instance, it will return NULL. Although this is Hotspot-specific behavior, it is not something required by the JNI specification. As such, crashes may occur in other JVM (Java Virtual Machine) implementations when using invalid jmethodID values.

Later in this article, I will focus on the Hotspot implementation of jmethodID and how invalid jmethodID values can still pose problems.

jmethodID and profilers

Unsurprisingly, the stack traces obtained via the GetStackTrace or GetAllStackTraces JNI functions have their stack frames represented by jmethodID values. However, for all threads except the current one, it is impossible to guarantee that the classes referenced in the stack trace via jmethodID will be strongly held, making the behavior more or less undefined.

In Hotspot, the implementation is hardened and is supposed to handle invalid jmethodID values gracefully, compensating for this hole in the specification.

However, there is a bug - JDK-8313816 - causing spurious JVM crashes when trying to resolve the jmethodID values captured as a stack trace sometime later after obtaining the stack trace. The bug is not easy to reproduce, and it took a significant amount of time to find a reliable reproducer. The reproducer is not perfect, but it is good enough to trigger the crashes reliably. Unfortunately, it is a test suite of an internal project that heavily uses Mockito and will trigger the crashes within a few tens of seconds.

The sad reality is that this crash makes any profiler implementation using the JVMTI functions to obtain stack traces unusable in an always-on, in-production manner.

Investigation

I must admit, this particular crash has preoccupied me for many months. Not completely understanding the underlying implementation, I took several wrong turns and even proposed a patch, which turned out to solve nothing. But all that time, I was convinced there was a subtle bug in the Hotspot implementation of jmethodid, while many people remained skeptical, attributing the crash to incorrect usage of jmethodid values.

What is that assert doing

The vast majority of the crashes were happening in Method::checked_resolve_jmethod_id() call. While examining the code I skimmed over this statement

assert(is_valid_method(o), "should be valid jmethodid");

just assuming that under normal circumstances this assert should never trigger. Well, that is until I ran the reproducer with fastdebug OpenJDK build which enabled all those asserts. And suddenly, I saw the root cause for all those various crashes - the captured jmethodid values are becoming invalid and when they are used there is a good chance of crashing JVM.

So, I set out on checking the source code of the Method::is_valid_method() function.

// Check that this pointer is valid by checking that the vtbl pointer matches
bool Method::is_valid_method(const Method* m) {
  if (m == nullptr) {
    return false;
  } else if ((intptr_t(m) & (wordSize-1)) != 0) {
    // Quick sanity check on pointer.
    return false;
  } else if (!os::is_readable_range(m, m + 1)) {
    return false;
  } else if (m->is_shared()) {
    return CppVtables::is_valid_shared_method(m);
  } else if (Metaspace::contains_non_shared(m)) {
    return has_method_vptr((const void*)m);
  } else {
    return false;
  }
}

Well, that’s a few cases where jmethodid can be invalid! But which ones are triggering when I am getting the follow-up crash? In order to do that, I resorted to ‘printf debugging’, inserting printf statements into each branch with a ‘unique’ id, so it would be easy to match the failed precondition with the code path leading to the crash.

Long story short - when has_method_vptr((const void*)m) returns false and such an invalid jmethodid is then used, it is almost certain that usage will crash the JVM. Well, we are trying to use a memory blob as an object instance, but it is missing the valid object header. It only makes sense that this attempt can lead to very bad things.

Hygiene is important

At this point, I formed a hypothesis that the affected jmethodID is pointing to a Method object which was reclaimed, and part or all of the memory originally occupied by that instance was overwritten.

The 100-point question is: when can a Method object be reclaimed? Well, it turns out the JVM is very strict about that. A Method can be reclaimed only when its containing class is unloaded. And when is a class allowed to be unloaded? Only when its associated classloader is not referenced anymore. A bit of code searching and following call sites led me to ClassLoaderData::unload(). And indeed, the jmethodIDs associated with the methods from that class loader are wiped in order to make sure they will not be pointing to deallocated objects.

  // Method::clear_jmethod_ids only sets the jmethod_ids to null without
  // releasing the memory for related JNIMethodBlocks and JNIMethodBlockNodes.
  // This is done intentionally because native code (e.g. JVMTI agent) holding
  // jmethod_ids may access them after the associated classes and class loader
  // are unloaded. The Java Native Interface Specification says "method ID
  // does not prevent the VM from unloading the class from which the ID has
  // been derived. After the class is unloaded, the method or field ID becomes
  // invalid". In real world usages, the native code may rely on jmethod_ids
  // being null after class unloading. Hence, it is unsafe to free the memory
  // from the VM side without knowing when native code is going to stop using
  // them.
  if (_jmethod_ids != nullptr) {
    Method::clear_jmethod_ids(this);
  }

So, this should work, right? Hotspot takes care of nulling the references to ensure no invalid memory can be accessed via dangling pointers, and unloading is conducted in a VM operation, guaranteeing that the native code and the unloading process never race. All parts are in place, and yet, it is still possible to trigger a JVM crash quite reliably. There must be something else at play.

Asserts and guarantees

You are probably familiar with the concept of code asserts - they should capture invariants and fail if these are broken. Since the invariants are supposed to hold at all times, usually the asserts are enabled only in ‘debug’ builds. This makes them extremely cheap in non-debug builds, because they are physically removed from the resulting machine code.

In the JVM codebase, in addition to assert, there is also guarantee. Guarantee works pretty much like an assert, with the difference that guarantee will trigger also in a non-debug build.

‘What are you getting at?’, you might ask now. The answer is simple - I needed a way to ‘annotate’ the JVM code dealing with jmethodIDs, and particularly, deallocating the Method instances with various invariant checks. But, since my reproducer is quite a large application, running it with a debug build became quite problematic because it became really, really slow. So, I used temporary guarantees instead.

In the InstanceKlass class, which represents Java class metadata (very simply put), there is an interestingly looking function deallocate_contents().

// This function deallocates the metadata and C heap pointers that the
// InstanceKlass points to.
void InstanceKlass::deallocate_contents(ClassLoaderData* loader_data)

This function, in turn, calls InstanceKlass::deallocate_methods(), which takes care of physically deallocating all the Method instances contained in the current InstanceKlass. That looks promising - the assumption is that once a Method instance is deallocated, the jmethodid pointing to that method should be updated to nullptr instead. This should be quite easy to express as an invariant via guarantee -

void InstanceKlass::deallocate_methods(ClassLoaderData* loader_data,
                                       Array<Method*>* methods) {
  if (methods != nullptr && methods != Universe::the_empty_method_array() &&
      !methods->is_shared()) {
    for (int i = 0; i < methods->length(); i++) {
      Method* method = methods->at(i);
      if (method == nullptr) continue;  // maybe null if error processing
      // Only want to delete methods that are not executing for RedefineClasses.
      // The previous version will point to them so they're not totally dangling
      assert (!method->on_stack(), "shouldn't be called with methods on stack");
      
      // >>> inserted code
      // try and resolve the jmethodid from the `Method` instance
      jmethodID jmid = method->find_jmethod_id_or_null();
      // <<<
      
      MetadataFactory::free_metadata(loader_data, method);
      
      // >>> inserted code
      // The method that was deallocated must not have a jmethodid pointing to it
      // Because class redefinitions/retransformations can introduce new method versions
      // a jmethodid pointing to the original version will be updated to point to newer version
      // making it necessary to also assert that the jmethodid is actually pointing to the method
      // being deallocated  
      guarantee(jmid == nullptr || *((Method**)jmid) == nullptr || *((Method**)jmid) != method, "jmethodid was not cleaned up: %p", (void*)jmid);
      // <<<
    }
    MetadataFactory::free_array<Method*>(loader_data, methods);
  }
}

Rebuilding the OpenJDK with the added guarantee and rerunning the reproducer quickly demonstrates that there is indeed an invariant violation. As expected, the guarantee fails, resulting in the following stack trace:

Stack: [0x000000016f708000,0x000000016f90b000],  sp=0x000000016f90a7b0,  free space=2057k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.dylib+0xa5def0]  VMError::report_and_die(int, char const*, char const*, char*, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long)+0x52c  (instanceKlass.cpp:551)
V  [libjvm.dylib+0xa5e64c]  VMError::report_and_die(Thread*, char const*, int, unsigned long, VMErrorType, char const*, char*)+0x0
V  [libjvm.dylib+0x2d9bb4]  print_error_for_unit_test(char const*, char const*, char*)+0x0
V  [libjvm.dylib+0x47da14]  InstanceKlass::deallocate_methods(ClassLoaderData*, Array<Method*>*)+0x14c
V  [libjvm.dylib+0x47dcfc]  InstanceKlass::deallocate_contents(ClassLoaderData*)+0x94
V  [libjvm.dylib+0x487b70]  InstanceKlass::purge_previous_version_list()+0xc4
V  [libjvm.dylib+0x25b08c]  ClassLoaderData::classes_do(void (*)(InstanceKlass*))+0x3c
V  [libjvm.dylib+0x25ed24]  ClassLoaderDataGraph::clean_deallocate_lists(bool)+0x7c
V  [libjvm.dylib+0x25ee08]  ClassLoaderDataGraph::walk_metadata_and_clean_metaspaces()+0x38
V  [libjvm.dylib+0xa64ad4]  VM_Operation::evaluate()+0xe4
V  [libjvm.dylib+0xa71458]  VMThread::evaluate_operation(VM_Operation*)+0xe4
V  [libjvm.dylib+0xa71c90]  VMThread::inner_execute(VM_Operation*)+0x28c
V  [libjvm.dylib+0xa7112c]  VMThread::run()+0xcc
V  [libjvm.dylib+0x9e7b5c]  Thread::call_run()+0xc8
V  [libjvm.dylib+0x8180e0]  thread_native_entry(Thread*)+0x118
C  [libsystem_pthread.dylib+0x6fa8]  _pthread_start+0x94
VM_Operation (0x0000000170252d60): CleanClassLoaderDataMetaspaces, mode: safepoint, requested by thread 0x000000011480a400

Down the rabbit hole

Hm, apparently, there is another code path that deallocates InstanceKlass contents (and transitively the contained methods) in addition to ClassLoaderData::unload(). It starts in ClassLoaderDataGraph and the method of interest is

void ClassLoaderDataGraph::clean_deallocate_lists(bool walk_previous_versions) {
  assert(SafepointSynchronize::is_at_safepoint(), "must only be called at safepoint");
  uint loaders_processed = 0;
  for (ClassLoaderData* cld = _head; cld != nullptr; cld = cld->next()) {
    // is_alive check will be necessary for concurrent class unloading.
>>> if (cld->is_alive()) { <<<<
      // clean metaspace
      if (walk_previous_versions) {
        cld->classes_do(InstanceKlass::purge_previous_versions);
      }
      cld->free_deallocate_list();
      loaders_processed++;
    }
  }
  log_debug(class, loader, data)("clean_deallocate_lists: loaders processed %u %s",
                                 loaders_processed, walk_previous_versions ? "walk_previous_versions" : "");
}

Yes! The methods are being deallocated for a class loader that is still alive. Therefore, the code taking care of all jmmethodids associated with this class loader is not called, and any deallocated method has the potential to leave a dangling pointer in the corresponding jmethodid value.

Let’s fix it!

Good, it seems like the fix is trivial. Let’s just do this

void InstanceKlass::deallocate_methods(ClassLoaderData* loader_data,
                                       Array<Method*>* methods) {
  if (methods != nullptr && methods != Universe::the_empty_method_array() &&
      !methods->is_shared()) {
    for (int i = 0; i < methods->length(); i++) {
      Method* method = methods->at(i);
      if (method == nullptr) continue;  // maybe null if error processing
      // Only want to delete methods that are not executing for RedefineClasses.
      // The previous version will point to them so they're not totally dangling
      assert (!method->on_stack(), "shouldn't be called with methods on stack");

      jmethodID jmid = method->find_jmethod_id_or_null();
      // Do the pointer maintenance before releasing the metadata, just in case
      // We need to make sure that jmethodID actually resolves to this method
      // - multiple redefined versions may share jmethodID slots and if a method
      //   has already been rewired to a newer version we could be removing reference
      //   to a still existing method instance
      if (jmid != nullptr && *((Method**)jmid) == method) {
        // dangling pointer; needs to be cleaned up
        *((Method**)jmid) = nullptr;
      }
      MetadataFactory::free_metadata(loader_data, method);
    }
    MetadataFactory::free_array<Method*>(loader_data, methods);
  }
}

And we are done. Honestly, I can’t believe the fix was so easy. I just had to update the jmethodID value to not point to Method instances that are known to be deallocated shortly.

What is the real cause?

As you probably expect, I left out a lot of wrong turns I took while investigating. Although not leading to the fix, they helped me understand the underlying cause, which is the class redefinition/retransformation. Do you remember InstanceKlass::purge_previous_versions from the failed guarantee stack trace? The ‘previous versions’ are older forms of redefined classes when some of the methods of those classes were running at the time they were redefined, and because of that, the JVM had to keep them around for a while. Then, from time to time, the JVM will check if it is possible to unload those class versions.

The tricky part is that the JVM will keep the jmethodID cache only in the ‘main’ class version, dynamically extending the cache to accommodate jmethodID values from obsolete but yet still active class versions. This can happen, e.g., when a class is redefined, the previous version is marked obsolete, but there are still some of its methods somewhere on the stack, and at that moment, we call GetStackTrace. New jmethodID values will be generated as required, and they will be placed in the jmethodID cache in the main version.

And here come the giggles - after an older class version has been purged, the jmethodID values are still present in the main class version cache but pointing to deallocated Method instances!

Ok, now the loop is closed, and I am pretty sure the proposed fix is correct and actually addresses the root cause.

There is still more to it

The way jmethodID values are stored for previous versions of a retransformed class also opens a whole new can of worms - requesting a jmethodID for an older method version may lead to cache resizing, which in turn will cause memory allocation. And remember, doing memory allocations in signal handlers is bad, mmkay? How is CPU and wallclock profiling generally done? Well, glad you asked - it’s by sending a signal to the JVM and letting the signal handler call AsyncGetCallTrace, which returns a stack trace comprised of, drumroll please, jmethodID values.

Sensible JVM profiler implementations try to avoid jmethodID cache resizes by eagerly requesting all the jmethodID values for all methods in the class upon class load. However, this strategy won’t work if the cache is resized because a previous method version jmethodID is requested.

Honestly, I don’t have a good solution for this yet. It might take some time to figure out how to address this issue.

Diagram time

For posterity, I am adding the schematic diagrams about how the jmethodID cache is organized for a redefined class.

Fig1: Overview of the jmethodID storage

Fig2: How the jmethodIDs are stored for a redefined class

Next steps

The next steps would be trying to create a JTREG test for the fix. I will have to look around for tests that involve class redefinitions with several versions of a redefined class. Additionally, I need to find other tests that trigger metaspace cleanup. This way, I can combine them to trigger the problematic behavior. However, if this turns out to be too complicated, I would rather propose the fix and aim to get it into the upcoming JDK updates as soon as possible.