Fix memory leak with FastThrow optimization by jpbempel · Pull Request #12254 · DataDog/dd-trace-java

jpbempel · 2026-08-21T11:53:34Z

What Does This Do

The fix is to clear the snapshot list after sending the snapshot and tagging the span. We also add a cap to the number of snapshots (256) added to the ThrowableState object.
we are filtering out also exceptions that have no stacktrace (typical from FastThrow ones).

Motivation

HotSpot JVM has a special optimization when some special exceptions are (implicitly) thrown like NullPointerException. The JIT will replace a throw with a newly created Exception that walk the stack to produce the stacktrace by a throw with a singleton instance of the exception without stacktrace.
This singleton exception is kept forever by the JVm contrary to usual way to throw exception to create the object at location of the exception.
Therefore the map snapshotsByThrowable in ExceptionProbeManager which maps Exception instance to ThrowableState object keep an instance forever with the along snapshots collected.
At some point the number of snapshots kept can lead to OOME.

Additional Notes

Contributor Checklist

Format the title according to the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any other useful labels
Avoid using close, fix, or any linking keywords when referencing an issue
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, migration, or deletion
Update public documentation with any new configuration flags or behaviors
Once approved, use merge queue to merge the PR

Jira ticket: DEBUG-6056

HotSpot JVM has a special optimization when some special exceptions are (implicitly) thrown like NullPointerException. The JIT will replace a throw with a newly created Exception that walk the stack to produce the stacktrace by a throw with a singleton instance of the exception without stacktrace. This singleton exception is kept forever by the JVm contrary to usual way to throw exception to create the object at location of the exception. Therefore the map snapshotsByThrowable in ExceptionProbeManager which maps Exception instance to ThrowableState object keep an instance forever with the along snapshots collected. At some point the number of snapshots kept can lead to OOME. The fix is to clear the snapshot list after sending the snapshot and tagging the span. We also add a cap to the number of snapshots (256) added to the ThrowableState object.

dd-octo-sts · 2026-08-21T11:53:48Z

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 15c924f41d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

datadog-prod-us1-4

Clearing the shared list leaves a reused FastThrow instance with an empty state, so later error spans lose snapshot and replay tags. The same list also has unsafe concurrent access, and its new check permits more than 256 entries.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 15c924f · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

datadog-prod-us1-4 · 2026-08-21T12:08:07Z

🎯 Code Coverage (details)
• Patch Coverage: 58.33%
• Overall Coverage: 58.41% (-0.21%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 4ad5cf2 | Docs | View more details | Give us feedback!}

pr-commenter · 2026-08-21T12:13:54Z

Debugger benchmarks

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
ci_job_date	1787565171	1787565497
end_time	2026-08-24T09:54:18	2026-08-24T09:59:45
git_branch	master	jpbempel/fix-er-fastthrow
git_commit_sha	`bd49cfc`	`4ad5cf2`
start_time	2026-08-24T09:52:52	2026-08-24T09:58:18

See matching parameters

	Baseline	Candidate
ci_job_id	1976023331	1976023331
ci_pipeline_id	132700191	132700191
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
git_commit_date	1787563654	1787563654

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 5 unstable metrics.

See unchanged results

scenario	Δ mean agg_http_req_duration_min	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p75	Δ mean agg_http_req_duration_p99	Δ mean throughput
scenario:noprobe	unstable [-16.331µs; +20.173µs] or [-5.257%; +6.494%]	unstable [-20.511µs; +38.712µs] or [-5.986%; +11.298%]	unstable [-28.565µs; +47.943µs] or [-8.015%; +13.453%]	unstable [-99.740µs; +82.561µs] or [-8.062%; +6.673%]	same
scenario:basic	same	same	same	unstable [+8.805µs; +246.088µs] or [+0.838%; +23.425%]	same
scenario:loop	same	same	same	same	same

Request duration reports for reports

gantt
    title reports - request duration [CI 0.99] : candidate=None, baseline=None
    dateFormat X
    axisFormat %s
section baseline
noprobe (342.635 µs) : 315, 370
.   : milestone, 343,
basic (324.05 µs) : 318, 330
.   : milestone, 324,
loop (8.093 ms) : 8031, 8156
.   : milestone, 8093,
section candidate
noprobe (351.736 µs) : 323, 380
.   : milestone, 352,
basic (325.228 µs) : 319, 332
.   : milestone, 325,
loop (8.093 ms) : 8030, 8156
.   : milestone, 8093,

baseline results

Scenario	Request median duration [CI 0.99]
noprobe	342.635 µs [315.021 µs, 370.25 µs]
basic	324.05 µs [318.395 µs, 329.706 µs]
loop	8.093 ms [8.031 ms, 8.156 ms]

candidate results

Scenario	Request median duration [CI 0.99]
noprobe	351.736 µs [323.084 µs, 380.388 µs]
basic	325.228 µs [318.847 µs, 331.609 µs]
loop	8.093 ms [8.03 ms, 8.156 ms]

dd-octo-sts · 2026-08-21T12:17:47Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.81 s	14.63 s	[+0.4%; +2.1%] (maybe worse)
startup:insecure-bank:tracing:Agent	13.56 s	13.72 s	[-1.9%; -0.4%] (maybe better)
startup:petclinic:appsec:Agent	16.88 s	16.79 s	[-0.6%; +1.6%] (no difference)
startup:petclinic:iast:Agent	16.95 s	16.97 s	[-0.9%; +0.7%] (no difference)
startup:petclinic:profiling:Agent	16.75 s	16.90 s	[-2.3%; +0.6%] (no difference)
startup:petclinic:sca:Agent	16.27 s	16.06 s	[-4.8%; +7.3%] (unstable)
startup:petclinic:tracing:Agent	15.73 s	16.09 s	[-6.3%; +1.8%] (no difference)

Commit: 4ad5cf26 · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

dudikeleti

It’s worth looking at the bot comments.

jpbempel · 2026-08-24T11:50:22Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-08-24T11:50:26Z

View all feedbacks in Devflow UI.

2026-08-24 11:50:26 UTC ℹ️ Start processing command /merge

2026-08-24 11:50:32 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 2h (p90).

2026-08-24 12:47:19 UTC ℹ️ MergeQueue: This merge request was merged

jpbempel requested a review from a team as a code owner August 21, 2026 11:53

jpbempel requested review from dudikeleti and removed request for a team August 21, 2026 11:53

jpbempel added comp: debugger Dynamic Instrumentation type: bug fix Bug fix labels Aug 21, 2026

chatgpt-codex-connector Bot reviewed Aug 21, 2026

View reviewed changes

Comment thread ...t/agent-debugger/src/main/java/com/datadog/debugger/exception/AbstractExceptionDebugger.java Outdated

Comment thread ...t/agent-debugger/src/main/java/com/datadog/debugger/exception/AbstractExceptionDebugger.java Outdated

datadog-prod-us1-4 Bot reviewed Aug 21, 2026

View reviewed changes

dudikeleti approved these changes Aug 21, 2026

View reviewed changes

fix issue with empty snapshot list

a882216

skrcode mentioned this pull request Aug 22, 2026

JAIPilot campaign evaluation: DataDog/dd-trace-java #12254 skrcode/dd-trace-java#1

Open

java-dependency-upgrade-fixer Bot mentioned this pull request Aug 22, 2026

Extract duplicated snapshot-cap literal in ExceptionProbeManager skrcode/dd-trace-java#2

Draft

filter out capturing snapshot for exception with no stacktrace

4ad5cf2

gh-worker-dd-mergequeue-cf854d Bot merged commit 17927ed into master Aug 24, 2026
801 of 803 checks passed

gh-worker-dd-mergequeue-cf854d Bot deleted the jpbempel/fix-er-fastthrow branch August 24, 2026 12:47

github-actions Bot added this to the 1.66.0 milestone Aug 24, 2026

Navigation Menu

Uh oh!

Fix memory leak with FastThrow optimization - #12254

Fix memory leak with FastThrow optimization#12254
gh-worker-dd-mergequeue-cf854d[bot] merged 3 commits into
masterfrom
jpbempel/fix-er-fastthrow

jpbempel commented Aug 21, 2026 •

edited

Loading

Uh oh!

dd-octo-sts Bot commented Aug 21, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

datadog-prod-us1-4 Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

datadog-prod-us1-4 Bot commented Aug 21, 2026 •

edited by datadog-prod-us1-5 Bot

Loading

Uh oh!

pr-commenter Bot commented Aug 21, 2026 •

edited

Loading

Uh oh!

dd-octo-sts Bot commented Aug 21, 2026 •

edited

Loading

Uh oh!

dudikeleti left a comment

Uh oh!

jpbempel commented Aug 24, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Aug 24, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jpbempel commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

dd-octo-sts Bot commented Aug 21, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

datadog-prod-us1-4 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

datadog-prod-us1-4 Bot commented Aug 21, 2026 • edited by datadog-prod-us1-5 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pr-commenter Bot commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Debugger benchmarks

Parameters

Summary

Uh oh!

dd-octo-sts Bot commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

dudikeleti left a comment

Choose a reason for hiding this comment

Uh oh!

jpbempel commented Aug 24, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jpbempel commented Aug 21, 2026 •

edited

Loading

datadog-prod-us1-4 Bot commented Aug 21, 2026 •

edited by datadog-prod-us1-5 Bot

Loading

pr-commenter Bot commented Aug 21, 2026 •

edited

Loading

dd-octo-sts Bot commented Aug 21, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Aug 24, 2026 •

edited

Loading