Skip to content

🪞 12053 - Fix concurrent span event recording in OTel shim - #12301

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 5 commits into
masterfrom
community-pr-12053
Aug 26, 2026
Merged

gh-worker-dd-mergequeue-cf854d[bot] merged 5 commits into
masterfrom
community-pr-12053

Conversation

@mcculls

@mcculls mcculls commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

This PR mirrors the changes from the original community contribution to enable CI testing with maintainer privileges.

Original PR: #12053
Original Author: @Meemaw
Original Branch: Meemaw/dd-trace-java:fix-otel-span-events-concurrency

Closes #12053


This is an automated mirror created to run CI checks. See tooling/mirror-community-pull-request.sh for details.

@mcculls mcculls added the type: bug fix Bug fix label Aug 26, 2026
@mcculls
mcculls requested a review from a team as a code owner August 26, 2026 09:32
@mcculls mcculls added the tag: community Community contribution label Aug 26, 2026
@mcculls
mcculls requested a review from a team as a code owner August 26, 2026 09:32
@mcculls mcculls added the inst: opentelemetry OpenTelemetry instrumentation label Aug 26, 2026
@mcculls
mcculls requested review from jordan-wong and mhlidd and removed request for a team August 26, 2026 09:32
@dd-octo-sts dd-octo-sts Bot added the tag: ai generated Largely based on code generated by an AI or LLM label Aug 26, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 17f0ff1ae6

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@datadog-prod-us1-3 datadog-prod-us1-3 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: PASS

More details

The event list uses one monitor for each write and for the copy before serialization, so concurrent event recording cannot corrupt the serialized list.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Datadog Autotest · Commit 17f0ff1 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

@datadog-prod-us1-3

datadog-prod-us1-3 Bot commented Aug 26, 2026

Copy link
Copy Markdown

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 69.74% (+10.95%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 25583ae | Docs | View more details | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.76 s 14.75 s [-0.6%; +0.7%] (no difference)
startup:insecure-bank:tracing:Agent 13.63 s 13.63 s [-0.9%; +0.9%] (no difference)
startup:petclinic:appsec:Agent 16.82 s 16.69 s [-0.2%; +1.7%] (no difference)
startup:petclinic:iast:Agent 16.90 s 16.92 s [-1.0%; +0.8%] (no difference)
startup:petclinic:profiling:Agent 16.66 s 16.76 s [-1.6%; +0.4%] (no difference)
startup:petclinic:sca:Agent 16.76 s 15.97 s [+0.7%; +9.2%] (maybe worse)
startup:petclinic:tracing:Agent 16.21 s 16.14 s [-0.3%; +1.2%] (no difference)

Commit: 25583ae2 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

Meemaw and others added 4 commits August 26, 2026 14:59
OtelSpan stored span events in a plain ArrayList that was mutated by
addEvent()/recordException() without synchronization and iterated at
span finish. When events are recorded from multiple threads (e.g.
GraphQL DataLoaders running on virtual threads) while the span is being
finished, the backing list gets corrupted, leaving a null hole that
triggers a NullPointerException in OtelSpanEvent.toTag. That NPE escapes
span finish (DDSpan.finishAndAddToTrace) and can surface to the
application as a request failure.

Guard all event-list mutations by synchronizing on the span, and copy
the list under lock at finish so serialization iterates a private
snapshot. The events field is volatile so onSpanFinished keeps a
lock-free fast path (single volatile read) for the common case of a
span with no events, avoiding overhead on the hot finish path.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…ock instead of copying.

This avoids the extra list allocation while still bounding the lock to a quick pointer swap.
@mcculls
mcculls force-pushed the community-pr-12053 branch from 7d0dee0 to ccabd93 Compare August 26, 2026 14:00
@mcculls
mcculls requested review from jordan-wong and removed request for jordan-wong August 26, 2026 16:10
@mcculls
mcculls added this pull request to the merge queue Aug 26, 2026
@dd-octo-sts

dd-octo-sts Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Aug 26, 2026

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-08-26 17:39:15 UTC ℹ️ Start processing command /merge


2026-08-26 17:39:20 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).


2026-08-26 18:39:32 UTC ℹ️ MergeQueue: This merge request was merged

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Aug 26, 2026
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit be059cb into master Aug 26, 2026
601 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the community-pr-12053 branch August 26, 2026 18:39
@github-actions github-actions Bot added this to the 1.66.0 milestone Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

inst: opentelemetry OpenTelemetry instrumentation tag: ai generated Largely based on code generated by an AI or LLM tag: community Community contribution type: bug fix Bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants