Route OpenLineage over the Agent's Unix Domain Socket URL by pawel-big-lebowski · Pull Request #12408 · DataDog/dd-trace-java

pawel-big-lebowski · 2026-09-04T11:33:59Z

What Does This Do

Adds a socket-only code path to AbstractDatadogSparkListener.getAgentHttpUrl(): when the resolved Agent URL uses the unix:// scheme, it is passed through unchanged so the OpenLineage HTTP transport sends lineage over the Unix Domain Socket. Every other configuration keeps the existing http://host:port behavior, so http(s) setups are unaffected.

String agentUrl = Config.get().getAgentUrl();
if (agentUrl != null && agentUrl.startsWith("unix:")) {
  return agentUrl;
}
// ...unchanged http://host:port fallback

Motivation

The Spark DJM integration derives the OpenLineage transport URL from the Agent host/port (http://host:port). In deployments where the Agent is reachable only over a Unix Domain Socket — e.g. Spark on Kubernetes / EMR-on-EKS, where the node Agent exposes unix:///var/run/datadog/apm.socket — that TCP URL is unreachable, so OpenLineage delivery fails with connection-refused while APM traces (which already use the socket) succeed. This routes OpenLineage over the same socket.

Additional Notes

Requires OpenLineage 1.54+, which adds unix:// support to its Java HTTP transport — see Support Unix Domain Socket URLs (unix://) in the Java HTTP transport OpenLineage/OpenLineage#4920 (Support Unix Domain Socket URLs (unix://) in the Java HTTP transport OpenLineage/OpenLineage#4920).
The change is intentionally minimal and gated on the unix: scheme so no existing (TCP) code path changes behavior.

Jira ticket: SDO-169

The Spark integration builds the OpenLineage HTTP transport URL from the Agent host and port (http://host:port). When the Agent is reachable only over a Unix Domain Socket (e.g. on Kubernetes nodes), that TCP URL is unreachable and lineage delivery fails with connection-refused. Add a socket-only code path to getAgentHttpUrl(): if the resolved agent URL uses the unix:// scheme, pass it through so OpenLineage sends events over the socket (requires OpenLineage 1.54+, which added unix:// support to its HTTP transport). Every other configuration keeps the existing http://host:port behavior unchanged, so http(s) setups are unaffected. Signed-off-by: pawel.leszczynski <[email protected]>

dd-octo-sts · 2026-09-04T11:34:11Z

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 341f0ad40d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

datadog-prod-us1-6

More details

The new unix: branch returns the configured socket URL without a change. Other Agent URLs still use the existing host and port path.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 341f0ad · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

dd-octo-sts · 2026-09-04T12:02:43Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.02 s	13.94 s	[-0.3%; +1.5%] (no difference)
startup:insecure-bank:tracing:Agent	12.94 s	12.95 s	[-0.7%; +0.6%] (no difference)
startup:petclinic:appsec:Agent	17.57 s	16.73 s	[+0.6%; +9.4%] (maybe worse)
startup:petclinic:iast:Agent	17.35 s	17.47 s	[-1.7%; +0.3%] (no difference)
startup:petclinic:profiling:Agent	17.32 s	17.27 s	[-1.0%; +1.7%] (no difference)
startup:petclinic:sca:Agent	17.54 s	16.81 s	[-0.2%; +8.9%] (no difference)
startup:petclinic:tracing:Agent	16.60 s	16.24 s	[-2.1%; +6.4%] (no difference)

Commit: 341f0ad4 · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

pawel-big-lebowski · 2026-09-07T07:02:12Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-09-07T07:02:17Z

View all feedbacks in Devflow UI.

2026-09-07 07:02:16 UTC ℹ️ Start processing command /merge

2026-09-07 07:02:22 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).

2026-09-07 07:55:19 UTC ℹ️ MergeQueue: This merge request was merged

pawel-big-lebowski requested review from a team as code owners September 4, 2026 11:34

pawel-big-lebowski requested review from ygree and removed request for a team September 4, 2026 11:34

pawel-big-lebowski added type: bug fix Bug fix inst: apache spark Apache Spark instrumentation labels Sep 4, 2026

chatgpt-codex-connector Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread ...k-common/src/main/java/datadog/trace/instrumentation/spark/AbstractDatadogSparkListener.java

datadog-prod-us1-6 Bot reviewed Sep 4, 2026

View reviewed changes

mobuchowski approved these changes Sep 4, 2026

View reviewed changes

vandonr approved these changes Sep 4, 2026

View reviewed changes

gh-worker-dd-mergequeue-cf854d Bot merged commit d2c0e37 into master Sep 7, 2026
604 of 611 checks passed

gh-worker-dd-mergequeue-cf854d Bot deleted the pawel.leszczynski/spark-openlineage-unix-socket-url branch September 7, 2026 07:55

github-actions Bot added this to the 1.66.0 milestone Sep 7, 2026

Navigation Menu

Route OpenLineage over the Agent's Unix Domain Socket URL - #12408

Route OpenLineage over the Agent's Unix Domain Socket URL#12408
gh-worker-dd-mergequeue-cf854d[bot] merged 1 commit into
masterfrom
pawel.leszczynski/spark-openlineage-unix-socket-url

pawel-big-lebowski commented Sep 4, 2026 •

edited by atlassian Bot

Loading

Uh oh!

dd-octo-sts Bot commented Sep 4, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

datadog-prod-us1-6 Bot left a comment

Uh oh!

dd-octo-sts Bot commented Sep 4, 2026

Uh oh!

pawel-big-lebowski commented Sep 7, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Sep 7, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

pawel-big-lebowski commented Sep 4, 2026 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Uh oh!

dd-octo-sts Bot commented Sep 4, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

datadog-prod-us1-6 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

dd-octo-sts Bot commented Sep 4, 2026

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

pawel-big-lebowski commented Sep 7, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pawel-big-lebowski commented Sep 4, 2026 •

edited by atlassian Bot

Loading

gh-worker-devflow-routing-ef8351 Bot commented Sep 7, 2026 •

edited

Loading