Skip to content

Only remove a Kafka header before injecting it if it's already there - #13181

Open
dougqh wants to merge 3 commits into
masterfrom
dougqh/kafka-inject-dedup-probe
Open

dougqh wants to merge 3 commits into
masterfrom
dougqh/kafka-inject-dedup-probe

Conversation

@dougqh

@dougqh dougqh commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

What Does This Do

When Kafka producers inject trace context, each header is now removed first only if it's already on the record. Before, every injected header went through headers.remove(key), which iterates with an iterator it allocates, though on an outgoing record the header is usually not there yet. Now lastHeader(key), which loops by index and doesn't allocate, probes first.

Behavior doesn't change:

  • Duplicate keys still collapse. The Datadog and W3C codecs both write ot-baggage- headers (span baggage, the end-to-end start time). A second write still replaces the first, leaving one header.
  • Read-only headers still throw. add throws IllegalStateException as remove did, so the producer's retry with a writable copy of the record still runs.

Motivation

A JFR profile of kafka-producer-perf-test under the agent showed ArrayList.iterator() at 4.5% of allocation, most of it from this setter, which runs for every injected header on every send.

KafkaInjectBenchmark (new) injects a span into new RecordHeaders (JDK 17, 3 forks, per inject):

spanBaggage before ns after ns before B after B
false 358.1 ± 2.1 342.3 ± 4.2 1728 1536
true 504.8 ± 8.9 508.0 ± 9.7 2400 2144

That's 192 B less per send, and 256 B with span baggage, where the replacing write still removes. The machine was busy (load 7–15), so treat the times as approximate; they're flat or slightly better.

Additional Notes

  • Why not add-only: dropping the removal on records that start with no headers would have saved the scan too, but it would leave duplicate ot-baggage- headers, since two codecs write them.
  • The value copy stays. value.getBytes(UTF_8) still copies each value. Removing it needs a way for codecs to write header values straight into bytes.
  • Tests: TextMapInjectAdapterTest, in both client modules, covers adding, replacing, setting the same key twice and read-only headers. It passes against the old setter too.

🤖 Generated with Claude Code

dougqh and others added 3 commits November 3, 2026 10:30
Removing iterates the headers with an iterator it allocates, once per
injected header, though the header is usually not there yet. Probe with
lastHeader, which doesn't allocate, and remove only on a hit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dougqh dougqh added tag: performance Performance related changes type: refactoring inst: kafka Kafka instrumentation tag: ai generated Largely based on code generated by an AI or LLM labels Nov 3, 2026
@dd-octo-sts

dd-octo-sts Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.82 s 14.74 s [-0.1%; +1.2%] (no difference)
startup:insecure-bank:tracing:Agent 13.56 s 13.69 s [-1.6%; -0.3%] (maybe better)
startup:petclinic:appsec:Agent 17.25 s 17.12 s [-0.0%; +1.5%] (no difference)
startup:petclinic:iast:Agent 17.08 s 17.00 s [-0.4%; +1.4%] (no difference)
startup:petclinic:profiling:Agent 16.76 s 16.96 s [-2.3%; -0.1%] (maybe better)
startup:petclinic:sca:Agent 17.12 s 16.96 s [+0.1%; +1.8%] (maybe worse)
startup:petclinic:tracing:Agent 16.21 s 16.19 s [-0.9%; +1.1%] (no difference)

Commit: 722d664e · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@datadog-datadog-prod-us1

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 68.82% (+9.05%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 722d664 | Docs | Give us feedback!

@dougqh
dougqh marked this pull request as ready for review November 3, 2026 16:25
@dougqh
dougqh requested review from a team as code owners November 3, 2026 16:25
@dougqh
dougqh requested review from ValentinZakharov and jandro996 and removed request for a team November 3, 2026 16:25
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-03T16:27:34.929069Z 722d664 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@datadog-datadog-prod-us1 datadog-datadog-prod-us1 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

Both Kafka adapters retain duplicate-header replacement and read-only exceptions, preserving the producer’s retry with writable headers.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 722d664

@pr-commenter

pr-commenter Bot commented Nov 3, 2026

Copy link
Copy Markdown

Kafka / consumer-benchmark

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master dougqh/kafka-inject-dedup-probe
git_commit_date 1793717677 1793720031
git_commit_sha 82b7ad5 722d664
See matching parameters
Baseline Candidate
ci_job_date 1793722111 1793722111
ci_job_id 2207145043 2207145043
ci_pipeline_id 149823280 149823280
cpu_model Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
jdkVersion 11.0.31 11.0.31
jmhVersion 1.36 1.36
jvm /usr/lib/jvm/java-11-openjdk-amd64/bin/java /usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs -Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=15002 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=15002 -Dhttp.nonProxyHosts=localhost *.localhost
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-pic8vicl 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-pic8vicl 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
vmName OpenJDK 64-Bit Server VM OpenJDK 64-Bit Server VM
vmVersion 11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu 11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results
scenario Δ mean throughput
scenario:not-instrumented/KafkaConsumerBenchmark.benchConsume unsure
[+1273.221op/s; +14019.943op/s] or [+0.543%; +5.982%]
scenario:only-tracing-dsm-disabled-benchmarks/KafkaConsumerBenchmark.benchConsume same
scenario:only-tracing-dsm-enabled-benchmarks/KafkaConsumerBenchmark.benchConsume same

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

inst: kafka Kafka instrumentation tag: ai generated Largely based on code generated by an AI or LLM tag: performance Performance related changes type: refactoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant