Skip to content

Build injected header values as ASCII bytes, starting with traceparent - #13182

Draft
dougqh wants to merge 9 commits into
dougqh/kafka-inject-dedup-probefrom
dougqh/ascii-value-builder
Draft

dougqh wants to merge 9 commits into
dougqh/kafka-inject-dedup-probefrom
dougqh/ascii-value-builder

Conversation

@dougqh

@dougqh dougqh commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

Injected propagation header values can now be built directly as ASCII bytes, so carriers that store bytes (Kafka headers) take them without a String in between. The W3C traceparent and the Datadog trace id, parent id, sampling priority and end-to-end start time are now built this way.

  • AsciiValueBuilder (components/context) appends chars, ASCII text, fixed-width hex and decimal straight into a byte[]. Sized to the value's length, toBytes() returns its own array without copying. toString() gives the String form.
  • AsciiDecimal writes a long as decimal digits into an array sized exactly, two digits per division as Long.toString does. A single-number header uses it directly, with no builder.
  • CarrierSetter.setAscii has two new default methods, one taking a builder and one taking byte[]. Both set the value's String form, so every existing setter keeps working unchanged. The Kafka setters override both to add the bytes to the header as they are.

Before, each value went through a String, often built from hex or decimal Strings and a StringBuilder, and the Kafka setter then encoded it again with getBytes. Now the bytes Kafka keeps are the only array.

Motivation

In a profile of kafka-producer-perf-test under the agent, writing header values was about 15% of the sending thread's CPU, and byte-array copies were about 22% of allocation. Most of that came from building header values as Strings and then encoding them to bytes.

KafkaInjectBenchmark injects into new RecordHeaders (JDK 17, 3 forks, per inject). Its shape parameter sets which span each inject is for: reused is one span every time (a best case for values cached on its ids), newTrace is a new trace per inject (a producer outside any request), and fanOut is a new child span of one trace per inject (a request sending many messages).

The traceparent step, measured on a reused span without span baggage:

before after
ns 337.2 ± 7.1 329.4 ± 15.7
B 1536 1280

And the Datadog headers on top of it:

shape baggage before ns after ns before B after B
reused no 322.8 ± 16.2 327.5 ± 5.8 1280 1179
reused yes 442.3 ± 5.4 485.7 ± 26.0 1952 1840
newTrace no 743.1 ± 6.6 742.7 ± 14.0 2414 2296
newTrace yes 932.5 ± 63.6 894.2 ± 12.9 3291 3096
fanOut no 465.6 ± 6.5 472.6 ± 5.0 1896 1784
fanOut yes 659.7 ± 2.3 660.1 ± 8.7 2816 2704

Together that's about 360 B less per send, with times unchanged within noise.

Additional Notes

  • No byte cache for the trace id. The String path caches the trace id's decimal form for the trace; the byte path formats it on each inject. A per-trace byte cache wouldn't help: Kafka keeps and exposes each header's array, so every header needs its own copy anyway, and copying 19 cached bytes costs about as much as writing 19 digits (compare fanOut with newTrace). String carriers keep the existing cache.
  • Single-use values: the carrier may keep the bytes it's given, so a builder must not be appended to after toBytes(), and an array passed to setAscii must not be reused.
  • ASCII only: propagation header values are ASCII, and the builder throws on a non-ASCII char rather than encode it wrongly. origin stays a String, since it can be user-supplied.
  • Other carriers: carriers that store Strings get the same values through the default methods, as one String built from the bytes.
  • Not yet converted: tracestate and x-datadog-tags, which go through the propagation tags codecs.
  • Perf review rubric: adds universal check Track 3rd party licenses #10, hidden cost at an API boundary: a per-item conversion to fit an API, or a convenience call that allocates or scans, on the line reviewers tend to treat as outside their control. It also adds Java entry J14 for String↔byte[] at carrier boundaries, with the Kafka patterns measured here and in Skip context extraction for Kafka records without headers #13177 and Only remove a Kafka header before injecting it if it's already there #13181. The rubric lists AsciiValueBuilder and AsciiDecimal as the fix, so they land together.

Stacked on #13181.

🤖 Generated with Claude Code

dougqh and others added 3 commits November 3, 2026 11:41
CarrierSetter gains setAscii, which by default sets the value's String form,
so a setter for a carrier that stores bytes can take them as built.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It was formatted through hex strings and a StringBuilder into a String,
then encoded again by byte carriers such as Kafka headers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dougqh dougqh added tag: performance Performance related changes type: refactoring comp: context propagation Trace context propagation inst: kafka Kafka instrumentation tag: ai generated Largely based on code generated by an AI or LLM labels Nov 3, 2026
A per-item conversion to fit an API, or a convenience call that allocates or
scans, sits on the line reviewers treat as outside their control. Add a
universal check to read the callee, and a Java entry for String/byte[]
conversions at carrier boundaries such as Kafka headers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dd-octo-sts

dd-octo-sts Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.97 s 14.00 s [-1.1%; +0.6%] (no difference)
startup:insecure-bank:tracing:Agent 12.92 s 13.08 s [-1.9%; -0.6%] (maybe better)
startup:petclinic:appsec:Agent 17.12 s 17.00 s [-0.0%; +1.5%] (no difference)
startup:petclinic:iast:Agent 16.91 s 17.05 s [-1.7%; +0.1%] (no difference)
startup:petclinic:profiling:Agent 16.85 s 16.61 s [+0.3%; +2.5%] (maybe worse)
startup:petclinic:sca:Agent 16.62 s 16.97 s [-6.5%; +2.4%] (no difference)
startup:petclinic:tracing:Agent 16.07 s 16.27 s [-2.2%; -0.3%] (maybe better)

Commit: 73dee017 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@datadog-datadog-prod-us1

This comment has been minimized.

dougqh and others added 5 commits November 3, 2026 13:04
Two digits per division, as Long.toString does, so a 19-digit id costs no
more than the JDK's formatting, and AsciiDecimal returns arrays sized to the
digits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A single-number value needs no builder: the setter takes its bytes, which
Kafka headers keep as they are.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e reused span

A reused span lets values cached on its ids be built once, which flatters
caching.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: context propagation Trace context propagation inst: kafka Kafka instrumentation tag: ai generated Largely based on code generated by an AI or LLM tag: performance Performance related changes type: refactoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant