tailnet: honor debug MTU for nested paths

[linonetwo]

Summary

• size padded discovery Ping/Pong packets from the explicitly configured TS_DEBUG_MTU
• when that packet budget is below the IPv6 netstack link minimum, clamp outbound TCP SYN/SYN-ACK MSS to the same budget
• keep the default 1280-byte inner MTU, 1310-byte discovery packet, and normal TCP behavior unchanged
• preserve valid TCP checksums after rewriting the MSS option

Stacked dependency

This draft is stacked on #128 (wgengine/netstack: keep IPv6 link MTU at least 1280). The first commit shown here is #128; the new behavior is isolated in commit 688ad269e. I will rebase this branch onto main after #128 is merged or otherwise resolved.

The dependency is intentional: an inner packet budget can be below 1280 on a nested encrypted path, but the dual-stack gVisor link endpoint still cannot advertise an IPv6 MTU below RFC 8200's minimum.

Deployment context

We found this in a Coder v2.34.0 Kubernetes deployment where the cluster nodes and user workstations are already connected by an encrypted Tailnet. Kubernetes uses host-gw Pod routing over the nodes' Tailscale interfaces, and the Coder CLI/agent then creates another userspace WireGuard tailnet inside that path.

Identifying names and addresses are omitted. The relevant topology is:

Coder desktop CLI
  -> existing host Tailnet / WireGuard path
  -> Kubernetes node Tailscale interface (MTU 1280)
  -> host-gw Pod network
  -> Coder workspace agent
  -> inner Coder userspace WireGuard/tailnet

The outer Tailnet peers were direct P2P. The inner Coder connection was not stable:

• the fixed 1310-byte discovery packet could not reliably traverse the remaining nested packet budget, so Coder selected DERP even though the outer Tailnet was direct;
• when a direct path was found, SSH/VS Code repeatedly reset or had extremely low TCP throughput because netstack advertised an MSS derived from 1280 rather than the smaller nested budget;
• setting TS_DEBUG_MTU=1182 alone expressed the correct empirical packet budget, but before wgengine/netstack: keep IPv6 link MTU at least 1280 #128 it also made the IPv6-capable gVisor link endpoint advertise the invalid value 1182.

With the logical link held at 1280, discovery sized to 1182 + 30 = 1212 bytes, and TCP MSS limited to 1182 - 40 - 20 = 1122, the same deployment established stable direct P2P over the existing private Tailnet. Repeated coder ping measurements were approximately 4-7 ms instead of a DERP/reconnect loop. The value 1182 is deployment-specific and is not introduced as a default by this PR.

This also matches the fragmentation and throughput behavior previously investigated in coder/coder#15523 for Coder running inside an outer VPN.

Behavior

• TS_DEBUG_MTU unset: unchanged (1280-byte inner MTU, 1310-byte padded discovery packet, no MSS rewrite).
• TS_DEBUG_MTU=1182: netstack link remains 1280 via wgengine/netstack: keep IPv6 link MTU at least 1280 #128; discovery packet is 1212 bytes; TCP MSS is capped at 1122.
• TS_DEBUG_MTU=1280: unchanged normal behavior.
• explicit values above 1280: discovery validates the configured larger packet size; MSS needs no additional rewrite because the netstack link already uses that MTU.
• discovery packet length is capped at the maximum IPv4 UDP payload size.

Both CLI and agent should use the same explicit value. This change controls discovery probes and TCP segmentation; applications sending large UDP datagrams still need to respect their path budget independently.

Why reuse TS_DEBUG_MTU?

This keeps the change opt-in and avoids introducing a deployment-specific default or a Coder-only environment variable into the networking fork. More importantly, it makes the existing debug knob internally consistent: the configured MTU now affects the path probe and the TCP packets generated for that path, while #128 preserves the separate IPv6 link-layer invariant.

If maintainers prefer a new supported configuration name instead of extending a debug knob, the implementation can be adjusted without changing the packet-sizing mechanics demonstrated here.

Tests

• go test ./disco ./wgengine/netstack -count=1

Coverage includes default, 1182, 1280, 1420, UDP payload capping, TCP-only filtering, SYN detection, MSS no-op behavior, and checksum validity after rewriting.