Skip to content

Add staged connection diagnosis for opaque dial failures#1114

Draft
rossnelson wants to merge 2 commits into
mainfrom
connection-error-diagnosis
Draft

Add staged connection diagnosis for opaque dial failures#1114
rossnelson wants to merge 2 commits into
mainfrom
connection-error-diagnosis

Conversation

@rossnelson

Copy link
Copy Markdown

What was changed

Connection failures now produce a classified, actionable error with an inline staged diagnosis instead of a bare context deadline exceeded or an empty message.

When client.DialContext fails, the CLI probes the target address in stages (DNS → TCP → TLS) using the same TLS configuration the client would use, classifies the root cause, and suggests one concrete fix:

$ temporal workflow list --address foo.bar.tmprl.cloud:7233 --tls
Error: failed connecting to Temporal server at foo.bar.tmprl.cloud:7233: TLS handshake failed: server requires client certificate (mTLS)

  Connecting to foo.bar.tmprl.cloud:7233
    ✓ DNS resolved (3 addresses)
    ✓ TCP connection established
    ✗ TLS handshake failed: server requires mTLS, no valid client certificate was provided

  This looks like a Temporal Cloud endpoint secured with mTLS. Provide client certificates:

    temporal workflow list \
        --address foo.bar.tmprl.cloud:7233 \
        --tls \
        --tls-cert-path YourCert.pem \
        --tls-key-path YourKey.pem

  Or configure once:

    temporal config set --prop tls.client_cert_path YourCert.pem
    temporal config set --prop tls.client_key_path YourKey.pem

Other classified cases: connection refused (with a temporal server start-dev hint on localhost:7233), DNS failure (naming the config profile if the address came from one), TLS↔plaintext mismatch in either direction (detected behaviorally, including an opportunistic anonymous TLS handshake when the client is plaintext), untrusted/mismatched server certificates, rejected API keys, unreadable TLS cert files (caught pre-dial, naming the exact path), and black-holed connections.

Design notes:

  • The probe runs only after a dial has already failed — zero cost on the happy path — and is capped at 3s (never exceeding --client-connect-timeout when set). TEMPORAL_CLI_DISABLE_CONNECT_DIAGNOSIS=1 opts out; interrupts skip it.
  • The rich text is the error's Error(), so the default Error: printing, JSON mode (no ANSI — rendering happens while color.NoColor is set), and the test harness all work unchanged. Unwrap() preserves the original error for errors.Is/As.
  • The first line embeds the familiar short cause (connection refused, context deadline exceeded, …) so scripts grepping stderr keep matching.
  • TLS misclassification degrades gracefully: unmatched failures still render the raw error in the ✗ stage, just without a suggestion. The Go alert strings the classifier depends on are pinned by unit tests against real crypto/tls servers.

Two adjacent paper cuts fixed along the way:

  • The descriptive causes the CLI already attaches via context.WithTimeoutCause (command timed out after 5s) were discarded; they now survive into the error output.
  • The default fail handler no longer masks --command-timeout expiry as program interrupted.

cliext.ClientOptionsBuilder gains read-only output fields (ResolvedAddress, ResolvedAddressSource, ResolvedProfileName, HasAPIKey) so the suggestion engine knows where the address came from, following the existing PayloadCodec output-field pattern.

Why?

Fixes #224 — a server requiring mTLS without client certs produced an empty error.
Fixes #851 — connection failures surfaced as a bare context deadline exceeded with no indication it was a connection problem (tctl gave a useful error).

Both trace to the same gap: the SDK's dial is lazy, so failures surface from the eager GetSystemInfo check and were returned bare from dialClient, while SDK connection logs go to a nop logger by default (--log-level defaults to never).

Checklist

  1. Closes [Feature Request] Show actual error message when mTLS certs are missing #224

  2. Closes Temporal cli gives context deadline exceeded. Tctl does not #851

  3. How was this tested:

  • New unit tests (connectdiag_test.go) run the classifier against real in-process listeners: a RequireAndVerifyClientCert TLS server, a plaintext server answered with TLS, a TLS server dialed in plaintext, refused ports, .invalid DNS, and an untrusted CA. These also act as canaries for the Go TLS alert strings.
  • New harness tests (client_test.go) assert the full rendered error for each scenario end-to-end through temporalcli.Execute, including JSON mode (no ANSI, empty stdout), the disable env var, unreadable cert files, and --command-timeout cause survival.
  • Manual verification against a live dev server: happy path unchanged (~0.1s), TLS-vs-plaintext mismatch, refused port, and DNS failure all render as designed.
  1. Any docs updates needed?
  • Possibly a mention of TEMPORAL_CLI_DISABLE_CONNECT_DIAGNOSIS if we document escape-hatch env vars; open to renaming or dropping it.

Notes for reviewers

  • No new command surface and no changes to commands.yaml; this is all in the dial path.
  • Follow-up ideas intentionally left out: a structured error object on stdout in -o json mode, a standalone temporal doctor command reusing the same prober, and surfacing the diagnosis in operator cluster health.
  • The standalone cliext module does not currently compile on its own at main (its go.mod pins envconfig v1.0.0, predating ClientConfigProfile.Authority); it builds through the root module's replace. Unrelated to this PR but worth a follow-up bump.

Connection failures now render a classified error with an inline
DNS/TCP/TLS diagnosis and one suggested fix, instead of a bare
'context deadline exceeded' or an empty message.

Fixes #224
Fixes #851
@CLAassistant

CLAassistant commented Jul 8, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Covers classifyGRPCError, connectSummary's grep-compatibility contract,
and an end-to-end case where the failing address comes from a config
profile (exercising the new cliext builder metadata).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Temporal cli gives context deadline exceeded. Tctl does not [Feature Request] Show actual error message when mTLS certs are missing

2 participants