Repository navigation
Conversation
When a TypeError, ReferenceError or RangeError raised in the CLI's own files ends or fails a command, send one Sentry envelope to Checkly's error ingest. Errors in user code, dependencies, failed checks, API and network errors are not reported. Errors are noted where they are thrown (BaseCommand.catch), printed (CommandStyle.longError and the sites that print only err.message) or emitted by the check runner, and sent from BaseCommand.finally, so commands that print their errors and exit are covered too. Wrapped errors are found through their cause chain. The report carries no paths outside the CLI, credentials, arguments, environment, host name or private package names. Only release builds report: the release workflow writes the DSN from the CHECKLY_CLI_ERROR_DSN repository variable. CHECKLY_NO_ERROR_REPORTING=1 or DO_NOT_TRACK=1 turns it off.
…report error-reporting.ts did four jobs in one file. It is now a folder: classify.ts decides what is a CLI bug, scrub.ts removes what names the user, event.ts builds the Sentry event and report.ts holds the run's state and sends. isCliBug now reads the stack with Sentry's Node parser, the same one the event uses, instead of its own regex, so classification and the reported in_app frames cannot disagree. Frames of code run through eval or new Function are traced to the file that ran it. Because the parser is loaded only when needed, noteCliError keeps a few candidate errors and the check runs when the command ends.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Affected Components
Notes for the Reviewer
The CLI had a
// TODO: we can add Sentry hereinBaseCommand.catch. This PR reports crashes caused by bugs in the CLI's own code to Checkly's own error tracking (the RUM ingest atrum.checklyhq.com, which speaks the Sentry envelope protocol). Nothing goes to sentry.io.What is reported. Only a
TypeError,ReferenceErrororRangeErrorwhose innermost stack frame is in the CLI's own files (not itsnode_modules), at most one per run. Errors in user code, dependencies, failed checks, API and network errors, and Node'sERR_INVALID_URLfor a bad user URL are not. Wrapped errors (new Error('…', { cause }), e.g. while loading a check file) are found through the cause chain, and only the inner error is sent. A user's invalid configuration that makes a construct crash is reported on purpose: bad config should produce a clear message, not a crash.Where it hooks in. There is no single place that sees every crash:
deploy,test,triggerand many other branches print the error and exit without rethrowing. So errors are noted where they are thrown (BaseCommand.catch), printed (CommandStyle.longError, plus the 7 sites that print onlyerr.message) or emitted byAbstractCheckRunner, and sent fromBaseCommand.finally, which oclif runs on every path before its handler prints and callsprocess.exit.What a report contains. Error type, scrubbed message and stack; CLI version, command id, CLI mode; OS, arch, runtime version. Scrubbing: frames outside the CLI become
<project file>; packages the CLI does not depend on directly become<dependency>; paths under cwd/home become<project file>/<home file>(segment boundaries, forward slashes,file://URLs, case-insensitive on Windows);cu_keys, Bearer/Basic tokens, URL userinfo,*token=/*secret=/*api_key=-style values and the values ofCHECKLY_API_KEY/CHECKLY_ACCOUNT_ID. No server name, user, argv or environment.Release only. The DSN is not in the repo. The release workflow writes it from the repository variable
CHECKLY_CLI_ERROR_DSNtopackages/cli/error-reporting.json(gitignored, infiles) for prerelease and release builds. Canary and local builds (0.0.0*,0.0.1-dev) never report. Until the variable is set, nothing is reported, so this is safe to merge first. The key is write-only and can be revoked in the ingest, as with every CLI that ships a crash-report key (Wrangler, Shopify, flyctl).Opt-out.
CHECKLY_NO_ERROR_REPORTING=1orDO_NOT_TRACK=1. Documented in the README ("Error reporting") andCLAUDE.md.Cost to the user.
@sentry/coreis loaded with dynamicimport()only when a report is sent. The report waits at most 3 s (timeoutplusAbortSignal.timeout) and never throws, so exit codes are unchanged. For an error that ends the command throughcatch, the report is sent before oclif prints the error, because oclif exits right after printing; this delay only happens on a real CLI bug.Not covered (follow-ups). Crashes in un-awaited callbacks and unhandled rejections (needs a
process.on('unhandledRejection')handler, which changes how the CLI exits).Code layout.
src/services/error-reporting/:classify.ts(is it a CLI bug; reads stacks with Sentry's Node parser, the one the event uses, and tracesevalframes to the file that ran them),scrub.ts(removes what names the user),event.ts(builds the event),report.ts(run state, DSN, sending).Testing. 24 unit tests in
error-reporting/__tests__/error-reporting.spec.tsusing a real local HTTP server and real.mjsfiles in an install directory with a space in its name (file-URL stacks with%20, as on Windows). Each new test fails with its bug restored. Full suite: 239 files, 4586 passed / 7 skipped; lint,tsc,test:typesclean. Manually: the packed tarball installed undermy installwith a bug injected indist/rest/checks.js→checks listexits 1, prints the error, sends one envelope; withDO_NOT_TRACK=1none; a thrown bug inwhoamialso sends one. The envelope was parsed by the ingest's own Gosentry.ParseEnvelope, and axios's User-Agent reachesrum.checklyhq.comthrough Cloudflare (401 invalid sentry_keywith a fake key). Windows is only covered by CI.Open question for Simo: keep
@sentry/coreor replace it? It adds 14.2 MB to every install (@sentry/core5.4 MB +@sentry/conventions8.8 MB). Todaycheckly@10.1.0installs 55 MB in 211 packages, so +26 %. We use onlyuuid4, a stack-line parser, DSN parsing and envelope serialization, which is about 40 lines withcrypto.randomUUID.New Dependency Submission
@sentry/core@^11.10.0(supports Node >=22.12, published 2026-10-28; one dependency,@sentry/conventions; no OpenTelemetry). It builds and serializes Sentry envelopes and parses Node stack lines, so the events match what the ingest's Sentry parser expects.@sentry/nodewas not used: it would auto-instrument the process and pull in OpenTelemetry. Whether to keep it at all is the open question above.