The Worker reads the current Redis link record, checks that it may open on the requested host and is active and unexpired, applies A/B and UTM rules, and returns a 302 with browser caching disabled. Analytics never decides whether a visitor reaches the destination:
- The link lookup is the only failure that answers 503. It has one quick retry and a three-second timeout per attempt.
- A tracked redirect normally waits for Cloudflare Queues to accept its event,
for at most 1.5 s. If the queue fails or is slow, the visitor is redirected
anyway and the event is kept after the response: two delayed sends (same
event ID, so ingest and Convex count it once), then an unresolved
failed-click archive under
failed-clicks/v1/archive/<main queue>/spool-<event ID>.jsonthat alerts through the operations check and replays with the tool below. An isolate stopped within those ~1.3 s before the archive is written can still lose that click. - The first-click-of-session marker has a 300 ms timeout and records
falsewhen the store is unavailable. - A link record that cannot produce a valid event redirects without tracking.
Links with tracking disabled skip event collection and the queue.
HEAD requests (link previewers, uptime tools, email scanners) get the same
redirect but are never counted. Clicks are marked is_bot for crawlers, HTTP
libraries, chat/social link previews, email and office link checks, requests
without a user agent, and browser prefetch/prerender (Sec-Purpose,
Purpose, X-Purpose, X-Moz); they are still stored and appear in bot
counts. In-app browsers where a person really clicked are not treated as bots.
Redirects send Accept-CH but not Critical-CH, which made Chromium repeat a
first visit to obtain the hints and so count it twice.
The producer writes { version: 1, event }. The normalized event has one generated
ID used unchanged as both request_id and idempotency_key. Caller-supplied
request IDs cannot merge separate visits. The original Cloudflare ray remains in
structured logs. The event's time and first-session flag are captured once, before
queueing, and are reused on retries. Session markers use atomic SET NX EX with
a 30-minute lifetime. The flag is a first-observation hint, not an exact count of
people; ingestion should deduplicate session IDs when calculating visitors.
The consumer sends every tracked event in a queue batch to ingest in one
POST <INGEST_ENDPOINT>/batch request ({ "events": [...] }, at most 100 per
request), authorized with INGEST_WRITE_SECRET. Invalid envelopes are archived
and events with tracking disabled are acknowledged without being sent. Ingest
must answer HTTP 200 with success: true and one result per event, in order, with
the matching index and idempotency_key and a status of recorded,
duplicate, ignored, invalid or conflict. A 200 means every recorded or
duplicate event is committed. Any other status, a malformed 200 or a network
failure retries every message in that request. A 404 means ingest does not have
the batch route yet: each event then goes to POST /ingest as before, which
requires a 202 with success: true, the same idempotency_key and
status: queued (extra fields such as committed are ignored).
Per event, recorded and duplicate update the Convex live view with the same
event ID, at most 10 Convex writes at a time to limit contention on busy links.
Bot events stay in durable analytics but do not update human live counts.
Only tracked clicks are sent, so ignored, invalid and conflict are all
retried and so reach the failed-click queue like a rejected single-event request.
A Convex failure retries only that message. Ingest must deduplicate IDs across
replay. Convex must deduplicate click writes and return one of recorded,
duplicate, link_deleted, tracking_disabled, or too_old. Unknown results
are retried. A deleted link cannot cause endless retries. Old history is retained
by ingest even if Convex rejects it outside its supported live replay window.
The main consumer receives up to 100 messages per batch, waits at most 5 seconds
to fill one, and runs at most 2 batches at once. Each message is acknowledged only
after its required deliveries finish. Failed messages retry after 10 seconds,
doubling to a one-hour cap, and move to the failed-events queue after 30 retries:
about 22.4 hours (80,710 seconds) of continuous ingest failure, up from about 4.4
hours with the previous 12 retries. Its separate consumer archives messages in
R2 before acknowledging them. There is no lossy direct-HTTP fallback. The
previous optional ANALYTICS_ENDPOINT/Tinybird fanout is removed; ingest is the
durable analytics authority. A new external sink must consume that retained
stream with its own deduplication and retry policy.
The destination decision and existing deterministic A/B assignment are preserved. The legacy assignment uses an IP-derived key. Changing that to include user agent or a cookie would move existing visitors between variants, so introduce an explicit assignment version for new experiments before making that change.
SHORT_LINK_HOSTS lists the comma-separated hosts that serve every link:
ndle.fyi,www.ndle.fyi in production and dev.ndle.fyi,www.dev.ndle.fyi in
development (wrangler.jsonc). Missing or empty means ndle.fyi,www.ndle.fyi.
Hosts are compared lowercase without a trailing dot. Any other host is a
customer custom domain and opens only links whose Redis record has the same
domain. A link with domain: null (default short domain) or another domain
gets the same 404 as a missing link and records no click. Custom-domain links
still open on the default hosts, so hosted QR codes and existing ndle.fyi
shares keep working. Records projected before Convex wrote domain lack the
field; they still open on any host until re-projected, and each such request on a
custom domain logs legacy_domain_projection at info level. The rollout is
complete when that log stops appearing.
Ingest secrets are scoped. INGEST_WRITE_SECRET authorizes POST /ingest (click
delivery and the repeated delivery in resolve). OPS_SECRET authorizes
/health/detailed, /health/metrics, /internal/events/receipt and other
/internal/* operator routes. Each falls back to the legacy shared API_SECRET
when unset or empty, so deploying without them changes nothing. Once both are
set, this Worker no longer uses API_SECRET.
IP_HASH_SECRET is optional. When set, the stored ip_hash is hex HMAC-SHA256
of the visitor IP, and session_id is derived from it; unset or empty keeps the
legacy plain SHA-256. A/B assignment is unchanged. Set a long random value with
pnpm exec wrangler secret put IP_HASH_SECRET (add --env dev for development).
Setting or rotating it changes every visitor's ip_hash and session_id once:
visitors active at the switch may have their first click of a session counted
once more, and hashes from before the switch do not match later ones.
Use the coordinated production cutover as the authoritative order. Provision and pause queue delivery, deploy and verify this durable producer, finish old in-flight requests, then drain/stop and rebuild ingest. Deploy the verified downstream contracts before resuming delivery. Do not stop the old backend while redirects still use the old direct-HTTP producer.
The guide contains the exact create/update, pause and resume commands, the required post-deploy pause-state check, retention/DLQ limits, and rollback gates. Rehearse on ndle-click-events-dev and ndle-click-events-failed-dev with separate credentials first. Development INGEST_ENDPOINT remains deliberately empty until a verified staging ingest URL is configured; never point it at production. Run the checks below before releasing, then measure queue-acceptance latency and catch-up under real traffic.
The failed queue is consumed separately from normal click delivery. Each message
is saved in the existing ndle-analytics R2 bucket through the
FAILED_CLICK_ARCHIVES binding. Development uses ndle-analytics-dev.
For an original queue name QUEUE and Cloudflare message ID MESSAGE, storage is:
failed-clicks/v1/archive/QUEUE/MESSAGE.json: immutable message body, its SHA-256 hash, event ID when valid, source queue/message ID/time, first observed attempt count, and archive time. JSON stores the exact decoded message, including extra version-1 fields; text and binary bodies are retained too. Unsupported structured values are never silently converted or acknowledged.failed-clicks/v1/unresolved/QUEUE/MESSAGE.json: the investigation marker.failed-clicks/v1/resolved/QUEUE/MESSAGE/UUID.json: retained resolution evidence.
Both the archive and marker must be read back and verified before acknowledgement. A crash between writes or before acknowledgement is safe to retry: the body and source stay unchanged even though the delivery attempt count increases. Malformed JSON events in the main queue go directly to this archive without repeated HTTP delivery attempts. An unknown queue fails the invocation instead of acknowledging its messages. These keys are private operational records, not user export files; do not add them to the archive file-access grants or an R2 expiry rule.
Before deploying this consumer, verify the ingest receipt endpoint and existing
delivery contract, then set both queues' retention to 14 days and read back the
settings. Rehearse with the equivalent -dev queue names first:
pnpm exec wrangler queues update ndle-click-events --message-retention-period-secs 1209600
pnpm exec wrangler queues update ndle-click-events-failed --message-retention-period-secs 1209600The failed consumer permits 100 retries. Storage failures back off from two minutes to twelve hours; an unexpected invocation failure defaults to twelve hours. This keeps the retry count from exhausting before the retention period. A Cloudflare or R2 outage lasting past queue retention can still lose an unarchived message; alerts and intervention before that deadline remain necessary. See Cloudflare's retry rules and retention configuration.
Use the operator tool with a token permitted to read/write this R2 bucket and read
and write Queues. Supply credentials through environment variables, never command
arguments. It uses the same R2 object HTTP operations as the installed Wrangler
CLI; an OAuth login without R2 object access is insufficient. Set
CLOUDFLARE_ACCOUNT_ID, CLOUDFLARE_API_TOKEN, FAILED_CLICK_BUCKET,
CLICK_EVENTS_QUEUE_NAME, and CLICK_EVENTS_FAILED_QUEUE_NAME for one environment.
Replay additionally requires CLICK_EVENTS_QUEUE_ID; the tool verifies its name.
Resolve records RECOVERY_OPERATOR; tracked events also require INGEST_ENDPOINT,
OPS_SECRET (receipt check) and INGEST_WRITE_SECRET (delivery), either of which
may fall back to API_SECRET, plus CONVEX_URL and SHARED_SECRET.
# Substitute the exact source queue and original Cloudflare message ID from logs
# or an unresolved R2 marker. This read-only command prints metadata, not the body.
node scripts/failed-clicks.mjs inspect QUEUE MESSAGE
# Optional fourth argument writes the private archive to a new 0600 local file.
node scripts/failed-clicks.mjs inspect QUEUE MESSAGE /private/path/archive.json
# After fixing the outage, republish the unchanged envelope to the matching main queue.
node scripts/failed-clicks.mjs replay QUEUE MESSAGE
# After delivery finishes, verify its actual durable receipt and Convex outcome.
node scripts/failed-clicks.mjs resolve QUEUE MESSAGEReplay does not clear the investigation marker. It preserves the original ID,
timestamp, owner and envelope, so repeated replay does not create new clicks.
Resolve requires ingest's authenticated /internal/events/receipt to confirm the
same committed ID, owner and timestamp, plus link ID while that raw row is retained.
Legacy timestamp precision differences remain unresolved for manual investigation.
It then repeats the idempotent delivery to validate the actual Convex terminal
outcome. A missing receipt, payload conflict, redirect, failed request or unknown
outcome leaves the marker in place. Events outside Convex's live window may resolve
with too_old; do not generate IDs to force old history into live counts.
For an immutable valid event with tracking_enabled=false, explicit resolve
records the operator and tracking_disabled decision without sending that event
to ingest or Convex. Its archive is still preserved.
Only after verified resolution evidence is stored does the tool remove that exact marker. The original archive and resolution evidence remain. A late queue delivery can conservatively recreate the marker and renew an alert; inspect and resolve it again. It cannot overwrite the archived body or clear a newer failure. Source ID and body hash are checked before removing a marker.
Malformed events cannot be blindly replayed or automatically resolved. After
investigation, an operator may use resolve-invalid QUEUE MESSAGE DECISION.json,
where the private JSON file contains operator and a clear reason (20–4000
characters). This records the decision without changing or deleting the source.
A valid event is refused by that path and requires the receipt checks above.
Any deliberate repair requires a separate review; this tool never invents a
corrected payload or changes ownership.
The production scheduled handler checks NDLE every five minutes, separately
from redirects and click delivery. Development has no schedule and alerts are
disabled. Set RESEND_API_KEY, OPS_ALERT_FROM and OPS_ALERT_TO with Wrangler
secrets; use a sending-only key restricted to the verified NDLE sender domain.
The detailed ingest health check sends OPS_SECRET (else API_SECRET).
OPS_ALERTS_ENABLED=false disables checks without changing redirects.
Email is sent when the main queue's oldest reported event is over five minutes
old, its backlog exceeds 10,000 messages or 100 MB, or the failed-click queue
contains any message. A bounded R2 listing also alerts while any archived failed
message is unresolved, even after the failed queue empties; an unreadable archive
raises its own alert. Checks also cover ingest readiness and detailed component
health, any failed ingest job, more than 1,000 waiting ingest jobs, monitoring
readiness, and a missing/invalid backup or a latest backup older than 26 hours.
The database, writer, archive and backup components are required. Older ingest
versions report queue and recovery; newer ones drop those and report the
event journal. Each of these is checked only when present, so either version
passes without false alerts.
The backup check reads only the latest manifest and the referenced object's
metadata; checksum verification remains the database owner's backup duty.
Each source, including each ingest health endpoint, has its own 12-second
deadline. HTTP requests have their own ten-second timeout and cannot follow
redirects. Ingest alerts distinguish a failed connection, unexpected HTTP
status, invalid response, timeout, or a service-reported failure. Known component
and recovery reason codes identify database, queue, writer, backup, archive,
journal and recovery problems. Logs include only fixed status, component and reason
fields; raw exception text, event bodies, owner data and credentials are omitted.
A check that throws or stalls logs "NDLE operations check failed" with the
check, the error name (TimeoutError for a stall) and, for Cloudflare binding
errors, only their numeric code.
An ingest health failure is checked once more after 15 seconds before sending one combined email. A healthy second sample clears that temporary health alert. Initial failed-job counts, recovery records needing investigation, queue, unresolved archive, backup and monitoring issues are always retained. When ingest health is healthy, those other alerts send without the confirmation delay. Both health samples are logged, including failures that recover.
Each email is written for a person, not a log reader (src/alert-email.ts).
Every problem says what was measured (counts and ages only), what it means for
visitors and data, whether anything is lost, numbered steps with links and
commands to check and fix it, and how to tell it is fixed. Problems that often
clear by themselves after a deploy say so. Times are shown in
OPS_ALERT_TIMEZONE (production: Asia/Kolkata) and UTC. Saved failed clicks
are listed with their exact inspect, replay and resolve commands.
Healthy checks send no email. The last emailed problems are saved in
operations/alert-state.json in the analytics bucket: an unchanged set is
repeated at most once an hour (with how long it has lasted), a changed set is
sent at once, and when every check passes again one all-clear email names what
was fixed.
Only real failures are emailed, and only once they last. The same file keeps when each problem was first seen, so a problem is emailed only if a check at least 4 minutes later (normally the next one) still finds it; link monitoring must be down for 30 minutes. A blip that clears by then sends neither an alert nor an all-clear. A check that gets no answer from Cloudflare (click queue, failed-click queue or failed-click archive) says nothing is broken, so it is only logged, however long it lasts; while it lasts an all-clear waits, because the queue can't be judged healthy. If the saved state can't be read, nothing can be timed and real problems email at once. Each email's provider idempotency key includes a hash of its body, so retries of the same email are sent once. If the saved state cannot be read, the email drops its changing numbers and uses an hourly key so reminders stay hourly. Provider rejection fails the scheduled run and is logged; acceptance is not proof that the recipient read the message. Observe Cron executions and provider delivery status after deployment. A failure of the alert provider or Cloudflare itself still requires an independent external check.
Queue age is based on Cloudflare's reported oldest timestamp. A small backlog without that timestamp cannot establish its age. Ingest's detailed endpoint does not expose the oldest internal job's age, and monitoring readiness does not establish per-link delivery freshness; those thresholds are not claimed by these checks. Existing failed monitoring jobs are retained for separate review.
Use Node.js 22 or newer, pnpm, and Bun. Local secrets go in .dev.vars, with only
development credentials. Reuse any existing development server for this repo.
pnpm install --frozen-lockfile
pnpm test
pnpm test:runtime
pnpm typecheck
pnpm exec biome check src
pnpm exec wrangler types --env-interface CloudflareBindings --check
pnpm deploy:dry-run
pnpm exec wrangler deploy --env dev --dry-runThe checks never deploy. Tests cover queue-acceptance ordering, redirects that
survive queue outages, stalls and session-store failures with the click kept
for replay, 503 only for a failed link lookup, ignored tracking, atomic session markers, downstream failures, repeated
IDs, terminal Convex outcomes, malformed events, redirect safety, domain binding,
keyed visitor hashes, scoped ingest secrets, batch delivery and its per-event
fallback, bounded Convex writes, and both ingest health shapes. CI also runs the actual workerd
runtime with isolated R2 and intercepted HTTP: archive/write
failures, crash recovery, verified acknowledgement, exact v1 replay, downstream
deduplication, receipt mismatches, retained resolution evidence, late redelivery,
and archived-failure alerts. The runtime is stopped in finally; tests never
contact production services or send real email.