Skip to content

fix(test): replace flaky PKCE distribution test with deterministic checks#258

Merged
wyattjoh merged 2 commits intomainfrom
fix-failing-test
May 5, 2026
Merged

fix(test): replace flaky PKCE distribution test with deterministic checks#258
wyattjoh merged 2 commits intomainfrom
fix-failing-test

Conversation

@rafa-thayto
Copy link
Copy Markdown
Contributor

@rafa-thayto rafa-thayto commented May 5, 2026

Summary

  • The statistical distribution test for generateCodeVerifier had a ~2% failure rate per CI run (example failure) — the 10% tolerance was ~3.6σ per character, and with 66 characters tested the Bonferroni-corrected probability was high enough to flake regularly
  • Replaced with two deterministic tests that prove the same property with zero flakiness:
    • Boundary mock test: spies on crypto.getRandomValues to feed byte 198 (at threshold, rejected) then byte 197 (below, accepted), verifying the off-by-one boundary is correct
    • Uniform mapping test: iterates all 256 byte values and proves the 198 accepted ones map exactly 3 per charset index — the mathematical invariant that rejection sampling guarantees

Test plan

  • bun test packages/cli-core/src/lib/pkce.test.ts — 11 pass, 0 fail
  • 10/10 consecutive runs stable at 6-7ms (faster than the original ~11ms statistical test)
  • Full test suite unaffected

@changeset-bot
Copy link
Copy Markdown

changeset-bot Bot commented May 5, 2026

⚠️ No Changeset found

Latest commit: c0b6bc3

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 5, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 742a04b7-8557-4c78-822f-a0c8986429c2

📥 Commits

Reviewing files that changed from the base of the PR and between 4e59fdb and c0b6bc3.

📒 Files selected for processing (1)
  • packages/cli-core/src/lib/pkce.test.ts

📝 Walkthrough

Walkthrough

Replaces a probabilistic PKCE distribution test with two deterministic tests. One test spies on and mocks crypto.getRandomValues to force boundary behavior, asserts it was called twice, and verifies generateCodeVerifier() yields a specific 43-character result. The second test deterministically verifies uniform mapping of accepted bytes into the charset by counting occurrences of byte % CHARSET.length across the accepted-byte range and asserting exact per-index counts. No public API or non-test code was changed.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title directly describes the main change: replacing a flaky PKCE distribution test with deterministic checks, which aligns with the changeset's core objective.
Description check ✅ Passed The description comprehensively explains the problem (2% flaky statistical test), the solution (two deterministic tests with specific verification approaches), and test results demonstrating stability.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@rafa-thayto rafa-thayto changed the title fix(test): widen PKCE distribution tolerance to prevent flaky CI fix(test): replace flaky PKCE distribution test with deterministic checks May 5, 2026
@wyattjoh wyattjoh merged commit 61d091a into main May 5, 2026
10 checks passed
@wyattjoh wyattjoh deleted the fix-failing-test branch May 5, 2026 16:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants