Skip to content

perf: replace regex parser with a charCode scanner - #81

Open
mcollina wants to merge 1 commit into
mainfrom
perf/charcode-scanner
Open

perf: replace regex parser with a charCode scanner#81
mcollina wants to merge 1 commit into
mainfrom
perf/charcode-scanner

Conversation

@mcollina

Copy link
Copy Markdown
Member

Summary

Rewrites the parser as a hand-rolled charCodeAt scanner driven by a single 64KB Uint8Array character-class table, replacing the mediaTypeRE/paramRE regular expressions. parse and safeParse now share one internal parseHeader that returns sentinel objects on failure — no duplicated logic, no try/catch.

Motivation: content-type@2.x/3.x (jshttp rewrite, 2026) had become 25–100% faster than this package on inputs with parameters. Its speed comes from dropping validation entirely (it accepts application/json foo, empty keys, \r\n in values, etc.). This PR closes the gap without relaxing validation.

Behaviour is unchanged

The scanner reproduces the exact grammar the previous regexes accepted, including their quirks (documented in the code):

  • media-type token excludes ` while parameter tokens include it
  • qdtext accepts VT (0x0b) rather than HTAB
  • only spaces (not tabs) are allowed after ;
  • whitespace around the media type uses String.prototype.trim()'s full Unicode set
  • duplicate parameters: last one wins

Error messages, result shape, null-prototype parameters, the frozen defaultContentType identity and all exports are preserved.

Verification: differential fuzzing against the previous implementation — two generators (random structural chars; well-formed headers with mutations), five seeds, ~19M inputs — 0 mismatches in result values, thrown messages, prototype, frozen-ness or object identity. Plus npm test (100% coverage, tstyche) and lint.

Benchmark (Node 24.18, ops/sec)

input before after content-type@3.0.0
application/json 13.9M 18.0M (1.3×) 13.0M
application/json; charset=utf-8 2.76M 4.87M (1.8×) 3.80M
application/json; charset="utf-8" 2.46M 4.89M (2.0×) 3.46M
multipart/form-data; boundary=----WebKit… 2.52M 3.43M (1.4×) 2.83M
text/html; charset=utf-8; foo=bar; baz="qu\"x" 0.99M 2.13M (2.2×) 1.85M

Implementation note: a first draft was slower than the regex on long inputs because charCodeAt(len)NaN was used as a typed-array index and upper |= undefined pushed V8's optimized loop onto generic paths. Every loop is now bounded by index < len and the table covers the full 0x10000 range so lookups never go out of bounds.

Also

  • Tests added for previously uncovered error paths (empty key, missing =, empty token value, bad/unterminated quoted-pair, Unicode whitespace around the type).
  • README benchmark section pointed at a nonexistent benchmarks/index.js; fixed the command and refreshed the numbers.

Checklist

🤖 Generated with Claude Code

https://claude.ai/code/session_01BHGGooWAqjNjuTqBMiytQU

Rewrite the parser as a hand-rolled charCodeAt scanner driven by a single
64KB Uint8Array character-class table, replacing the mediaTypeRE/paramRE
regular expressions. parse and safeParse now share one internal parseHeader
that returns sentinel objects on failure, so there is no duplicated logic
and no try/catch.

Behaviour is unchanged: the scanner reproduces the exact grammar accepted
by the previous regexes (including their quirks, which are documented in
the code), verified by differential fuzzing against the previous
implementation over ~19M inputs with zero mismatches. Error messages,
result shape, null-prototype parameters and the frozen defaultContentType
identity are all preserved.

Throughput on Node 24 (ops/sec, before -> after):

  application/json                                 13.9M -> 18.0M (1.3x)
  application/json; charset=utf-8                   2.76M -> 4.87M (1.8x)
  application/json; charset="utf-8"                 2.46M -> 4.89M (2.0x)
  multipart/form-data; boundary=----WebKit...       2.52M -> 3.43M (1.4x)
  text/html; charset=utf-8; foo=bar; baz="qu\"x"    0.99M -> 2.13M (2.2x)

Also adds tests for the previously untested error paths (empty key,
missing "=", empty token value, bad/unterminated quoted-pair, Unicode
whitespace around the media type) and fixes the README benchmark command.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BHGGooWAqjNjuTqBMiytQU

@Tony133 Tony133 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@blakeembrey

blakeembrey commented Aug 21, 2026

Copy link
Copy Markdown

Just a heads up that the "including their quirks" are bugs. E.g. media type should allow backticks, qdtext should not be VT but HTAB, and whitespace include HTAB. Dropping .trim() might also boost the perf further.

Finally last parameter wins is a choice but does differ from the official node and content-type behavior. It might be nice to align so the ecosystem is consistent 🤷

new (require('util').MIMEType)('text/html; charset=1; charset=2').params.get('charset') //=> 1

@mcollina

Copy link
Copy Markdown
Member Author

@blakeembrey thanks, all addressed in #82 (stacked on this one, semver-major).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants