Fix scanning issues related to Unicode escapes in RegExp group names and refactor identifier scanning

[graphemecluster (graphemecluster)]

Per microsoft/typescript-go#4881 (review), this is a port of the unmerged #61042, which fixes that leading Unicode escapes and surrogate pair escapes are not parsed and normalised in regular expression group names and result in TS1514.
By this fix, identifier scanning is refactored and unified across normal, private, JSX, JSDoc identifiers and RegExp group names.

• The first commit is a faithful port of Fix scanning issues related to Unicode escapes in identifiers #61042 including the 6 test cases there. The issue detailed in Text between Unicode escapes within an identifier is skipped #61043 (which the old PR also fixes) does not exist in TypeScript Go, and some parts of the old PR are already implemented or are no longer relevant, so not everything is ported.

• The second commit contains some more refactoring (centralising surrogate pair logic) and a tiny optimisation (?) for private identifiers.

Fixes #63995
Closes #61043

[Copilot]

Pull request overview

Fixes Unicode-escape scanning and normalization across identifiers and RegExp group names.

Changes:

• Unifies standard, JSX, JSDoc, private, and RegExp identifier scanning.
• Supports leading escapes and surrogate-pair escapes in RegExp group names.
• Adds compiler tests and updated baselines.

Reviewed changes

Copilot reviewed 47 out of 53 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tsc/internal/scanner/scanner.go	Refactors identifier scanning and escape handling.
tsc/internal/scanner/regexp.go	Uses RegExp-aware identifier scanning.
tsc/testdata/tests/cases/conformance/jsx/unicodeEscapesInJsxtags.tsx	Extends JSX escape coverage.
tsc/testdata/tests/cases/compiler/unicodeEscapesInNames03.ts	Adds property and type-name cases.
tsc/testdata/tests/cases/compiler/unicodeEscapesInNames01.ts	Adds multi-escape identifier cases.
tsc/testdata/tests/cases/compiler/unicodeEscapesInJSDoc.ts	Tests escaped JSDoc names.
tsc/testdata/tests/cases/compiler/regularExpressionGroupNameUnicodeEscapes.ts	Tests RegExp group-name escapes.
tsc/testdata/tests/cases/compiler/extendedUnicodeEscapeSequenceIdentifiers.ts	Tests multiple extended escapes.
tsc/testdata/baselines/reference/conformance/unicodeEscapesInJsxtags.types	Updates JSX type baseline.
tsc/testdata/baselines/reference/conformance/unicodeEscapesInJsxtags.symbols	Updates JSX symbol baseline.
tsc/testdata/baselines/reference/conformance/unicodeEscapesInJsxtags.js	Updates JSX emit baseline.
tsc/testdata/baselines/reference/conformance/unicodeEscapesInJsxtags.errors.txt	Updates JSX diagnostics.
tsc/testdata/baselines/reference/compiler/unicodeEscapesInNames03(target=esnext).types	Adds ESNext type baseline.
tsc/testdata/baselines/reference/compiler/unicodeEscapesInNames03(target=esnext).symbols	Adds ESNext symbol baseline.
tsc/testdata/baselines/reference/compiler/unicodeEscapesInNames03(target=esnext).js.map	Adds ESNext source maps.
tsc/testdata/baselines/reference/compiler/unicodeEscapesInNames03(target=esnext).js	Adds ESNext emit baseline.
tsc/testdata/baselines/reference/compiler/unicodeEscapesInNames03(target=es2015).types	Adds ES2015 type baseline.
tsc/testdata/baselines/reference/compiler/unicodeEscapesInNames03(target=es2015).symbols	Adds ES2015 symbol baseline.
tsc/testdata/baselines/reference/compiler/unicodeEscapesInNames03(target=es2015).js.map	Adds ES2015 source maps.
tsc/testdata/baselines/reference/compiler/unicodeEscapesInNames03(target=es2015).js	Adds ES2015 emit baseline.
tsc/testdata/baselines/reference/compiler/unicodeEscapesInNames01(target=esnext).types	Updates ESNext types.
tsc/testdata/baselines/reference/compiler/unicodeEscapesInNames01(target=esnext).symbols	Updates ESNext symbols.
tsc/testdata/baselines/reference/compiler/unicodeEscapesInNames01(target=esnext).sourcemap.txt	Updates ESNext source-map record.
tsc/testdata/baselines/reference/compiler/unicodeEscapesInNames01(target=esnext).js	Updates ESNext emit.
tsc/testdata/baselines/reference/compiler/unicodeEscapesInNames01(target=es2015).types	Updates ES2015 types.
tsc/testdata/baselines/reference/compiler/unicodeEscapesInNames01(target=es2015).symbols	Updates ES2015 symbols.
tsc/testdata/baselines/reference/compiler/unicodeEscapesInNames01(target=es2015).js	Updates ES2015 emit.
tsc/testdata/baselines/reference/compiler/unicodeEscapesInJSDoc.types	Updates JSDoc types.
tsc/testdata/baselines/reference/compiler/unicodeEscapesInJSDoc.symbols	Updates JSDoc symbols.
tsc/testdata/baselines/reference/compiler/unicodeEscapesInJSDoc.js	Updates JSDoc emit.
tsc/testdata/baselines/reference/compiler/shebangError.types.diff	Adds an unregistered diff artifact.
tsc/testdata/baselines/reference/compiler/shebangError.types	Updates shebang recovery types.
tsc/testdata/baselines/reference/compiler/shebangError.js.diff	Adds an unregistered diff artifact.
tsc/testdata/baselines/reference/compiler/shebangError.js	Updates shebang recovery emit.
tsc/testdata/baselines/reference/compiler/shebangError.errors.txt.diff	Adds an unregistered diff artifact.
tsc/testdata/baselines/reference/compiler/shebangError.errors.txt	Updates shebang diagnostics.
tsc/testdata/baselines/reference/compiler/regularExpressionGroupNameUnicodeEscapes.types	Adds RegExp type baseline.
tsc/testdata/baselines/reference/compiler/regularExpressionGroupNameUnicodeEscapes.symbols	Adds RegExp symbol baseline.
tsc/testdata/baselines/reference/compiler/regularExpressionGroupNameUnicodeEscapes.js	Adds RegExp emit baseline.
tsc/testdata/baselines/reference/compiler/regularExpressionGroupNameUnicodeEscapes.errors.txt	Adds RegExp diagnostics.
tsc/testdata/baselines/reference/compiler/manyCompilerErrorsInTheTwoFiles.types.diff	Adds an unregistered diff artifact.
tsc/testdata/baselines/reference/compiler/manyCompilerErrorsInTheTwoFiles.types	Updates recovery types.
tsc/testdata/baselines/reference/compiler/manyCompilerErrorsInTheTwoFiles.js.diff	Adds an unregistered diff artifact.
tsc/testdata/baselines/reference/compiler/manyCompilerErrorsInTheTwoFiles.js	Updates recovery emit.
tsc/testdata/baselines/reference/compiler/manyCompilerErrorsInTheTwoFiles.errors.txt	Updates recovery diagnostics.
tsc/testdata/baselines/reference/compiler/extendedUnicodeEscapeSequenceIdentifiers.types	Updates extended-escape types.
tsc/testdata/baselines/reference/compiler/extendedUnicodeEscapeSequenceIdentifiers.symbols	Updates extended-escape symbols.
tsc/testdata/baselines/reference/compiler/extendedUnicodeEscapeSequenceIdentifiers.js	Updates extended-escape emit.

Files not reviewed (2)

• tsc/testdata/baselines/reference/compiler/unicodeEscapesInNames01(target=es2015).js: Generated file
• tsc/testdata/baselines/reference/compiler/unicodeEscapesInNames01(target=esnext).js: Generated file

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[graphemecluster (graphemecluster)]

All checks passed, so I am leaving the diff files here. If they are to be removed, feel free to do so on my behalf.

[Jake Bailey (jakebailey)]

We definitely don't want the diffs, we just don't have anything that checks for extra baselines (my mistake for not retaining that)