P2728: loosen wording to admit chunked (SIMD) implementations#237
Merged
Conversation
The wording previously pinned the transcoding iterator to decoding one code point per read: buf_ was sized to hold exactly one transcoded code point, and base() and iterator equality were specified directly in terms of the exposition-only members, making the buffer anchor observable and blocking an as-if chunked implementation. Loosen it so an implementation may transcode chunks of input at a time, e.g. with SIMD: - Make buf_'s capacity an unspecified constant, buffer-capacity, of at least 4 / sizeof(ToType), and widen buf_index_ and to_increment_ accordingly. Chunking is permitted only when the underlying range models forward_range; read-ahead on a single-pass range is destructive and therefore observable. - Allow read() to transcode an implementation-chosen number n >= 1 of consecutive input subsequences per invocation, provided the result fits in buf_. The choice of n is unobservable and may vary call to call: an implementation typically transcodes a fixed-size window of code units trimmed back to whole input subsequences. Substitution of Maximal Subparts applies per input subsequence exactly as before. - Allow read-reverse() to chunk symmetrically (n subsequences ending at current_), specify explicitly that it leaves current_ at the beginning of the first subsequence it transcoded, and note that the chunk's starting boundary is locatable by bounded backward scanning. - Respecify base() and iterator equality positionally -- in terms of the position of the current input subsequence and the offset of the current element within its transcoded code units -- instead of by memberwise comparison, since iterators denoting the same element can hold different buffer anchors once buffers are chunked. - Make operator-- skip an ill-formed input subsequence's three-unit replacement-character encoding as a unit in _or_error views with char8_t output, mirroring operator++: with chunked buffers those code units can sit mid-buffer, and the skip preserves the canonical first-unit position that positional equality relies on. - Add a "SIMD support" design discussion section: iterator size and ABI implications of the buffer capacity, base() implementation strategy, validation-plus-scalar-fallback for SMS error handling, interactivity considerations, preliminary UTF-16 to UTF-8 performance numbers from the prototype std::simd kernel (enolan_simd4 branch), and why a view cannot approach bulk transcoding speed. Update the changelog. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ac3cf3d to
7d9aefd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The wording previously pinned the transcoding iterator to decoding one code point per read: buf_ was sized to hold exactly one transcoded code point, and base() and iterator equality were specified directly in terms of the exposition-only members, making the buffer anchor observable and blocking an as-if chunked implementation. Loosen it so an implementation may transcode chunks of input at a time, e.g. with SIMD: