Avoid using the terms "latin1" and "iso-8859-1" for isomorphic encoding/decoding

[domenic]

09878c7 made me aware that the functions latin1toString, latin1fromString propagate the confusion between "latin1" and "isomorphic" that we see in a lot of the JavaScript ecosystem, and have tried to help combat in whatwg/encoding@36fb4e7.

In short, the "latin1" encoding specified in the ISO-8859-1 spec does not provide any encodings for the bytes 0x00 to 0x1F or 0x7F to 0x9F. So a proper latin1 decoder would never return those bytes, and a proper latin1 decoder would throw when given those bytes.

In practice, nobody does this, and we have either:

• Libraries following the windows-1252 mapping (TextEncoder/TextDecoder, the entire web platform, Node.js's modern standard library);
• or libraries following the isomorphic decoding / encoding (a lot of C++ code, Node.js's old Buffer API).

This creates a lot of confusion when people expect one of these interpretations and get the other.

My strong suggestion is to never mention the terms latin1 or iso-8859-1 in public APIs, since they mean windows-1252 for people who read standards and mean something else (usually isomorphic encoding) for people who are coming from certain C++ codebases. (I think V8 is the original source of the confusion, at least in the Node.js ecosystem.) Instead, use the standard and non-overloaded term "isomorphic".

I realize this is a breaking change and might not be one you want to take on, but I thought I should file it, in the interest of making this the best encoding/decoding library for JS.

[ChALkeR]

In short, the "latin1" encoding specified in the ISO-8859-1 spec does not provide any encodings for the bytes 0x00 to 0x1F or 0x7F to 0x9F. So a proper latin1 decoder would never return those bytes, and a proper latin1 decoder would throw when given those bytes.

iso-8859-1 from single-byte.js follows https://unicode.org/Public/MAPPINGS/ISO8859/8859-1.TXT, which maps every possible byte value to Unicode.

Also https://icu4c-demos.unicode.org/icu-bin/convexp?conv=ISO-8859-1&s=IANA.

Also, mapping 0x00-0xff to U+0000/U+00FF is something which has many usecases and is also known as "binary", including in HTML spec.
Perhaps the shorthand methods could be renamed to binary from latin1 (while keeping the deprecated aliases until a major).

Not sure what to do with encodings though

[domenic]

iso-8859-1 from single-byte.js follows https://unicode.org/Public/MAPPINGS/ISO8859/8859-1.TXT, which maps every possible byte value to Unicode.

Also https://icu4c-demos.unicode.org/icu-bin/convexp?conv=ISO-8859-1&s=IANA.

Yep, there are a lot of possible mappings for the bytes which the ISO-8859-1 spec didn't map. You've found one of them; other people have found others.

Perhaps the shorthand methods could be renamed to binary from latin1 (while keeping the deprecated aliases until a major).

That would work, although I think "isomorphic" is probably better since it's a standard term.

Not sure what to do with encodings though

Not sure exactly which problem you're referring to here...

[ChALkeR]

the ISO-8859-1 spec

@domenic It's more complex than just the mapping in that single file

"1. Scope" section of that spec says:

ISO-8859-1 spec explicitly permits using it in conjunction with ISO/IEC 6429 (sample) (and only that).

That is the part of the standard which defines the ranges 0x00-0x1F (C0 control), 0x80-0x9F (C1 control) and 0x7F (not sure where the spec puts it, C0/C1/Cx, citations disagree on that, the original spec is behind a paywall)

ISO-8859-1 doesn't explicitly define those ranges because it defers to ISO/IEC 6429.

A decoder following that is per ISO-8859-1 specification.

there are a lot of possible mappings for the bytes which the ISO-8859-1 spec didn't map.

What are the others?

[domenic]

I agree that if you augment ISO-8859-1 with more specs, whether they're ISO/IEC 6429, or windows-1252, or any other encoding, you can come up with answers for the other code points. Different software has chosen different augmentation specs. As you point out, ISO-8859-1 has one "should"-level suggestion, but it's not always followed, and not mandated. (Especially since accessing that augmentation spec costs money, whereas accessing the Encoding Standard is free.)

[ChALkeR]

@domenic WHATWG Encoding spec does not define iso-8859-1, iso-8859-9, iso-8859-11 encodings.
It implements those as mappings to other encodings.

single-byte.js is lower-level, does not claim to match WHATWG Encoding Spec for those mappings.
In fact, it supports no mappings.
You likely came over it looking for encoders, but for that, see #37 which awaits a review.
For other (non-URL) use-cases, there would be a different unified encode() API which will follow Encoding Spec mappings.

While using encoding.js, encoding-lite.js, or utf8.js (if you just need UTF-8) is the recommended way in general, supporting old encodings is valuable for other, less common usecases.
Background: I did an npm scan to check what iconv-lite is used for.

Perhaps what should be done is to more clearly document that single-byte.js is not following the Encoding spec and is likely not something one should look for when implementing web-related things, and discourage using it in general in the modern world. Those extra encodings are needed for some compatibility purposes though.

In that context, it would be very confusing to implement iso-8859-1 without newlines (which are in C0). Or as windows-1252. As it is not windows-1252 per it's own specification (which suggests mapping via C0/C1) or per Unicode or ICU mapping (which all match to that).

Moreover, I'm considering adding other Unicode.org mappings to single-byte/multi-byte under different names from the current ones, which would also mismatch WHATWG Encoding aliases. That won't affect encoding.js, obviously.

---

I.e. the solution for this is likely make the warning better, not remove the encoding, not remove C0/C1 from encoding, and not make it an alias for windows-1252.

---

As for isomorphic name, that's again something WHATWG defines (side note: it still uses the term binary for it in some cases), and could be defined under that name in either encoding.js or whatwg.js (see #37), so users won't have to load single-byte to get it.

Upd: perhaps latin1* helpers should be indeed deprecated in favor of isomorphic* in encoding to reduce confusion.

[domenic]

@domenic WHATWG Encoding spec does not define iso-8859-1, iso-8859-9, iso-8859-11 encodings.
It implements those as mappings to other encodings.

Sure, just like it doesn't define "utf8", it only defines "utf-8". In practice, labels are what software accepts (including what your library accepts); the fact that the spec has factored itself to have one canonical "name" and many "labels" is basically an editorial choice.

I.e. the solution for this is likely make the warning better, not remove the encoding, not remove C0/C1 from encoding, and not make it an alias for windows-1252.

Upd: perhaps latin1* helpers should be indeed deprecated in favor of isomorphic* in encoding to reduce confusion.

Yes, I think using a non-ambiguous name is the best route, and is the one I suggested in the OP of this issue :).

[ChALkeR]

I added an extra warning in 7d82bcf

I think what remains is isomorphic* support in /encoding.js.
Or is there anything else?

---

For context: even adding legacy cp* encodings that are different from windows-* is not completely out of question for single-byte.js. That import is not to be intended a throw-in easy-to-use high-level API, most usecases should not load it.

latin1* alias in it was an oversight though.

There might be a better high-level API later, but those two imports are not it (which is also why they don't auto-detect single/multi-byte encoding and why it was inconvenient to use them as generic encoders).

[ChALkeR]

I'm thinking about just removing single-byte and multi-byte from readme (but keeping them in the detailed docs with a warning) once better APIs are finalized.

Which should resolve this, I guess.

[ChALkeR]

Adding in #68 under isomorphicEncode / isomorphicDecode name.

[ChALkeR]

@domenic latin1 part is mostly done.

1. isomorphicEncode / isomorphicDecode were added to encoding-lite.js (and encoding.js)
2. latin1toSting / latin1fromString are deprecated
3. Readme no longer lists/promotes single-byte.js and multi-byte.js, though full documentation includes them with suggestions to use other / higher-level APIs.

While everything actionable on the current state is done, I'm not closing this yet.

I'm still unsure what to do with iso-8859-1 name
There are usecases for encoders, not just decoders, including for testing stuff
Encoding API doesn't cover that

Also there are usecases for legacy encodings on the server/app/script-side, which expect iso-8859-1 supported in the original/historic sense.

I'm considering adding an api that is a mix of what iconv is doing, but with windows-*, utf-16 and multi-byte encodings fixed - that would still be compatible to iconv-lite usecase.
And that API should have iso-8859-1 as isomorpic.