[scottanderson]

Problem

std::mem::read_string returned raw bytes. Under a non-UTF-8 #pragma encoding, the UI then read those bytes as UTF-8. For example, Mac Roman 48 69 6E 6B D5 73 ("Hink’s") showed as garbage, because 0xD5 took the s with it.

Under such a pragma, a str out variable could also crash the process. A str variable holds the text assigned to it, not data bytes, but its text went through the codec again. If it did not decode, the error was thrown inside a scope guard's destructor, and that ended the process. A [[transform]] that fails on an out variable crashes the same way, with no pragma at all.

Summary

read_string now returns UTF-8 text. It has a new optional 4th parameter, encoding:

read_string(address, size, section, encoding)
  encoding empty           -> use #pragma encoding
  encoding is "UTF-8"      -> raw bytes, as before
  no encoding set          -> raw bytes, as before
  no codec set (plcli)     -> raw bytes, as before
  otherwise                -> codec.decode(bytes, encoding)
    bytes do not decode    -> E0004 "invalid byte sequence for encoding '...'"

• [[encoding]] does not apply, because read_string reads no field.
• The empty-name check is done here, not in the host. ImHex's codec reads an empty name as the hex editor's encoding, not as UTF-8.
• Patterns with no pragma do not change.
• The comment on the function named an endian parameter. It has none. The 3rd parameter is the section.

The second commit fixes the out-variable crash:

 PatternString::getValue()
-  decode(stored bytes, pragma encoding)      // also for a str variable
+  if section is Heap or PatternLocal
+    return stored bytes                      // a str variable holds text
+  decode(stored bytes, pragma encoding)

 Evaluator::evaluate() ON_SCOPE_EXIT
-  outValues[name] = pattern->getValue()      // a throw here calls std::terminate
+  try   outValues[name] = pattern->getValue()
+  catch log "Cannot read out variable '...'"

Evidence

New tests, each with a small fake codec:

Test	Checks
ReadStringNoPragma	raw bytes, including invalid UTF-8; the codec is never called
ReadStringDecoded	decoded under the pragma; "" uses the pragma; "UTF-8" gives raw bytes
ReadStringNoCodec	raw bytes under a pragma with no codec
ReadStringUndecodableFail	fails with invalid byte sequence for encoding 'ascii'
OutVariableEncoding	a str out variable keeps "café" under a pragma that cannot hold é
OutVariableError	an out variable whose transform fails is left out; the others are kept

• Before: OutVariableEncoding and OutVariableError abort (std::terminate, exit 134).
  After: all 85 tests pass.

The "ImHex Patterns Tests" job runs all 311 pattern tests, and they pass. The job then fails in generate_encodings.py --check. That check fails on ImHex-Patterns master too, with the same 6 problems. The fix is in the ImHex-Patterns PR under "Depends on".

Merge Danger

Door: two-way

A pattern that uses read_string under a non-UTF-8 #pragma encoding now gets decoded text, not raw bytes. To read binary data there, it must pass "UTF-8". Reverting gives back the old behavior.

Blast Radius: small

Only patterns with a non-UTF-8 #pragma encoding that call read_string, or that use str variables, see a change. No pattern in ImHex-Patterns master uses #pragma encoding.

Depends on

This PR fixes the generate_encodings.py --check failure in the "ImHex Patterns Tests" job.

• encodings: Add Python 3.14's new aliases to EXTRA_ALIASES ImHex-Patterns#590

Needed by

The ImHex PR updates the submodule and tests with the real codec. The ImHex-Patterns PR adds the encoding parameter to the std::mem::read_string wrapper.

• impr: Decode std::mem::read_string through the pattern's encoding ImHex#2943
• includes/std: Add an encoding parameter to std::mem::read_string ImHex-Patterns#604

Related

The Hotline patterns use #pragma encoding macintosh and read_string, and need this change.

• patterns/hotline: Add Hotline client bookmark and resource file patterns ImHex-Patterns#578