Skip to content

unicode fix - #723

Closed
xermicus wants to merge 1 commit into
mainfrom
cl/unicode-escape-zero-padding
Closed

xermicus wants to merge 1 commit into
mainfrom
cl/unicode-escape-zero-padding

Conversation

@xermicus

@xermicus xermicus commented Nov 9, 2026

Copy link
Copy Markdown
Member

Closes #708

Signed-off-by: Cyrill Leutwiler <bigcyrill@hotmail.com>
@xermicus
xermicus requested review from elle-j and jubnzv November 9, 2026 12:15

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably unify the escape decoding logic and just have it owned by the yul crate's String (most of what's in the current String::unescape).

E.g. the loop logic from String::unescape could be shared and probably replace all of:

  • crates/yul/src/parser/statement/expression/literal.rs:

let mut hex_string = String::with_capacity(BYTE_LENGTH_WORD * 2);
let mut index = 0;
loop {
if index >= string.len() {
break;
}
if string[index..].starts_with('\\') {
index += 1;
if string[index..].starts_with('x') {
hex_string.push_str(&string[index + 1..index + 3]);
index += 3;
} else if string[index..].starts_with('u') {
let codepoint_str = &string[index + 1..index + 5];
let codepoint =
u32::from_str_radix(codepoint_str, BASE_HEXADECIMAL).map_err(
|error| {
anyhow::anyhow!(
"Invalid codepoint `{}`: {}",
codepoint_str,
error
)
},
)?;
let unicode_char = char::from_u32(codepoint).ok_or_else(|| {
anyhow::anyhow!("Invalid codepoint {}", codepoint)
})?;
for byte in unicode_char.to_string().bytes() {
hex_string.push_str(format!("{byte:02x}").as_str());
}
index += 5;
} else if string[index..].starts_with('t') {
hex_string.push_str("09");
index += 1;
} else if string[index..].starts_with('n') {
hex_string.push_str("0a");
index += 1;
} else if string[index..].starts_with('r') {
hex_string.push_str("0d");
index += 1;
} else if string[index..].starts_with('\n') {
index += 1;
} else {
hex_string
.push_str(format!("{:02x}", string.as_bytes()[index]).as_str());
index += 1;
}
} else {
hex_string
.push_str(format!("{:02x}", string.as_bytes()[index]).as_str());
index += 1;
}
}
hex_string

  • crates/newyork/src/from_yul.rs:

let mut hex = std::string::String::with_capacity(64);
let mut index = 0;
let bytes = string.as_bytes();
while index < bytes.len() {
if bytes[index] == b'\\' {
index += 1;
if index >= bytes.len() {
break;
}
match bytes[index] {
b'x' => {
if index + 2 < bytes.len() {
hex.push(bytes[index + 1] as char);
hex.push(bytes[index + 2] as char);
}
index += 3;
}
b'u' => {
if index + 4 < bytes.len() {
let code_point_string = &string[index + 1..index + 5];
if let Ok(codepoint) =
u32::from_str_radix(code_point_string, 16)
{
if let Some(character) = char::from_u32(codepoint) {
let mut buffer = [0u8; 4];
let encoded = character.encode_utf8(&mut buffer);
for byte in encoded.bytes() {
hex.push_str(&format!("{byte:02x}"));
}
}
}
}
index += 5;
}
b't' => {
hex.push_str("09");
index += 1;
}
b'n' => {
hex.push_str("0a");
index += 1;
}
b'r' => {
hex.push_str("0d");
index += 1;
}
b'\n' => {
index += 1;
}
other => {
hex.push_str(&format!("{other:02x}"));
index += 1;
}
}
} else {
hex.push_str(&format!("{:02x}", bytes[index]));
index += 1;
}
}
hex

@xermicus

Copy link
Copy Markdown
Member Author

will be addressed in #724

@xermicus xermicus closed this Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

resolc compiles a Yul \uXXXX escape below U+0800 with extra zero bytes

2 participants