Skip to content

gh-150474: Fix RFC 2231 charset handling in the email header parser - #157761

Open
lpyu001 wants to merge 6 commits into
python:mainfrom
lpyu001:fix-issue-150475
Open

lpyu001 wants to merge 6 commits into
python:mainfrom
lpyu001:fix-issue-150475

Conversation

@lpyu001

@lpyu001 lpyu001 commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

While writing test cases, I found that the for loop here iterates over a Value token list whose only child is the Attribute (or QuotedString) containing the terminal, for example:

token = Value([
    Attribute([
        ValueTerminal("us-ascii", "extended-attrtext"),
        CFWSList([...]),
    ])
])

As a result, the loop never reaches the extended-attrtext terminal, so the break condition is never satisfied. Parameter.charset is therefore taken from the enclosing token and includes any CFWS following the charset. It also means that the loop's intended purpose—retyping the extended-attrtext terminal as attrtext—cannot take effect.

I decided to fix this by first descending into the child token that contains the charset, then retyping the actual extended-attrtext terminal as attrtext. The charset value is taken from the child token's stripped_value, so any surrounding CFWS is preserved in the parse tree but excluded from Parameter.charset.

get_parameter() iterated over the outer Value node when looking for the extended-attrtext terminal that contains the RFC 2231 charset. Since the terminal is nested inside an Attribute or QuotedString, the loop never reached it.

Descend into the charset container before retyping the terminal as attrtext, and use stripped_value for Parameter.charset so trailing CFWS is not included.
@lpyu001
lpyu001 requested a review from a team as a code owner September 18, 2026 15:55
@lpyu001 lpyu001 changed the title gh-150475: Fix RFC 2231 charset handling in the email header parser gh-150474: Fix RFC 2231 charset handling in the email header parser Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant