[io] Windows legacy mode mistakenly ignores the device encoding

[eryksun]

BPO	42261
Nosy	@pfmoore, @vstinner, @tjguk, @zware, @eryksun, @zooba

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-11-04.15:20:48.448>
labels = ['interpreter-core', 'type-bug', '3.8', '3.9', 'expert-IO', '3.10', 'OS-windows']
title = 'Windows legacy I/O mode mistakenly ignores the device encoding'
updated_at = <Date 2020-11-04.16:50:47.706>
user = 'https://github.com/eryksun'

bugs.python.org fields:

activity = <Date 2020-11-04.16:50:47.706>
actor = 'eryksun'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core', 'Windows', 'IO']
creation = <Date 2020-11-04.15:20:48.448>
creator = 'eryksun'
dependencies = []
files = []
hgrepos = []
issue_num = 42261
keywords = []
message_count = 5.0
messages = ['380329', '380330', '380333', '380335', '380343']
nosy_count = 6.0
nosy_names = ['paul.moore', 'vstinner', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue42261'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

Linked PRs

• gh-86427: Use the console code page for the stdio encoding on Windows #155415
• gh-86427: Determine the stdio encoding for every standard stream on Windows #155416
• gh-86427: Replace PyConfig.stdio_encoding with three separate members #155417
• [3.15] gh-86427: Use the console code page for the stdio encoding on Windows (GH-155415) #156602
• [3.14] gh-86427: Use the console code page for the stdio encoding on Windows (GH-155415) #156603
• [3.13] gh-86427: Use the console code page for the stdio encoding on Windows (GH-155415) #156604

[eryksun]

In Python 3.8+, legacy standard I/O mode uses the process code page from GetACP instead of the correct device encoding from GetConsoleCP and GetConsoleOutputCP. For example:

C:\>chcp 850
Active code page: 850
C:\>set PYTHONLEGACYWINDOWSSTDIO=1

C:\>py -3.7 -c "import sys; print(sys.stdin.encoding)"
cp850
C:\>py -3.8 -c "import sys; print(sys.stdin.encoding)"
cp1252
C:\>py -3.9 -c "import sys; print(sys.stdin.encoding)"
cp1252

This is based on config_init_stdio_encoding() in Python/initconfig.c, which sets config->stdio_encoding via config_get_locale_encoding(). Cannot config->stdio_encoding be set to NULL for default behavior?

Computing this ahead of time would require separate encodings config->stdin_encoding, config->stdout_encoding, and config->stderr_encoding. And _Py_device_encoding would have to be duplicated as something like config_get_device_encoding(PyConfig *config, int fd, wchar_t **device_encoding).