Skip to content

Python: write and read chat history files as UTF-8 - #14745

Open
Hyacinth-of-Security (lxcxjxhx) wants to merge 1 commit into
microsoft:mainfrom
lxcxjxhx:fix/chat-history-utf8-io
Open

Hyacinth-of-Security (lxcxjxhx) wants to merge 1 commit into
microsoft:mainfrom
lxcxjxhx:fix/chat-history-utf8-io

Conversation

@lxcxjxhx

Copy link
Copy Markdown

What

Three text-mode open() calls in shipped code rely on the platform default encoding. On Windows that is the ANSI code page (GBK on a zh-CN machine, cp1252 on en-US), so:

  • ChatHistory.store_chat_history_to_file() writes a file that is not UTF-8: a history containing héllo — 世界 lands on disk as h\xa8\xa6llo \xa1\xaa \xca\xc0\xbd\xe7 — GBK bytes for the same text. That file cannot be read on Linux/macOS or inside a container, where open() defaults to UTF-8: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa8, and any character outside the ANSI code page raises UnicodeEncodeError when writing.
  • ChatHistory.load_chat_history_from_file() is the matching half, so the pair only round-trips on the machine that wrote it.
  • OnnxGenAICompletionBase._load_genai_settings reads genai_config.json from a model directory without an encoding — those configs come from the model package and are UTF-8.

Change

-        with open(file_path, "w") as local_file:
+        with open(file_path, "w", encoding="utf-8") as local_file:

-        with open(file_path) as file:
+        with open(file_path, encoding="utf-8") as file:

-            with open(json_gen_ai_config) as file:
+            with open(json_gen_ai_config, encoding="utf-8") as file:

Three lines, no behaviour change on POSIX (UTF-8 is already the default there).

Verification

Windows 11 / CPython 3.13, semantic_kernel installed editable from python/:

# before: file is not UTF-8
utf-8 decodable: False -> 'utf-8' codec can't decode byte 0xa8 in position 218

# after: file is UTF-8 and the content is byte-identical to the input
utf8 decode ok: True
text matches original: True
round trip matches: True

python/tests/unit/contents/test_chat_history.py: 76 passed before and after (the suite does not pin an encoding, so it does not notice the difference — the fix is about interoperability, not about a failing test).

Scope

Only python/semantic_kernel/. A scan of the tree finds 32 more text-mode opens without an encoding, but they are in tests, samples and scripts; I kept this PR to the three in the shipped package so it stays reviewable in one pass. Happy to send the test/sample sweep separately if you want it.

Copilot AI balanced review requested due to automatic review settings November 7, 2026 18:18
@semantic-kernel-automation semantic-kernel-automation Bot added the python Pull requests for the Python Semantic Kernel label Nov 7, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Existing tests would still pass if explicit UTF-8 encodings were removed from ONNX initialization and chat-history persistence, so targeted regression coverage is needed.

2 open issues
What changed in this PR

This PR makes file encoding explicit in the Python package, improving chat-history portability and ensuring ONNX model configuration files are read as UTF-8.

Changes:

  • Use UTF-8 when writing and reading chat-history files.
  • Use UTF-8 when reading ONNX genai_config.json files.
File Description
python/​semantic_kernel/​contents/​chat_history.py Specifies UTF-8 for history file reads and writes.
python/​semantic_kernel/​connectors/​ai/​onnx/​services/​onnx_gen_ai_completion_base.py Specifies UTF-8 for model configuration reads.

Review effort: Balanced

try:
json_gen_ai_config = os.path.join(ai_model_path + "/genai_config.json")
with open(json_gen_ai_config) as file:
with open(json_gen_ai_config, encoding="utf-8") as file:
"""
json_str = self.serialize()
with open(file_path, "w") as local_file:
with open(file_path, "w", encoding="utf-8") as local_file:

This branch was successfully deployed

1 active deployment
github-app-auth — c2141d1f Deployed Nov 7, 2026 by lxcxjxhx via add_label #29627
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Pull requests for the Python Semantic Kernel

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants