Description
QuestionAnsweringTrainer (Microsoft.ML.TorchSharp, Roberta/QATrainer.cs) cannot produce an answer that contains characters above U+00FF. For Korean, Chinese, Japanese, Cyrillic and similar text, the predicted answer comes out with those characters missing, so exact match is unreachable whatever the model learned.
Cause
The predicted answer is not sliced out of the context string. It is decoded from the span's token ids:
// QATrainer.cs (main), in the mapper that fills PredictedAnswersBuffer
_parent.Tokenizer.Decode(
_parent.Tokenizer.RobertaModel().ConvertOccurrenceRanksToIds(contextIds).ToArray()
.AsSpan(predictStart - questionLength - 2, predictEnd - predictStart).ToArray())The tokenizer comes from TokenizerExtensions.GetInstance, which calls EnglishRobertaTokenizer.Create(...) with the default filterUnsupportedChars = true. On that path EnglishRobertaTokenizer.MapIdToToken maps each token character through UnicodeToByte and appends the byte value as a char. Every output character is therefore ≤ U+00FF, and multi-byte UTF-8 sequences are never reassembled into the original characters.
Reproduction
- Train
mlContext.MulticlassClassification.Trainers.QuestionAnswer(...) on a small SQuAD-format dataset whose contexts and answers are in Korean (any non-Latin script shows the same behavior).
- Transform a test set and read the
Answer column.
Observed with 26 training rows and 10 epochs: training completes normally, but all 8 predicted answers contain only the digits, ASCII punctuation and spaces of their spans, with every Hangul character missing, e.g. "12 , 1 100". Character-level F1 is about 0.05–0.08 and exact match is 0.
Expected behavior
The predicted answer is the corresponding substring of the context, in whatever script the context uses.
Suggested fix
Map the predicted token span back to character offsets in the original context (the tokenizer already provides offsets via EncodeToTokens) and return context.Substring(start, length), instead of decoding the ids. Alternatively, decode byte-level tokens to bytes and UTF-8-decode them (filterUnsupportedChars: false with proper byte reassembly).
Environment
- Microsoft.ML 5.0.0, Microsoft.ML.TorchSharp 0.23.0 (behavior also matches the current
main source)
- .NET 10, Windows 11, CPU
Description
QuestionAnsweringTrainer(Microsoft.ML.TorchSharp,Roberta/QATrainer.cs) cannot produce an answer that contains characters above U+00FF. For Korean, Chinese, Japanese, Cyrillic and similar text, the predicted answer comes out with those characters missing, so exact match is unreachable whatever the model learned.Cause
The predicted answer is not sliced out of the context string. It is decoded from the span's token ids:
The tokenizer comes from
TokenizerExtensions.GetInstance, which callsEnglishRobertaTokenizer.Create(...)with the defaultfilterUnsupportedChars = true. On that pathEnglishRobertaTokenizer.MapIdToTokenmaps each token character throughUnicodeToByteand appends the byte value as achar. Every output character is therefore ≤ U+00FF, and multi-byte UTF-8 sequences are never reassembled into the original characters.Reproduction
mlContext.MulticlassClassification.Trainers.QuestionAnswer(...)on a small SQuAD-format dataset whose contexts and answers are in Korean (any non-Latin script shows the same behavior).Answercolumn.Observed with 26 training rows and 10 epochs: training completes normally, but all 8 predicted answers contain only the digits, ASCII punctuation and spaces of their spans, with every Hangul character missing, e.g.
"12 , 1 100". Character-level F1 is about 0.05–0.08 and exact match is 0.Expected behavior
The predicted answer is the corresponding substring of the context, in whatever script the context uses.
Suggested fix
Map the predicted token span back to character offsets in the original context (the tokenizer already provides offsets via
EncodeToTokens) and returncontext.Substring(start, length), instead of decoding the ids. Alternatively, decode byte-level tokens to bytes and UTF-8-decode them (filterUnsupportedChars: falsewith proper byte reassembly).Environment
mainsource)