.Net: [codex] Fix TextChunker orphan merge token count

[pragnyanramtha]

Summary

Fixes #13713. TextChunker.SplitPlainTextParagraphs now checks the configured token counter before merging a short final paragraph into the previous paragraph. This prevents the orphan-paragraph balancing step from creating a chunk that exceeds the requested token limit when a custom token counter is used.

Root Cause

The orphan merge logic compared the number of whitespace-delimited words in the last two paragraphs against adjustedMaxTokensPerParagraph. That word count can be lower than the actual token count reported by the provided TokenCounter, so the merge could produce an oversized chunk.

Change

• Build the candidate merged paragraph using the existing paragraph strings.
• Call GetTokenCount(mergedParagraph, tokenCounter) before merging.
• Add a regression test using a length-based token counter to cover the oversized merge case.

Validation

• PATH=/tmp/dotnet:$PATH dotnet test dotnet/src/SemanticKernel.UnitTests/SemanticKernel.UnitTests.csproj --filter FullyQualifiedName~TextChunkerTests
  • Passed: 40, Failed: 0, Skipped: 0

Full repository test suite was not run because the focused TextChunker unit tests cover the changed behavior.

[github-actions]

Automated Code Review

Reviewers: 4 | Confidence: 93% | Result: All clear

Reviewed: Correctness, Security Reliability, Test Coverage, Design Approach

---

Automated review by pragnyanramtha's agents

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.