Flag bootstrap-only CLI options when set in testconfig.json

[Evangelink]

Description

A few platform CLI options are intentionally read off CommandLineParseResult before IConfiguration (and therefore testconfig.json) is built, so they cannot be honored from JSON:

• --config-file – chicken-and-egg, it tells the system where to find the JSON
• --diagnostic, --diagnostic-output-directory, --diagnostic-file-prefix, --diagnostic-verbosity, --diagnostic-synchronous-write – needed before the config system is up so we can log config-build itself

Before this change, declaring any of these under commandLineOptions in testconfig.json was silently ignored, which is confusing.

Fix

CommandLineOptionsValidator now keeps a small BootstrapOnlyOptions set built from PlatformCommandLineProvider constants and runs a JSON-only validation pass right after the unknown-option check. When any bootstrap-only option appears under commandLineOptions in testconfig.json, startup fails fast with:

In testconfig.json under 'commandLineOptions': The option 'config-file' is read during platform bootstrap (before the testconfig.json file is loaded) and must therefore be passed on the command line rather than declared under 'commandLineOptions' in testconfig.json.

Behavior:

• Matches case-insensitively (Config-File flagged just like config-file), consistent with how testconfig.json keys are handled everywhere else.
• Flags both value entries and disabled entries ("diagnostic": false) — either way the user clearly intended the option to take effect via JSON, which it never does.
• Aggregates every offending entry in one pass so users can fix them all in one go.
• Setting bootstrap-only options on the CLI itself remains fully supported.

Tests

Added under JsonCommandLineOptionsTests:

• Validator_JsonBootstrapOnlyOption_Fails — [DataRow] over all six bootstrap-only options.
• Validator_JsonBootstrapOnlyOption_CaseInsensitive_Fails
• Validator_JsonBootstrapOnlyOption_DisabledEntry_StillFails
• Validator_CliBootstrapOnlyOption_Allowed
• Validator_JsonNonBootstrapOption_Allowed (non-regression for normal JSON options)
• Validator_JsonMultipleBootstrapOnlyOptions_AllReported

All 321 CommandLine~ unit tests pass locally.

Localization

Added JsonCommandLineOptionIsBootstrapOnlyErrorMessage to PlatformResources.resx and regenerated all *.xlf files via dotnet msbuild Microsoft.Testing.Platform.csproj /t:UpdateXlf.

Fixes #8829

[Copilot]

Pull request overview

This PR improves Microsoft.Testing.Platform’s testconfig.json validation by detecting “bootstrap-only” CLI options (options read before configuration is loaded) when they’re incorrectly specified under commandLineOptions, and failing fast with a clear, localized error message instead of silently ignoring them.

Changes:

• Added a bootstrap-only option validation pass to CommandLineOptionsValidator (case-insensitive, aggregates all offending JSON entries).
• Added unit tests covering bootstrap-only JSON failures (including casing, disabled entries, aggregation) and ensuring CLI usage remains allowed.
• Introduced a new localized resource string (and updated corresponding .xlf files).

Show a summary per file

File	Description
test/UnitTests/Microsoft.Testing.Platform.UnitTests/Configuration/JsonCommandLineOptionsTests.cs	Adds coverage for bootstrap-only options being rejected in testconfig.json and allowed on CLI.
src/Platform/Microsoft.Testing.Platform/CommandLine/CommandLineOptionsValidator.cs	Implements bootstrap-only JSON validation and integrates it into the validation pipeline.
src/Platform/Microsoft.Testing.Platform/Resources/PlatformResources.resx	Adds the new error message resource for bootstrap-only JSON option usage.
src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.zh-Hant.xlf	Adds localized entry for the new error message.
src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.zh-Hans.xlf	Adds localized entry for the new error message.
src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.tr.xlf	Adds localized entry for the new error message.
src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.ru.xlf	Adds localized entry for the new error message.
src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.pt-BR.xlf	Adds localized entry for the new error message.
src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.pl.xlf	Adds localized entry for the new error message.
src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.ko.xlf	Adds localized entry for the new error message.
src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.ja.xlf	Adds localized entry for the new error message.
src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.it.xlf	Adds localized entry for the new error message.
src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.fr.xlf	Adds localized entry for the new error message.
src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.es.xlf	Adds localized entry for the new error message.
src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.de.xlf	Adds localized entry for the new error message.
src/Platform/Microsoft.Testing.Platform/Resources/xlf/PlatformResources.cs.xlf	Adds localized entry for the new error message.

Copilot's findings

• Files reviewed: 16/16 changed files
• Comments generated: 1

[Evangelink]

🧪 Test quality grade — PR #9055

6 new test methods graded across 1 file (JsonCommandLineOptionsTests.cs). All tests are clean: clear AAA structure, meaningful multi-faceted assertions, descriptive names, and well-chosen data-driven parameterization. No issues found that warrant follow-up.

Δ	Test	Grade	Band	Notes
new	JsonCommandLineOptionsTests. Validator_ CliBootstrapOnlyOption_ Allowed	A	90–100	Clear AAA; proves the supported CLI path remains valid; Assert.IsTrue with message parameter aids failure diagnostics.
new	JsonCommandLineOptionsTests. Validator_ JsonBootstrapOnlyOption_ CaseInsensitive_ Fails	A	90–100	Clear AAA; verifies OrdinalIgnoreCase matching; both failure status and the mixed-case option name appear in the error message.
new	JsonCommandLineOptionsTests. Validator_ JsonBootstrapOnlyOption_ DisabledEntry_ StillFails	A	90–100	Clear AAA; confirms isDisabled=true does not exempt a bootstrap-only option; comment explains the edge-case rationale.
new	JsonCommandLineOptionsTests. Validator_ JsonBootstrapOnlyOption_ Fails	A	90–100	Clear AAA; data-driven across all 6 bootstrap-only option keys; checks failure status and three distinct error-message signals.
new	JsonCommandLineOptionsTests. Validator_ JsonMultipleBootstrapOnlyOptions_ AllReported	A	90–100	Clear AAA; verifies error aggregation — all three option names must appear in the combined error message.
new	JsonCommandLineOptionsTests. Validator_ JsonNonBootstrapOption_ Allowed	A	90–100	Clear AAA sanity guard; verifies the new bootstrap-only check does not regress general JSON option support.

_{This advisory comment was generated automatically. Grades are heuristic
and informational — they do not block merging. Re-run with
/grade-tests.}

🤖 Automated content by GitHub Copilot. Posted via a maintainer's GitHub token, so it appears under their account — the account owner did not write or approve this content personally. Generated by the Grade Tests on PR (on open / sync) workflow. · 241.3 AIC · ⌖ 13.2 AIC · [◷]( · ◷)