[SDTEST-3827] Abort full discovery when no tests are skippable#75
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f4da31442b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| skippedTests = tp.fetchTestsToSkip(tiaSkippingEnabled) | ||
| tp.planReport.SkippableTestsCount = skippedTests.Count() | ||
|
|
||
| if tiaSkippingEnabled && len(skippedTests.tiaSkippableTests) == 0 { |
There was a problem hiding this comment.
When TIA is enabled but Datadog returns zero TIA-skippable tests, this cancels full discovery even if fetchTestsToSkip just populated skippedTests.disabledTests from Test Management. In that scenario (Test Management enabled with disabled tests but no TIA skips), a still-running full discovery is aborted and the planner falls back to fast file discovery, so recordFullDiscoveryResults never calls skippableTests.Contains to mark disabled tests as skipped or drop fully skipped files. Please gate this on the combined skip set (or on len(skippedTests.disabledTests) == 0) rather than TIA-only skips.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Test suite is supposed to have a very small number of disabled tests, it is not useful to run long tests discovery process to be able to find the test suites that would be fully disabled - the chances of this happening are close to zero.
TIA is another story - for some commits we are going to skip a lot of tests, up to 100% - so if we skip tests, we always want to run tests discovery.
anmarchenko
changed the title
E2E Test Report: SUCCESSTested by: Shepherd Agent (autonomous QA for Datadog Test Optimization) Test Environment
Results
Verification DetailsAll successful runs used mockdog scenarios where TIA/test skipping was enabled and the skippable-tests endpoint returned an empty list ( Observed planner evidence:
Stashed output checks:
Plan outputs:
Issues Found
Datadog UI VerificationNot applicable for this verification pass. These were local mockdog-backed planner runs; no Datadog UI data was expected. Backend interactions were verified through mockdog HTTP caches and debug logs. Test Methodology
This E2E test was performed by Shepherd - autonomous QA agent for Datadog Test Optimization. |
2 participants
What
Abort long-running full test discovery when TIA/test skipping is enabled but Datadog returns no TIA-skippable tests for the run. In that case planning falls back to fast test-file discovery. If full discovery completes successfully before that cancellation takes effect, its results are still used.
Why
Full discovery is only needed to map TIA-skippable tests onto local test files. When the skippable set is empty, continuing full discovery adds planning latency without changing which tests should run.
E2E testing
Run
ddtest planin a repository with TIA/test skipping enabled where the skippable-tests response is empty. Verify the planner logsNo TIA-skippable tests found for this run, cancelling full test discovery, writes the plan from fast-discovered test files, and still completes successfully.