Speed up Code Coverage Jobs by paulmedynski · Pull Request #3907 · dotnet/SqlClient

paulmedynski · 2026-01-21T15:58:00Z

Description

The code coverage pipeline job is inefficient, and can be greatly streamlined. We don't need to publish separate uploads to CodeCov for MDS .NET, .NET Framework, and AKV.

Testing

PR runs will validate the new consolidation steps.
I will manually verify the Azure DevOps test results and coverage, and CodeCov reports.

Copilot

Pull request overview

This pull request optimizes CI/CD pipeline execution by separating unit and integration tests to reduce redundant test execution. The PR introduces a testType parameter that controls which test suites run in different pipeline contexts.

Changes:

Added testType parameter ('All', 'Unit', or 'Integration') to pipeline templates
Modified PR pipelines to run only relevant test suites (Unit tests in Project builds, Integration tests in Package builds)
Updated test execution conditionals to respect the new testType parameter

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
eng/pipelines/sqlclient-pr-project-ref-pipeline.yml	Sets testType to 'Unit' for project reference builds, restricting to unit and functional tests
eng/pipelines/sqlclient-pr-package-ref-pipeline.yml	Sets testType to 'Integration' for package reference builds, restricting to manual tests
eng/pipelines/dotnet-sqlclient-ci-core.yml	Defines testType parameter with default 'All' for CI pipelines to continue running all tests
eng/pipelines/common/templates/steps/run-all-tests-step.yml	Implements conditional test execution based on testType parameter
eng/pipelines/common/templates/stages/ci-run-tests-stage.yml	Propagates testType parameter through stage template
eng/pipelines/common/templates/jobs/ci-run-tests-job.yml	Propagates testType parameter through job template

eng/pipelines/common/templates/stages/ci-run-tests-stage.yml

eng/pipelines/common/templates/steps/run-all-tests-step.yml

eng/pipelines/common/templates/jobs/ci-run-tests-job.yml

codecov · 2026-01-21T17:06:00Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 63.85%. Comparing base (3ffbba7) to head (2fa8826).
⚠️ Report is 9 commits behind head on main.

❗ There is a different number of reports uploaded between BASE (3ffbba7) and HEAD (2fa8826). Click for more details.

HEAD has 3 uploads less than BASE

Flag BASE (3ffbba7) HEAD (2fa8826)

netfx 1 0

netcore 1 0

addons 1 0

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #3907       +/-   ##
===========================================
- Coverage   74.53%   63.85%   -10.68%     
===========================================
  Files         266      282       +16     
  Lines       42915    66380    +23465     
===========================================
+ Hits        31987    42388    +10401     
- Misses      10928    23992    +13064

Flag	Coverage Δ
PR-SqlClient-Project	`63.85% <ø> (?)`
addons	`?`
netcore	`?`
netfx	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

benrr101

Overall, probably a good change. The main issue I have right now is the lumping of functional tests with unit tests. My understanding of them were that they were not unit tests in theory, they were just the simple integration tests that validate they do the most basic stuff. As it turns out a lot of those are unit tests, but not the majority. I was working on the side a bit to move the unit tests from the functional tests and into the unit test project. Ultimately I think we're aligned that we want: unit test, local integration test, remote integration test, stress test, and fuzz test projects. And this change is kinda orthogonal to those goals, without getting in the way of them. So go for it 😆

I think another low-ish hanging fruit is to separate the unit tests and functional tests as separate jobs, like the test sets. That way we don't need to repeat unit and functional tests four times for each platform. They're fast, yes, but not instantaneous, and I imagine we could shave a decent amount of time off a build doing it.

paulmedynski · 2026-01-21T18:05:22Z

You're correct on all counts. I chose to combine the running of Unit and Functional tests because they're mostly unit with some local integration sprinkled in, but mainly because their combined run time is close to the Manual ones. This gives us the most bang-for-the-buck in terms of speeding up the PR runs right now.

We also shouldn't be running Unit and Functional tests in any of the Azure SQL configurations - another future optimization.

eng/pipelines/dotnet-sqlclient-ci-core.yml

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

eng/pipelines/dotnet-sqlclient-ci-core.yml

priyankatiwari08

Looks good to me. Manual tests in itself takes quite significant amount of time to run. So, this segregation should help us in longer run.

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

eng/pipelines/sqlclient-pr-project-ref-pipeline.yml

- Project runs will perform unit tests (i.e. MDS Unit and Functional suites). - Package runs will perform integration tests (i.e. MDS Manual suite).

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

- Use a temp dir to avoid cluttering up the pooled agents. - Combine all merging/converting into a few files as possible. - Use the modern features of the dotnet tools to avoid parallel jobs and loops. - Combined all MDS reports into a single upload, and same for AKV.

eng/pipelines/dotnet-sqlclient-ci-core.yml

paulmedynski · 2026-01-23T21:01:44Z

eng/pipelines/common/templates/jobs/ci-code-coverage-job.yml

    variables:
-      netFxDir: $(Build.SourcesDirectory)\coverageNetFx
-      netCoreDir: $(Build.SourcesDirectory)\coverageNetCore
+      # Use a temp directory that is cleaned up after each job runs.  This helps


I will run the pipeline in debug mode to make sure this temp dir lives on a partition/mount with sufficient space.

Confirmed:

AGENT_TEMPDIRECTORY=/home/vsts/work/_temp Filesystem Size Used Avail Use% Mounted on /dev/root 72G 54G 19G 75% /

The temp dir is on the root partition with plenty of space.

paulmedynski · 2026-01-23T21:03:52Z

eng/pipelines/common/templates/jobs/ci-code-coverage-job.yml

-            ${{ else }}:
-              targetPath: $(netCoreDir)
+
+      # Download all of the coverage reports from the test jobs.


I greatly simplified things here. The old tasks went out of their way to separate the MDS .NET and .NET Framework coverage and reports. Now that we've merged the codebases, I can't see why we would care about that. So the new approach is to merge all of the test results together, generate Cobertura reports for MDS and AKV, and then upload them as separate entities. Thoughts?

- Removed unnecessary parameters related to test and coverage publishing.

Copilot

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated no new comments.

paulmedynski · 2026-01-29T13:39:04Z

eng/pipelines/common/templates/jobs/ci-build-nugets-job.yml

      nuspecPath: 'tools/specs/Microsoft.Data.SqlClient.nuspec'
      OutputDirectory: $(packagePath)
-      generateSymbolsPackage: false
+      generateSymbolsPackage: true


Developers will want symbols packages now that PDBs aren't embedded in the NuGet package.

paulmedynski · 2026-01-29T13:40:55Z

eng/pipelines/common/templates/jobs/ci-run-tests-job.yml

        dotnetx86RootPath: $(dotnetx86RootPath)
        operatingSystem: ${{ parameters.operatingSystem }}

-  - ${{ if eq(parameters.publishTestResults, true) }}:


We always want to publish test results.

paulmedynski · 2026-01-29T13:42:09Z

eng/pipelines/common/templates/steps/build-and-run-tests-netcore-step.yml

      --no-build
      -v n
      --filter "category!=failing&category!=flaky"
-      --collect "Code Coverage"


This file is only used in Official pipelines, so no need for coverage reporting.

paulmedynski · 2026-01-29T13:42:21Z

eng/pipelines/common/templates/steps/build-and-run-tests-netfx-step.yml

      --no-build
      -v n
      --filter "category!=failing&category!=flaky"
-      --collect "Code Coverage"


This file is only used in Official pipelines, so no need for coverage reporting.

paulmedynski · 2026-01-29T13:43:03Z

eng/pipelines/common/templates/steps/ci-prebuild-step.yml

      artifact: ${{parameters.artifactName }}
-      patterns: '**/*.nupkg'
-      targetPath: $(Pipeline.Workspace)/${{parameters.artifactName }}
+      patterns: '**/*.*nupkg'


Might as well download the symbols packages too.

paulmedynski · 2026-01-29T13:49:19Z

eng/pipelines/dotnet-sqlclient-ci-core.yml

-      # Include the code coverage job if the build type is Project.
+
+      # Include the code coverage job for Project reference builds.  Coverage
+      # can't be collected properly when we use Package references.


Ok maybe it can, but I went down several rabbit holes and couldn't get coverage from the MDS assembly when it was referenced via package in the tests. We're probably better off adding proper coverage to Unit and Functional tests, and then never using Manual tests to collect coverage.

paulmedynski · 2026-01-29T13:52:39Z

src/Microsoft.Data.SqlClient/add-ons/Directory.Build.props

    <TargetsWindows Condition="'$(OSGroup)'=='Windows_NT'">true</TargetsWindows>
    <TargetsUnix Condition="'$(OSGroup)'=='Unix'">true</TargetsUnix>
    <ReferenceType Condition="'$(ReferenceType)'==''">Project</ReferenceType>
-    <AllowedOutputExtensionsInPackageBuildOutputFolder>$(AllowedOutputExtensionsInPackageBuildOutputFolder);.pdb</AllowedOutputExtensionsInPackageBuildOutputFolder>


We shouldn't be publishing PDBs with our NuGet packages. We already publish separate symbols packages.

paulmedynski · 2026-01-29T13:53:56Z

src/Microsoft.Data.SqlClient/tests/Directory.Build.props

+      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
+      <Link>CodeCoverage.runsettings</Link>
+    </None>
+    <None Include="$(MSBuildThisFileDirectory)tools\Microsoft.Data.SqlClient.TestUtilities\xunit.runner.json">


All of the tests need this file, so it made sense to copy it once here rather than multiple times in each project.

paulmedynski · 2026-01-29T13:55:01Z

...t.Data.SqlClient/tests/tools/Microsoft.Data.SqlClient.TestUtilities/CodeCoverage.runsettings

+            <!-- Include only product assemblies for coverage -->
+            <ModulePaths>
+              <Include>
+                <ModulePath>.*Microsoft\.Data\.SqlClient\.dll$</ModulePath>


This reduces the amount of coverage data generated during test runs to just those assemblies that we care about.

paulmedynski · 2026-01-29T13:55:26Z

...t.Data.SqlClient/tests/tools/Microsoft.Data.SqlClient.TestUtilities/CodeCoverage.runsettings

Copilot generated this file for me. It seems like a good idea.

Copilot

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated 1 comment.

eng/pipelines/common/templates/jobs/ci-code-coverage-job.yml

Copilot

Pull request overview

Copilot reviewed 18 out of 21 changed files in this pull request and generated 5 comments.

...t.Data.SqlClient/tests/tools/Microsoft.Data.SqlClient.TestUtilities/CodeCoverage.runsettings

...crosoft.Data.SqlClient/tests/FunctionalTests/Microsoft.Data.SqlClient.FunctionalTests.csproj

eng/pipelines/common/templates/stages/ci-run-tests-stage.yml

eng/pipelines/common/templates/jobs/ci-code-coverage-job.yml

build.proj

paulmedynski · 2026-02-10T18:03:45Z

eng/pipelines/common/templates/steps/run-all-tests-step.yml

          -p:TF=${{ parameters.targetFramework }}
          -p:TestSet=${{ parameters.testSet }}
          -p:ReferenceType=${{ parameters.referenceType }}
-          -p:AbstractionsPackageVersion=${{ parameters.abstractionsPackageVersion }}


Unit tests only use Project mode, so no need for package versions.

paulmedynski · 2026-02-10T18:04:09Z

eng/pipelines/common/templates/steps/run-all-tests-step.yml

+          -p:Platform=${{ parameters.platform }}
          -p:Configuration=${{ parameters.buildConfiguration }}
          -p:Filter="category=flaky"
+          -p:CollectCodeCoverage=false


No coverage for flaky tests.

...crosoft.Data.SqlClient/tests/FunctionalTests/Microsoft.Data.SqlClient.FunctionalTests.csproj

…icate-runs

Copilot

Pull request overview

Copilot reviewed 21 out of 24 changed files in this pull request and generated no new comments.

paulmedynski requested a review from a team as a code owner January 21, 2026 15:58

Copilot AI review requested due to automatic review settings January 21, 2026 15:58

paulmedynski added Area\Tests Issues that are targeted to tests or test projects Area\Engineering Use this for issues that are targeted for changes in the 'eng' folder or build systems. labels Jan 21, 2026

Copilot started reviewing on behalf of paulmedynski January 21, 2026 15:58 View session

Copilot AI reviewed Jan 21, 2026

View reviewed changes

eng/pipelines/common/templates/stages/ci-run-tests-stage.yml Outdated Show resolved Hide resolved

eng/pipelines/common/templates/steps/run-all-tests-step.yml Outdated Show resolved Hide resolved

eng/pipelines/common/templates/jobs/ci-run-tests-job.yml Outdated Show resolved Hide resolved

benrr101 previously approved these changes Jan 21, 2026

View reviewed changes

paulmedynski dismissed benrr101’s stale review via 546a0eb January 21, 2026 18:20

paulmedynski commented Jan 21, 2026

View reviewed changes

eng/pipelines/dotnet-sqlclient-ci-core.yml Outdated Show resolved Hide resolved

Copilot AI review requested due to automatic review settings January 22, 2026 00:29

Copilot started reviewing on behalf of paulmedynski January 22, 2026 00:29 View session

Copilot AI reviewed Jan 22, 2026

View reviewed changes

eng/pipelines/dotnet-sqlclient-ci-core.yml Outdated Show resolved Hide resolved

eng/pipelines/dotnet-sqlclient-ci-core.yml Outdated Show resolved Hide resolved

priyankatiwari08 self-assigned this Jan 22, 2026

priyankatiwari08 previously approved these changes Jan 22, 2026

View reviewed changes

paulmedynski dismissed priyankatiwari08’s stale review via e170d4d January 22, 2026 18:35

Copilot AI review requested due to automatic review settings January 22, 2026 18:46

Copilot started reviewing on behalf of paulmedynski January 22, 2026 18:46 View session

Copilot AI reviewed Jan 22, 2026

View reviewed changes

eng/pipelines/sqlclient-pr-project-ref-pipeline.yml Outdated Show resolved Hide resolved

eng/pipelines/sqlclient-pr-project-ref-pipeline.yml Outdated Show resolved Hide resolved

apoorvdeshmukh previously approved these changes Jan 23, 2026

View reviewed changes

PR pipeline runs will separate which tests they run:

523250b

- Project runs will perform unit tests (i.e. MDS Unit and Functional suites). - Package runs will perform integration tests (i.e. MDS Manual suite).

Copilot AI review requested due to automatic review settings January 23, 2026 13:52

paulmedynski dismissed apoorvdeshmukh’s stale review via 523250b January 23, 2026 13:52

paulmedynski force-pushed the dev/paul/duplicate-runs branch from 190fcb7 to 523250b Compare January 23, 2026 13:52

Copilot started reviewing on behalf of paulmedynski January 23, 2026 13:53 View session

Copilot AI reviewed Jan 23, 2026

View reviewed changes

paulmedynski commented Jan 23, 2026

View reviewed changes

- Moved code coverage job to Linux and streamlined it even further.

8b7da18

- Removed unnecessary parameters related to test and coverage publishing.

Copilot AI reviewed Jan 29, 2026

View reviewed changes

Reverted changes to blame config.

6ef1ea2

paulmedynski commented Jan 29, 2026

View reviewed changes

paulmedynski changed the title ~~[DRAFT] Separate Unit and Integration Test Runs~~ [DRAFT] Speed up Code Coverage Jobs Jan 29, 2026

paulmedynski changed the title ~~[DRAFT] Speed up Code Coverage Jobs~~ Speed up Code Coverage Jobs Jan 29, 2026

paulmedynski marked this pull request as ready for review January 29, 2026 21:04

Copilot AI review requested due to automatic review settings January 29, 2026 21:04

Copilot started reviewing on behalf of paulmedynski January 29, 2026 21:04 View session

Copilot AI reviewed Jan 29, 2026

View reviewed changes

eng/pipelines/common/templates/jobs/ci-code-coverage-job.yml Show resolved Hide resolved

cheenamalhotra previously approved these changes Jan 29, 2026

View reviewed changes

paulmedynski mentioned this pull request Feb 4, 2026

Create separate flaky tests pipeline #3936

Draft

paulmedynski added 2 commits February 10, 2026 11:35

Removed stale comments from PR pipelines.

d85a1f3

Merge branch 'main' into dev/paul/duplicate-runs

327ea6d

paulmedynski dismissed cheenamalhotra’s stale review via 327ea6d February 10, 2026 17:46

Merge branch 'main' into dev/paul/duplicate-runs

ec25bc9

Copilot AI review requested due to automatic review settings February 10, 2026 17:49

Copilot started reviewing on behalf of paulmedynski February 10, 2026 17:50 View session

Copilot AI reviewed Feb 10, 2026

View reviewed changes

paulmedynski commented Feb 10, 2026

View reviewed changes

Addressed Copilot feedback.

2fa8826

apoorvdeshmukh previously approved these changes Feb 13, 2026

View reviewed changes

mdaigle previously approved these changes Feb 13, 2026

View reviewed changes

Merge branch 'main' of github.com:dotnet/SqlClient into dev/paul/dupl…

5d2d5f4

…icate-runs

Copilot AI review requested due to automatic review settings February 13, 2026 21:44

mdaigle dismissed stale reviews from apoorvdeshmukh and themself via 5d2d5f4 February 13, 2026 21:44

Copilot started reviewing on behalf of mdaigle February 13, 2026 21:44 View session

mdaigle approved these changes Feb 13, 2026

View reviewed changes

Copilot AI reviewed Feb 13, 2026

View reviewed changes

cheenamalhotra approved these changes Feb 13, 2026

View reviewed changes

Conversation

paulmedynski commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

benrr101 left a comment

Choose a reason for hiding this comment

Uh oh!

paulmedynski commented Jan 21, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

priyankatiwari08 left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

paulmedynski commented Jan 21, 2026 •

edited

Loading

codecov bot commented Jan 21, 2026 •

edited

Loading