Skip to content

ci: simplify fan-out and promote sharded macOS coverage - #3276

Merged
Hillary Mutisya (hillary-mutisya) merged 12 commits into
mainfrom
dev/hillarym/ci-build-smoke-profiling
Nov 10, 2026
Merged

Hillary Mutisya (hillary-mutisya) merged 12 commits into
mainfrom
dev/hillarym/ci-build-smoke-profiling

Conversation

@hillary-mutisya

@hillary-mutisya Hillary Mutisya (hillary-mutisya) commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Promote scheduled full macOS Node 24/26 coverage to four balanced, complete-package lanes per runtime, while retaining ordinary Ubuntu/Windows sharding and minimal macOS PR compatibility coverage.
  • Retire the legacy matrix, duplicate unsharded memory controls and experiment-named production entrypoints. Add one fail-closed TypeScript CI aggregate, retaining the six existing required-check aliases until a coordinated ruleset migration.
  • Bound nightly execution and owned-process cleanup; improve subprocess diagnostics, native-runtime handling and structured failure reporting without masking failed assertions.
  • Narrow offline memory/provider fixtures and replace fragile polling with bounded, state-labelled waits while retaining deterministic recovery, privacy and restart coverage.

Motivation

Full macOS runs were taking hours and sometimes outliving their useful test results. Rollout-only execution paths also made the PR graph harder to understand. Hosted qualification exposed additional fixture and reporting problems that obscured failures or coupled deterministic tests to unrelated maintenance work.

This branch addresses those reliability and orchestration costs without dropping the existing test inventory or increasing retry allowances. The accepted rollout does not depend on another one-off performance-comparison campaign.

Behavior and configuration

  • Scheduled full macOS coverage, and manual macos_qualification=true, build once per runtime and run all 82 package scripts across four lanes. Each lane contains one heavy package: dispatcher, default agent provider, memory service or memory MCP server. Package scripts remain sequential within a lane, with existing lifecycle/chained checks and retries preserved.
  • Full macOS lanes retain a 180-minute workspace budget, shared by first pass and retries, inside a 240-minute job limit. Deadlines/cancellation terminate the owned process tree and remain blocking infrastructure failures. Ordinary PR and local execution defaults are unchanged.
  • Remove the legacy matrix and fanout_experiment / fanout_include_controls selectors. Production entrypoints are now ci-fanout.yml and ciFanout.mjs.
  • Optional build/native caches and compact artifacts remain off by default; artifact compression remains 6. Remove injected GitHub setup/artifact timing instrumentation; use native job/step timestamps instead.
  • Azure smoke retains the full-workspace production build. smokeBuildScope=smoke-closure remains an explicit opt-in measurement, preserving CLI/shell checks and authentication/fork protections. Install the pinned standalone pnpm and scope store/concurrency configuration to the project.
  • Preserve Node 22 package-engine support; CI qualification remains Node 24/26. This does not claim Node 22 CI qualification.
  • Add missing structured Node/Jest failure reporting and bounded CLI subprocess diagnostics, preserving main's stdin closure and killed/non-numeric child-exit rejection. The CLI child deadline stays 30 seconds; existing retry budgets are unchanged.
  • Offline fixture changes remove unrelated setup/maintenance prerequisites, retain the maintenance being tested, reuse a closed recovery baseline and bound both polling and individual reads. The shared wait defaults to 30 seconds; recovery stages share a 45-second budget within the unchanged 60-second test deadline. No provider-backed endpoint is introduced into these tests.

Reviewer guide

  1. Workflow and aggregate, production reusable workflow, and runbook: coverage selection, fail-closed qualification, temporary aliases, configuration and ordered migration.
  2. Workspace lanes, partition/collector contracts and ciFanout tests: complete package/script inventory, source/toolchain provenance and no missing or duplicate work.
  3. Workspace harness, process/command helpers and reporter regressions: bounded cleanup, captured failures and unchanged retry semantics. Native pipe/runtime and generated-file fixtures add attributable startup/setup diagnostics.
  4. Memory recovery tests, shared wait helpers and service/MCP consumers: all five interruption phases, exact canonical/redirect/index state, no repeated synthesis or duplicate versions, privacy deletion and restart idempotency remain asserted. Source-following and provider fixture changes similarly isolate the contract under test.

Validation

Current PR head: dff20dc543bcc2ee546c8b373b0525dd209de35e.

Main integration and current-head validation

PR creation exposed one conflict with main in the workflow CLI test. With owner approval, 3c11e6491 was merged into the feature branch without rewriting history. The resolution preserves main's stdin closure, killed/signal handling and original error cause alongside this branch's bounded diagnostics and shared Jest reporter. One equivalent timeout regression is retained rather than duplicating both versions.

At dff20dc54, the full build passed (64.969 seconds), all 12 CLI tests and 62 shared-config/reporter/retry contracts passed, and full-branch formatting plus all four committed-HEAD ratchets passed against origin/main (merge-base 3c11e6491). The new PR-triggered checks must still finish.

The following larger memory selections and branch dispatches qualified the preceding implementation head, not the subsequent main-integration commit.

Hosted sanity at preceding implementation head 3f0efc4ae

Coverage Evidence Result
Ordinary Ubuntu/Windows and macOS compatibility 42906632864 Passed: 86 successful jobs, two intentionally skipped; strict qualification and TypeScript CI green
Full sharded macOS Node 24/26 42906632875 Passed: 18 successful jobs, six intentionally skipped; strict qualification and TypeScript CI green

Both dispatches used disabled optional caches/compact artifacts and compression 6. They are hosted offline CI qualification, not provider-backed live integration. PR-triggered checks and external smoke/.NET/policy checks are separate and must still complete.

Local implementation validation at 3f0efc4ae

Windows, Node 26.13.0, standalone pnpm 11.9.0; existing dependencies, no Corepack. Import the PowerShell profile before commands. Local package scripts used pnpm_config_verify_deps_before_run=false to avoid an implicit dependency reinstall; focused Jest runs used CI=true.

  • pnpm run build from ts: passed (83.221 seconds).
  • From ts/packages/memory/service:
    pnpm run jest-esm --runInBand --testPathPattern='(waitForTestState|wikiConsolidationRecovery|wikiConsolidationTemporal|sourceFollowing|evidenceWindow|evidenceWindowUpgrade|graphService|schemaViews|snapshotReuse|timeline|informationalWiki|knowledgeAnswering|topicService|viewBuilds|viewExtensions|viewIndexConcurrency|viewSynthesisEvidence)[.]spec[.]js$'
    All 347 tests in 17 suites passed. Two additional complete wikiConsolidationRecovery.spec runs passed all six cases each.
  • From ts/packages/memory/mcp-server:
    pnpm run jest-esm --runInBand --testPathPattern='(viewBuildProtocol|extensionViewProtocol|viewInventoryProtocol)[.]spec[.]js$'
    All nine tests in three suites passed.
  • Repository changed-file formatting and all four ratchets passed against freshly fetched origin/main, after the final commit:
    node tools\scripts\prettier-changed.mjs --base origin/main
    pnpm run code-lint -- --ratchet --base origin/main
    pnpm run code-complexity -- --ratchet --base origin/main --cyclomatic 25 --cognitive 30 --new-file-cyclomatic 25 --new-file-cognitive 30
    pnpm run code-circular -- --ratchet --base origin/main --exceptions-file tools\scripts\code\circular-baseline-exception.json
    pnpm run code-debt -- --gate --base origin/main

Earlier targeted reporter, retry, lane/inventory, process-cleanup and fixture regressions also passed during implementation; the two final hosted qualifications exercise the pre-merge implementation head. No tests were newly skipped.

Risks, limitations and rollout

  • Do not remove the six build_ts (OS, Node) aliases in this PR. Merge under the existing required contexts, verify the deployed aggregate on ordinary main/PR/merge-group runs, then coordinate replacement of only those six contexts with TypeScript CI. Preserve all unrelated required checks and policy. Remove aliases in a separate follow-up after the ruleset cutover.
  • No branch protection, required-check ruleset, Azure service connection or approval policy has been changed. No merge or auto-merge is requested.
  • Before ruleset cutover, rollback is a source revert with existing aliases retained. After cutover, coordinate the matching workflow and required-context restoration; do not leave required checks without implementations.
  • The final ordinary and full-macOS runs are correctness/reliability qualification, not matched p50/p95 performance evidence. Earlier failed controls and cross-commit timing observations are not presented as causal speedup measurements.
  • The original isolated Windows recovery stall and Ubuntu runner disconnect were not causally established. Fixture isolation, bounded reads and diagnostics address the identified test weaknesses without speculative production tuning.
  • Documentation-only fast paths, source/toolchain/native provenance, inventory collectors, full-macOS root-script/memory-quality coverage and ordinary memory-shard failure semantics remain intact.

Preserve late dispatcher cleanup, enforce a shared workspace deadline with process-tree cleanup, and keep original failures blocking. Add macOS-only branch qualification with the nightly budgets without changing PR defaults.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Compare real CLI entry paths, retain import-only behavior, and add stage-specific credential-safe fixture process diagnostics without retries. Preserve the existing optional pnpm availability probe.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Measure exact-source setup, artifact and build stages without changing test inventories or production cache defaults. Add an Azure full-versus-CLI/shell dependency-closure comparison, preserving authentication and test gates. Bootstrap pinned standalone pnpm rather than enabling Corepack.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Avoid requiring a user-global pnpm bin directory on fresh Windows/Linux agents. Configure the store and CPU-based workspace concurrency explicitly at project scope, and exercise the real pinned pnpm CLI in an isolated regression that preserves user configuration.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Separate profiling records from native probe mocks and cover both profiling modes through bounded retries. Canonicalize aliased deadline fixtures and report ingestion state and progress without increasing the polling budget.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retain actual ingestion, current-revision assertions and indexed retrieval limits without generating an unrelated wiki. Preserve the dedicated source-following lifecycle scenarios.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Restore uninstrumented setup, artifact operations and build commands while retaining the existing native-cache/retry policies, coverage and reliability fixes. Keep Azure smoke profiling and validate that inherited profiling flags cannot instrument GitHub artifacts.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Run four full-package macOS lanes per runtime with sequential packages and one heavy workload per lane. Preserve exact package scripts, retries, roots, approved deadlines and fail-closed collectors. Keep ordinary PR coverage and nightly scheduling unchanged, with the existing manual legacy rollback.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use full-package macOS lanes for scheduled coverage, retire legacy and duplicate control execution, and introduce the TypeScript CI aggregate with temporary required-check aliases. Preserve optional cache settings and Node 22 support while qualifying Node 24/26.

Wire structured Node test failures into existing retries and retain bounded native pipe startup diagnostics without weakening readiness or identity assertions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use the shared Jest configuration so CLI assertion failures reach the existing workspace retry harness. Reject killed or timed-out CLI child processes with bounded stdout/stderr and signal diagnostics instead of falsely returning exit zero; preserve the 30-second deadline and dry-run assertions.

Add real killed-child and Jest failure-artifact regressions. Hosted Ubuntu runner communication loss remains an unconfirmed infrastructure failure, not a basis for speculative resource tuning.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reuse a closed consolidation baseline and exclude unrelated discovery while retaining real interrupted-effect recovery, privacy and restart assertions. Bound shared polling and individual reads, preserve last-state diagnostics, and reuse durable-build waits across service and MCP fixtures.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Merge current main without rewriting the validated feature history. Preserve main stdin closure and process-error causes alongside bounded subprocess diagnostics and the shared failure reporter, retaining one timeout regression.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@hillary-mutisya
Hillary Mutisya (hillary-mutisya) merged commit b833369 into main Nov 10, 2026
109 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant