Skip to content

refactor(ci): parallelize native qualification and promote verified release candidates - #2879

Open
Daniel Meppiel (danielmeppiel) wants to merge 23 commits into
mainfrom
danielmeppiel-release-pipeline-audit
Open

refactor(ci): parallelize native qualification and promote verified release candidates#2879
Daniel Meppiel (danielmeppiel) wants to merge 23 commits into
mainfrom
danielmeppiel-release-pipeline-audit

Conversation

@danielmeppiel

@danielmeppiel Daniel Meppiel (danielmeppiel) commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator

refactor(ci): parallelize native qualification and promote verified release candidates

TL;DR

Fold both release-engineering phases into this PR: parallelize qualification, test and publish the exact built archives, remove redundant source-test work, and prebuild docs/wheels without publishing privileges. Two successful read-only GitHub pipelines now show 34m41s -> 24m53s: 9m48s / 28.26% shorter, with 22.77% higher unweighted runner time. The Windows installer blocker is resolved, but production sharding remains unchanged; the earlier three sharding rehearsals remain rejected.

Important

Measured proposed source: 1687380d74cc255519b2ab93ced5ac8744fdd54b. This is an observed successful pair, not a controlled population estimate: retries exceeded the declared budget, an extra run overlapped, and images varied. Runner records were transparently reconciled from original artifacts; a subsequent fail-closed reporting repair is not included in the historical timing. Signing/publishing were excluded. The earlier 22-29m planning envelope remains withdrawn.

Problem (WHY)

Motivation: the maintainer requested lower latency without weakening safeguards in an agent-developed repository with limited routine human review. Ten bounded test/CI investigations informed the changes; estimates were not treated as measurements.

Historical baseline: run 34104956270, SHA 8c2e0d9c352e2ed0e8c56b40063a63e1dd4a1937.

Approach (WHAT)

  • Start native units/builds independently and each consumer after its own build.
  • Require source checks and every applicable native result before qualification.
  • Package once, verify and execute those bytes, then promote only exact-SHA/version trusted full candidates.
  • Share the integration executor and immutable timing snapshot; retain production native Unix lanes at one shard/four workers pending whole-DAG proof and rollout approval.
  • Overlap MQ smoke with integration and read-only docs/wheel builds with release qualification; keep publication behind successful release creation.
  • Remove only disproven prerequisites and unnecessary AST work, retaining original scenarios and failure guards.
  • Measure matched executions; reject omissions, overlapping shards, new skips, failures, or identity/environment drift.

Implementation (HOW)

Files Intent and scope
.github/workflows/release-platform.yml, .github/workflows/release-integration.yml Independent native paths; shared integration executor; required fan-in; verified candidate extraction; no unaccepted ARM rollout.
.github/workflows/build-release.yml, scripts/release-platforms.json Canonical platform policy, fresh/warm orchestration, early read-only prebuilds, guarded publishers.
.github/workflows/docs.yml, .github/workflows/docs-build.yml, .github/workflows/pypi-distributions.yml Separate reusable read-only builders from Pages/PyPI authority; preserve standalone docs behavior.
scripts/package_release.py Archive once; bind SHA/version/digests, enforce extraction safety, verify executable identity.
scripts/release-candidate.cjs, scripts/release-candidate.test.cjs Exact qualification and immutable promotion protocol; transitive workflow fingerprints; malformed/tampered/rerun negatives.
.github/workflows/ci-source-performance.yml, .github/workflows/ci-release-performance.yml Opt-in, read-only hosted comparisons; current ARM candidate; experimental 2x3 against 1x4; shared cold snapshot; no release authority.
.github/workflows/ci-release-rehearsal.yml, .github/workflows/release-unit.yml All-five-platform/source/read-only-builder rehearsal; canonical native units; opt-in ARM/Linux integration and Intel unit pairs; unchanged production defaults.
.github/workflows/ci-release-wallclock.yml, scripts/release_wallclock.py, corresponding unit contracts Frozen read-only full-DAG comparison; exact native/observation artifact IDs, fail-closed runner evidence, and actual attempt-specific GitHub clocks.
.github/workflows/ci-windows-installer-proof.yml, scripts/windows/run-installer-proof.ps1, corresponding unit contracts Same-archive long/short-root diagnostic; real native stderr controls; reject empty/skipped installer evidence.
.github/actions/performance-cohort/action.yml, scripts/performance_cohort.py, scripts/performance-cohort.cjs Three actual runners agree before pytest; bounded wait; current-attempt identity/artifacts; no mixed-image timings accepted.
scripts/release_rehearsal.py, tests/unit/scripts/test_release_rehearsal.py, tests/unit/test_release_rehearsal_workflow.py Exact proof/job binding; full native/source DAG, observed native gate delays, unchanged bottlenecks, cost/tail sensitivity and rejecting mutations.
tests/unit/scripts/test_performance_cohort.py, scripts/performance-cohort.test.cjs Reject missing/stale/duplicate/image-drift evidence, invalid runner traces, API failure and timeout before expensive tests.
scripts/compare_test_runs.py, scripts/pytest_performance_evidence.py Exact original-node/outcome equivalence, environment/lock fingerprints, complete shard coverage; concrete pytest 9 subtests emitted by the controller without changing worker scheduling IDs.
scripts/architecture_linter/checks/lexical_shared.py Use cached lexical reads instead of constructing ASTs for grep-equivalent duplicate scans.
tests/unit/scripts/test_architecture_runner.py, tests/integration/test_architecture_owner_rule_mutations.py Parser/index counters, malformed/override/exemption/error cases, two missing guard mutations, cheap PR-time completeness assertion.
scripts/windows/test-install-script.ps1, tests/utils/windows_installer_candidate.py Candidate install/reinstall/launcher/upgrade fixtures; older release only seeds upgrade; loopback archive integrity.
tests/integration/test_windows_installer_launchers.py, tests/unit/test_windows_installer_candidate.py Real candidate installer wiring and negative input/digest/archive/version/server cases.
tests/unit/test_windows_installer_launchers.py, tests/unit/install/test_mcp_registry_config_layer.py Short nonexistent installer fixture root with actual constructor regression; temporary registry cache cleared at teardown with a lifecycle regression.
tests/unit/test_console_utils.py Restore parent-package module attributes as well as sys.modules after fallback probes; Rich/Colorama identity regressions catch the shard-exposed leak.
scripts/windows/test-release-validation.ps1, scripts/windows/test-dependency-integration.ps1, tests/unit/test_release_validation_prerequisites.py Explicit API-only public validation retains dependency checks without PAT/Models aliases; actual PowerShell negatives preserve private/inference credential boundaries.
pyproject.toml, uv.lock Explicit packaging dev dependency for candidate version comparison; no package-version changes.
scripts/build-binary.sh, scripts/windows/build-binary.ps1 Remove unused UPX work; enforce Windows build-SHA injection.
scripts/test-integration.sh Selected runtime setup, explicit candidate path, benchmark plugin module entrypoint, ASCII status output.
tests/conftest.py, tests/integration/conftest.py, tests/unit/test_runtime_prerequisite_gate.py Root-registered strict prerequisites that work during parent-directory collection.
tests/integration/test_mcp_env_var_copilot_e2e.py, tests/unit/test_mcp_runtime_prerequisite.py Remove false Copilot requirement, retaining all three configuration scenarios, binary prerequisite and HOME group.
.github/workflows/ci.yml, .github/workflows/ci-integration.yml, .github/workflows/ci-runtime.yml Source evidence, Windows two-worker gate, MQ overlap/exact archives, runtime smoke opt-ins.
.github/actions/pytest-timing/action.yml, tests/unit/test_pytest_timing_workflows.py Shared immutable hints; real cold/weighted pytest-split/xdist partition proofs; attempt-scoped artifacts.
.github/workflows/merge-gate.yml, .github/scripts/ci/merge_gate_wait.sh, tests/unit/test_merge_gate_wait.py, tests/quality/test_ci_topology.py Include required authorities; reject ambiguous/skipped/neutral results; follow reusable lifecycle execution.
tests/integration/test_mcp_registry_e2e.py, tests/unit/test_mcp_registry_collection.py No collection-time HTTP; one lazy shared health probe; unchanged selected cases.
tests/unit/test_unix_installer_checksums.py python -S for stdlib-only fake tools; all 102 scenarios retained.
tests/unit/scripts/test_package_release.py, tests/unit/scripts/test_compare_test_runs.py Real archive roundtrips, malformed proof negatives, original-ID interoperability and exact outcome comparison.
tests/unit/test_platform_contract_workflow.py, tests/unit/test_release_performance_workflow.py, tests/unit/test_release_publication_overlap.py Executed gates, producer/consumer ABI, artifact layout, retry isolation, publication failure/skip simulations and permission boundaries.
tests/unit/test_ado_live_workflow_contract.py, tests/unit/test_docs_contract_workflow.py, tests/unit/test_security_workflow_contracts.py, tests/unit/test_windows_compat_gate_workflow.py Preserve auth/live selection, docs authority, privileged-job isolation and Windows selection.
tests/unit/test_windows_signing_contract.py, tests/unit/test_windows_unit_diagnostics.py Preserve signing and native diagnostics after extraction.
.apm/instructions/cicd.instructions.md, .github/instructions/cicd.instructions.md, apm.lock.yaml Canonical/deployed CI instruction and deployment-record parity.
docs/src/content/docs/contributing/development-guide.md, docs/src/content/docs/contributing/integration-testing.md Qualification, runtime selection, sharding, recovery, measurement and opt-in probe instructions.

Review anchors: native executor, publication, proof checker.

No application dependency or production installer interface changes. No new CHANGELOG entry: internal release/test orchestration; upstream release notes remain unchanged.

Diagrams

The artifact/authority pipeline shows independent evidence converging before publication; full/Windows-only consumers remain conditional, and prebuilding never grants publishing permission.

flowchart LR
    P["plan"] -->|"fresh full"| S["candidate-checks"]
    P -->|"fresh"| U["native unit-tests"]
    P -->|"fresh"| B["native build and package"]
    subgraph Native["Per-platform evidence"]
        B --> I["integration-tests-shard"]
        I --> F["Integration Tests fan-in"]
        B --> V["release-validation"]
        B --> W["windows-installer"]
        U --> G["Native Candidate Gate"]
        F --> G
        V --> G
        W --> G
    end
    S --> Q["candidate-ready"]
    G --> Q
    Q --> E["verify-candidate"]
    P -->|"trusted exact-SHA candidate"| E
    E --> A["10 verified assets"]
    A --> R["create-release"]
    P -->|"stable tag"| DB["build-docs-artifact"]
    P -->|"stable public tag"| WB["build-pypi-distributions"]
    DB --> DP["deploy-docs"]
    R --> DP
    WB --> WP["publish-pypi"]
    R --> WP
    classDef new stroke-dasharray: 5 5;
    class P,B,I,F,G,Q,E,A,DB,WB new;
Loading

Trade-offs

  • Elapsed time versus runner cost: paired rehearsal runs add allocations and cohort waits; this is not proposed production duration. Native gates remain in the DAG using observed dependency-to-verdict delays held constant across modeled worlds.
  • Cold proof, not cached success: all ARM variants consume one empty immutable snapshot. No fresh timing-cache writer or warm-balancing improvement is claimed; every test still executes.
  • Fail-closed recovery: missing/expired automatic candidates rebuild; inconsistent evidence/API failures stop. Whole rehearsals use fresh label-triggered run IDs: their proofs are attempt-scoped, but canonical source CI artifacts still have bare names. Never mix attempts.
  • Bounded optimization: retain expensive transport mutations, runtime cold/digest paths, Hypothesis examples, native platforms and independent compiler cases. No global facts cache or shared mutable HOME redesign.
  • Evidence limits: architecture ABBA is source-rule execution, not binary timing. Failed sharding rehearsals remain invalid. The new observed full-DAG pair includes qualification/packaging/verification but excludes signing/publication; retries, overlap, image variation and post-run evidence reconciliation are disclosed rather than normalized away. Scenario 6 is a unit/static workflow contract, not executed publication evidence.

Benefits

  1. Hosted Windows: three matched comparisons retained all 498 scenarios/outcomes (496 passed, two skipped). Pytest improved 21.4-37.4%; full jobs saved 52-106s. Latest: 246.037s -> 154.062s pytest and 286s -> 180s job elapsed.
  2. Controlled lexical cases: six matched scenarios in ABBA order, 24 direct executions: 34.490s -> 10.525s / 69.5% lower. The five duplicate-heavy scenarios alone improved 80.6%, eliminating hundreds of parses/indexes per case.
  3. Checksum fixture: identical 102 scenarios 250.65s -> 221.44s, saving 29.21s / 11.65% locally. Neither this nor source Windows is separately added to release savings.
  4. Hosted integration sharding: Linux improves 44.62%, 43.11%, 40.74% across three matched pairs with all 9620 outcomes; ARM improves 28.78%, 33.05% with all 9622 outcomes. ARM spends 23.21-31.99% more pytest runner time; Linux varies from 11.11% less to 18.46% more. Lane gains are not additive.
  5. Actual full-DAG pair: frozen main 2,081s versus proposed 1,493s, saving 588s / 28.26% through successful terminal verification. Unweighted executed-job time rises 6,614s -> 8,120s. This replaces a model-only answer for the measured read-only scope, not the rejected sharding result.

Validation

Current review candidate

7227d255170f29f01119651b0f17fdc4af45fd0a folds the actual panel's signing-order/cross-run regression traps, explicit API-only Windows dependency path, diagnostic retention, and observation comparability checks. Local results: 61 Node tests, 267 targeted Python tests, 88 observation tests after formatting, 24 killed/restored guard mutations, and the current seven-part lint plus architecture boundary check. Nonpublishing qualification 34337498500 succeeded: 38 required jobs, five platform archive/executable bindings, and original proof/index ZIP digests and JSON contents verified. Tested merge c5563f9ad159a06235c0de6024bb66397a7ae186 and source head 7227d255170f29f01119651b0f17fdc4af45fd0a have exactly equal tree 697913c4b45e72bb498fc4dc9dc82d3fddc52d68. The required gate and CodeQL succeeded. Final actual panel resolves all 15 prior items; its ship_now stance is code advice only. Human codeowner/last-push review remains outstanding. Original timing comparability is inconclusive: six Windows observation fingerprints exactly match CRLF-rendered canonical Git bytes; no proof was normalized and no timing benefit is inferred. Prior all-five run 34331009260 remains history for f4a4a02b3 and equal-tree merge 0bb0f7c1c217a8b56eb1abf92e8317520c4c138d (tree c862763ab944c0119688d845f13194b3ad30ae7b). Neither run proves production signing/publication or grants tagging/merge approval. Adapted #2882 owns lifecycle identity/receipt integration and the full MQ source-evidence gate; this root fold does not change that interface.

Actual wall-clock comparison

Full reconciled report: clocks, artifact recovery, scenario inventories, expensive tests, rejected attempts, and protocol deviations.

Side Successful fresh attempt 1 Actual UTC creation -> terminal completion Elapsed
Frozen main f8df1b751 34221564076, controller 6b76f97e5 11:36:14 -> 12:10:55 on 2026-09-08 34m41s
Proposed 1687380d7 34228763968 12:54:25 -> 13:19:18 on 2026-09-08 24m53s

All 14 baseline / 43 proposed required jobs plus their terminal jobs succeeded. Both original proofs omitted separately uploaded runner records. Immutable artifact ZIP digests and recorded run/source identities were independently checked; original proofs remain unchanged, and the unchanged strict comparator passes explicitly derived reconciled proofs containing 9 baseline / 15 proposed records. Production of an empty-record proof now fails at the producer, with immutable observation downloads wired into the terminal.

The reporting-only follow-up at aa850fabc passed 62 targeted contracts; the candidate Node suite passed 39 cases. Canonical local lint/format/duplication and boundary guards passed. This is local evidence, not a claim that the follow-up was timed in the historical full run.

All four Unix integration lanes retain every original JUnit ID with no passed-to-skipped transition and add one case each. Native unit lanes grow 22,450 -> 22,841 cases each: 39 old topology-contract IDs are replaced and 430 added; no original product-unit ID is missing. Windows integration retains job/log evidence rather than claiming JUnit parity. Proposed installer: one passed case, zero skips.

The declared four-run budget was exceeded: six full-workflow IDs and eight attempt executions occurred; mixed-attempt retries are rejected. The selected proposed run overlaps an unselected PR-label run, and Linux ARM integration images differ. No further runs were authorized in that measurement campaign. The current review qualification above was separately authorized and does not reset or retroactively accept that campaign. These limitations qualify the observed 28.26%, not the arithmetic or the fact that both selected workflows succeeded.

Earlier bounded experiments (retained, not retroactively accepted)

Hosted Windows proofs: 34151563754, 34152045787, and 34153648403. Same-image pairing, exact scenarios/outcomes and the unchanged 5% threshold passed in all three. Latest executed merge SHA: 5730579ab67fd0bf92c1a034c9940097d7ba2c1b.

Source CI passed at 05c3aa563, e93483c26, and 3aa6eb958. The first ARM probe failed before pytest because the benchmark upload flattened release-assets/; it is excluded. e93483c26 repaired the envelope and added a rejecting mutation.

Both earlier ARM probes (2x2, 2x3) retained 9622 identical outcomes but mixed macOS images. They correctly failed comparison. Production was restored to 1x4; that revision's source CI passed.

The predeclared whole-rehearsal plan and live results compare ARM integration 1x4/2x3, Linux x64 integration 1x4/2x4 and Intel units 1xauto/2x4. Per-lane threshold: 15%; combined modeled DAG: 10%, target 30%; maximum three full attempts.

Attempt 1 has valid Linux evidence, but ARM image mismatch, Intel JUnit capture and Windows controls invalidated the whole result. Attempt 2 has valid ARM/Linux pairs; installer root reuse and registry cache leakage were then repaired at de21774b4. Final attempt 34167328643, merge 76f4e903604961741c34e87b8e7ff3fe21223270, independently repeats both integration wins but fails Intel's console import-state probe and the Windows candidate installer. Three attempts are exhausted; no combined DAG is accepted.

The console-helper repair at 1f9a5295f has passing source CI 34168861498, not a replacement Intel performance proof. The later same-archive Windows proof 34210978528 reproduced the long-root failure and passed the full short-root installer harness with zero skips; production now uses that short pytest-owned root without warning suppression. The earlier green-but-skipped diagnostic 34210113252 remains rejected.

Hosted JUnit identifies the expensive families: owner-rule mutations 322.080 case-seconds, config lifecycle 185.565s, required lifecycle state machine 122.510s, and frontmatter/BOM 114.296s. Architecture totals 27.78% of baseline summed case elapsed, not release wall time. No cases/examples were cut. Permanent empirical report preserves the comparisons, rejections and profiling method; architecture proof includes the exact ABBA harness and raw rows.

Local commands and measured source evidence

Canonical Ruff over src/, tests/, the architecture linter and the two new proof scripts:

All checks passed!
1857 files already formatted

Canonical pylint R0801 command:

Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

bash scripts/lint-auth-signals.sh:

[*] Rule A: get_bearer_provider boundary (any reference)
[*] Rule B: git ls-remote auth-delegated annotation
[+] auth-signal lint clean

Focused final native, timing, auth, publication, Windows and probe contracts:

310 passed in 39.62s

Final repair set, including both fixture-isolation leaks, actual Windows-root construction and proof-plugin tests under outer xdist:

161 passed in 16.81s

Full corrected owner mutation matrix:

343 passed in 184.65s (0:03:04)

Node candidate/cohort trust: 47 passed, zero failures. YAML/2100-line/path guards, auth/architecture boundaries and actionlint passed. Source CI 34162508726 passed at 8b752a2fe before the full rehearsal.

The earlier architecture full-family pair (337 cases, both failed: 212.677s -> 304.880s) is not a speed proof. The successful 343-case run proves correctness, not same-case whole-family speedup. Its JUnit attributes 97.914s of case time to transport guards, still roughly 3.0-3.4s each.

The six-case ABBA comparison invokes run_selected_rules(...) with the same mutations and equivalent failure assertions as the integration cases. All 24 direct executions passed; the exact harness and JSONL are retained. No parser-cache redesign was added.

Scenario Evidence

# Scenario (user promise) Principle(s) Test(s) proving it Type
1 Failed, skipped, cancelled or ambiguous checks cannot qualify or pass the gate. Governed by policy tests/unit/test_merge_gate_wait.py; tests/unit/test_platform_contract_workflow.py; tests/quality/test_ci_topology.py integration
2 Windows must install the candidate, including upgrade from an older source, without accepting empty/skipped evidence. DevX (pragmatic as npm), Secure by default tests/integration/test_windows_installer_launchers.py; tests/unit/test_windows_installer_candidate.py; tests/unit/test_windows_installer_proof_workflow.py e2e
3 Tampered archives, wrong versions and escaping paths cannot be published. Secure by default tests/unit/scripts/test_package_release.py; scripts/release-candidate.test.cjs integration
4 Sharding retains every selected scenario exactly once, even with cold hints. Governed by policy tests/unit/test_pytest_timing_workflows.py; tests/unit/scripts/test_compare_test_runs.py integration
5 Faster results with omitted cases or new skips are rejected. Governed by policy tests/unit/scripts/test_compare_test_runs.py; tests/unit/test_release_performance_workflow.py integration
6 Early docs/wheel workflow conditions deny publication after failed or skipped qualification. Secure by default tests/unit/test_release_publication_overlap.py unit (static workflow contract; not executed publication)
7 Runtime prerequisites fail visibly when genuinely needed; config-only cases need no Copilot install. Multi-harness support, DevX (pragmatic as npm) tests/unit/test_runtime_prerequisite_gate.py; tests/unit/test_mcp_runtime_prerequisite.py integration
8 Unrelated collection makes no registry requests; checksum negatives stay covered. OSS / community-driven, Secure by default tests/unit/test_mcp_registry_collection.py; tests/unit/test_unix_installer_checksums.py integration
9 Architecture mutations still fail for the right guard; new guards cannot omit their case silently. Governed by policy tests/integration/test_architecture_owner_rule_mutations.py; tests/unit/scripts/test_architecture_runner.py integration
10 A different runner image or late unchanged native gate cannot manufacture an overall win. Governed by policy tests/unit/scripts/test_performance_cohort.py; tests/unit/scripts/test_release_rehearsal.py::test_late_required_native_authorities_cannot_be_dropped_to_manufacture_a_win integration
11 Public Windows validation retains dependency checks without copying API-only job tokens into PAT/Models aliases. Secure by default tests/unit/test_release_validation_prerequisites.py integration
12 Shard order cannot retain a temporary MCP registry after its fixture ends. Governed by policy tests/unit/install/test_mcp_registry_config_layer.py::test_project_fixture_clears_registry_cache_when_its_config_scope_ends; original resolver default tests component
13 Import fallback probes cannot leave a fake console module on the parent package. Governed by policy tests/unit/test_console_utils.py::TestImportFallbacks::test_fallback_probe_restores_both_import_paths; original double-check-lock test component
14 Missing runner observations cannot produce a successful comparison proof. Governed by policy tests/unit/scripts/test_release_wallclock.py::test_verify_rejects_missing_observations_before_emitting_a_proof; tests/unit/test_release_wallclock_workflow.py component

How to test

  • Install frozen dev/build dependencies; run the canonical lint chain and focused Scenario Evidence files.
  • Run node --test scripts/release-candidate.test.cjs; expect fresh/warm, identity, tampering and retry negatives to pass.
  • With a new authorized evaluation budget, apply ci-performance for small pairs or ci-release-rehearsal for the full DAG; require exact outcomes, matched actual runners and the documented thresholds. Remove labels before unrelated pushes.
  • On native Windows, supply the documented candidate/baseline archives, versions and hashes; execute the installer launcher tests against those fixtures.
  • After merge, inspect a full native qualification before tagging. The measured read-only pair does not authorize publishing its artifacts; use only an exact-SHA/version trusted candidate. Production retry rules do not override a benchmark's fresh-attempt-only protocol.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

…candidates

Preserve native coverage, test the Windows candidate, bind promotion to immutable evidence, harden merge gates, and record safe timing snapshots.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

It makes broad, security-sensitive changes across release qualification and CI gating that warrant careful human verification in GitHub Actions.

Review tier: Lite
Findings: 1 Medium severity

New issues introduced by this change (1)
Severity Finding
Medium severity tests/​utils/​windows_installer_candidate.py — This new test helper imports packaging.version.Version, but packaging is not declared in…
What changed in this PR

This PR refactors the CI/release qualification pipeline to decouple native unit tests, builds, integration, isolated validation, and installer verification, and to promote only release artifacts that are bound to immutable, verified qualification evidence.

Changes:

  • Adds a reusable per-platform workflow (release-platform.yml) that runs native units/build first, then lets integration/validation/installer consumers depend only on the platform build artifact.
  • Introduces candidate packaging + promotion authorities (scripts/package_release.py, scripts/release-candidate.cjs) that package once, verify archive identity/digests, enforce attempt-scoped artifact identities, and fail closed on ambiguity.
  • Hardens CI/test gates (merge-gate required-check polling, runtime prerequisite strictness, Windows installer candidate fixture + contracts, pytest timing snapshotting) and updates contributor docs accordingly.
File Description
tests/​utils/​windows_installer_candidate.py Adds verified Windows installer candidate fixtures and a loopback server used by installer E2E.
tests/​unit/​test_windows_unit_diagnostics.py Updates the workflow contract for Windows unit-test diagnostics in the new reusable workflow.
tests/​unit/​test_windows_signing_contract.py Updates signing-step contract assertions to the new per-platform workflow job layout.
tests/​unit/​test_windows_installer_candidate.py Adds cross-platform unit proofs for candidate-only Windows installer fixture inputs and integrity.
tests/​unit/​test_unix_installer_checksums.py Ensures fake tool invocations are stdlib-only (python -S) for determinism and speed.
tests/​unit/​test_runtime_prerequisite_gate.py Adds unit coverage for failing closed when selected runtime prerequisites are missing in strict CI mode.
tests/​unit/​test_pytest_timing_workflows.py Adds contract tests for immutable timing snapshots and disjoint shard selection guarantees.
tests/​unit/​test_platform_contract_workflow.py Reworks hosted workflow contracts for parallel native qualification, artifact identity, and publication gating.
tests/​unit/​test_merge_gate_wait.py Adds hermetic tests for the merge-gate polling script’s fail-closed semantics and identity checks.
tests/​unit/​test_mcp_registry_collection.py Verifies MCP registry E2E module import/collection does not attempt network access.
tests/​unit/​test_ado_live_workflow_contract.py Updates ADO-live selection contracts to the extracted platform workflow and catalog.
tests/​unit/​scripts/​test_package_release.py Adds unit coverage for “package once” archives and verify/extract safety checks.
tests/​quality/​test_ci_topology.py Updates required-check topology assertions to require Lint + ratchets in both PR and merge-group contexts.
tests/​integration/​test_windows_installer_launchers.py Switches installer E2E to candidate-archive + loopback fixture model with request auditing.
tests/​integration/​test_mcp_registry_e2e.py Avoids repeated live health probes by caching and using a fixture for conditional skips.
tests/​integration/​conftest.py Adds --strict-runtime-prerequisites to fail collection when a selected runtime is missing (instead of skipping).
scripts/​windows/​test-install-script.ps1 Refactors installer tests to target the verified candidate fixture (no published-release destination), adds tamper rejection + candidate hash assertions.
scripts/​windows/​build-binary.ps1 Hardens build SHA injection (CRLF-safe) and fails closed when injection doesn’t apply.
scripts/​test-integration.sh Makes runtime provisioning selectable (APM_TEST_RUNTIMES) and enforces strict runtime prerequisite collection in CI.
scripts/​release-platforms.json Introduces the canonical platform catalog and per-platform integration-marker policy.
scripts/​release-candidate.cjs Implements candidate planning/qualification/verification logic with immutable artifact IDs, config hashing, and fail-closed evidence checks.
scripts/​package_release.py Implements archive creation, digest sidecars/metadata, safe extraction, and binary identity verification used by validation and publication.
scripts/​build-binary.sh Removes UPX provisioning messaging/logic for consistency with the release packaging policy.
docs/​src/​content/​docs/​contributing/​integration-testing.md Updates integration-testing documentation to reflect the new qualification, runtime selection, strict runtime prerequisites, and artifact provenance rules.
docs/​src/​content/​docs/​contributing/​development-guide.md Updates contributor guidance to reflect the gate check authority and candidate reuse constraints.
.github/​workflows/​release-platform.yml Adds the reusable native lane (units/build + consumers + per-platform “Native Candidate Gate”).
.github/​workflows/​merge-gate.yml Requires Lint + architecture ratchets in both contexts and documents the stronger poll semantics.
.github/​workflows/​ci.yml Adds workflow_call support and publishes timing/JUnit artifacts for shards; runs release-candidate JS tests.
.github/​workflows/​ci-runtime.yml Supplies missing E2E/integration opt-ins for runtime smoke.
.github/​workflows/​ci-integration.yml Freezes one shared integration timing snapshot and publishes shard timing/JUnit artifacts; removes UPX provisioning.
.github/​scripts/​ci/​merge_gate_wait.sh Refactors to single-snapshot polling, enforces GitHub Actions identity and ambiguity fail-closed semantics, and rejects non-success conclusions.
.github/​instructions/​cicd.instructions.md Synchronizes CI/CD rules with the new workflow topology, candidate provenance, and artifact handling.
.github/​actions/​pytest-timing/​action.yml Adds a composite action for timing restore and immutable snapshot publication (restore-only, no rolling writes).

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread tests/utils/windows_installer_candidate.py
…ployment

Register strict runtime options before whole-tree collection and regenerate the CI instruction self-deployment from its canonical source.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Preserve exact candidate and scenario identity, bind rerun artifacts to attempts, avoid AST work in lexical guards, and add opt-in controlled performance probes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@danielmeppiel Daniel Meppiel (danielmeppiel) added the ci-performance Opt in to read-only before/after CI performance comparisons. label Sep 7, 2026
Match the native artifact envelope instead of flattening release-assets during upload. Add a regression for the failed hosted probe layout.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@danielmeppiel

Copy link
Copy Markdown
Collaborator Author

Phase-two architecture performance evidence

Permanent raw evidence for the lexical optimization in this PR. This is a source-only, direct run_selected_rules(...) microbenchmark, not 24 pytest invocations, not candidate-binary timing, and not a whole-release speedup.

Six fixed mutation scenarios, ABBA order, two repeats each: 34.490s baseline -> 10.525s optimized (69.5% lower). All 24 scenario assertions passed, with zero report failures. The five duplicate-heavy scenarios drop from 477 parses/indexes each to 1-13; frontmatter remains the broad-parse control.

Original source label: c026361e2-plus-lexical-and-missing-guard-cases, Python 3.12, frozen existing environment, PYTHONHASHSEED=0. Baseline restores only the original duplicate-definition helper in memory before importing its consumers; the source, lock and selected mutations otherwise stay fixed across the four fresh processes. The separately executed corrected pytest matrix passed 343 tests in 184.65s.

The earlier failed 337-case full-family pair (212.677s -> 304.880s) is explicitly excluded. Its slowdown was not reproduced, but its cause is not established. No whole-family speedup is inferred from the later successful matrix.

Raw 24 scenario executions, outcomes and counters
{"abba_index": 1, "batch_elapsed_seconds": 17.454663875018014, "blamed_count": 1, "broken_input_stages": [], "elapsed_seconds": 2.8384222500026226, "failure_count": 0, "guard_id": "contracts-tooling-cached-policy-shape", "metrics": {"ast_visits": 649730, "max_parses_per_file": 1, "max_reads_per_file": 1, "max_tree_index_builds_per_file": 1, "parse_attempts": 477, "read_attempts": 484, "total_seconds": 2.7854032089817338, "tree_index_builds": 477, "tree_index_cache_hits": 0}, "mode": "baseline_file_facts", "nodeid": "tests/integration/test_architecture_owner_rule_mutations.py::test_owner_rule_catches_its_guard_mutation[contracts-tooling-cached-policy-shape]", "outcome": "passed", "rule_id": "contracts-tooling-cached-policy-shape", "violation_count": 1}
{"abba_index": 1, "batch_elapsed_seconds": 17.454663875018014, "blamed_count": 1, "broken_input_stages": [], "elapsed_seconds": 3.034132166998461, "failure_count": 0, "guard_id": "contracts-tooling-frontmatter-yaml", "metrics": {"ast_visits": 649730, "max_parses_per_file": 1, "max_reads_per_file": 1, "max_tree_index_builds_per_file": 1, "parse_attempts": 477, "read_attempts": 484, "total_seconds": 2.974740584002575, "tree_index_builds": 477, "tree_index_cache_hits": 0}, "mode": "baseline_file_facts", "nodeid": "tests/integration/test_architecture_owner_rule_mutations.py::test_owner_rule_catches_its_guard_mutation[contracts-tooling-frontmatter-yaml]", "outcome": "passed", "rule_id": "contracts-tooling-frontmatter-yaml", "violation_count": 1}
{"abba_index": 1, "batch_elapsed_seconds": 17.454663875018014, "blamed_count": 1, "broken_input_stages": [], "elapsed_seconds": 2.8665437919844408, "failure_count": 0, "guard_id": "install-deployment-frozen-mutation-eligibility", "metrics": {"ast_visits": 649730, "max_parses_per_file": 1, "max_reads_per_file": 1, "max_tree_index_builds_per_file": 1, "parse_attempts": 477, "read_attempts": 484, "total_seconds": 2.813187583000399, "tree_index_builds": 477, "tree_index_cache_hits": 0}, "mode": "baseline_file_facts", "nodeid": "tests/integration/test_architecture_owner_rule_mutations.py::test_owner_rule_catches_its_guard_mutation[install-deployment-frozen-mutation-eligibility]", "outcome": "passed", "rule_id": "install-deployment-frozen-mutation-eligibility", "violation_count": 1}
{"abba_index": 1, "batch_elapsed_seconds": 17.454663875018014, "blamed_count": 1, "broken_input_stages": [], "elapsed_seconds": 2.808129540993832, "failure_count": 0, "guard_id": "install-deployment-mcp-ownership-migration", "metrics": {"ast_visits": 649730, "max_parses_per_file": 1, "max_reads_per_file": 1, "max_tree_index_builds_per_file": 1, "parse_attempts": 477, "read_attempts": 484, "total_seconds": 2.754094999982044, "tree_index_builds": 477, "tree_index_cache_hits": 0}, "mode": "baseline_file_facts", "nodeid": "tests/integration/test_architecture_owner_rule_mutations.py::test_owner_rule_catches_its_guard_mutation[install-deployment-mcp-ownership-migration]", "outcome": "passed", "rule_id": "install-deployment-mcp-ownership-migration", "violation_count": 1}
{"abba_index": 1, "batch_elapsed_seconds": 17.454663875018014, "blamed_count": 1, "broken_input_stages": [], "elapsed_seconds": 2.7469971659884322, "failure_count": 0, "guard_id": "install-deployment-prospective-dry-run-plan", "metrics": {"ast_visits": 649730, "max_parses_per_file": 1, "max_reads_per_file": 1, "max_tree_index_builds_per_file": 1, "parse_attempts": 477, "read_attempts": 484, "total_seconds": 2.6794039589876775, "tree_index_builds": 477, "tree_index_cache_hits": 0}, "mode": "baseline_file_facts", "nodeid": "tests/integration/test_architecture_owner_rule_mutations.py::test_owner_rule_catches_its_guard_mutation[install-deployment-prospective-dry-run-plan]", "outcome": "passed", "rule_id": "install-deployment-prospective-dry-run-plan", "violation_count": 1}
{"abba_index": 1, "batch_elapsed_seconds": 17.454663875018014, "blamed_count": 1, "broken_input_stages": [], "elapsed_seconds": 2.8505892080138437, "failure_count": 0, "guard_id": "install-deployment-source-plan", "metrics": {"ast_visits": 649730, "max_parses_per_file": 1, "max_reads_per_file": 1, "max_tree_index_builds_per_file": 1, "parse_attempts": 477, "read_attempts": 484, "total_seconds": 2.793726540985517, "tree_index_builds": 477, "tree_index_cache_hits": 0}, "mode": "baseline_file_facts", "nodeid": "tests/integration/test_architecture_owner_rule_mutations.py::test_owner_rule_catches_its_guard_mutation[install-deployment-source-plan]", "outcome": "passed", "rule_id": "install-deployment-source-plan", "violation_count": 1}
{"abba_index": 2, "batch_elapsed_seconds": 5.5390263329900336, "blamed_count": 1, "broken_input_stages": [], "elapsed_seconds": 0.6256675420154352, "failure_count": 0, "guard_id": "contracts-tooling-cached-policy-shape", "metrics": {"ast_visits": 8875, "max_parses_per_file": 1, "max_reads_per_file": 1, "max_tree_index_builds_per_file": 1, "parse_attempts": 1, "read_attempts": 484, "total_seconds": 0.6130847919848748, "tree_index_builds": 1, "tree_index_cache_hits": 0}, "mode": "optimized_lexical_lines", "nodeid": "tests/integration/test_architecture_owner_rule_mutations.py::test_owner_rule_catches_its_guard_mutation[contracts-tooling-cached-policy-shape]", "outcome": "passed", "rule_id": "contracts-tooling-cached-policy-shape", "violation_count": 1}
{"abba_index": 2, "batch_elapsed_seconds": 5.5390263329900336, "blamed_count": 1, "broken_input_stages": [], "elapsed_seconds": 2.45171933301026, "failure_count": 0, "guard_id": "contracts-tooling-frontmatter-yaml", "metrics": {"ast_visits": 649730, "max_parses_per_file": 1, "max_reads_per_file": 1, "max_tree_index_builds_per_file": 1, "parse_attempts": 477, "read_attempts": 484, "total_seconds": 2.402815333014587, "tree_index_builds": 477, "tree_index_cache_hits": 0}, "mode": "optimized_lexical_lines", "nodeid": "tests/integration/test_architecture_owner_rule_mutations.py::test_owner_rule_catches_its_guard_mutation[contracts-tooling-frontmatter-yaml]", "outcome": "passed", "rule_id": "contracts-tooling-frontmatter-yaml", "violation_count": 1}
{"abba_index": 2, "batch_elapsed_seconds": 5.5390263329900336, "blamed_count": 1, "broken_input_stages": [], "elapsed_seconds": 0.5407385829894338, "failure_count": 0, "guard_id": "install-deployment-frozen-mutation-eligibility", "metrics": {"ast_visits": 8088, "max_parses_per_file": 1, "max_reads_per_file": 1, "max_tree_index_builds_per_file": 1, "parse_attempts": 2, "read_attempts": 484, "total_seconds": 0.5392124580102973, "tree_index_builds": 2, "tree_index_cache_hits": 0}, "mode": "optimized_lexical_lines", "nodeid": "tests/integration/test_architecture_owner_rule_mutations.py::test_owner_rule_catches_its_guard_mutation[install-deployment-frozen-mutation-eligibility]", "outcome": "passed", "rule_id": "install-deployment-frozen-mutation-eligibility", "violation_count": 1}
{"abba_index": 2, "batch_elapsed_seconds": 5.5390263329900336, "blamed_count": 1, "broken_input_stages": [], "elapsed_seconds": 0.504773833003128, "failure_count": 0, "guard_id": "install-deployment-mcp-ownership-migration", "metrics": {"ast_visits": 15300, "max_parses_per_file": 1, "max_reads_per_file": 1, "max_tree_index_builds_per_file": 1, "parse_attempts": 5, "read_attempts": 484, "total_seconds": 0.5029639170097653, "tree_index_builds": 5, "tree_index_cache_hits": 0}, "mode": "optimized_lexical_lines", "nodeid": "tests/integration/test_architecture_owner_rule_mutations.py::test_owner_rule_catches_its_guard_mutation[install-deployment-mcp-ownership-migration]", "outcome": "passed", "rule_id": "install-deployment-mcp-ownership-migration", "violation_count": 1}
{"abba_index": 2, "batch_elapsed_seconds": 5.5390263329900336, "blamed_count": 1, "broken_input_stages": [], "elapsed_seconds": 0.5211999999883119, "failure_count": 0, "guard_id": "install-deployment-prospective-dry-run-plan", "metrics": {"ast_visits": 7836, "max_parses_per_file": 1, "max_reads_per_file": 1, "max_tree_index_builds_per_file": 1, "parse_attempts": 3, "read_attempts": 484, "total_seconds": 0.5194111670134589, "tree_index_builds": 3, "tree_index_cache_hits": 0}, "mode": "optimized_lexical_lines", "nodeid": "tests/integration/test_architecture_owner_rule_mutations.py::test_owner_rule_catches_its_guard_mutation[install-deployment-prospective-dry-run-plan]", "outcome": "passed", "rule_id": "install-deployment-prospective-dry-run-plan", "violation_count": 1}
{"abba_index": 2, "batch_elapsed_seconds": 5.5390263329900336, "blamed_count": 1, "broken_input_stages": [], "elapsed_seconds": 0.6020510410016868, "failure_count": 0, "guard_id": "install-deployment-source-plan", "metrics": {"ast_visits": 43658, "max_parses_per_file": 1, "max_reads_per_file": 1, "max_tree_index_builds_per_file": 1, "parse_attempts": 13, "read_attempts": 484, "total_seconds": 0.595757874980336, "tree_index_builds": 13, "tree_index_cache_hits": 0}, "mode": "optimized_lexical_lines", "nodeid": "tests/integration/test_architecture_owner_rule_mutations.py::test_owner_rule_catches_its_guard_mutation[install-deployment-source-plan]", "outcome": "passed", "rule_id": "install-deployment-source-plan", "violation_count": 1}
{"abba_index": 3, "batch_elapsed_seconds": 5.495421583997086, "blamed_count": 1, "broken_input_stages": [], "elapsed_seconds": 0.5644712499924935, "failure_count": 0, "guard_id": "contracts-tooling-cached-policy-shape", "metrics": {"ast_visits": 8875, "max_parses_per_file": 1, "max_reads_per_file": 1, "max_tree_index_builds_per_file": 1, "parse_attempts": 1, "read_attempts": 484, "total_seconds": 0.5629522079834715, "tree_index_builds": 1, "tree_index_cache_hits": 0}, "mode": "optimized_lexical_lines", "nodeid": "tests/integration/test_architecture_owner_rule_mutations.py::test_owner_rule_catches_its_guard_mutation[contracts-tooling-cached-policy-shape]", "outcome": "passed", "rule_id": "contracts-tooling-cached-policy-shape", "violation_count": 1}
{"abba_index": 3, "batch_elapsed_seconds": 5.495421583997086, "blamed_count": 1, "broken_input_stages": [], "elapsed_seconds": 2.5544120000267867, "failure_count": 0, "guard_id": "contracts-tooling-frontmatter-yaml", "metrics": {"ast_visits": 649730, "max_parses_per_file": 1, "max_reads_per_file": 1, "max_tree_index_builds_per_file": 1, "parse_attempts": 477, "read_attempts": 484, "total_seconds": 2.503135792008834, "tree_index_builds": 477, "tree_index_cache_hits": 0}, "mode": "optimized_lexical_lines", "nodeid": "tests/integration/test_architecture_owner_rule_mutations.py::test_owner_rule_catches_its_guard_mutation[contracts-tooling-frontmatter-yaml]", "outcome": "passed", "rule_id": "contracts-tooling-frontmatter-yaml", "violation_count": 1}
{"abba_index": 3, "batch_elapsed_seconds": 5.495421583997086, "blamed_count": 1, "broken_input_stages": [], "elapsed_seconds": 0.5248760420072358, "failure_count": 0, "guard_id": "install-deployment-frozen-mutation-eligibility", "metrics": {"ast_visits": 8088, "max_parses_per_file": 1, "max_reads_per_file": 1, "max_tree_index_builds_per_file": 1, "parse_attempts": 2, "read_attempts": 484, "total_seconds": 0.5232549580105115, "tree_index_builds": 2, "tree_index_cache_hits": 0}, "mode": "optimized_lexical_lines", "nodeid": "tests/integration/test_architecture_owner_rule_mutations.py::test_owner_rule_catches_its_guard_mutation[install-deployment-frozen-mutation-eligibility]", "outcome": "passed", "rule_id": "install-deployment-frozen-mutation-eligibility", "violation_count": 1}
{"abba_index": 3, "batch_elapsed_seconds": 5.495421583997086, "blamed_count": 1, "broken_input_stages": [], "elapsed_seconds": 0.5034217910142615, "failure_count": 0, "guard_id": "install-deployment-mcp-ownership-migration", "metrics": {"ast_visits": 15300, "max_parses_per_file": 1, "max_reads_per_file": 1, "max_tree_index_builds_per_file": 1, "parse_attempts": 5, "read_attempts": 484, "total_seconds": 0.5014852079912089, "tree_index_builds": 5, "tree_index_cache_hits": 0}, "mode": "optimized_lexical_lines", "nodeid": "tests/integration/test_architecture_owner_rule_mutations.py::test_owner_rule_catches_its_guard_mutation[install-deployment-mcp-ownership-migration]", "outcome": "passed", "rule_id": "install-deployment-mcp-ownership-migration", "violation_count": 1}
{"abba_index": 3, "batch_elapsed_seconds": 5.495421583997086, "blamed_count": 1, "broken_input_stages": [], "elapsed_seconds": 0.4908202909864485, "failure_count": 0, "guard_id": "install-deployment-prospective-dry-run-plan", "metrics": {"ast_visits": 7836, "max_parses_per_file": 1, "max_reads_per_file": 1, "max_tree_index_builds_per_file": 1, "parse_attempts": 3, "read_attempts": 484, "total_seconds": 0.4890739999827929, "tree_index_builds": 3, "tree_index_cache_hits": 0}, "mode": "optimized_lexical_lines", "nodeid": "tests/integration/test_architecture_owner_rule_mutations.py::test_owner_rule_catches_its_guard_mutation[install-deployment-prospective-dry-run-plan]", "outcome": "passed", "rule_id": "install-deployment-prospective-dry-run-plan", "violation_count": 1}
{"abba_index": 3, "batch_elapsed_seconds": 5.495421583997086, "blamed_count": 1, "broken_input_stages": [], "elapsed_seconds": 0.6410540840006433, "failure_count": 0, "guard_id": "install-deployment-source-plan", "metrics": {"ast_visits": 43658, "max_parses_per_file": 1, "max_reads_per_file": 1, "max_tree_index_builds_per_file": 1, "parse_attempts": 13, "read_attempts": 484, "total_seconds": 0.6248712920059916, "tree_index_builds": 13, "tree_index_cache_hits": 0}, "mode": "optimized_lexical_lines", "nodeid": "tests/integration/test_architecture_owner_rule_mutations.py::test_owner_rule_catches_its_guard_mutation[install-deployment-source-plan]", "outcome": "passed", "rule_id": "install-deployment-source-plan", "violation_count": 1}
{"abba_index": 4, "batch_elapsed_seconds": 17.514485708001303, "blamed_count": 1, "broken_input_stages": [], "elapsed_seconds": 2.7862215409986675, "failure_count": 0, "guard_id": "contracts-tooling-cached-policy-shape", "metrics": {"ast_visits": 649730, "max_parses_per_file": 1, "max_reads_per_file": 1, "max_tree_index_builds_per_file": 1, "parse_attempts": 477, "read_attempts": 484, "total_seconds": 2.736373082996579, "tree_index_builds": 477, "tree_index_cache_hits": 0}, "mode": "baseline_file_facts", "nodeid": "tests/integration/test_architecture_owner_rule_mutations.py::test_owner_rule_catches_its_guard_mutation[contracts-tooling-cached-policy-shape]", "outcome": "passed", "rule_id": "contracts-tooling-cached-policy-shape", "violation_count": 1}
{"abba_index": 4, "batch_elapsed_seconds": 17.514485708001303, "blamed_count": 1, "broken_input_stages": [], "elapsed_seconds": 3.0431924579897895, "failure_count": 0, "guard_id": "contracts-tooling-frontmatter-yaml", "metrics": {"ast_visits": 649730, "max_parses_per_file": 1, "max_reads_per_file": 1, "max_tree_index_builds_per_file": 1, "parse_attempts": 477, "read_attempts": 484, "total_seconds": 2.9902510830143, "tree_index_builds": 477, "tree_index_cache_hits": 0}, "mode": "baseline_file_facts", "nodeid": "tests/integration/test_architecture_owner_rule_mutations.py::test_owner_rule_catches_its_guard_mutation[contracts-tooling-frontmatter-yaml]", "outcome": "passed", "rule_id": "contracts-tooling-frontmatter-yaml", "violation_count": 1}
{"abba_index": 4, "batch_elapsed_seconds": 17.514485708001303, "blamed_count": 1, "broken_input_stages": [], "elapsed_seconds": 2.9645252500195056, "failure_count": 0, "guard_id": "install-deployment-frozen-mutation-eligibility", "metrics": {"ast_visits": 649730, "max_parses_per_file": 1, "max_reads_per_file": 1, "max_tree_index_builds_per_file": 1, "parse_attempts": 477, "read_attempts": 484, "total_seconds": 2.9145757909864187, "tree_index_builds": 477, "tree_index_cache_hits": 0}, "mode": "baseline_file_facts", "nodeid": "tests/integration/test_architecture_owner_rule_mutations.py::test_owner_rule_catches_its_guard_mutation[install-deployment-frozen-mutation-eligibility]", "outcome": "passed", "rule_id": "install-deployment-frozen-mutation-eligibility", "violation_count": 1}
{"abba_index": 4, "batch_elapsed_seconds": 17.514485708001303, "blamed_count": 1, "broken_input_stages": [], "elapsed_seconds": 2.908012875006534, "failure_count": 0, "guard_id": "install-deployment-mcp-ownership-migration", "metrics": {"ast_visits": 649730, "max_parses_per_file": 1, "max_reads_per_file": 1, "max_tree_index_builds_per_file": 1, "parse_attempts": 477, "read_attempts": 484, "total_seconds": 2.8573549590073526, "tree_index_builds": 477, "tree_index_cache_hits": 0}, "mode": "baseline_file_facts", "nodeid": "tests/integration/test_architecture_owner_rule_mutations.py::test_owner_rule_catches_its_guard_mutation[install-deployment-mcp-ownership-migration]", "outcome": "passed", "rule_id": "install-deployment-mcp-ownership-migration", "violation_count": 1}
{"abba_index": 4, "batch_elapsed_seconds": 17.514485708001303, "blamed_count": 1, "broken_input_stages": [], "elapsed_seconds": 2.74130629200954, "failure_count": 0, "guard_id": "install-deployment-prospective-dry-run-plan", "metrics": {"ast_visits": 649730, "max_parses_per_file": 1, "max_reads_per_file": 1, "max_tree_index_builds_per_file": 1, "parse_attempts": 477, "read_attempts": 484, "total_seconds": 2.6776912079949398, "tree_index_builds": 477, "tree_index_cache_hits": 0}, "mode": "baseline_file_facts", "nodeid": "tests/integration/test_architecture_owner_rule_mutations.py::test_owner_rule_catches_its_guard_mutation[install-deployment-prospective-dry-run-plan]", "outcome": "passed", "rule_id": "install-deployment-prospective-dry-run-plan", "violation_count": 1}
{"abba_index": 4, "batch_elapsed_seconds": 17.514485708001303, "blamed_count": 1, "broken_input_stages": [], "elapsed_seconds": 2.901462291018106, "failure_count": 0, "guard_id": "install-deployment-source-plan", "metrics": {"ast_visits": 649730, "max_parses_per_file": 1, "max_reads_per_file": 1, "max_tree_index_builds_per_file": 1, "parse_attempts": 477, "read_attempts": 484, "total_seconds": 2.84993866700097, "tree_index_builds": 477, "tree_index_cache_hits": 0}, "mode": "baseline_file_facts", "nodeid": "tests/integration/test_architecture_owner_rule_mutations.py::test_owner_rule_catches_its_guard_mutation[install-deployment-source-plan]", "outcome": "passed", "rule_id": "install-deployment-source-plan", "violation_count": 1}
Portable reproduction harness

Only the output-directory assignment below differs from the recorded inline harness. The original saved harness SHA-256 is 47f1de5073deecf3ca5a4349c4b014799fd79aad63d232930992df1e3d1b4c72. Save this as lexical-abba-harness.py outside the checkout; run from the PR checkout using the same frozen dev environment and a new PERF_OUTPUT directory. Do not overwrite the recorded evidence. Each output row records the corresponding test node as a scenario identifier; the harness calls the rule runner directly.

PERF_OUTPUT="$(mktemp -d)" PYTHONHASHSEED=0 uv run --frozen --no-sync python -c "$(cat /path/to/lexical-abba-harness.py)"
import importlib, json, os, pathlib, subprocess, sys, time
art=pathlib.Path(os.environ["PERF_OUTPUT"])
art.mkdir(parents=True, exist_ok=True)
jsonl=art/"lexical-abba-duplicate-subset-c026361e2.jsonl"
summary=art/"lexical-abba-duplicate-subset-c026361e2-summary.json"
child_code = r"""
import json, sys, time
mode=sys.argv[1]
case_ids=sys.argv[2].split(",")
if mode == "baseline_file_facts":
    import scripts.architecture_linter.checks.lexical_shared as ls
    def old_duplicate_definition_lines(provider, *, rule_id, prefix, pattern, owner, message, respect_exempt):
        findings=[]
        for path in ls.python_paths(provider, under=prefix):
            if path == owner:
                continue
            facts = provider.file_facts(path)
            if facts.read_error is not None:
                continue
            for number, line in enumerate(facts.lines, start=1):
                if respect_exempt and ls.EXEMPT_MARKER in line:
                    continue
                match = pattern.search(line)
                if match is not None:
                    findings.append(ls.violation(rule_id, path, message, line=number, column=match.start()+1))
        return findings
    ls.duplicate_definition_lines = old_duplicate_definition_lines
import tests.integration.test_architecture_owner_rule_mutations as tm
from scripts.architecture_linter.runner import run_selected_rules
cases_by_guard={case.guard_id: case for case in tm.MUTATIONS}
rows=[]
for guard_id in case_ids:
    case=cases_by_guard[guard_id]
    node=f"tests/integration/test_architecture_owner_rule_mutations.py::test_owner_rule_catches_its_guard_mutation[{case.guard_id}]"
    started=time.perf_counter()
    report=run_selected_rules(tm.ROOT, (case.rule_id,), source_overrides={case.path: tm._mutate(case)})
    elapsed=time.perf_counter()-started
    blamed=[v for v in report.violations if v.rule_id == case.rule_id]
    broken=[f.stage for f in report.failures if f.stage in (f"read:{case.path}", f"parse:{case.path}")]
    rows.append({
        "mode": mode,
        "guard_id": case.guard_id,
        "rule_id": case.rule_id,
        "nodeid": node,
        "elapsed_seconds": elapsed,
        "outcome": "passed" if (blamed and not broken) else "failed",
        "violation_count": len(report.violations),
        "blamed_count": len(blamed),
        "failure_count": len(report.failures),
        "broken_input_stages": broken,
        "metrics": {
            "read_attempts": report.metrics.read_attempts,
            "parse_attempts": report.metrics.parse_attempts,
            "ast_visits": report.metrics.ast_visits,
            "tree_index_builds": report.metrics.tree_index_builds,
            "tree_index_cache_hits": report.metrics.tree_index_cache_hits,
            "max_reads_per_file": report.metrics.max_reads_per_file,
            "max_parses_per_file": report.metrics.max_parses_per_file,
            "max_tree_index_builds_per_file": report.metrics.max_tree_index_builds_per_file,
            "total_seconds": report.metrics.total_seconds,
        },
    })
print(json.dumps(rows, sort_keys=True))
"""
case_ids="contracts-tooling-cached-policy-shape,contracts-tooling-frontmatter-yaml,install-deployment-frozen-mutation-eligibility,install-deployment-mcp-ownership-migration,install-deployment-prospective-dry-run-plan,install-deployment-source-plan"
order=["baseline_file_facts","optimized_lexical_lines","optimized_lexical_lines","baseline_file_facts"]
all_rows=[]
jsonl.write_text("", encoding="utf-8")
for idx, mode in enumerate(order, start=1):
    started=time.perf_counter()
    proc=subprocess.run([sys.executable,"-c",child_code,mode,case_ids], text=True, capture_output=True, env=dict(os.environ, PYTHONHASHSEED="0"))
    elapsed=time.perf_counter()-started
    if proc.returncode != 0:
        rec={"abba_index":idx,"mode":mode,"returncode":proc.returncode,"elapsed_seconds":elapsed,"stdout":proc.stdout[-4000:],"stderr":proc.stderr[-4000:]}
        jsonl.open("a",encoding="utf-8").write(json.dumps(rec,sort_keys=True)+"\n")
        print(json.dumps(rec,indent=2,sort_keys=True))
        sys.exit(proc.returncode)
    rows=json.loads(proc.stdout)
    for row in rows:
        row["abba_index"]=idx
        row["batch_elapsed_seconds"]=elapsed
        jsonl.open("a",encoding="utf-8").write(json.dumps(row,sort_keys=True)+"\n")
        all_rows.append(row)
by_mode={}
for row in all_rows:
    by_mode.setdefault(row["mode"],[]).append(row)
summary_data={"artifact":str(jsonl),"order":order,"case_ids":case_ids.split(","),"rows":len(all_rows),"all_passed":all(r["outcome"]=="passed" for r in all_rows),"by_mode":{}}
for mode, rows in by_mode.items():
    summary_data["by_mode"][mode]={
        "case_runs":len(rows),
        "total_case_seconds":sum(r["elapsed_seconds"] for r in rows),
        "mean_case_seconds":sum(r["elapsed_seconds"] for r in rows)/len(rows),
        "mean_parse_attempts":sum(r["metrics"]["parse_attempts"] for r in rows)/len(rows),
        "mean_ast_visits":sum(r["metrics"]["ast_visits"] for r in rows)/len(rows),
        "mean_tree_index_builds":sum(r["metrics"]["tree_index_builds"] for r in rows)/len(rows),
        "mean_read_attempts":sum(r["metrics"]["read_attempts"] for r in rows)/len(rows),
    }
summary.write_text(json.dumps(summary_data,indent=2,sort_keys=True),encoding="utf-8")
print(json.dumps(summary_data,indent=2,sort_keys=True))

@danielmeppiel Daniel Meppiel (danielmeppiel) removed the ci-performance Opt in to read-only before/after CI performance comparisons. label Sep 7, 2026
Two-worker cold sharding was not accepted: outcome parity held but hosted images differed and raw timing was below target. Retry three workers per shard without relaxing proof checks.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@danielmeppiel Daniel Meppiel (danielmeppiel) added ci-performance Opt in to read-only before/after CI performance comparisons. and removed ci-performance Opt in to read-only before/after CI performance comparisons. labels Sep 7, 2026
Both bounded hosted comparisons preserved every case but failed strict runner-image identity. Retain one four-worker release shard and the read-only two-shard experiment; do not claim unaccepted ARM savings.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@danielmeppiel

Copy link
Copy Markdown
Collaborator Author

Phase two: measured results and expensive-test evidence

Phase two is folded into this PR, not a separate PR. Performance acceptance
uses exact original pytest node IDs/outcomes, candidate bytes, source SHA,
Python/runner identity and the dependency lock. The minimum execution
improvement remains 5%; failed and non-comparable experiments are recorded,
not filtered away.

Accepted measurements

Change Baseline Proposed Measured reduction Scope
Windows compatibility, run 34151563754 219.834s 172.845s 21.4% Pytest execution; 498 identical cases/outcomes
Windows repeat, run 34152045787 235.721s 159.155s 32.5% Pytest execution; same 498 cases/outcomes
Windows third run, 34153648403 246.037s 154.062s 37.4% Pytest execution; same 498 cases/outcomes
Six architecture scenarios, ABBA 34.490s 10.525s 69.5% 24 direct rule executions, not the whole suite
Unix checksum fixture 250.65s 221.44s 11.65% Same 102 locally executed scenarios

The Windows jobs, including setup/upload but excluding initial allocation,
fell from 259s to 207s, 268s to 193s, and 286s to 180s:
52-106s saved per job. All three paired Windows experiments used
image 20260824.214.3, Windows Server
2025, four CPUs, Python 3.12.10, and identical lock fingerprints. All three
retained 496 passes and two skips, with zero missing/extra/changed outcomes.

Hosted evidence: first Windows,
Windows repeat,
third Windows.
The complete architecture raw data and exact portable harness are preserved
in this PR comment.
The corrected 343-case architecture matrix passed separately; that proves
correctness, not a whole-matrix speedup.

Read-only documentation/wheel builders also completed successfully
(first run: 56s/15s). That demonstrates their execution without publishing
privileges, not a measured production-publication saving.

ARM decision

Deferred from production at c41bd4bdf. The final bounded three-worker
comparison 34153648594
also mixed runner images. Baseline and proposed shard one received Darwin
25.6.0/image 20260831.0337.3; proposed shard two received Darwin 25.5.0/image
20260728.0273.1. All 9622 outcomes match, but the strict comparator rejected
the run. No image check or threshold was relaxed.

Raw pytest times were 1167.573s baseline versus 458.564s/562.505s proposed;
the apparent 51.8% gain is not accepted controlled evidence. The earlier
2x2 experiment had shown only 3.19% raw gain. Rather than repeatedly rerun
until favorable numbers appear, retain production 1x4 and leave
experimental 2x3 available only in the opt-in read-only probe.

Candidate archive SHA256 for that final experiment:
c8a3b093ddeeaf77d19bb636edb644aedf6197b96d01635644c8b2bdeac3e0b7.
Executed merge SHA: 5730579ab67fd0bf92c1a034c9940097d7ba2c1b.

Whole-release accounting

Historical release: 39m16s. Retain the conservative phase-one fresh
forecast 32m12s-34m12s, or 5m04s-7m04s / 13-18% modeled savings.
The previous 22-29m phase-two envelope depended on accepted ARM savings
and is withdrawn. Windows, lexical microbenchmarks and read-only builder
times are not added to release elapsed. A complete optimized production
release has not been timed, so there is no measured end-to-end total.

Experiments rejected, not hidden

  1. Run 34151564107 failed before pytest because the probe upload flattened
    release-assets/. The exact-candidate consumer rejected the layout.
    e93483c26 repaired the probe envelope and added a rejecting mutation.
  2. Run 34152045985 passed every execution with identical 9622 original
    outcomes: 9468 passed, 152 skipped, two xfailed. Baseline received Darwin
    25.5.0/image 20260728.0273.1; both proposed shards received Darwin
    25.6.0/image 20260831.0337.3. The comparator correctly failed and
    produced no success artifact.
  3. That second run was also unconvincing on raw timing: 965.834s baseline
    versus 934.997s/728.843s proposed, only 3.19% maximum-execution
    reduction. Summed pytest runner elapsed rose 72.3%. These numbers
    are diagnostic only, not a controlled improvement.
  4. The initial architecture full-family pair failed on a missing mutation
    case and slowed from 212.677s to 304.880s. It is not speed evidence.
    Two genuinely missing registry cases were added; no case was removed.

The different ARM/Windows lock hashes are exactly LF versus CRLF checkout
bytes, confirmed by hashing the committed bytes with only that conversion.
There was no demonstrated dependency-resolution drift. Same-platform
comparisons still require equal raw lock hashes.

Expensive tests: actual hosted costs

Source: ARM run 34152045985,
baseline JUnit, executed merge SHA
3cded367062588a7f2ac1528bc297ef3aac5249e. Candidate archive SHA256:
f644763af2b82513d3d55f668b7e9e0d7a4039792de1743c79f3ad30c51b9dd2.

These are summed JUnit case elapsed times, including fixtures, not CPU
seconds and not release elapsed time. The 9622 cases total 3612.882
case-seconds across four workers. Architecture families consume
1003.757 case-seconds / 27.78%.

Family Cases Case-seconds Engineering decision
Architecture owner-rule mutations 343 322.080 Keep every guard mutation; remove only unnecessary parsing
Config-surface lifecycle contract 43 185.565 Retain command-boundary and independent lifecycle failures
Required lifecycle state machine 28 122.510 No reduced transitions or Hypothesis examples
Architecture frontmatter/BOM 12 114.296 Broad-parse control; do not extrapolate lexical-only gains here
Architecture install compound mutations 59 108.943 Prefer scoped guard execution over omitted mutation rows
Artifactory install 8 108.734 Hosted-only anomaly needs phase profiling, not shorter timeouts
Architecture contract compound mutations 5 97.766 Keep independent multi-guard failure evidence
Consume-source ref/cache matrix 20 92.145 Preserve stale-ref, cache and replay cases
Architecture linter entrypoint 21 77.401 Preserve real CLI and corrupted-repository execution

The slowest individual baseline case is the generated lifecycle
state-machine sequence, 54.410s. The architecture wrapper's first
fixture consumer is 47.673s; that includes expensive fixture work, not
just its assertion.

Cold two-shard assignment balanced counts, not work: 4811 cases each,
but 1751.607s versus 1367.447s in summed case elapsed. The full-entrypoint
module has expensive module-scoped fixtures that loadgroup reuses within
each worker; splitting it across outer jobs recreates those fixtures in
both jobs. Its proposed total became 165.898s, versus 77.401s in
baseline. Source confirms the duplication mechanism; image mismatch means
the entire increase cannot be attributed to it experimentally.

Three Artifactory cases each cost about 35s on the hosted runner.
An isolated local profile retained all eight cases and passed in 3.49s;
the slow hosted behavior did not reproduce. Therefore no retry, timeout,
archive-security assertion, or auth-redirect check was weakened based on
an unproven cause.

All three formerly Copilot-gated configuration cases passed in both ARM
variants against the candidate. Their binary prerequisite and HOME
isolation remain; only the false need to install Copilot was removed.

Reproduce and inspect

Download the exact run's performance-evidence-* and test-results-*
artifacts. The repository's scripts.compare_test_runs rejects mismatched
identities/outcomes and incomplete or overlapping shard membership.
Original node IDs come from apm_performance_nodeid JUnit properties,
never by stripping the xdist @group suffix heuristically.

To reproduce the family ranking from each downloaded JUnit:

import collections
import xml.etree.ElementTree as ET

families = collections.defaultdict(lambda: [0, 0.0])
for case in ET.parse("test-results/integration-shard-1.xml").getroot().iter("testcase"):
    prop = case.find("./properties/property[@name='apm_performance_nodeid']")
    node = prop.get("value")
    row = families[node.split("::")[0]]
    row[0] += 1
    row[1] += float(case.get("time", "0"))
for name, (count, seconds) in sorted(families.items(), key=lambda item: -item[1][1]):
    print(f"{seconds:9.3f}s {count:5d} {name}")

Raw hosted artifacts have 30-day retention. This comment preserves the
key measurements and methodology beyond artifact expiry. No production
release, merge or tag was triggered to obtain these proofs.

Compare ARM and Linux integration plus Intel units against complete unchanged native/source authorities. Match actual runner identities before pytest and retain exact scenario outcomes and observed native gate delays in the DAG model. Production sharding remains unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@danielmeppiel

Daniel Meppiel (danielmeppiel) commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator Author

Whole-pipeline sharding model and hosted proof

Final result: three full attempts completed; required failures prevent
any overall speedup claim. Independent matched pairs demonstrate
40.7-44.6% lower Linux integration time and 28.8-33.1% lower ARM integration
time. Production sharding is unchanged; see the final empirical decision.

Decision boundary

Run a read-only, five-platform release-qualification rehearsal at one PR
merge SHA. Preserve production policy until comparable evidence exists.
No release, tag, deployment, signing secret, PAT, test relocation, omitted
scenario, reduced example count, or weaker comparison threshold.

The experiment tests three independent latency levers together, because
ARM alone can expose an Intel-unit or Linux-integration bottleneck.

Surface Current policy / baseline Experimental proposal
macOS ARM integration One runner, four pytest workers Two runners, three workers each
Linux x64 integration One runner, four workers Two runners, four workers each
macOS Intel unit tests One runner, -n auto Two runners, four workers each
Other native unit/integration/validation/installer paths Existing policy Unchanged, executed as common controls
Source CI, docs build, wheels Existing required/read-only builders Unchanged, executed as common controls

Integration pairs consume exactly the same once-built native archive and
its metadata. Unit pairs use identical frozen source/dependencies. Cold
immutable scheduling hints are shared; pass results are never cached.

Why these three surfaces

Historical run 34104956270 provides actual stage durations:

Stage Observed execution time Role after ARM improves
ARM integration 24m27s Original bottleneck
Intel units 14m03s Likely next bottleneck; not ARM's 7m52s units
Linux x64 integration 13m09s (13m45s whole job) Another likely floor
Linux ARM integration 10m59s Possible floor after all three changes
Intel focused integration 6m39s Retained native coverage
ARM units 7m52s Below the other observed floors
Windows units 5m19s Retained; candidate installer measured separately

These are historical observations, not predictions for today's source or
secret-free rehearsal. JUnit case-seconds are not CPU work and are not
divided by worker count to manufacture a guaranteed speedup.

Critical-path model

For each variant, measure completion of every required source/native
authority from one common workflow origin. Builds and units are independent;
integration and isolated validation depend only on their native build.

flowchart LR
    O["one rehearsal SHA / origin"] --> S["source CI"]
    O --> U["five native unit lanes"]
    O --> B["five native builds / exact archives"]
    B --> V["five isolated validations / Windows installer"]
    B --> I["native integration"]
    U --> UP["Intel: baseline versus two shards"]
    I --> IP["ARM and Linux x64: baseline versus two shards"]
    O --> D["read-only docs / wheels"]
    S --> R["compare evidence-ready critical paths"]
    V --> R
    UP --> R
    IP --> R
    D --> R
Loading

Let C be completion of all unchanged required work. Let A, L, and U
be completion of the ARM, Linux x64, and Intel native authorities after
selecting the corresponding paired integration/unit variant. Retain their
build, other tests, isolated validation and installer prerequisites.
Use timestamp maxima, not sums of parallel jobs:

  • Baseline: max(C, A_baseline, L_baseline, U_baseline) - origin.
  • ARM-only: max(C, A_proposed, L_baseline, U_baseline) - origin.
  • ARM + Linux: max(C, A_proposed, L_proposed, U_baseline) - origin.
  • Combined: max(C, A_proposed, L_proposed, U_proposed) - origin.

For a sharded surface, test completion is the latest shard completion.
Retain every native integration fan-in and Native Candidate Gate: measure
each actual dependency-to-verdict delay, then hold that delay constant while
propagating the selected variant through its native DAG. Unchanged native
authorities retain their actual completion timestamps and can become the
floor. Only the additional experiment-specific both-world checkers are
excluded, not production gates. Reject missing/failed gates or a gate that
purports to finish before its prerequisites.

Proposed native-gate timestamps are therefore modeled with observed delays,
not independently measured. Show this assumption alongside each result,
and show the limiting authority in every scenario. Report actual
queue/setup-inclusive test-job spans, pytest controller times, native
fan-in delays and summed selected runner time separately.

Both variants coexist in one rehearsal. The workflow itself waits for all
variants, so its overall duration is NOT the proposed release duration.
The timestamp reconstruction is an empirical qualification-work model.
Production qualification-recording, asset verification, signing, deployment,
and privileged publication remain outside this unprivileged measurement.
Show additional unchanged-tail sensitivity at 0/60/120/180 seconds; do not
claim a measured production release saving.

Environmental control before expensive execution

Each paired surface has three actual test runners: baseline, proposed-1,
proposed-2. After checkout/Python/candidate preparation, each publishes an
attempt-scoped identity record and waits at a bounded cohort barrier.

All three must agree on source SHA, selection, candidate bytes (integration),
Python, OS/kernel, architecture, CPU count, runner image and lock bytes.
Missing, duplicate, malformed or mismatched evidence stops that cohort
before pytest. The identity check applies to the actual test jobs, not
earlier probe jobs allocated on unrelated machines.

The existing post-run exact-case/outcome checker remains unchanged in its
acceptance semantics. Preflight success never substitutes for final proof.

Empirical protocol and acceptance

  1. Validate the orchestration, barrier, strict report parser and DAG model
    locally, including missing/failing/skipped jobs, old attempts, identity
    mismatches, overlapping shards, omissions and a new slower bottleneck.
  2. Run one full hosted rehearsal through a deliberate opt-in label.
    Read-only reusable workflows perform real source/native work.
  3. Require all common control jobs and every baseline/proposed shard to
    succeed. Compare every original scenario and outcome, separately by
    paired surface. No arbitrary declared-omission facility is added.
  4. Require at least 15% paired execution improvement for each proposed
    surface before recommending it. Require at least 10% additional
    evidence-ready improvement for the combined proposal; target 30%.
    A faster individual lane with less than 10% overall is insufficient.
  5. If the first valid rehearsal passes, run one confirmatory rehearsal.
    Maximum three full attempts, including environment-inconclusive attempts.
    Select on image validity only, never discard a valid unfavorable timing.
    Functional/parity failures trigger diagnosis, not a relaxed acceptance.
  6. Publish both favorable and unfavorable results, raw artifacts, exact
    source/candidate identities, remaining bottlenecks and runner-cost tradeoff.
    Do not change production sharding merely because the plan looks promising.

At the earlier modeled 32m12s-34m12s baseline, 10% means 3m13s-3m25s;
30% means 9m40s-10m16s. The rehearsal must establish its own baseline rather
than treat that prior model as measured fact.

Implementation ownership and current state

  • Native executor: reuse production unit/integration/validation behavior;
    add isolated, opt-in paired execution and keep production defaults.
  • Parent: actual-runner cohort barrier, read-only rehearsal orchestration,
    strict comparison/DAG report, contributor docs, validation and publishing
    results to the existing PR refactor(ci): parallelize native qualification and promote verified release candidates #2879.
  • No source-test relocation or inventory cache change is included.

Implementation is pushed to PR #2879 at
8b752a2fe75667be43b467594528a11f4e8df9b3. The cohort, native executor,
read-only orchestration and revised whole-DAG model are complete. Local
validation passed 310 targeted cases, 47 Node cases, workflow validation and
the canonical lint/authority guards. Hosted source CI passed before launch.

Independent review caught an omitted native-gate floor before any experiment
was launched. The model now retains and propagates every production native
gate; the reviewer's 1100-second gate counterexample now correctly yields
zero overall gain. The repaired model and native extraction have no
outstanding high-confidence review findings.

The real GitHub attempt-specific APIs were checked: completed jobs expose
run ID, attempt, runner name and timestamps; a real rerun has its own
run_started_at, rather than charging the earlier attempt to the new run.
Each cohort record now captures actual runner/job identity and a UTC
timestamp so the final report can bind it to the executed job.

Attempt 1: 34162882041

Executed at merge SHA 0c1577c26ed9dca0c197603576bdee6125eb1316.
Full result: rejected; no combined-DAG speedup can be claimed.

  • Linux x64 integration has a valid same-image, same-candidate comparison:
    all 9620 original scenarios/outcomes match (9466 passed, 152 skipped,
    2 xfailed). Pytest 1034.616s -> 573.003s, 44.62% lower. Full baseline
    job 1085s, slower proposed job 616s; actual completion advanced 457s
    (7m37s), including different allocation times. Summed pytest runner time
    1034.616s -> 919.698s, not a CPU-work estimate.
  • ARM integration preflight rejected July/August runner image mixing.
    All three expensive pytest steps were skipped; no ARM time is credited.
  • Intel unit pytest completed with 22645 original cases versus 11323 +
    11322 across shards, but all three proof captures rejected inflated JUnit
    summary counts. Those raw timings are not accepted performance evidence;
    investigate the producer rather than relax the comparator.
  • Windows public validation exposed a prerequisite wiring mismatch:
    GITHUB_API_TOKEN was supplied but only PAT/Models variables were accepted.
    The other three public golden scenarios passed. Correct the prerequisite
    without aliasing the read-only job token into private/Models credentials.
  • Windows candidate installer failed to load the staged Python DLL at a
    252-character path. The same archive runs under the short validation path.
    Shorten only the harness-generated temporary prefix, retaining spaces,
    &, all checksum/launcher/reinstall/upgrade assertions. Native rerun must
    establish whether that repair resolves the loader failure.

The opt-in label has been removed, preventing accidental extra runs. Attempt
1 counts toward the three-attempt cap. Fix the instrumentation/control
defects before attempt 2; do not retry an unchanged broken harness or remove
the failing authorities. Production sharding remains unchanged.

Corrective commit before attempt 2

2b3d7f6ed fixes the three observed harness/control defects. The Intel cause
is pytest 9's JUnit treatment of passed unittest.subTest reports: summary
counts increase while child assertions are folded into their parent node.
The opt-in plugin now emits concrete parent-relative ordinal subtest nodes
in the controller, without mutating xdist worker scheduling IDs. Real serial,
worksteal, loadgroup and two-shard probes prove equal inventory; failures and
new skips remain rejected. An outer-xdist 132-case repair set passed. No
hosted evidence was regenerated with local identity and the comparator was
not relaxed.

The exact Windows archive contains 50 DLLs. Under the original harness root,
42 DLL paths reach or exceed 260 characters (maximum 284); the short fixture
root lowers the maximum to 246 and retains the adversarial install-prefix
characters. This supports the path-headroom correction but does not replace
the next native execution.

The public API prerequisite has real PowerShell negative tests proving an
API-only job token cannot enable inference or populate PAT/Models aliases.
Canonical lint and authority gates passed before the corrective push.
Attempt 2 uses this new SHA consistently across every baseline/proposal;
performance policy and acceptance thresholds are unchanged.

Attempt 2: 34165233237

Executed at merge SHA b925aea03fa4d3c39ec22062f84556979e3b0bf9.
Full result: rejected; no combined-DAG speedup is accepted. All nine actual
paired runners matched within their respective three-runner cohorts.

Independently valid lane Baseline pytest Slowest proposed shard Reduction Summed pytest runner time
ARM integration 1x4 -> 2x3 998.545s 711.138s 28.78% 998.545s -> 1230.309s (+23.21%)
Linux x64 integration 1x4 -> 2x4 976.924s 555.815s 43.11% 976.924s -> 929.138s (-4.89%)

ARM preserved all 9622 outcomes (9468 passed, 152 skipped, 2 xfailed) on
image 20260831.0337.3, Darwin 25.6.0, Python 3.12.10, three CPUs.
Its baseline job took 1047s; the proposed critical job took 779s. Actual
latest test-job completion advanced 268s (4m28s). Linux preserved all 9620
outcomes (9466 passed, 152 skipped, 2 xfailed) on image 20260831.293.1.
Its baseline job took 1029s; the proposed critical job took 604s. Actual
latest completion advanced 426s (7m06s). These are separate lane gains;
they are not additive and do not establish full-pipeline savings.

The Intel JUnit producer correction worked: completed pytest reports now
capture concrete subtests without relaxing strict inventory checks.
However, proposed shard 1 exposed two actual default-registry failures.
The registry-config fixture reset its process-global cache only at setup;
teardown restored the config path while leaving the temporary registry
cached. Running the original config-writing test followed by the two
original default-resolution tests reproduced exactly 1 pass / 2 failures
locally. Resetting the cache in finally repairs the leak without changing
resolver precedence or skipping scenarios. A direct fixture-lifecycle
regression now also asserts the cache is empty after teardown.

Windows public isolated validation passed with the API-only token. The
short installer prefix exposed a harness bug: mktemp("wi") already exists,
but IsolatedApmEnvironment.create() correctly refuses an existing root.
The repair uses the short nonexistent child mktemp("wi") / "i"; the
cross-platform regression now both verifies the exact harness expression
and executes the actual constructor. The shared refusal-to-reuse-root
contract and all native installer assertions remain intact. Native loader
success still requires the next Windows execution.

All available proofs and the complete job inventory are preserved in
the run artifacts; local strict comparisons are arm-comparison.json and
linux-comparison.json. The failed Intel pair is not accepted or repaired
post hoc. Only one full attempt remains under the preregistered cap.
Its repair set passed 72 serial cases and 121 cases under outer xdist,
including real nested proof-plugin execution; canonical lint and quality
ratchets passed. Independent scoped repair review found no high-confidence
issues. A fresh label-triggered run, not a same-run rerun, avoids the
canonical source CI's existing non-attempt-scoped artifact names.

Attempt 3: 34167328643

The final full attempt completed with failed required controls after
corrective commit de21774b4ac28480c0548475ce9ebc7da8dc33c3, executing merge
SHA 76f4e903604961741c34e87b8e7ff3fe21223270.
Ordinary source CI 34166994416
passed before launch. The opt-in label was removed immediately after the
rehearsal plan executed; the branch remained fixed until it completed.
All nine actual runners matched within their three paired cohorts.

Both integration pairs passed exact original-case/outcome comparison again:

Final independently valid lane Baseline pytest Slowest proposed shard Reduction Summed pytest runner time
ARM integration 1x4 -> 2x3 882.657s 590.898s 33.05% 882.657s -> 1164.991s (+31.99%)
Linux x64 integration 1x4 -> 2x4 903.466s 535.391s 40.74% 903.466s -> 1070.270s (+18.46%)

ARM retained all 9622 outcomes; Linux retained all 9620. In actual
setup-inclusive jobs, ARM advanced latest completion by 273s (4m33s):
933s baseline versus 639s + 659s proposed jobs. Linux advanced by 375s
(6m15s): 965s baseline versus 590s + 578s proposed jobs. Summed hosted
job spans increased by 365s on ARM and 203s on Linux in this attempt.
Neither measure is a billed-cost estimate or fixed CPU-work measurement.

Why the full result is rejected

The registry-cache defect from attempt 2 did not recur. Intel proposed
shard 1 instead exposed an independent import-state leak in
tests/unit/test_console_utils.py: an optional-dependency probe restored
sys.modules but left apm_cli.utils.console pointing at the fake
Rich-unavailable module on its parent package. The original fallback test
followed by the original double-check-lock test reproduced 1 pass / 1 fail
locally. New Rich/Colorama import-identity regression cases both failed
before repair. The helper now restores both import paths in finally.
The complete related repair set passed 161 cases under outer xdist.
The post-attempt repair is 1f9a5295f20688773361b97cfbc92efc93e5898b; it
does not retroactively validate the failed hosted Intel pair.
Its ordinary source CI 34168861498
passed; that is separate correctness evidence, not a native performance
comparison or a successful Windows installer result.

The shortened Windows fixture now reaches candidate download, checksum
verification, extraction and binary smoke. Its current built candidate is
still rejected during smoke with RequestsDependencyWarning: Unable to find acceptable character detection dependency (chardet or charset_normalizer). Public isolated validation passes, but the required
installer does not, so the Windows native authority remains failed.

Candidate artifact candidate-1-apm-windows-x86_64 contains the actual
charset_normalizer compiled extensions; calling this simply a missing
package would be incorrect. Complete archive inspection found 182 members,
152 files and 21 .pyd files. The full fixture TestRoot is still 81
characters; the longest staged file path reaches 270, and the detector's
md__mypyc.cp312-win_amd64.pyd reaches 265 during temporary extraction.
Windows PowerShell 5.1's native-stderr handling with
$ErrorActionPreference = "Stop" and path-sensitive loading are plausible
contributors, not independently proven root causes. No warning is
suppressed, production installer behavior is not speculatively changed,
and no failed control is waived.

Final empirical decision

The preregistered three-full-attempt budget is exhausted. All three full
attempts failed required controls; there is no accepted combined-DAG
number, no demonstrated 10% overall minimum, and no demonstrated 30%
overall target
. No optimized production release was timed.

The independent integration evidence is repeatable:

Proposal Accepted hosted pairs Pytest reduction across every valid pair Pytest runner-time change
Linux x64 1x4 -> 2x4 3 44.62%, 43.11%, 40.74% -11.11%, -4.89%, +18.46%
ARM 1x4 -> 2x3 2 28.78%, 33.05% +23.21%, +31.99%
Intel units 1xauto -> 2x4 0 Not accepted Failed/incomplete evidence, not a cost comparison

Do not average unrelated job clocks into a release total or discard the
third Linux pair's unfavorable runner-cost result. The measurements support
Linux integration as the strongest first rollout candidate, and ARM as a
latency-for-runner-time tradeoff. They do not establish that either, or their
combination, clears the required overall threshold: another authority can
become the bottleneck. Intel sharding remains unqualified.

Production sharding stays unchanged. The next safety prerequisite is a
focused native Windows diagnosis, then a clean Intel paired result and
all-controls-successful whole-DAG evidence under a newly authorized budget.
The Windows failure also prevents presenting this PR as release-ready.

The whole-DAG implementation and rejecting counterexamples remain available
for that evaluation; its production gates were not removed to obtain an
attractive number. Baseline/proposed proof artifacts, actual runner cohorts,
and complete attempt-specific job inventories remain attached to the three
linked workflow runs. Local copies and strict per-lane JSON comparisons
are retained in the session artifacts.

@danielmeppiel Daniel Meppiel (danielmeppiel) added ci-release-rehearsal Opt-in read-only matched-runner whole-release sharding rehearsal and removed ci-release-rehearsal Opt-in read-only matched-runner whole-release sharding rehearsal labels Sep 7, 2026
Emit concrete unittest subtest cases in the xdist controller without changing worker scheduling or relaxing JUnit parity. Accept read-only public API credentials without promoting them to PAT/Models aliases. Reserve native DLL path headroom in the installer fixture while retaining adversarial quoting and all original checks.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Comment thread tests/unit/test_release_validation_prerequisites.py Fixed
@danielmeppiel Daniel Meppiel (danielmeppiel) added ci-release-rehearsal Opt-in read-only matched-runner whole-release sharding rehearsal and removed ci-release-rehearsal Opt-in read-only matched-runner whole-release sharding rehearsal labels Sep 7, 2026
Keep the Windows installer harness short while providing an uncreated isolation root, and clear temporary MCP registry configuration from the process cache on fixture teardown. Preserve native assertions and strict benchmark controls.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@danielmeppiel Daniel Meppiel (danielmeppiel) added ci-release-rehearsal Opt-in read-only matched-runner whole-release sharding rehearsal and removed ci-release-rehearsal Opt-in read-only matched-runner whole-release sharding rehearsal labels Sep 7, 2026
Repair the import-state leak exposed by the last Intel sharding rehearsal. Preserve both sys.modules and parent-package identity, with regressions for rich and colorama fallbacks. Do not relax performance acceptance or production gates.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@danielmeppiel Daniel Meppiel (danielmeppiel) added ci-windows-installer-proof Opt in to read-only native Windows installer diagnostics and removed ci-windows-installer-proof Opt in to read-only native Windows installer diagnostics labels Sep 8, 2026
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@danielmeppiel Daniel Meppiel (danielmeppiel) added ci-windows-installer-proof Opt in to read-only native Windows installer diagnostics and removed ci-windows-installer-proof Opt in to read-only native Windows installer diagnostics labels Sep 8, 2026
Add read-only, non-promotable full-DAG proof with strict archive, job, source identity, and actual GitHub clock validation. Preserve production job authorities and default test scheduling.

Use the empirically proven short Windows installer temp root and reject empty or skipped installer evidence.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Create a fresh push event after the failed mixed-attempt rerun so the proof workflow can run as attempt 1 with matching artifact names.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Download exact native observation artifact IDs before terminal verification and use the canonical record validator at proof production. Preserve the measured history rather than accepting green-but-incomplete proof files.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@danielmeppiel

Daniel Meppiel (danielmeppiel) commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator Author

Actual release wall-clock result

Observed improvement: 34m41s to 24m53s, saving 9m48s (28.26%). Both selected GitHub workflows succeeded. This is a measured read-only release-preparation comparison, not an estimate or a timed signing/publication run.

Metric Frozen-main baseline Proposed release pipeline Difference
Workflow creation to successful terminal completion 2,081s / 34m41s 1,493s / 24m53s 588s / 28.26% shorter
Run start to terminal completion 2,081s 1,493s 588s shorter
Sum of executed job elapsed time, unweighted 6,614s / 110m14s 8,120s / 135m20s 1,506s / 22.77% higher
Required nonterminal jobs 14/14 succeeded 43/43 succeeded No required job waived
Terminal job Succeeded Succeeded Actual GitHub timestamps

Runner elapsed time is not billable cost: OS multipliers and pricing are not applied. More parallelism and stronger source qualification buy lower latency at higher total runner time.

Exact executions and clock

Side Fresh attempt-1 run Tested source SHA Controller/API head SHA
Baseline 34221564076 f8df1b751efc30b32dc01b125616b51f777b4c81 6b76f97e539cbe9c5513c4a7ed338b06976b84dd
Proposed 34228763968 1687380d74cc255519b2ab93ced5ac8744fdd54b 1687380d74cc255519b2ab93ced5ac8744fdd54b

The primary clock is Verify wall-clock artifacts.completed_at - workflow.created_at, including setup, inter-job queues, artifact handling, and the terminal job:

  • Baseline: 2026-09-08T11:36:14Z to 2026-09-08T12:10:55Z.
  • Proposed: 2026-09-08T12:54:25Z to 2026-09-08T13:19:18Z.
  • Both run_started_at values equal created_at; this is why the secondary clock matches.

Included: five native platforms, each side's unit/integration selections, binary builds, isolated validation, installer coverage, native archive packaging/verification, docs, wheel/sdist, and the proposed source qualification. Excluded: signing/notarization, publishing, deployment, private-service acceptance, and post-publication checks. Production integration sharding remains 1x4.

Evidence recovery and fail-closed repair

The original terminal proofs verified five native archives/sidecars and the docs/wheel/sdist artifacts, but incorrectly contained zero runner records: the terminal jobs did not download the separately retained observation artifacts. The strict comparator correctly rejected those originals.

The parent independently downloaded the original observation artifacts by immutable artifact ID, verified each downloaded ZIP against the GitHub API's SHA-256 digest, and checked every recovered runner record against the actual run, attempt, source, controller, and API-head identities. This recovered 9 baseline records (8 native plus the terminal record) and 15 proposed native records.

The original hosted proof files were preserved byte-for-byte. Separate, explicitly labelled post-run reconciled proofs retain the original proof digest and per-artifact provenance. The unchanged strict comparator then passed, using the predeclared 14/43 required-job name sets and unmodified attempt-specific GitHub timestamps. No GitHub environment was spoofed; no missing runner record was invented.

Original hosted proof artifacts: baseline 10055150182; proposed 10057846905. Reconciliation inputs, SHA-256 manifests, original and derived proofs, raw API responses, JUnit inventories, and the comparator output are retained in this session's final-wallclock-audit/ artifact directory.

The subsequent PR repair selects/downloads exact native observation artifact IDs before verification and rejects absent or malformed records at proof production, not only at comparison. This reporting-only repair follows the measured commit; it is not represented as having run in the historical 24m53s execution. No additional full run was launched to conceal the gap.

That follow-up is pushed on #2879 at aa850fabc5cbf56d0df6a2b8baedc1a766aa484d; its source CI 34261969889 succeeded. The duplicate draft #2904 was closed after folding its checksum repair into #2879; its branch and measured-run history were preserved.

Scenario coverage

Native lane Baseline JUnit cases Proposed JUnit cases Original scenario result
Linux x64 integration 9,621 9,622 No original ID missing; no passed-to-skipped transition
Linux ARM integration 9,621 9,622 No original ID missing; no passed-to-skipped transition
macOS Intel integration 209 210 No original ID missing; no passed-to-skipped transition
macOS ARM integration 9,623 9,624 No original ID missing; no passed-to-skipped transition
Native units, each of five platforms 22,450 22,841 39 old topology-contract IDs replaced; 430 added; no passed-to-skipped transition
Proposed Windows installer N/A for exact same candidate 1 passed, 0 skipped Full candidate installer harness executed

All 39 replaced unit identities are in tests/unit/test_platform_contract_workflow.py, whose contracts were rewritten for the new reusable workflow topology. No original product-unit identity is missing. The 98 unit cases per platform without an original-node property were compared by their retained JUnit class/name identity, rather than dropped from the inventory.

Windows integration uses the existing PowerShell wrapper without JUnit passthrough; its successful job/log and recorded runner are retained, but this report does not claim exact JUnit scenario parity for that lane. Legacy baseline smoke reports contain 9 skips on each Linux/Windows build, whereas the proposed built-candidate core reports contain 7 passes per native platform; those are different selections, not falsely equivalent cases.

The focused Windows root proof 34210978528 used the same built archive: the long-root installer case failed; the short-root case passed with zero skips. The earlier green-but-skipped run 34210113252 remains rejected.

Expensive tests and the remaining critical path

The proposed macOS ARM integration job takes 20m11s and finishes last. Its native gate completes at 13:18:07Z; native evidence and terminal verification finish 71 seconds later. Docs and wheel builds finish near the start and are no longer the tail.

The baseline tail includes the combined macOS Intel job, then final Windows isolated validation, packaging, queued late docs/wheel builds, and the terminal verifier. The benefit is observed at the whole-workflow boundary, not calculated by adding independent lane percentages.

From proposed macOS ARM JUnit, the largest families by summed testcase elapsed time are:

Family Aggregate case seconds
Architecture owner-rule mutations 455.125
Architecture install-compound mutations 174.954
Configuration surface lifecycle 163.516
Architecture linter entrypoint 137.636
Architecture contract-compound mutations 126.019
Required lifecycle state machine 121.663

The clean architecture shell-wrapper case alone reports 91.378s; the generated lifecycle reference-model case reports 81.033s. These are genuine optimization targets, not permission to delete assertions, reduce examples, or share mutable fixture state. Summed case seconds overlap across workers and must not be added to wall-clock savings.

Intel unit hotspots include Unix install ownership (178.595 case-seconds), checksum fixtures (157.246s), and public release metadata (138.046s), but Intel units are not the final bottleneck in the selected proposed run.

Rejected executions and protocol deviations

Run Outcome and treatment
34219761800 Baseline failed: observer plugin was not importable by console pytest. Repaired only the adapter's virtualenv exposure; frozen product/test source unchanged.
34221540445 Proposed failed at native evidence capture on Windows CRLF checksum sidecars. The bounded digest/name-preserving repair and regressions are folded into this PR.
34225097927 Initial proposed attempt failed at Intel unit artifact upload. Two subsequent attempts failed on mixed-attempt evidence/fresh-attempt guards. None is a speed proof.
34228759315 Successful PR-label run, but not selected: PR merge context and overlap with the selected immutable-push run.

The orchestration exceeded the declared four-run cap: six unique full-workflow run IDs and eight attempt executions occurred. It also launched an early proposed run alongside the baseline retry and an extra label run alongside the final push run. These are coordination failures, not compliant execution of the original experimental plan; no more runs are authorized.

Runner images were not uniformly matched: for example, Linux ARM integration moved from 20260831.111.1 to 20260907.118.1. Cache/order effects and the overlapping run are not normalized away. The reported 28.26% is therefore an observed successful pair, not a protocol-conformant controlled estimate, a statistical confidence interval, or a guarantee for every release. The original three-attempt sharding experiment remains closed and unsuccessful; this result does not retroactively validate it or authorize sharding rollout.

Resolve CodeQL alert 208 by reporting in-process ordinal comparison booleans instead of SHA256 fingerprints of synthetic tokens. Keep absence, preservation, and non-disclosure assertions unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@danielmeppiel

Copy link
Copy Markdown
Collaborator Author

Docs sync advisory

Verdict: no_change * Pages affected: 0 * LLM calls: 1/15

No additional user-facing corpus changes are required: this candidate changes release orchestration and test tooling, not CLI/API or dependency-parser contracts. The existing contributing-guide updates are included in the expanded-scope release panel. This is documentation-impact classification, not native qualification or publication evidence.

@danielmeppiel

Daniel Meppiel (danielmeppiel) commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator Author

APM Review Panel: ship_now

PR #2879's retained reassessment resolves the prior bounded concerns while preserving exact-archive promotion and separate publication authority.

cc Sergio Sisternes (@sergio-sisternes-epam) -- a fresh advisory pass is ready for your review.

Evidence-only update after synthesis. Run 34337498500, attempt 1, succeeded with run head 7227d255170f29f01119651b0f17fdc4af45fd0a and tested source/controller merge c5563f9ad159a06235c0de6024bb66397a7ae186. The coordinator read the parent's content-verification record and original proof: both commits have full tree 697913c4b45e72bb498fc4dc9dc82d3fddc52d68; this equality does not rename the binary's embedded merge SHA. Parent verification records 38 required successful jobs, five archive/executable proof-index agreements, two exact artifact ZIP digests/member sets, 13 configuration plus uv.lock Git hashes, and all 15 observer roles. Fresh head-associated gate and CodeQL checks succeeded; parent reports no open CodeQL alerts. These observed results supersede the pending-native statements in the preserved original synthesis and persona summaries below.

Qualification is not comparability or authorization. The original proof remains promotable:false and observation_assessment.status=inconclusive: six Windows worktree hash mismatches, across three observers and two files, exactly match CRLF-rendered canonical Git bytes in the parent's diagnosis. The new guard retains those mismatches; neither the proof nor its assessment was normalized. The parent's verification-only helper initially expected a complete assessment and was corrected after byte diagnosis; no product, test or source bytes changed. No controlled performance, production signing/publication or human approval follows from qualification success. Reviews remain comments only, with codeowner and last-push approval outstanding; requested reviewer is Sergio Sisternes (@sergio-sisternes-epam) and workflow CODEOWNER is author Daniel Meppiel (@danielmeppiel). Mergeability is not permission to merge or bypass review. No roster was relaunched and no new panel surface was created.

At candidate 7227d255170f29f01119651b0f17fdc4af45fd0a, tree 697913c4b45e72bb498fc4dc9dc82d3fddc52d68, against frozen base f8df1b751efc30b32dc01b125616b51f777b4c81, all nine schema-valid reviews resolve their 15 prior items: 12 recommendations and three nits. The architect's procedural-design endorsement was not a bug repair. No remaining findings emerged within this bounded reassessment; cumulative schema respawns remain zero, without resetting the driver. tests/unit/test_windows_signing_contract.py:70 now requires assert build_idx < sign_idx < smoke_idx < package_idx < upload_idx, with mutations moving signing after smoke, packaging and upload. scripts/release-candidate.test.cjs:668-737 exercises real qualification of source A/attempt 1 and verification from distinct tag B, checks exact archive/sidecar bytes and attempt-specific job requests, and tests source-attempt, artifact ID/digest, fingerprint, status and conclusion drift. Its assert.equal(fs.existsSync(options.outputRoot), false); protects against producing publication assets on those failures. These close demonstrated regression gaps without adding another release authority.

The actual caller at .github/workflows/release-platform.yml:307-312 keeps the job token in GITHUB_API_TOKEN, leaves the wallclock PAT empty and selects PublicApiOnly. The caller/helper preserves all four public dependency checks and failure propagation while retaining default private/inference credential requirements. tests/unit/test_release_validation_prerequisites.py:323-332 checks assert record["SelectedChecks"] == expected_checks, null PAT/Models/general-token aliases in public mode, existing-credential preservation and sentinel non-disclosure. scripts/windows/run-installer-proof.ps1:387-406 persists raw stdout/stderr and exit/timeout metadata before JUnit validation, records validation errors and rethrows. scripts/release-candidate.cjs:346-366,746-763 distinguishes automatic rebuilding requiring no action from explicitly selected unavailable candidates and explains fresh/reuse planning. Canonical guidance at integration-testing.md:429-457 specifies trusted sources and separate publication recovery; it deliberately does not promise unsupported publisher-only retries. Contributor-policy duplication, current review anchors and row 6's static-publication evidence wording are corrected.

scripts/release_wallclock.py:726-757,911-1030,1221-1253 retains schema/time/environment metadata, validates identities and timeline bounds, and recomputes indexed fifteen-role native comparability in linear work. Missing roles or environment drift cannot manufacture matched evidence; controlled_acceptance: false remains explicit. The toy auto-worker cap is two; production remains one shard/four workers. Parent executions -- not reviewer reruns -- report 61 Node passes, 267 targeted passes in 121.20s, 88 post-format passes in 2.88s after formatting-only changes, 24 killed/restored mutations, and seven-part lint plus architecture checks passing. These support the local contracts, not production signing or publication. This fold changes no workflow_call ABI or CI/MQ/lifecycle dependencies; #2882 separately owns additive lifecycle receipts/full-MQ integration, with composition parent-reported rather than a missing root feature.

Dissent. There is no substantive remaining disagreement. Test Coverage's integration-with-fixtures classification is more precise than Supply Chain's unit label for cross-run promotion: qualifier, verifier and filesystem are real; GitHub responses are mocked. Signing-order and publication-condition evidence remains static.

Aligned with: Secure by default; Governed by policy; OSS community-driven; Pragmatic as npm.

Growth signal. Keep the story about auditable qualification, not guaranteed speed: 34m41s to 24m53s belongs to historical source 1687380, with 22.77% higher unweighted runner time, protocol deviations and signing/publication excluded.

Panel summary

Persona B R N Takeaway
Python Architect 0 0 0 Final fold preserves centralized promotion, independent native qualification and separate publisher authority. No remaining architecture concern; final native evidence is pending.
CLI Logging Expert 0 0 0 Both prior logging findings are resolved: rejected JUnit retains raw diagnostics, and automatic candidate recovery no longer prescribes manual reruns.
DevX UX Expert 0 0 0 Recovery guidance and fresh/reuse planning messages resolve both prior UX findings; publication retries remain deliberately fail-closed and operator-owned.
Supply Chain Security 0 0 0 Cross-run verifier coverage is resolved; no remaining high-confidence concern in the reviewed delta. Native qualification remains pending.
OSS Growth Hacker 0 0 0 Stale anchors are resolved. Historical timing remains qualified, and the current release-authority narrative does not claim new-head production speed or approval.
Auth Expert 0 0 0 API-only caller/helper separation resolves the prior finding; default credential behavior and public dependency checks remain intact. Native qualification remains pending.
Doc Writer 0 0 0 Both original documentation findings are resolved. Recovery guidance preserves publication boundaries; final-head native qualification remains pending.
Test Coverage 0 0 0 All three prior findings are resolved; new helper and comparability regression traps are present. Parent logs support local coverage, not final-head native signing or publication.
Performance Expert 0 0 0 Both prior performance findings are resolved at 7227d25: observation assessment is linear and explicit, toy auto-workers are bounded, and historical timings remain historical.

B = blocking-severity findings, R = recommended, N = nits.
Counts are signal strength, not gates. The maintainer ships.

Architecture

classDiagram
 direction LR
 class Candidate["scripts/release-candidate.cjs"]
 Candidate : plan() qualify() verify()
 class Native[".github/workflows/release-platform.yml"]
 class Archive["scripts/package_release.py"]
 Archive : package() verify_extract()
 class Evidence["release-candidate-evidence.json"]
 Evidence : run_id run_attempt head_sha config_hashes required_jobs platforms
 class Release[".github/workflows/build-release.yml"]
 class Builders["docs-build.yml / pypi-distributions.yml"]
 Release ..> Candidate : promotion authority
 Release ..> Native : native qualification
 Release ..> Builders : read-only prebuilds
 Native ..> Archive : package and checked extraction
 Candidate ..> Evidence : writes and verifies JSON
 note for Candidate "Functional core / imperative shell; module, not runtime class"
 note for Native "Fan-out/fan-in: independent units/build; applicable results join at gate"
 class Candidate:::touched
 classDef touched fill:#fff3b0,stroke:#d47600
Loading
flowchart TD
 E["Tag / schedule / dispatch: .github/workflows/build-release.yml"] --> P["[I/O][NET] scripts/release-candidate.cjs plan()"]
 P --> R{"Reusable candidate_run_id?"}
 R -->|no| U["[EXEC] release-platform.yml unit-tests / release-unit.yml"]
 R -->|no| B["[EXEC] release-platform.yml build: build-binary; optional sign-binary.ps1; test_core_smoke.py"]
 B --> A["[FS][NET] package_release.py package(); upload candidate-attempt-binary"]
 A --> C["[EXEC] verify_extract(); full-validation integration/validation; Windows installer when applicable"]
 U --> G["release-platform.yml gate: require applicable successes"]
 C --> G
 G --> Q["[I/O][NET] qualify(): source CI plus native evidence"]
 Q -->|tag| V["[I/O][NET] verify(): source run/attempt/jobs, fingerprints, artifact IDs/digests"]
 Q -->|schedule or dispatch| S["[NET] Retain attempt-scoped evidence; no publication"]
 R -->|yes, tag| V
 V --> K{"All checks and verifyLocalArchiveBytes() succeed?"}
 K -->|no| F["Throw; failed Actions job; no publication"]
 K -->|yes| W["[FS][NET] Exact assets upload; create-release"]
 P -->|stable tag; PyPI also public| D["[EXEC][NET] docs-build.yml / pypi-distributions.yml read-only builds"]
 W --> J{"Release and corresponding build succeeded?"}
 D --> J
 J -->|yes| PUB["[NET] deploy-docs / publish-pypi"]; J -->|no| N["No corresponding publication"]
Loading

Recommendation

Code advisory only: no further bounded code changes are recommended. Native run 34337498500 was pending/in progress at briefing; its success is not presumed. Prior f4 qualification 34331009260, equal-tree merge 0bb0/c862 and old CodeQL results are historical, not new-head proof. Human codeowner/last-push approval, security approval and production signing/publication authority are not conferred. The parent can append execution evidence to existing comment 5599634294.


Full per-persona findings

Python Architect

No findings.

CLI Logging Expert

No findings.

DevX UX Expert

No findings.

Supply Chain Security

No findings.

OSS Growth Hacker

No findings.

Auth Expert

No findings.

Doc Writer

No findings.

Test Coverage

No findings.

Performance Expert

No findings.

This panel is advisory. It does not block merge. Re-apply the
panel-review label after addressing feedback to re-run.

@danielmeppiel Daniel Meppiel (danielmeppiel) removed the ci-release-wallclock Opt-in wall-clock release proof label Sep 9, 2026
Preserve API-only Windows dependency validation and failure diagnostics; strengthen immutable promotion/signing regressions and runner observation comparability without changing native scheduling or publication authority.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@danielmeppiel Daniel Meppiel (danielmeppiel) added ci-release-wallclock Opt-in wall-clock release proof and removed ci-release-wallclock Opt-in wall-clock release proof labels Sep 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants