Copilot/apmlifecycle phase2 interactions - #2813
Copilot/apmlifecycle phase2 interactions#2813Daniel Meppiel (danielmeppiel) wants to merge 21 commits into
Conversation
Co-authored-by: danielmeppiel <51440732+danielmeppiel@users.noreply.github.com>
…fecycle-test-comprehensiveness Co-authored-by: danielmeppiel <51440732+danielmeppiel@users.noreply.github.com>
Co-authored-by: danielmeppiel <51440732+danielmeppiel@users.noreply.github.com>
Co-authored-by: danielmeppiel <51440732+danielmeppiel@users.noreply.github.com>
Co-authored-by: danielmeppiel <51440732+danielmeppiel@users.noreply.github.com>
Co-authored-by: danielmeppiel <51440732+danielmeppiel@users.noreply.github.com>
…phase2-interaction-arrays Co-authored-by: danielmeppiel <51440732+danielmeppiel@users.noreply.github.com>
Co-authored-by: danielmeppiel <51440732+danielmeppiel@users.noreply.github.com>
…phase2-interaction-arrays Co-authored-by: danielmeppiel <51440732+danielmeppiel@users.noreply.github.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The integration covering-array module docstring is now incorrect (still describes “sparse pairs”) and should be updated to match the catalog-derived full routing-cell coverage.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Lite
Findings: 1
New issues introduced by this change (1)
| Severity | Finding |
|---|---|
 |
tests/integration/test_primitive_target_covering_array.py — The module docstring still says the covering array uses "sparse pairs" rather than exercising every… |
What changed in this PR
Adds Phase 2 APMLifecycle “interactions” coverage by turning the primitive/target coverage matrix into a catalog-derived routing-cell covering array, then introducing a fast PR-time ratchet to prevent coverage drift and runtime blowups as KNOWN_TARGETS evolves.
Changes:
- Introduces a subprocess-free quality ratchet (
test_lifecycle_interaction_coverage.py) that asserts full catalog coverage, unique row IDs, and a max-catalog-size ceiling. - Refactors the integration covering array to derive catalog rows from
KNOWN_TARGETS, adds artifact write-set assertions, and encodes an explicit “second-pass convergence” known gap. - Updates integration-testing docs + lifecycle bug ledger to describe and track the new routing-cell coverage and the Copilot user-instructions known gap; wires the new ratchet into CI.
| File | Description |
|---|---|
| tests/quality/test_lifecycle_interaction_coverage.py | Adds a fast PR-time ratchet validating routing-cell coverage and bounded growth. |
| tests/integration/test_primitive_target_covering_array.py | Derives catalog rows from KNOWN_TARGETS, adds artifact write-set assertions, and models a known idempotency gap. |
| tests/fixtures/lifecycle_bug_ledger.json | Records the new Copilot user-instructions second-pass convergence known gap. |
| docs/src/content/docs/contributing/integration-testing.md | Documents routing-cell coverage + the two known gaps (lockfile replacement + Copilot instructions). |
| .github/workflows/ci.yml | Adds the new lifecycle interaction ratchet test to the PR-time ratchet job. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| from tests.utils.artifact_snapshot import ( | ||
| ArtifactSnapshot, | ||
| ArtifactSnapshotSet, | ||
| assert_only_snapshot_paths_changed, | ||
| assert_snapshot_changes_within, | ||
| assert_snapshot_set_unchanged, | ||
| assert_unchanged, | ||
| ) |
Preserve the test-only continuation with exact-write oracles, deterministic replay, actual interaction witnesses, and tracked unresolved product regressions. Phase 2 remains incomplete; mutation work continues separately. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Exercise three child-runtime production mutations against green baselines, recompute ordered mutation receipts, and enforce separate JUnit-backed mutation completeness without granting interaction coverage exceptions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use manifest targets, saved configuration, then existing audit detection without inferring historical install overrides from ownership. Reuse live target profiles in warm and cold replay, fail unavailable intent explicitly, and keep old file and directory claims in comparison after target contraction. Add hermetic source-CLI and component regressions, authority mutations, and audit/config documentation. No lockfile schema or experimental manifest changes. Fixes #2816 Refs #2813 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Reject malformed current target intent, isolate or refuse native replay, retain contracted native claims for comparison, and keep audit configuration reads non-creating. Add real CLI boundary controls without modifying published v0.1 conformance. Refs #2816, #2813. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
APM Review Panel:
|
| Persona | B | R | N | Takeaway |
|---|---|---|---|---|
| Python Architect | 0 | 2 | 0 | The pure intent/observed-state split is sound; strengthen interaction cache-coherence evidence and move the shared executor out of the collected test module. |
| CLI Logging Expert | 0 | 1 | 0 | Outcome summaries and pytest-veto diagnostics are clear; malformed interaction receipts still lose artifact and testcase context. |
| DevX UX Expert | 0 | 0 | 0 | Read-only review of 468756b: command intent, refusal guidance and scope handling are coherent. No new DevX findings. No tests run; partial evidence does not establish Phase 2 acceptance. |
| Supply Chain Security | 0 | 0 | 0 | Exact-head source review found no additional containment/provenance concerns; no tests run. Known reporter/waiver repairs, final immutable receipts, and full campaign/runtime acceptance remain pen... |
| OSS Growth Hacker | 0 | 0 | 0 | No material growth or contributor-friction findings at 468756b: current evidence, historical results, and pending acceptance are clearly distinguished. |
| Doc Writer | 0 | 2 | 0 | Pinned guidance distinguishes current evidence from final acceptance; consolidate campaign details and align the waiver wording with the pending reporter fix. |
| Test Coverage | 0 | 1 | 0 | ApmLifecycle contracts are added/adjusted; per-transition law obligations need an independent regression trap. Native execution and final P6 acceptance remain unverified. |
| Performance Expert | 0 | 1 | 0 | One advisory snapshot optimization; no production hot-path change. Hosted full-campaign W_C-W_B <= 600s against an explicit comparable baseline remains unproven. |
B = blocking-severity findings, R = recommended, N = nits.
Counts are signal strength, not gates. The maintainer ships.
Top 5 follow-ups
- [Test Coverage] Independently enumerate the exact law set for every legal state/transition combination in the existing parametrized model contract. -- The exact-head probe showed that removing SOURCE only on repair leaves the existing 66 cases and three corpus validations satisfied. Assert exact equality against independently authored expectations and include that omission as a negative control. Carry the contract through fix(install): deploy skills through home directory aliases #2876's expanded 26-transition, three-scope model seam rather than replacing it with ordinary CLI rows.
- [Python Architect] Require independent primary materialization, .apm-pin, and applicable cache-identity observations before source-coherence credit. -- The source-read contract gap means lock graph and commit checks currently receive broader law credit than their observations establish. Add a component corruption control that keeps valid lock/native deployment data while altering primary source bytes or pin identity; account explicitly for advanced tags, local sources, and intentionally absent cold caches.
- [Doc Writer] Align the known-gap wording within the parent's existing reporter/pair-waiver correction. -- This is a companion edit to work already owned by the parent, not a duplicate implementation task. State that the exact Copilot record earns no successful pair credit and that all 514 pairs still require successful witnesses elsewhere, leaving zero uncovered pairs.
- [CLI Logging Expert] Include artifact path and classname::name in malformed interaction-receipt errors. -- The exact-head in-memory probe observed contextless JSON and field-validation failures. Preserve the original exception as the cause and add focused contracts for both malformed JSON and invalid fields so maintainers can identify the offending shard and testcase directly.
- [Doc Writer] Separate full-campaign guidance from the PR-time smoke overview within the existing contributor page. -- A dedicated linked heading makes execution and evidence boundaries easier to follow without adding a new page. Preserve pytest exit statuses, immutable candidate provenance, reporter limitations, pending controls, and the distinction between the two-module local recipe and final qualification.
Architecture
Before: merge base
classDiagram
direction LR
class RuleBasedStateMachine
class _LifecycleReferenceModel {
+install()
+audit_tampered()
+durable_state_matches_reference_model()
}
class _ModelFixture {
+create()
}
class _Scenario
class ApmLifecycleRunner {
+run()
}
class ArtifactSnapshotSet {
+capture()
}
class LifecycleStateSnapshot {
+capture()
}
class _Row
class test_primitive_target_covering_array {
<<Module>>
+_capture_state()
+_assert_materialized()
+_assert_removed()
}
class LockFile
RuleBasedStateMachine <|-- _LifecycleReferenceModel
_LifecycleReferenceModel *-- _ModelFixture
_ModelFixture o-- _Scenario
_Scenario o-- ApmLifecycleRunner
_LifecycleReferenceModel ..> ArtifactSnapshotSet
test_primitive_target_covering_array ..> _Row
test_primitive_target_covering_array ..> ApmLifecycleRunner
test_primitive_target_covering_array ..> LifecycleStateSnapshot
test_primitive_target_covering_array ..> LockFile
note for _LifecycleReferenceModel "Scheduling, mutable intent, CLI actions and assertions share one class"
note for test_primitive_target_covering_array "Native deployment checks follow observed ledger records"
After: reviewed head
classDiagram
direction LR
class RuleBasedStateMachine
class _LifecycleSearch
class LifecycleModel {
<<ValueObject>>
+declared bool
+materialized bool
+clean bool
}
class ObservedState {
<<ValueObject>>
+artifacts ArtifactSnapshotSet
+lock_commits tuple
+cache_revision str
}
class TransitionObservation {
<<ValueObject>>
}
class LifecycleDriver {
+apply()
+replay()
+evidence()
}
class ApmLifecycleRunner {
+scenario()
+run()
}
class ArtifactSnapshotSet {
+capture()
}
class RoutingRow {
<<ValueObject>>
}
class SourceFixture {
<<ValueObject>>
}
class RoutingExpectation {
<<ValueObject>>
}
class InteractionOracle {
+observe()
+assert_routing()
+evaluated()
+assert_finished()
}
class CaseExecution {
<<ValueObject>>
}
class test_primitive_target_covering_array {
<<Module>>
+execute_row()
+_execute_row()
+_execute_row_body()
+_assert_provenance()
}
class lifecycle_interaction_report {
<<Module>>
+read_executions_from_root()
+main()
}
class LockFile
RuleBasedStateMachine <|-- _LifecycleSearch
_LifecycleSearch *-- LifecycleDriver
LifecycleDriver *-- LifecycleModel
LifecycleDriver ..> TransitionObservation
TransitionObservation o-- ObservedState
ObservedState o-- ArtifactSnapshotSet
LifecycleDriver ..> ApmLifecycleRunner
test_primitive_target_covering_array ..> ApmLifecycleRunner
test_primitive_target_covering_array ..> InteractionOracle
InteractionOracle o-- RoutingRow
InteractionOracle o-- SourceFixture
InteractionOracle ..> RoutingExpectation
InteractionOracle ..> ArtifactSnapshotSet
InteractionOracle ..> LockFile
test_primitive_target_covering_array ..> CaseExecution
lifecycle_interaction_report ..> CaseExecution
note for LifecycleDriver "Composition with dependency injection: command and declaration callbacks"
note for test_primitive_target_covering_array "Composition with dependency injection: shared runner entrypoint for ordinary and mutant execution"
note for lifecycle_interaction_report "Collect-then-render: observed receipts are aggregated and persisted before completeness checks"
class _LifecycleSearch:::touched
class LifecycleModel:::touched
class ObservedState:::touched
class TransitionObservation:::touched
class LifecycleDriver:::touched
class ApmLifecycleRunner:::touched
class RoutingRow:::touched
class SourceFixture:::touched
class RoutingExpectation:::touched
class InteractionOracle:::touched
class CaseExecution:::touched
class test_primitive_target_covering_array:::touched
class lifecycle_interaction_report:::touched
classDef touched fill:#fff3b0,stroke:#d47600
Before: merge-base routing execution
flowchart TD
A["pytest: tests/integration/test_primitive_target_covering_array.py::test_primitive_target_covering_array"] --> B["[FS] IsolatedApmEnvironment.create; LocalPackageFactory.create"]
B --> C{"row.dynamic_refusal?"}
C -->|yes| D["[EXEC] _run_dynamic_refusal: baseline install; requested target install"]
D --> E["Require returncode != 0 and assert_unchanged"]
C -->|no| F["[I/O] _capture_state: LifecycleStateSnapshot.capture"]
F --> G["[EXEC] _run: install; require exit 0"]
G --> H["[I/O] _assert_materialized: _deployment_records; read ledger-selected paths"]
H --> I{"row.widen_targets or row.narrow_targets?"}
I -->|yes| J["[FS] _set_project_targets / _write_user_manifest"]
J --> K["[EXEC] _run: install changed targets"]
I -->|no| L["[EXEC] _run: uninstall"]
K --> L
L --> M["[I/O] _assert_removed: ledger empty and prior recorded paths absent"]
After: reviewed interaction and report boundaries
flowchart TD
A["pytest: tests/integration/test_primitive_target_covering_array.py::test_primitive_target_covering_array"] --> B["execute_row: ApmLifecycleRunner.scenario"]
X["pytest: tests/integration/test_lifecycle_mutation_campaign.py::test_lifecycle_production_mutation"] --> Y["[EXEC] Inject runner for lifecycle_mutations.py::child_main"]
Y --> C
B --> C["_execute_row: _canonical_native_roots"]
C --> D["[FS] _execute_row_body: author sources, seed neighbors and co-owners"]
D --> E["expected_routing: authored sources plus catalog capabilities"]
E --> F["[I/O] InteractionOracle.observe / transition: capture before and after"]
F --> G["[EXEC] action: ApmLifecycleRunner.run"]
G --> H["_result: validate receipt before evaluated"]
H --> I{"operation"}
I -->|refusal| J["Require exit 1 and authored target diagnosis; unchanged=True"]
I -->|audit| K["assert_ci_audit_result: clean exit 0 or tampered exit 1; attributed completed checks"]
I -->|install / update / uninstall| L["Require exit 0"]
L --> M["[I/O] InteractionOracle.assert_routing; co-owner checks"]
M --> N["[I/O] _assert_provenance: lock graph and commits; credit source law"]
J --> O["InteractionOracle.evaluated"]
K --> O
N --> O
O --> P{"remaining row actions?"}
P -->|tagged update| Q["[EXEC] repositories.commit / advance_tag; then action(update)"]
Q --> F
P -->|warm / cold / tamper / target change / cleanup| R["[FS] Observed fixture changes; subsequent action through same boundary"]
R --> F
P -->|none| S["InteractionOracle.assert_finished(required_transitions(row))"]
S --> T["[I/O] _evidence / _write_evidence; pytest lifecycle_execution property"]
T --> U["[I/O] tests/utils/lifecycle_interaction_report.py::main: ET.parse; read_executions_from_root"]
U --> V{"JUnit failure, error or skipped?"}
V -->|yes| W["Replace receipt status; preserve diagnostics; no interaction credit"]
V -->|no| Z["coverage_report: check action and law requirements"]
W --> Z
Z --> AA["[FS] Persist interaction report; write_mutation_report when requested"]
AA --> AB["assert_complete_mutation_evidence / assert_complete_evidence when requested"]
AB --> AC["Current known-gap pair allowance remains here; parent-owned correction pending"]
Recommendation
The current candidate is not ready for Phase 2 delivery. Recommend folding the bounded framework proof fixes now, led by independent per-transition law obligations, while retaining all seven findings at recommended severity. Do not launch another partial native campaign. Reserve P6 qualification until the runtime and expanded-model successors form the final combined candidate, followed by the complete host-coordinated campaign with immutable provenance and actual HOSTED W_C-W_B <= 600s against an explicit comparable baseline. Genuine human/specification requirements remain pending. This advisory is not final phase authorization.
Full per-persona findings
Python Architect
-
[recommended] Check primary materialization and cache identity before crediting source coherence at
tests/integration/test_primitive_target_covering_array.py:604
The ledger defines source.ref_cache_coherent as agreement between materialized content, resolved references, and cache identity. _assert_provenance checks the primary dependency graph and resolved commits in the lockfile, then credits that entire law. Its materialization checks cover only persistent co-owners. The primary packages' .apm-pin identities and materialized source bytes are not checked here; _row_permissions merely authorizes those paths, while the warm/cold controls inspect cache presence. Consequently, this specific law receipt does not establish its full declared invariant, even though subsequent commands may detect some corruptions incidentally. The model driver already distinguishes these observations in law_source_coherent. This is a recommended evidence improvement, not a demonstrated production regression or a request to run the deferred campaign. Source inspection only; no tests were executed.
Suggested: Supply _assert_provenance with independently authored primary materialization expectations, including the expected revision and source bytes after tag advancement. Validate those observations before appending source.ref_cache_coherent, respecting intentionally absent cold caches and local-source semantics. Add a component corruption control that preserves valid lock/native deployment data while changing a primary pin or materialized source file and verifies that no source-law credit is emitted.
Proof (missing, integration-with-fixtures):tests/integration/test_lifecycle_interaction_oracle_contract.py-- proves: Install, tamper, audit, repair, and prune agree about package state, including the source identity behind interaction evidence. [portability-by-manifest,devx] -
[recommended] Give the shared injected row executor a utility-module home at
tests/integration/test_primitive_target_covering_array.py:703
The collected test module now owns the reusable execution boundary, native-root isolation, fixture publication, receipt validation, provenance checks, and evidence writing. The ordinary routing test, mutation campaign, and oracle contracts all consume this implementation; the latter two import its private helpers. In particular, test_lifecycle_mutation_campaign.py imports _execute_row directly, while numerous contract tests depend on its private implementation structure. This is already shared framework code rather than a test-local helper, so leaving its ownership in a collected test module makes reuse and the pending model/global integration unnecessarily coupled to test collection and private APIs. The injected runner and shared observation boundary themselves are good choices; no new hierarchy is needed.Design patterns
- Used in this PR: Dataclass-as-value-object -- LifecycleModel, ObservedState, SourceFixture, RoutingRow, and CaseExecution separate intent, observations, and receipts.
- Used in this PR: Composition with dependency injection -- LifecycleDriver receives command/declaration callbacks, and _execute_row receives ApmLifecycleRunner for ordinary and mutated execution.
- Used in this PR: Collect-then-render -- observed evaluations become CaseExecution receipts that lifecycle_interaction_report.main aggregates and persists before completeness assertions.
- Pragmatic suggestion: Module extraction -- move the existing shared executor into tests/utils without introducing a base class, plugin registry, or a second implementation.
Suggested: Extract the shared execution functions into a test utility module with one supported injected-runner entrypoint. Leave parametrization, pytest properties, and the strict xfail in the integration test. Route mutation and component callers through the same entrypoint, preserving native-root isolation, independent intent/observation, and the existing behavioral and static boundary contracts.
CLI Logging Expert
- [recommended] Identify the shard and testcase when interaction receipt parsing fails. at
tests/utils/lifecycle_interaction_report.py:47
At the reviewed commit, malformed JSON and execution_from_mapping validation errors escape without the input path or testcase identity. In the four-shard aggregation this leaves maintainers searching artifacts to locate the rejected receipt, unlike the contextual pytest-veto diagnostics. A cheap in-memory probe using the exact-head function definitions produced only 'JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)' and 'ValueError: Execution duration_seconds must be a number'. This verifies diagnostic behavior only; no native campaign or final acceptance evidence was run.
Suggested: Wrap JSON decoding and execution_from_mapping validation with a contextual ValueError naming path and classname::name, preserving the original exception as the cause. Add focused malformed-JSON and invalid-field contracts asserting that both artifact and testcase remain visible.
DevX UX Expert
No findings.
Supply Chain Security
No findings.
OSS Growth Hacker
No findings.
Auth Expert -- inactive
The exact diff touches lifecycle test utilities, integration/quality tests and fixtures, .github/workflows/ci.yml, .github/workflows/ci-integration.yml, and docs/src/content/docs/contributing/integration-testing.md, without production src/ or credential/authorization semantic changes.
Doc Writer
- [recommended] Update the report-boundary wording alongside the pending pair-waiver fix at
docs/src/content/docs/contributing/integration-testing.md:54
Source-read: line 54 accurately describes this pinned implementation, whose assert_complete_evidence permits uncovered pairs belonging to the known-gap row. However, that is broader than the approved row-only exception and the zero-uncovered final obligation stated at line 76. This is the documentation companion to the reporter fix already owned by the parent, not a separate implementation finding or a request for another partial campaign.
Suggested: When folding the reporter fix, state that the exact known-gap record remains uncredited and does not waive pair coverage: all 514 pairs must have successful witnesses elsewhere. Until then, identify the current permissiveness explicitly as a limitation, not the approved exception. - [recommended] Separate full-campaign instructions from the PR-time smoke overview at
docs/src/content/docs/contributing/integration-testing.md:48
Source-read: the page grows from 3,721 to 4,376 whitespace-delimited words, with no compensating documentation cuts in the reviewed diff. Most additions sit under 'Lifecycle Smoke (PR-time required check)', although the documented routing, replay/search, and mutation executions explicitly run outside that required selection. This makes a bounded smoke-test overview carry detailed campaign accounting and final-candidate review instructions, weakening progressive disclosure and discoverability.
Suggested: Keep a short smoke-selection summary and its local command in Tier 3. Group the full-campaign recipe and evidence boundaries under a dedicated heading on this existing page, linking from Tier 3. Consolidate repeated witness-credit qualifications while preserving the exit-status, provenance, pending-controls, and final-acceptance distinctions.
Test Coverage
- [recommended] Independently assert the complete law set for every legal model transition. at
tests/integration/test_lifecycle_model_contract.py:286
At reviewed head 468756b, the transition contract checks only that applicable laws belong to LAWS and include four unconditional laws. Replay assertions derive their per-step expectations from applicable_laws itself; corpus validation checks the union across a sequence. The corruption contracts directly exercise SOURCE on install, not its applicability on repair. Exact-head git grep for applicable_laws and _LAW_TRANSITIONS confirmed these coverage paths. A read-only, in-memory probe executed the existing 66 transition-contract cases and three literal corpus validations: both the baseline and an altered applicability function omitting SOURCE only on repair were accepted. This was a selected pure/component probe, not a pytest-suite or native-CLI run. Consequently, removing repair's source/cache verification can silently weaken the harness while these regression traps remain green.
Suggested: Extend test_transition_function_is_total_over_reachable_states with independently authored expected law sets for every legal state/transition combination. Assert exact equality rather than subset membership. Include a negative control that removes SOURCE only from repair and demonstrates that the independent expectation rejects it. Carry this contract forward when integrating the expanded model seam; reuse the existing parametrized test rather than adding an expensive lifecycle campaign.
Proof (missing, unit):tests/integration/test_lifecycle_model_contract.py::test_transition_function_is_total_over_reachable_states-- proves: Install, tamper, audit, repair, and prune agree about package state. [devx,portability-by-manifest]
assert applicable_laws(state, transition) == independently_expected_laws[state, transition]
Performance Expert
- [recommended] Reuse same-transition snapshots instead of repeatedly walking and hashing all five roots. at
tests/integration/test_primitive_target_covering_array.py:1040
Static inspection of exact head 468756b shows five complete captures around an ordinary clean reinstall: the caller's before capture at line 1040, transition before/after captures at lifecycle_interaction_oracle.py:336-340, assert_routing's capture at line 397, and _assert_reinstall's capture at test_primitive_target_covering_array.py:630. Each capture traverses project, user, cache, sources and repositories, sorting paths and reading regular-file bytes. For F entries and B bytes, this repeats O(F log F + B) work five times around one command, including unchanged Git objects. Sharing the two boundary snapshots can remove three redundant traversals and their byte reads without narrowing observation. This is a source-derived constant-factor opportunity, not a measured dominant phase or demonstrated hosted budget violation; the selected PR-body timings do not establish either.
Suggested: Expose immutable before/after snapshots from the observed action and pass them explicitly to routing and idempotency assertions while no intervening mutation occurs. Preserve fresh captures across every CLI/fixture mutation and preserve full-root hashing; standalone assertion callers should still capture when no snapshot is supplied. Add a cheap injected-runner guard counting captures and bytes read at N and 10*N fixture files: require two boundary captures per action and approximately linear byte-read growth, while retaining neighboring-write and tamper rejection tests. Keep any hosted performance claim pending the authorized complete campaign and comparable baseline, not another partial native run.
This panel is advisory. It does not block merge. Re-apply the
panel-review label after addressing feedback to re-run.
Docs sync advisoryVerdict: no_change * Pages affected: 0 * LLM calls: 1/15 * Took: not recorded The canonical index's deterministic L0 gate classifies all 20 changed paths at published The contributor edit is already included: source-CLI and independent exit-status guidance, fixture-derived routing/ownership, strict audit/refusal attribution, tagged-update action credit, and explicit final-acceptance boundaries. The PR body now clearly separates current evidence from its preserved historical This classification is not functional verification, a final panel, combined acceptance, hosted B/C evidence, or human/specification approval. The same advisory surface is retained; no companion PR or new documentation panel was needed. Execution and revision history
This |
Address the r1 CEO audit and native-ownership follow-ups with exact CI audit results, source-derived owners, contained native leaves and legitimate shared-package fixtures. Protect unowned neighbors and surviving contributions without expanding the existing Copilot exception; genuine production failures remain visible. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Address the r1 CEO mutation-evidence follow-ups by binding reached effects to the intended skill, roots and command project, retaining actual child cwd and preserving rejection diagnostics. Add strict counterexamples and real traceback controls so unrelated failures cannot earn mutation credit. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Address the r1 CEO replay, cache-witness and reporting follow-ups without shrinking the campaign. Pin reviewed obligation content, parse each shard once, preserve rejected and independently valid evidence, and provision only runtime dependencies in report fan-in. Document source identity, independent exit statuses and the stronger final acceptance boundary. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Fix the eighteen Lifecycle Smoke setup failures in run 34064612830: the new component helper passed the developer environment to Git and only worked where an author identity happened to be configured. Reuse IsolatedApmEnvironment for local repository setup and prove both missing and ambient identity/configuration cannot leak into the fixture. Native production failures remain unwaived. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Require complete attributed audit receipts at the shared replay/search consumer and reject opposite-state or unrelated failures before outcome credit. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Scope native roots at the injected executor boundary, including mutation callers. Require exact target-specific refusal outcomes and preserve full-root no-write observation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Enforce tagged-update action evidence, retain diagnostic text rather than traceback-bearing exceptions, and document the bounded framework contracts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
harden(lifecycle): make interaction coverage executable and auditable
Current checkpoint: 2026-09-07
Warning
Framework preparation, not Phase 2 completion or delivery approval. Current published head:
468756be6d1767784ccf29d475ea286a019170d0, based on actual main045f78cd71e56b9029e1b4193f7e4d4a46b00086. The complete8499f498erecord below is historical, including its commands, counts, failure outcomes, and links.R2 now requires complete, independently attributed model-audit receipts and actual tag advancement for tagged updates. The shared injected executor isolates Claude/Hermes roots, checks target-specific refusal status/diagnosis before credit, and preserves no-write observation; report aggregation retains diagnostic text rather than prior XML trees. Contributor guidance reflects these boundaries. No production
src/changes were added.The actual R2 panel reviewed
03384f72; its five bounded follow-ups are folded. A focused custom implementation review found no high-confidence remaining defects in the exact committed delta. This is not a new final panel or a qualifying human review.Current evidence: all canonical/strict pre-push gates, PR CI, and the aggregate gate passed. Hosted Lifecycle Smoke recorded 523 passed/1 skipped. The PR remains
MERGEABLE / BLOCKED; this is not hosted merge-group B/C evidence.Exact-head source-CLI output from the 14-module contract/model/mutation selection, six existing configured-root/refusal/tag rows, and runtime-only mutation reporting, respectively:
These are test-node results, not 583 interaction pairs or the historical six-file command. The targeted native report covers only 110/514 pairs; 118 scheduled cases were intentionally not run here. Raw model evidence includes three replays and bounded search, retaining seven empty/unexecuted attempts without credit.
Current reproduction and remaining dependency
Final obligations remain 5,040 legal inputs, 124 main rows, 514 covered/zero uncovered pairs, three named triples, three source mutants, exact model/replay/search/scope and four ordinary startup controls, plus 123 accepted records and the exact uncredited Copilot gap. Genuine human/specification requirements and actual hosted
W_C-W_B <= 600sremain open. No enqueue, merge, new waiver, or Phase 3 work is authorized by this checkpoint.Historical 8499f49 checkpoint - preserved verbatim, not current reproduction guidance
harden(lifecycle): make interaction coverage executable and auditable
TL;DR
Continue #2809 with a test-only lifecycle checkpoint: a pure intent model, deterministic replay, catalog-derived interaction cases, exact-write oracles, and three isolated production-behavior mutations.
The changes replace permissive write allowances and planned-only coverage with assertions at observed action boundaries.
They also expose unresolved product behavior rather than converting it into successful coverage.
Problem (WHY)
These are observed harness weaknesses; the linked Agent Skills excerpts are design anchors, not evidence of APM behavior.
Approach (WHAT)
Implementation (HOW)
All links below are pinned to the published checkpoint. The diff contains 20 test, fixture, CI, and contributor-documentation files.
tests/utils/lifecycle_model.pytests/fixtures/lifecycle_transition_corpus.jsontests/integration/test_lifecycle_model_contract.pytests/utils/lifecycle_model_driver.pylifecycle_model_executionevidence.tests/integration/test_generated_lifecycle_state_machine.pytests/utils/apm_lifecycle_runner.pytests/integration/test_apm_lifecycle_runner_contract.pytests/utils/lifecycle_interactions.pytests/utils/lifecycle_interaction_oracle.pytests/integration/test_primitive_target_covering_array.pytests/integration/test_lifecycle_interaction_oracle_contract.pytests/utils/lifecycle_interaction_report.pytests/utils/lifecycle_mutations.pytests/integration/test_lifecycle_mutation_campaign.pytests/integration/test_lifecycle_mutation_campaign_contract.pytests/quality/test_lifecycle_interaction_coverage.pytests/fixtures/lifecycle_bug_ledger.json.github/workflows/ci-integration.yml.github/workflows/ci.ymldocs/src/content/docs/contributing/integration-testing.mdDiagrams
The interaction pipeline compares observed receipts with independent obligations and saves its report before gating; dashed nodes mark new or tightened contracts.
flowchart LR subgraph Intent["Independent test intent"] C["catalog_rows(): live static catalog"] P["124 cases, valid pairs and named triples"] F["expected_routing(): source-authored expectations"] C --> P end subgraph Execution["Observed interaction execution"] R["execute_row(): CLI and fixture writes"] O["InteractionOracle: full-root observations and laws"] R --> O end subgraph Evidence["Execution evidence"] J["read_executions(): JUnit witnesses"] D["Report main(): persist coverage JSON"] G["assert_complete_evidence(): reject incomplete runs"] J --> D --> G end P --> R F --> O O --> J P --> D classDef new stroke-dasharray: 5 5; class P,F,O,J,G,D new;Trade-offs
lifecycle_smoke and not lifecycle_merge_group; the routing campaign stays outside it. The ten-minute incremental hosted merge-group budget is an unproven exit criterion, not a measured benefit.Benefits
Validation
Published-checkpoint contracts, rerun after merging main:
Exact command and output: 241 fast contracts
Output from the command above:
The pre-push lint mirror also passed after merging main: Ruff check/format, pylint R0801, auth/architecture boundaries, YAML I/O, portable-relative-path, and file-length guards.
Selected lifecycle campaign at the published head, four split groups with two workers per group:
Aggregate: 373 passed, 2 failed, 1 strict xfailed. Main-row evidence: 122 executed, 1 known gap, 1 failed, 0 missing records.
Executed coverage is 510/514 pairs; the three named triples have 9, 3, and 6 successful witnesses, respectively.
The four uncovered pairs combine grok-cloud with local source, absent ref, absent cache, and audit command.
python -m tests.utils.lifecycle_interaction_reportwith both completeness flags over those four shard XML files:--require-mutationspasses;--require-completestill fails oninteraction-b8852029871d9058(#2816), after preserving both reports.The second failing pytest node is the focused user-relative transitive regression (#2815); it is additional to the 124 main rows.
The separate strict xfail expresses the unresolved Copilot user-instruction idempotency law.
An independent reviewer adjudicated the three production call paths and original receipts; the coordinator then hardened accounting and reran the campaign. Wrong-target detection is via missing authorized output after reconciliation, not a claim to observe every transient write.
Scenario Evidence
Failing and expected-failing rows below are regression traps, not proof that the product promise is satisfied.
tests/integration/test_generated_lifecycle_state_machine.py::test_lifecycle_corpus_replaytests/integration/test_primitive_target_covering_array.py::test_primitive_target_covering_array(the three named transition rows)tests/integration/test_lifecycle_interaction_oracle_contract.py::test_exact_ancestors_reject_neighbor_overwrite;::test_shared_config_corruption_rejected_despite_exact_file_permission, plus the real routing rowstests/quality/test_lifecycle_interaction_coverage.py::test_generated_rows_never_count_as_executed_coverage;::test_junit_result_overrules_earlier_success_evidencetests/integration/test_primitive_target_covering_array.py::test_local_transitive_command_scope_parity[user-relative-regression]tests/integration/test_primitive_target_covering_array.py::test_primitive_target_covering_array[interaction-b8852029871d9058]tests/integration/test_primitive_target_covering_array.py::test_copilot_user_instructions_desired_idempotencytests/integration/test_lifecycle_mutation_campaign.py::test_lifecycle_production_mutation(all three mutants)How to test
uv sync --frozen --extra dev; use the resulting source CLI withexport APM_E2E_TESTS=1 APM_BINARY_PATH="$PWD/.venv/bin/apm"in a POSIX shell.uv run --frozen --extra dev --no-sync python \ -m tests.utils.lifecycle_interaction_report \ --junit lifecycle-junit.xml --revision "$(git rev-parse HEAD)" \ --output lifecycle-interactions.json \ --mutation-output lifecycle-mutations.json --require-mutations --require-completeStill required for Phase 2: resolve the unwaived product-contract failures under separate authorization, demonstrate full executed pair coverage after those fixes, measure incremental hosted runtime against the ten-minute budget, and obtain explicit phase-exit approval.
No OpenAPM-observable production behavior changes or release-note entry are included in this checkpoint.
Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com
Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com