fix(install): preserve generated aggregate ownership and rebuilds

[Daniel Meppiel (danielmeppiel)]

fix(install): preserve and rebuild shared Copilot aggregate ownership

TL;DR

Record every contributor to generated Copilot user instructions with the final combined-content digest, then reconstruct the file from surviving package sources and genuine root authoring on uninstall. Preserve ordinary last-writer collision behavior and refuse unsafe cleanup. The directly coupled warm-codec repair accumulates ordered owners before freezing records, avoiding repeated growing-owner copies without changing receipt semantics.

Important

Aggregate-only successor to #2820, not a claim to resolve all of #2815, #2816, or #2818. Local-source admission and acquisition trust belong to #2919. This PR does not activate OpenAPM v0.2, change the v0.1 assessment, ratify specification text, or authorize merge.

Problem (WHY)

• The original reproduction retained two authored sections but recorded only the last contributor. Component mutation reproduces the ownership loss with 50 contributors.
• An older survivor-only receipt can omit the package being removed even while its generated section remains. Deleting the surviving-snapshot guard reproduces Unrecorded removed-owner body survived in the installed CLI.
• The old warm reconstruction algorithm copies growing owner lists: at 50/500 contributors the no-root mutation constructs 1,325/125,750 owner memberships, failing the work bound.
• [!] Failed cleanup/rebuild completes package and declaration removal first; documentation and terminal recovery must not imply rollback, complete success, or a force bypass.

These are executable failure cases, not speculative expansion of the original fix. The evidence-first scope follows Agent Skills' "Real-world failure cases and their resolutions".

Approach (WHAT)

• Extend existing reconciliation only for declared aggregate paths; keep other collisions last-writer.
• Hash stable shared output once, accumulate all contributors, and project completed receipts.
• Rebuild from validated surviving sources and root instructions through existing authorization/scanning.
• Restore Git source identity using one shared helper, without granting acquisition trust.
• Propagate confirmed removal into canonical, local and virtual-root compatibility state.
• Accumulate legacy owners in insertion order, move repeats to the end, and freeze each final record once.

Implementation (HOW)

Runtime paths below are under src/apm_cli/; test and documentation paths are repository-relative.

Surface	Change
models/apm_package.py	Shared Git identity restoration helper and installed-package caller; prerequisite f1a605862.
install/sources.py	Shared cached caller in 63b361990, whose parent is f1a605862; no trust fields added.
core/deployment_state.py	Explicit shared-path claims and ordered aggregate owner accumulation.
core/deployment_ledger.py	Root cleanup snapshots, synchronized virtual-root projection, and bounded warm legacy reconstruction.
install/phases/lockfile.py	Final shared hashes, one projection, cleanup invalidation, and current root hash preservation.
install/manifest_reconcile.py	Protect files still claimed by genuine root authoring.
integration/instruction_integrator.py	Discover aggregate paths and route whole-file cleanup through canonical hash/containment checks.
commands/uninstall/engine.py	Restore survivor/root content, including old last-writer receipts; expose typed aggregate failures.
commands/uninstall/cli.py, commands/uninstall/lockfile_state.py	Scope-correct partial-removal recovery and confirmed-removal propagation.
install/services.py, install/helpers/security_scan.py	Suppress ineffective force hints during reconstruction without relaxing scanning.
.apm/architecture/owners/install-deployment.json	Attribute aggregate state and integration changes to existing canonical owners.
scripts/architecture_linter/checks/install_base_integrator_and_contraction.py, install_deployment_analyzers.py	Register/check aggregate routing and confirmed-removal forwarding.
tests/integration/test_architecture_copilot_aggregate.py, test_architecture_owner_rule_mutations.py	Preserve ten aggregate bypass witnesses and register the aggregate mutation.
tests/integration/test_copilot_aggregate_uninstall.py	Thirteen real installed-CLI lifecycle cases, including persisted legacy receipt and both unowned sentinels.
tests/unit/core/test_deployment_state.py	Fourteen codec ordering/hash/native-row semantic controls.
tests/unit/install/phases/test_lockfile_cross_package_reconcile.py	Attachment and warm 50/500 work counts, final digest, and invalidation controls.
tests/unit/install/test_cached_source_identity.py, tests/unit/test_aggregate_installed_source.py	Cached/fresh Git identity and local/registry nonconversion controls.
tests/unit/test_uninstall_engine_helpers.py	Sanitized project/global recovery and truthful failed outcomes.
tests/unit/install/test_install_copilot_user_instructions.py, tests/unit/scripts/test_architecture_runner.py	Relative output-location assertion and registered-rule inventory.
tests/spec_conformance/test_aggregate_lockfile_reqs.py	Genuine existing req-lk-020 fresh-path and req-lk-016 hash-envelope witnesses.
CONFORMANCE.json, CONFORMANCE.md	Regenerated inventory adds those two witnesses; requirement text/status unchanged.
docs/src/content/docs/reference/cli/uninstall.md, reference/targets-matrix.md	Retained root receipts, whole-file authoring, and recovery after partial removal.
packages/apm-guide/.apm/skills/apm-usage/commands.md, CHANGELOG.md	Concise matching recovery guidance and user-visible fix entry.

The cached caller restores identity, not acquisition trust. The warm controls count actual codec construction through existing-local-state preservation.

Diagrams

Authored inputs pass through existing safety owners; successful reconstruction records shared ownership, while refusal retains prior receipts.

flowchart LR
    subgraph Sources["Authored inputs"]
        P["Surviving package instructions"]
        R["Root .apm instructions"]
    end
    subgraph Validation["Canonical safety owners"]
        C["remove_stale_deployed_files: hash and containment"]
        S["integrate_package_primitives: authorization and scan"]
    end
    subgraph Materialization["Generated output"]
        A["InstructionIntegrator: write generated aggregate"]
        L["DeploymentLedgerCodec: shared owners and final digest"]
        F["AggregateIntegrationError: retain prior receipts"]
    end
    P --> S
    R --> S
    C -->|"safe removal"| S
    C -->|"refused"| F
    S -->|"accepted sources"| A
    S -->|"source failure"| F
    A --> L
    classDef new stroke-dasharray: 5 5;
    class C,L,F new;

Loading

Trade-offs

• Whole-file generation, not editable regions. Author in .apm/instructions/; back up/move refused generated output before regeneration.
• Narrow sharing, not universal sharing. Only declared aggregates retain all owners; ordinary collision semantics are unchanged.
• Local ordered accumulation, not a new cache/framework. Private codec changes preserve move-to-end, active owner, locator, truthy-hash fallback, native passthrough and service rows.
• Operation counts, not latency claims. The warm guard measures codec record/owner construction, not every builder operation or a packaged-binary speedup.
• Candidate-specific evidence. Native execution uses this checkout's installed editable CLI in isolated homes, not a frozen binary. Historical broader-suite runs are not relabelled as final-head evidence.

Benefits

1. Every contributing owner records the final on-disk aggregate digest.
2. Uninstall removes even an unrecorded old owner while preserving exact surviving sections and identity.
3. Empty eligible content removes stale aggregate receipts; genuine root-only receipts keep the lockfile.
4. Warm codec construction keeps two final records at both 50 and 500 contributors, with owner work growing below 15x.

Validation

Local qualification head: 6ccd935af3bd8bd02cbd889db742f41f3c030896; included main: e38261c5db4d893d6ddebc3925742e4e3bd2ba74.
Runtime source is byte-identical to preserved 475550d0; subsequent commits regenerate conformance inventory and make assertion-preserving test lint repairs.
Frozen uv.lock versions and hashes were restored through the existing trusted enterprise feed; UV_NO_SYNC=1 prevents redundant dependency restoration, not test execution.

Exact-head local execution excerpts and mutation limits

Focused aggregate, model, cached-source, contraction, codec, recovery and architecture selectors via uv run --frozen --extra dev pytest -p no:cacheprovider -q -s:

110 passed in 13.81s

Restored regression selectors, including all 14 codec semantic controls, using the same runner:

43 passed in 12.43s

APM_BINARY_PATH="$PWD/.venv/bin/apm" uv run --frozen --extra dev pytest -p no:cacheprovider -q -s tests/integration/test_copilot_aggregate_uninstall.py:

13 passed in 107.44s (0:01:47)

All 13 native cases have zero failures/errors/skips. Eight isolated component mutations were killed, plus one separate installed-CLI legacy negative. The latter preserves original function globals/classes/references and fails the removed-owner-body assertion, not setup/import.

The two-method old-codec mutation through the existing warm route:

2 failed, 14 passed in 5.22s

Both failures are the owner-work bound. Restored no-root counts are 100 -> 1000; root counts are 102 -> 1002; both keep two record constructions.

BASE_REF=e38261c5db4d893d6ddebc3925742e4e3bd2ba74 bash tests/spec_conformance/mode_b_detector.sh:

[+] mode_b: spec-concurrent edit detected; orphan_check owns this PR

The concurrent edit consists of two genuine existing-v0.1 test witnesses, not a waiver or new normative clause. Orphan binding passes. Generated inventory was refreshed after the full conformance run (204 passed, 2 skipped at 475550; those existing skips are not native aggregate skips).

Canonical seven-part lint mirror plus architecture boundaries on the exact final head:

All checks passed!
1847 files already formatted
YAML I/O, relative_to and 2100-line guards passed
[+] auth-signal lint clean
All seven lint gates and architecture boundaries passed

The deterministic owner detector found four touched owners. Schema-valid v2 evidence maps each to executed exact-head functional tests; the semantic verifier re-detected the same base/head and reported terminal_evidence_required: true, not the blocked-status bypass.

Hosted Spec conformance passed at this SHA, including Mode B, the complete conformance suite and clean regenerated artifacts. The former spec-citation failure is resolved without a waiver. Human approval and any normative ratification are separate from this result.

Hosted CI, required gate, CodeQL, NOTICE and Docs workflows also succeeded. The existing CEO continuation accepts the implementation with no remaining in-scope finding; the draft/code-owner/last-push and human ship decision remain outstanding.

Scenario Evidence

Lifecycle below means tests/integration/test_copilot_aggregate_uninstall.py::test_generated_copilot_aggregate_lifecycle; every listed native row passed at the exact head above.

#	Scenario (user promise)	Principle(s)	Test(s) proving it	Type
1	All authored contributors are recorded	Governed by policy	Lifecycle [initial-owners] (original #2820 regression trap)	e2e
2	Removing one package keeps surviving content/state	DevX	Lifecycle [removed-section], [survivor-state]	e2e
3	Surviving Git source identity stays resolved	Vendor-neutral	Lifecycle [survivor-identity]; cached/fresh identity tests	e2e/integration
4	Headerless user content remains unowned	Secure by default	Lifecycle [collision]	e2e
5	Genuine root instructions survive package removal	DevX	Lifecycle [root]	e2e
6	Edited generated output is refused with usable recovery	Secure by default, DevX	Lifecycle [edited]	e2e
7	Unsafe source is refused without false success	Secure by default	Lifecycle [unsafe-survivor]	e2e
8	Empty, missing or ineligible survivors leave no stale aggregate	Governed by policy	Lifecycle [survivor-empty], [survivor-missing], [survivor-ineligible]	e2e
9	Removing root authoring and the last package leaves no empty lock	Governed by policy	Lifecycle [root-missing]	e2e
10	Upgrade from an old last-writer receipt removes unrecorded old content	Governed by policy	Lifecycle [legacy-last-writer] plus guard-deletion negative	e2e
11	Large shared receipts avoid repeated growing-owner construction	DevX	tests/unit/install/phases/test_lockfile_cross_package_reconcile.py::test_aggregate_attachment_work_is_linear, ::test_warm_aggregate_root_carry_work_is_linear	integration

How to test

• Create this checkout's frozen development environment and install its own CLI; do not borrow another checkout's editable environment.
• Run the aggregate conformance, codec, source identity and architecture modules listed above; expect passing controls and current hash envelopes.
• Run all 13 lifecycle cases with this checkout's APM_BINARY_PATH and a unique pytest --basetemp; expect no skips and unchanged unowned sentinels.
• Run the canonical seven-part lint mirror, architecture boundary check, and full conformance regeneration; expect no diagnostics or generated-artifact diff.
• Review the final SHA, existing advisory and hosted checks. A human must release draft status, obtain eligible nonauthor code-owner/last-push approval, decide the scoped ship/specification question, then enqueue merge under repository rules.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[Daniel Meppiel (danielmeppiel)]

APM expert advisory panel

Recommendation: ship_with_followups. Implementation accepted; human ship/approval gates remain.

Final continuation of the existing outer4 advisory, not a new panel or roster. This updates the prior needs_rework at the same comment. Candidate: 6ccd935af3bd8bd02cbd889db742f41f3c030896, including main e38261c5db4d893d6ddebc3925742e4e3bd2ba74, published to the original PR branch.

Prior findings are closed

Finding	Disposition at the final head
Missing legacy last-writer lifecycle proof	Added persisted survivor-only receipt case; all 13 native positives pass. Removing the production surviving-snapshot guard fails Unrecorded removed-owner body survived, not setup/import.
Warm aggregate reconstruction cost	Approved bounded codec fold implemented and qualified. Owner work at 50/500 contributors is 100/1000 without root and 102/1002 with root, with two record constructions at both sizes. Restoring both old methods produces two scaling failures while 14 semantic controls pass.
Uninstall reference drift	Retained CDO-approved patch documents root-owned receipts and failure after package/declaration removal, with scope-correct recovery.
Red specification gate	Two genuine existing req-lk-016/req-lk-020 witnesses plus regenerated inventory; hosted Spec conformance succeeds, including the complete suite and clean generated diff. No waiver, normative change or v0.2 activation.
Stale owner/evidence claims	Exact-head deterministic detection finds four owners; schema-valid v2 evidence and full semantic verification cover all four, with functional/static/mutation guardrails.

Exact-head local evidence: 110 focused tests, 43 restored controls, 13 installed-CLI native positives with zero errors/failures/skips; eight component mutations killed plus the separate installed legacy negative; canonical seven lint gates and architecture boundaries pass. These groups are not a claim of unique whole-suite coverage.

Hosted success observed on the same SHA: CI, required gate, Spec conformance, CodeQL, NOTICE and Docs. This closes the CEO continuation's pending-CI condition; no automated implementation or CI work remains.

The actual CEO continuation accepts the implementation and reports no remaining in-scope code finding. Retained Architecture, Performance and Supply Chain source follow-ups support the bounded codec change; no security finding is overridden. The actual test-coverage continuation confirms the lint helper/closure repairs preserve all 13 obligations and original assertions. No new abstraction, trust authority or registry schema was introduced.

Evidence and scope limits

Native qualification is this checkout's own installed editable CLI in isolated homes, not a packaged binary. Warm measurements count codec record/owner construction, not elapsed time or all install work. Original aggregate runtime lineage is preserved through 475550; later commits regenerate conformance inventory and repair test lint only. Historical 2a60 native12 and earlier broad runs are not relabelled as final-head evidence.

The docs classifier and growth invocation failures remain explicitly disclosed. The continuation's tool-less classifier also failed; bounded orchestrator source/diff/index fallback is not portrayed as a successful specialist review. Retained CDO acceptance remains authoritative for the adopted prose. Source identity restoration is not acquisition trust; #2919 owns that separate scope.

The original outer4/Copilot2/CI2 history is preserved. Later user authorization removed administrative stopping conditions; it did not grant approval, merge or specification-ratification authority.

Required human actions

The driver has marked this PR ready for review and requested Sergio Sisternes (@sergio-sisternes-epam) under explicit maintainer authorization. Readback confirms the same qualified head and green checks; no human approval has been supplied.

1. The final human reviewer accepts the scoped existing-v0.1 witness mapping and ship decision, without treating this as v0.2 ratification.
2. Sergio Sisternes (@sergio-sisternes-epam) reviews and approves the latest push as the eligible nonauthor code owner.
3. After required checks and approvals, the maintainer enqueues the PR through the required merge queue. Do not bypass gate or CodeQL policy.

This is an advisory recommendation, not a merge approval. The owner-gate verifier's full-path success certifies its evidence contract, not the outstanding human permissions.

[Daniel Meppiel (danielmeppiel)]

Docs sync advisory

Verdict: in_place, CDO agree; adopted * Pages affected: 1 * LLM calls/phases: 11/15 (conservative inherited accounting including failed invocations and bounded fallbacks)

The canonical uninstall reference now states that root-owned receipts can retain the lockfile after the last package is removed. It also distinguishes aggregate cleanup/rebuild failure after package and declaration removal, and links to scope-correct recovery in the existing Copilot targets reference. Existing target-cleanup, hook and LSP paragraphs remain unchanged. No TOC or normative specification change was made.

The original writer, deterministic Python verifier, editorial review and CDO accepted this two-location correction. It was committed at 2be4e4a2136d198d43438a5eee8a220bfe9b00ee and remains unchanged at published 6ccd935af3bd8bd02cbd889db742f41f3c030896.

Continuation scope: the later codec accumulation preserves the public contract; the conformance regeneration adds two existing-requirement witnesses without changing requirement text/status. The last three commits regenerate inventory and perform assertion-preserving test lint repairs. These deltas need no additional page or TOC change. This is a narrow continuation conclusion, not a claim that the whole PR had no documentation impact.

Execution limits: the original classifier reported unavailable tools and the original growth invocation could not resolve its artifact path. The continuation classifier also lacked file/command tools. Those invocations are failures, not successful specialist reviews; the orchestrator recorded bounded source/diff/index fallbacks, without another roster. The retained successful panel returns were schema-validated.

Current evidence: all 13 installed-editable-CLI lifecycle cases pass at 6ccd935af3bd8bd02cbd889db742f41f3c030896, with zero failures/errors/skips, including partial-removal recovery and persisted legacy last-writer receipts. The earlier 12-case 2a60d629 result is historical only. This advisory is neither human code-owner approval nor specification ratification.

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved critical cleanup and moderate aggregate ownership and error-reporting issues remain.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Lite
Findings: 1 · 1

New issues introduced by this change (2)

Severity	Finding
	src/​apm_cli/​integration/​instruction_integrator.py — This aggregate cleanup passes an empty hash map when the legacy receipt has no…
	CHANGELOG.md — The Unreleased entry ends with (#2820), which identifies the earlier foundation PR rather than…

What changed in this PR

This PR repairs shared Copilot instruction ownership and uninstall reconstruction.

Changes:

• Tracks aggregate contributors and final content hashes.
• Rebuilds validated survivor/root content with safer cleanup and recovery.
• Adds identity fixes, ledger updates, tests, architecture checks, documentation, and conformance updates.

Open findings:

• Critical — Hashless legacy receipts can bypass edited-file cleanup protection.
• Moderate — --only installs may drop claims for unselected dependencies.
• Moderate — Removing root instructions can preserve stale aggregate ownership.
• Moderate — Verbose aggregate errors can expose chained exception details.
• Nit — Changelog attribution should reference #2923, not #2820.

File	Summary
tests/​unit/​test_uninstall_engine_helpers.py	Tests recovery and truthful failure outcomes.
tests/​unit/​test_aggregate_installed_source.py	Tests installed-source identity safeguards.
tests/​unit/​scripts/​test_architecture_runner.py	Verifies architecture rule registration.
tests/​unit/​install/​test_install_copilot_user_instructions.py	Tests aggregate output locations.
tests/​unit/​install/​test_cached_source_identity.py	Tests cached and fresh Git identity.
tests/​unit/​install/​phases/​test_lockfile_cross_package_reconcile.py	Tests aggregate hashes, claims, and reconstruction work bounds.
tests/​unit/​core/​test_deployment_state.py	Tests codec ordering, hashing, and row semantics.
tests/​spec_conformance/​test_aggregate_lockfile_reqs.py	Adds aggregate lockfile conformance witnesses.
tests/​integration/​test_copilot_aggregate_uninstall.py	Covers installed-CLI aggregate lifecycle scenarios.
tests/​integration/​test_architecture_owner_rule_mutations.py	Registers aggregate mutation coverage.
tests/​integration/​test_architecture_copilot_aggregate.py	Tests aggregate architecture boundaries.
src/​apm_cli/​models/​apm_package.py	Restores shared Git source identity.
src/​apm_cli/​integration/​instruction_integrator.py	Handles aggregate discovery and cleanup.
src/​apm_cli/​install/​sources.py	Applies cached source identity restoration.
src/​apm_cli/​install/​services.py	Controls reconstruction scan hints.
src/​apm_cli/​install/​phases/​lockfile.py	Persists aggregate hashes, claims, and invalidation state.
src/​apm_cli/​install/​manifest_reconcile.py	Protects genuine root-authored files.
src/​apm_cli/​install/​helpers/​security_scan.py	Suppresses ineffective force hints during reconstruction.
src/​apm_cli/​core/​deployment_state.py	Tracks ordered shared aggregate ownership.
src/​apm_cli/​core/​deployment_ledger.py	Reconstructs receipts and projects compatibility state.
src/​apm_cli/​commands/​uninstall/​lockfile_state.py	Propagates confirmed removal state.
src/​apm_cli/​commands/​uninstall/​engine.py	Rebuilds aggregates and reports typed failures.
src/​apm_cli/​commands/​uninstall/​cli.py	Handles partial-removal recovery.
scripts/​architecture_linter/​checks/​install_deployment_analyzers.py	Checks removal forwarding.
scripts/​architecture_linter/​checks/​install_base_integrator_and_contraction.py	Checks aggregate routing.
packages/​apm-guide/​.apm/​skills/​apm-usage/​commands.md	Adds recovery guidance.
docs/​src/​content/​docs/​reference/​targets-matrix.md	Documents aggregate ownership behavior.
docs/​src/​content/​docs/​reference/​cli/​uninstall.md	Documents aggregate recovery.
CONFORMANCE.md	Regenerates conformance documentation.
CONFORMANCE.json	Regenerates the conformance inventory.
CHANGELOG.md	Records the user-visible fix.
.apm/​architecture/​owners/​install-deployment.json	Registers ownership boundaries.

Suppressed comments (3)

src/apm_cli/commands/uninstall/cli.py:144

• AggregateIntegrationError deliberately exposes only sanitized phase/path facts, but this unconditional traceback also emits its chained cause in verbose mode. Source validation or cleanup exceptions can put raw paths and source/error text in that traceback, defeating the sanitization promised by the new exception; keep the traceback detail for generic errors only.

        logger.verbose_detail(f"Integration cleanup failed: {type(sync_err).__name__}: {sync_err}")
        logger.verbose_detail(traceback.format_exc().rstrip())

src/apm_cli/install/phases/lockfile.py:535

• When an existing user-scope aggregate was also recorded under ., removing the root instructions leaves current_aggregates empty, but this union copies the old aggregate back into the new local compatibility view. post_deps_local is skipped for USER scope, so the stale root owner and hash survive and later uninstall treats package-only output as root-owned. Filter prior local state so aggregate paths are carried forward only when they are present in current_aggregates.

            local_files = sorted(
                set(self.ctx.existing_lockfile.local_deployed_files) | current_aggregates
            )

src/apm_cli/install/phases/lockfile.py:250

• This path also runs for --only installs, where package_keys contains the merged existing dependencies but current_claims contains only the selected packages. For a shared aggregate, reconcile_package_claims therefore drops the prior claims for unselected packages; because a selected package still produces a canonical batch, apply_to_lockfile then projects empty deployed_files for those survivors and loses their ownership. Carry forward shared claims for unselected dependencies (or preserve their canonical rows) before projecting the merged ledger.

        package_claims = DeploymentReconciler.reconcile_package_claims(
            package_keys=package_keys,
            current_claims=current_claims,
            prior_files=prior_files_by_package,
            prior_hashes=prior_hashes_by_package,
            shared_paths=shared_paths,

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.