Skip to content

Strengthen verification evidence and native context checks - #64

Merged
kentwelcome merged 2 commits into
mainfrom
refine/verification-evidence-gates
Oct 29, 2026
Merged

kentwelcome merged 2 commits into
mainfrom
refine/verification-evidence-gates

Conversation

@kentwelcome

@kentwelcome kentwelcome commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

What changes

Follow-up to #63, preserving the ownership model: the handoff owns the journey, Behavior Diff owns comparison, and the existing runner owns execution.

  • Complete assessment evidence: frozen scenarios can nominate exact decision-artifact paths. The existing bounded collector captures their saved contents with provenance, rejects unsafe/private inputs, and makes missing or truncated evidence explicit. A completion claim cannot replace a required artifact or a recorded authoritative source read.
  • Enforce a required filesystem boundary: the existing Codex adapter can select a native permission profile on supported macOS hosts. Actual configuration and access canaries run before model workers start. Unsupported/conflicting routes fail closed; no automatic Full Access, credential copying, custom launcher, or permission expansion. Trials remain ineligible until observation finishes; violations and unavailable markers exclude their raw evidence and eligible counts. Ordinary comparisons retain the existing route.
  • Retain available preparer provenance: private native records and reviewed input are bound to the freeze and checked for drift. The supported Codex export establishes represented supplied input and completion. Resolved per-turn model and complete historical-context evidence remain explicitly unavailable, not invented compliant receipts.

Updates canonical preparation guidance, architecture/flow documentation, deterministic regressions and existing CI wiring. No new orchestration database or model executor.

Verification

  • Full repository deterministic suite passed, including hook/runner, progress, decisions, report schema/rendering, source/CI provenance, shared-data, isolation, Jev, release and human-evaluation contracts. Existing suite-reported skips retained.
  • Focused suites: native boundary 12 passed; native preparation 5 passed; recorded evidence 32 passed; shared data 31 tests, 1 skipped.
  • CI-mode native-boundary suite passed with the local runner smoke skipped; that smoke separately passed outside CI. The initial CI run was canceled to correct this policy finding before continuing.
  • Entire hook/runner suite also passed with macOS Bash 3.2 forced through PATH.
  • Real native Codex 0.165 non-model smoke: configuration validated; workspace write/read allowed; nominated and outside reads denied.
  • Actual run-trial.sh smoke with real native preflight and a synthetic malformed event stream: nonzero exit, boundary still unavailable, zero assessment records. No model invoked.
  • Saved-file collection and native provenance API smokes passed; changed preparer input rejected.
  • Whole-repository Ruff 0.16.5 and shfmt 3.14.0 checks, Python syntax and patch whitespace checks passed.
  • Two independent read-only AI reviewers approved the complete final tree. All three review rounds are recorded in PR comments.

Limits

This is not a fresh paid end-to-end model journey. Native filesystem enforcement does not prove isolation from injected instructions, connectors, browser/cloud state or every other context channel. Lack of an observed attempt is not proof of clean context. Captured artifacts do not prove authorship or real external actions. Native exports do not authenticate themselves or prove the resolved execution model.

No live/paid model run, release, or merge performed. This PR does not touch the instruction-file paths that trigger the paid dogfood workflow.

Reviewed commit: 5808dbe9bd538ce4e806f7e9c237a5bb5e53bc04
Reviewed tree: d16cd5ecd5afc1fb5b9afea592e666dac9577708
Base: 7839bd02bdb8ced4733c2f9884547ecaac2e5637
DCO sign-off included.

Signed-off-by: Kent Huang <kent@infuseai.io>
@kentwelcome

Copy link
Copy Markdown
Contributor Author

Independent AI review — round 1

Base and working-tree HEAD: 7839bd02bdb8ced4733c2f9884547ecaac2e5637. Reviewed snapshot: unpublished pre-fix local changes, before the Bash 3.2 and pending-observation corrections. No separate review commit existed.

  • Standards reviewer: REVISE. P1: empty optional argument-array expansion fails under Bash 3.2, breaking ordinary comparisons. P1: preparation prematurely marked trials eligible; normalization failure could leave partial evidence assessable before observation completed.
  • Specification reviewer: APPROVE. Traced G1 artifact evidence, G2 boundary integration and G3 private provenance through their consumers. No material finding, decision or optional suggestion.
  • Disposition: both P1 findings reproduced and fixed in 71e25af50511bd4713851bb8d1d9a22cf407263e. Guarded array expansion preserves the ordinary route. Prepared trials now remain unavailable/unknown with observation_pending until observation succeeds. A failing-before/passing-after regression covers exclusion.
  • Scope audit: additions stay within existing evidence, native execution and freeze responsibilities; no alternate executor, orchestration database, automatic Full Access, credential copying or paid CI mechanism. No unauthorized decisions identified.

Parent verification: focused evidence/provenance/boundary tests and real non-model native canaries. Subsequent complete verification and re-review are recorded separately. These are independent AI-subagent findings, not human or GitHub approval submissions.

@kentwelcome

Copy link
Copy Markdown
Contributor Author

Independent AI review — round 2

Base: 7839bd02bdb8ced4733c2f9884547ecaac2e5637.
Reviewed complete staged tree: 128cc12e5ae8946fe424c27d456e6eced42f79b0, now published as 71e25af50511bd4713851bb8d1d9a22cf407263e.

Standards: APPROVE. Specification: APPROVE. Both independently reviewed the entire 21-file scope, not just the fixes. No remaining material findings, decisions or optional suggestions. Both confirmed the two round-1 P1 corrections, configuration validation ordering, private provenance handling, bounded artifact evidence and shared eligibility filtering.

Scope audit: ownership remains handoff/journey, Behavior Diff/comparison, existing runner/execution. No new orchestration database, alternative model executor, automatic custom fallback, credential copying or Full Access. Added deterministic tests and canonical documentation remain in scope.

Parent verification passed: full repository deterministic suite; whole-repository Ruff and shfmt; Python syntax; patch whitespace; complete hooks/runner suite under Bash 3.2; actual native config/access canaries; saved-file and native-record API smokes. Actual runner + synthetic malformed native stream exited nonzero, retained unavailable boundary status and emitted zero assessment records. No model call.

Limits: no fresh paid model journey; filesystem enforcement does not prove all-context isolation; unavailable per-turn model/history provenance remains unavailable. Review approval does not authorize merge. These are independent AI reviews, not human approvals.

@kentwelcome

Copy link
Copy Markdown
Contributor Author

Post-review finding — CI policy correction

Published head: 71e25af50511bd4713851bb8d1d9a22cf407263e; base: 7839bd02bdb8ced4733c2f9884547ecaac2e5637.

A subsequent policy check found that the new native-boundary test suite included a local runner smoke. The CI step would invoke behavior-diff.sh through that test, contrary to AGENTS.md invariant 5, even though this smoke rejects the unsupported route before any model call.

Disposition: confirmed in-scope defect, correction pending verification and independent re-review. The current CI run received a cancellation request. The local correction skips this runner smoke whenever CI is set; deterministic module tests remain enabled. No model invocation occurred in the exercised smoke. Earlier approval does not close this new finding. The correction will be verified and reviewed before publication; no merge is authorized.

Signed-off-by: Kent Huang <kent@infuseai.io>
@kentwelcome

Copy link
Copy Markdown
Contributor Author

Independent AI review — round 3 and published correction

Base: 7839bd02bdb8ced4733c2f9884547ecaac2e5637.
Reviewed complete tree: d16cd5ecd5afc1fb5b9afea592e666dac9577708.
Published and remotely confirmed head: 5808dbe9bd538ce4e806f7e9c237a5bb5e53bc04.

Standards: APPROVE. Specification: APPROVE. Both independent AI reviewers re-read the complete staged scope. No material findings, decisions or optional suggestions remain.

The post-review CI defect is fixed and verified: the workflow sets CI=1 and the runner-smoke decorator skips before its body executes. Module-level boundary tests remain enabled. The runner smoke remains local-only. The superseded CI run is confirmed canceled; the new head's CI result will be recorded separately.

Parent checks for this correction: CI=1 native-boundary suite ran 12 tests with the one local-only smoke skipped; the targeted actual-runner smoke passed outside CI; Ruff passed. Prior full-suite and runtime evidence applies to unchanged production code. An additional real native non-model check also denied access to a file in an enclosing Git repository outside a nested trial workspace.

Scope audit: bounded artifact evidence, optional native filesystem checks, private preparer provenance and deterministic regressions stay within existing responsibilities. No new executor/database, automatic Full Access, credential copying, paid model run, or automatic fallback. Prior Bash 3.2 and premature-marker fixes remain intact. No permanent mechanism was removed.

Limits remain explicit: this is not a new paid end-to-end journey; filesystem enforcement is not full-context isolation; resolved per-turn model and historical context provenance remain unavailable where the native host cannot establish them. Both commits carry DCO sign-offs. No merge performed or authorized.

@kentwelcome

Copy link
Copy Markdown
Contributor Author

Final verification

Published head 5808dbe9bd538ce4e806f7e9c237a5bb5e53bc04 matches the independently approved tree.

CI run 40817945121 completed successfully: Format — success; Unit — success. The earlier run remains canceled and is not counted as successful verification.

All review findings are resolved; the final complete-scope standards and specification reviews both approve. No paid model journey, release or merge performed.

@kentwelcome
kentwelcome merged commit ce9eab4 into main Oct 29, 2026
2 checks passed
@kentwelcome
kentwelcome deleted the refine/verification-evidence-gates branch October 29, 2026 07:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant