Skip to content

memory: reduce repeated topic curation with frozen preparation - #3282

Merged
Hillary Mutisya (hillary-mutisya) merged 2 commits into
mainfrom
dev/hillarym/memory-topic-curation
Nov 11, 2026
Merged

Hillary Mutisya (hillary-mutisya) merged 2 commits into
mainfrom
dev/hillarym/memory-topic-curation

Conversation

@hillary-mutisya

Copy link
Copy Markdown
Collaborator

Summary

  • Add opt-in compact wire IDs, prior-concept delta responses, and two-worker frozen topic preparation followed by serialized, validated reconciliation.
  • Preserve exact evidence/membership checks, ordered window receipts, checkpoint recovery, cancellation/draining, and native publication/freshness guards.
  • Harden retained-input topic replay with maintained history reads, pinned model metadata, absolute deadlines, validated partial outcomes, and the corrected diagnostic organization handoff.

Motivation

Serial topic curation repeatedly sends and rewrites already-admitted concepts. Speculative preparation can also become invalid when earlier decisions change its prior-concept frontier. Prepare independent raw packets against frozen inputs instead, then reconcile their proposals against current admitted concepts before installing a complete window.

Behavior and configuration

All three new treatments are off by default:

Environment variable Behavior
TYPEAGENT_MEMORY_TOPIC_PRIOR_DELTAS Omit unchanged prior concepts from responses; the host retains their exact objects, including audited summaries.
TYPEAGENT_MEMORY_TOPIC_COMPACT_IDS Remap packet-local wire IDs and schema enums reversibly; evidence, prose, membership, and durable IDs stay unchanged.
TYPEAGENT_MEMORY_TOPIC_FROZEN_PREPARATION Prepare two raw-only packets concurrently, then serialize reconciliation and complete-window installation. Requires explicit concurrency 2.

These are also programmatic topicCuration settings and replay CLI switches. Effective treatment settings participate in policy/replay identity. Watermark continuation accepts explicit frozen two-worker execution, not ordinary speculative concurrency or the native Decision engine. Source correction/removal, forgetting, eligibility, and current-publication checks remain fail-closed.

Automatic maintenance may recapture a changed topic head once during admission; repeated churn still fails stale. Explicit callers retain strict expected-head semantics.

The replay organization handoff drops captured native packet-receipt assertions from its cloned diagnostic input. It does not disable production receipt validation or certify native watermark installation.

Reviewer guide

  1. Curation orchestration, frozen preparation, and wire remapping.
  2. Host integration in fileMemoryService.ts, viewHostOptions.ts, and topicGeneration.ts.
  3. Replay driver, replay documentation, and regression tests.

Validation

Windows, Node 26.13.0, standalone pnpm 11.9.0. Final reviewed head: 1748afb8767aa3ed86780da91f34f621dee368ec. The branch starts from refreshed main; unrelated model-migration work is already merged and is not included here.

Check Result
Full pnpm run build from ts Pass
Nine focused memory-service Jest suites 169/169 pass
Three topic replay/options Node test suites 43/43 pass; no skips/cancellations
Pinned changed-file formatting Pass
Committed-HEAD lint, complexity, circular and test-debt gates All pass: zero added lint/complexity violations; cycles 210 to 210; no focused/newly skipped tests

The service suites are topicWorkReuse, topicBuilds, topicSnapshot, topicCuration, topicGeneration, topicLookup, topicFrozenPreparation, topicCurationScheduler, and viewHostOptions, run with pnpm run jest-esm --runInBand --runTestsByPath against their corresponding dist\test\*.spec.js paths.

Replay tests and PR gates, from ts:

node --test .\tools\memory-maintenance-qualification\topic-curation-replay.test.mjs .\tools\memory-maintenance-qualification\topic-decision-replay.test.mjs .\tools\memory-maintenance-qualification\topic-curation-options.test.mjs
pnpm run prettier:changed
pnpm run code-lint -- --ratchet --base origin/main
pnpm run code-complexity -- --ratchet --base origin/main --cyclomatic 25 --cognitive 30 --new-file-cyclomatic 25 --new-file-cognitive 30
pnpm run code-circular -- --ratchet --base origin/main --exceptions-file tools\scripts\code\circular-baseline-exception.json
pnpm run code-debt -- --gate --base origin/main

Tests cover exact receipt replay and cold resume, append/correction/forget behavior, audited-prior preservation, unknown compact IDs and bounded repair, sibling drainage, deadline enforcement before initialization/repairs/organization, metadata redaction/binding drift, and diagnostic organization without weakening native receipt checks. The captured-watermark regression is awaited so its lifetime cannot escape its parent test.

Live diagnostic results

One sequential, fixed-order control/candidate pair reused 2,853 raw topic nodes from 45 previously ingested chats; no new ingestion. Both arms used the same configured GPT-6 Luna model, unchanged provider-default reasoning, and independently matched 300-second HTTP request budgets under one unchanged absolute deadline. Earlier 60-second timeout failures were preserved, not mixed into this pair.

Metric Control Candidate
Curation including setup 76m32s 39m12s
Curation model-method calls 134 100
Prompt tokens 1,553,127 1,564,108
Completion tokens 371,313 257,085
Final concepts 269 201
Admitted / explicitly deferred raw nodes 2,559 / 294 2,554 / 299

Candidate completed curation and actual organization in 40m03s. Curation latency was 48.8% lower, with 25.4% fewer calls and 30.8% fewer completion tokens; prompt tokens increased 0.7%.

Control's initial organization and bounded repair both returned hierarchy cycles, so the original control arm failed. A separate 1m03s, two-call live organization retry succeeded from its exact completed checkpoint, with zero repeated curation calls and no overlap with candidate inference. The original failed attempt remains a failure, not an uninterrupted successful pipeline.

Both final snapshots passed native snapshot/curation validation and accounted for all 2,853 inputs, with zero missing/duplicate/unknown assignments. Measurements were collected on the development snapshot before publication onto refreshed main; the final PR tree is rebuilt and offline-tested separately. No additional inference was authorized or started for PR preparation.

Risks, limitations and rollout

  • This is a combined-treatment, single-pair observation, not individual-factor causal attribution, a native required-readiness result, or semantic/answer-quality qualification.
  • Candidate produced 68 fewer concepts and deferred five additional raw nodes. Structural coverage cannot distinguish useful consolidation from over-merging; semantic review remains a release gate.
  • HTTP transport settings are not covered by the model-binding hash and must be matched independently. Production timeout defaults are unchanged.
  • No experimental defaults are promoted. Roll back by unsetting the three treatment variables; changed policy/progress identities fail closed rather than mixing incompatible checkpoints.
  • Corpus payloads, private archives, credentials, and machine-local links are intentionally excluded.

Add opt-in prior deltas, compact wire IDs and two-worker frozen preparation with serialized, validated reconciliation and ordered watermark receipts.

Harden retained-input replay with model bindings, absolute deadlines, validated partial progress and the corrected diagnostic organization handoff. Preserve default behavior and native receipt validation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep deadline cancellation artifacts and persistence failures unchanged while bringing the replay lifecycle within the committed-branch complexity budget.

Validation: full workspace build and all 43 replay/options tests pass; the 169 service tests passed on the unchanged service implementation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@hillary-mutisya
Hillary Mutisya (hillary-mutisya) merged commit 3317b2a into main Nov 11, 2026
109 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant