Automatically rebuild stale indexes in the background after engine upgrades

[bompus]

Problem

After an upgrade changes extraction or resolution output, an existing graph can remain on the old behavior even when every source file is unchanged. End users should not need to read release notes, understand which fixes affect stored index content, or choose between sync and index to benefit from the new engine.

Could CodeGraph automatically rebuild a stale index in the background when that project is next opened/queried, while continuing to serve the last completed index until its replacement is ready?

Existing foundation

Checked upstream main at 374b3b4209d93d8be5bc77ca487ba027e76e0758 on 2026-09-08. The latest published release listed by GitHub is v1.6.0 (2026-08-26). This is a feature proposal based on source inspection, not a claim of a newly reproduced release regression.

• EXTRACTION_VERSION already distinguishes extracted-content changes from SQLite schema migrations. Its documented purpose is currently to drive a reindex recommendation/reminder.
• getIndexBuildInfo() / isIndexStale() already compare the stored extraction version with the running engine. Full indexing stamps that version; incremental sync deliberately does not advance it.
• feat(cli): add codegraph upgrade self-update + stale-index re-index hint #710 introduced this detection and the reminder. does it auto reindex when any file content changes ? #525 covers automatic updates when source files change; this proposal covers an engine change while source files remain unchanged. CG-33/CG-35: converge incremental sync with a full rebuild #1526 improves incremental/full-rebuild convergence after source edits, which is a related but different trigger.
• fix(mcp): reopen the database when it's replaced on disk instead of serving a deleted inode (#925) #940 provides related groundwork for reopening a replaced database in a running server on POSIX. Its implementation explicitly excludes Windows, so the handover needs platform-specific validation rather than assuming an open database can be replaced everywhere.

Proposed behavior

1. Extraction/resolver changes that require existing content to be regenerated advance the extraction version, including fixes that change stored output. CLI/UI-only releases do not cause a rebuild.
2. On the next normal project open/use, detect an older or missing extraction stamp and start a background rebuild into a separate replacement database. Continue answering from the last completed, readable index with an explicit stale/updating status. Users should not need a manual command or confirmation, and the expensive rebuild should not block ordinary queries.
3. Concurrent clients share one rebuild. After it succeeds, catch up any intervening source changes and coordinate the switch to the replacement database and its fresh query caches. Publish the new stamp only after success. On failure, retain the old usable index and report the failure; never expose a partially built replacement as current. A short coordinated handover is acceptable, particularly where Windows file locking requires it. If no usable old index exists or its schema cannot be read safely, the nonblocking fallback does not apply.
4. Apply the same compatibility principle to the separate session-transcript index, using its own reader/index version. Idle projects and temporary indexes are refreshed if and when they are used again; an upgrade does not need to discover and rebuild every database on the machine.

This would eliminate the need for downstream upgrade scripts to maintain a global database inventory and force full reindexes after every deployment.

Suggested regression proof

• Build an index with an older extraction stamp, leave source contents/hashes unchanged, and open it with the newer engine. Hold the background rebuild at a test barrier and prove queries still answer from the old index with an updating status. Release it and prove queries reflect the new extraction without reconnecting or running a manual command.
• Open the same stale project from two clients and assert that only one rebuild runs.
• Inject a rebuild failure and verify that the previous usable index survives and the compatibility stamp is not advanced; a later attempt can recover.
• Change a source file during the rebuild and verify the replacement includes the change at handover. Validate connection/cache handover on Windows as well as POSIX.
• Reopen a current index and verify that no full rebuild runs. Cover the corresponding unchanged-transcript case for the session index.

[inth3shadows]

This reproduces as a live bug today, not only as a missing feature — and the cheap half of it is one call. Posting here rather than opening a duplicate, since your reading of the foundation is already exactly right.

sync says "Already up to date" about an index the same tool says needs a re-index

Reproduced just now, twice: once on published 1.6.0 (the bundled runtime), once on a build of main @ 3ed73bc under Node 22. Same result both times.

Set an index's stamp behind the running engine (any real upgrade does this; here it is done directly so the repro needs one binary, not two):

node -e 'const {DatabaseSync}=require("node:sqlite");
const db=new DatabaseSync(".codegraph/codegraph.db");
db.prepare("update project_metadata set value=? where key=?").run("25","indexed_with_extraction_version");'

status is honest about it:

"index": { "builtWithExtractionVersion": 25, "currentExtractionVersion": 26,
           "reindexRecommended": true, "state": "complete" }

sync is not:

●  Already up to date

And with a real edit in the tree, sync does its incremental work, reports success, and still leaves the bulk on extractor 25:

●  Added: 1 — 2 nodes in 118ms
"index": { "builtWithExtractionVersion": 25, "currentExtractionVersion": 26, "reindexRecommended": true }

So the two signals a user reaches for disagree, and the one that runs the work is the one that is wrong. Nothing in the sync path ever prints the recommendation status already computed.

What it cost me

I hit this before finding your issue. A fixed extractor became the default on PATH; the index was synced after the fix and still carried every fabricated call edge that fix removes, because none of the files carrying them had been edited in between — the index was newer than the fix and contained none of its output. A downstream report went out saying the bug was "still present after the fix." It wasn't.

The trap worth stating explicitly: the release version cannot be the signal. A synced index and a freshly built one of the same commit both report indexed_with_version: 1.6.0 while their extraction versions are 25 and 26. A fork, or a patch release, can change extraction without moving the release string. builtWithExtractionVersion is the only field that discriminates — an argument for the design you propose, not against it.

Enforcement is the gap, not knowledge

Re-verified against main @ 3ed73bc. You cite 374b3b4; the three functions involved are byte-identical between the two revisions (getChangedFiles, isIndexStale, collectGitStatus — hashed at both revs), so nothing here has drifted since you filed.

isIndexStale() (src/index.ts:1238) has exactly two callers in the tree:

src/bin/codegraph.ts:1041       the `status` command
src/ui-server/api/stats.ts:113  the viewer

Neither is sync. And the invariant sync violates is already written down, at src/index.ts:686:

// Stamp the index with the engine that built it, so `codegraph status`
// and `codegraph upgrade` can recommend a re-index when the running
// engine produces richer extraction than the one on disk. Only on a
// real full index — a sync touches a subset, so it must NOT advance the
// extraction stamp (the bulk would still be stale). See extraction-version.ts.

The codebase already knows a sync leaves the bulk stale, deliberately withholds the stamp for that reason, computes isIndexStale() from it, and surfaces reindexRecommended — then sync runs without ever asking. extraction-version.ts even names this precise case as a bump reason: "a resolver fix that materially changes which edges exist."

Suggested staging

The background rebuild you describe is the right end state, but it needs a serving-while-rebuilding story and a concurrency story. There is a subset that closes the silent-wrong-answer window immediately and is independently useful:

Have sync consult isIndexStale() and either escalate to a full re-extract or refuse with a pointer to index. The signal, the threshold and the stated intent are all already in the tree — wiring, not new machinery — and it makes the failure loud instead of silent while the background design is worked out.

Happy to send that as a PR if you want it split that way.

Environment for both runs: Linux x86_64 (WSL2); published 1.6.0 via its bundled runtime, and main @ 3ed73bc built from source on Node 22.23.2 (the source build's Node gate rejects Node 26).

---

Filed separately as #1829, because it is a different mechanism in a different file: status.pendingChanges reports zero for a change that is committed but unindexed — so the other signal a user would reach for to check this one is also unreliable.

[bompus]

Thanks for the reproduction and for splitting the silent-sync case from the background rebuild.

The cheap half you described is already up as #1842: sync consults the existing extraction-staleness signal and refuses with a codegraph index recovery path instead of printing “Already up to date.” It does not escalate sync into a full re-extract, and it does not implement the background rebuild this issue asks for.

The committed-but-unindexed status signal you filed as #1829 is #1843.

[inth3shadows]

Appreciated — and thank you for pointing me at both PRs rather than leaving me to find them. #1842 is the right scope for this half: refusing is more conservative than the escalate-to-full-reindex shape I had, and it keeps the decision with the person running the command.

One question on it rather than an objection: --quiet exits 1 as well. That is the git-hook path (codegraph sync -q), so on a repo whose hook is wired to fail the commit on a non-zero exit, the first commit after an upgrade would be blocked by a signal the user cannot see — quiet suppresses the recovery instruction that explains it. Exiting 0 in quiet mode, or writing the one line to stderr before exiting 1, would keep the signal without that edge. Your call; you know the hook wiring better than I do.

I have left the background-rebuild question here alone — that is what this issue is actually about, and neither PR touches it.

[bompus]

Good catch, and taken: #1842 now writes the reason as one line on stderr under --quiet and still exits 1, so a hook that fails on a non-zero exit shows the user why instead of blocking silently (cb8dbdc3, quiet-mode tests updated to assert exactly one stderr line and empty stdout).

For what it is worth, the hooks codegraph installs itself never hit this: they run codegraph sync in the background with output discarded, so they cannot fail a commit. The edge is a user-authored hook, which is exactly the one that needs the line.

Agreed the background rebuild is a separate question; leaving that to this issue.