Bundle a commit-pinned instance snapshot in the toolbox instead of downloading openMINDS_instances at runtime #128

Problem

The pipeline branch clones openMINDS_instances at whatever commit main is at and bakes only the instance names into CONTROLLED_INSTANCES on each generated controlled term class. The instance data is fetched separately at runtime: InstanceLibrary downloads the whole openMINDS_instances repository zip into userpath on first use, scans roughly 15k files per version into a table, and getControlledInstance falls back to raw GitHub URLs per file. The freshness check calls the GitHub commit API when the library is created.

This has several consequences:

• The name list in the class and the data on disk come from different commits. A term added upstream is rejected by the class until the next rebuild. A term removed upstream passes validation and then fails on the raw file fallback.
• First use requires network access. Offline users, CI runners, and proxied networks get warnings or failures. The API check is rate limited when unauthenticated.
• Users download every instance version, when the terminologies, licenses, and content types they mostly need are about 11 MB.
• A metadata document written today can reference terms that another user's downloaded library does not have. Results are not reproducible across installations.

Upstream version convention

Schemas and instances follow different rules, and this decides what "reproducible" can mean.

• Schema versions are pinned. versions.json on the openMINDS pipeline branch maps every numbered version (v1.0 to v5.0) to a specific commit of each module repository. Only latest floats. Commits that touch old version folders are the central build re-rendering the same pinned schemas, not semantic changes.
• Instance version folders are not points in time. A folder means "this term rendered against that schema version and namespace". The autopopulation workflow in openMINDS_instances runs on every push to main and copies any added or modified instance into every other version folder that has the matching top-level folder. v3.0, v4.0, v5.0 and latest are one library rendered four ways.
• v1.0 to v3.0 use the openminds.ebrains.eu instance namespace, v4.0 onwards use openminds.om-i.org. Every id therefore differs between v3.0 and later versions.
• There are no v1.0 or v2.0 instance folders upstream.
• Neither openMINDS nor openMINDS_instances has tags or releases.

Consequence: a reproducible instance library can only be a snapshot pinned to an openMINDS_instances commit SHA. The SHA is the sole identity of the snapshot.

Proposal

Ship the instance library inside the toolbox as a per-model-version snapshot pinned to a specific openMINDS_instances commit. Offline and reproducible use is the default. Updating from upstream becomes an explicit opt-in.

Pipeline branch

• Record the openMINDS_instances commit SHA used by the build and emit it as a generated constant per model version.
• Emit aggregated instance data per model version and per type, for example instances/v4.0/controlledTerms/Species.json with all instances of that type. Split parcellation entity types by atlas subgroup as upstream does if single files get too large.
• Store plain JSON in git, pretty-printed with stable key order and instances sorted by id, so a rebuild produces a readable diff. This is roughly 50 files per version instead of 15k. Gzipped or MAT storage in git would be opaque to diff.
• Generate CONTROLLED_INSTANCES from the same clone so names and data cannot drift.
• Emit a functionSignatures.json per version for the controlled terms package with choices= on the constructor's first argument, so instance names tab complete in constructor calls.
• Extend apply-new-build.sh to copy the snapshot into main alongside the generated types.
• v1.0 and v2.0 get no bundled instances, mirroring upstream. They are not synthesised from v3.0.

Main branch

• Resolver order: user overlay in userpath, only if the user opted in, then the bundled snapshot, then an error naming the term and type.
• Load a type's JSON lazily on first use and cache it in memory. No directory scan at session start. No network call unless the user asks for an update.
• Replace the download-on-construct behaviour of InstanceLibrary and the raw GitHub fallback in getControlledInstance with the bundled resolver. Keep downloadRepository only behind the opt-in update function.
• Add an opt-in update function that downloads upstream main or a given SHA into the overlay folder and records the SHA.
• listControlledInstances reads from the bundle plus the overlay.
• Preserve instance IRIs as ids when a type has no bundled library. As read from the source, deserializeFromName parses a full instance IRI into a name, fails to match an empty CONTROLLED_INSTANCES, warns, and never sets the id. Old Knowledge Graph exports for v1.0 and v2.0 carry full IRIs, so this is the common path for those versions. Warn only on bare names.

Sizes

Measured on a clone with three instance versions. Upstream now also has v5.0, so the bundle will hold four.

Decisions

• Installed footprint: accepted. Plain JSON in a resources folder off the MATLAB path, roughly 70 to 85 MB on disk for four versions (21.7 MB per version pretty-printed, 17.9 MB minified), in exchange for no first-use extraction and no cache. About 50 aggregated files per version, not the 15k JSON-LD files. To confirm on the first build: the .mltbx download stays small because the archive is zip-compressed, expected around 1 MB per version.
• v1.0 and v2.0: no bundled instances. The resolver preserves IRIs for those versions instead.
• latest: its own snapshot, not an alias of the newest numbered version. The diff is small and a separate copy avoids special cases.
• Deduplicating the snapshot by storing content once and rendering ids per namespace is possible, since the versions are one library rendered per schema version, but it couples the resolver to the autopopulation rendering rules. Not in the first pass.

Open questions

• Verify that choices= from a static method expression in functionSignatures.json resolves for package class constructors.
• Tab completion on property assignment (subject.species = "hom<Tab>) worked when controlled terms were enumeration classes. Enumerations are not viable: closed set, 10,712 of 15,041 instance names are not valid identifiers, 436 exceed 63 characters, and all members are constructed eagerly. Investigate whether any other mechanism gives completion on assignment to string-typed properties.
• Scheduled rebuild to bump the pinned SHA, or a manual bump per release?

Tests

• Pipeline: the aggregated JSON for a type contains exactly the files in the clone, CONTROLLED_INSTANCES equals the ids in the aggregated file, the SHA constant matches the clone HEAD, and v1.0 and v2.0 produce no instance files.
• Main: a bundled term resolves with no network, an unknown term errors cleanly, the overlay wins over the bundle when present, a type is loaded once, a JSON-LD round trip for a bundled instance matches the previous download-based result, and a full instance IRI on a v1.0 or v2.0 controlled term keeps its id without a warning.

🤖 Generated with Claude Code