feat(laddr-import): one-shot migration script from laddr mysqldump#24
Merged
Conversation
Custom line-streaming parser for the narrow CREATE TABLE / INSERT INTO grammar laddr dumps use; avoids loading the whole SQL file into memory. Includes a 10-row synthetic fixture covering all migrated entities (people, projects, memberships, updates, buzz, tags, tag-assignments) plus one dropped table (member_checkins) to verify it's skipped without error. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…alization
Per-table translator functions that map a raw laddr row into a v1
record (Zod-shape, not yet validated). Mints UUIDv7s, threads them
through cross-table id maps for FK resolution, and emits warnings for
soft-fixed conditions (invalid slugs, unknown stages, dropped context
classes).
Normalization rules pulled from data-model.md:
- slugs slugified + dedupe-with-suffix when source doesn't match the
new regex (default per plan; document in Notes)
- stage TitleCase → lowercase, unknowns fall back to "commenting"
- tag handles like "topic.transit" split into namespace + slug
- ContextClass "...Project" / "...Person" → taggableType
- Email + password hash routed to PrivateProfile +
LegacyPasswordCredential separately (never in the public side)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The orchestrator (importer.ts) drives FK-respecting passes — tags → people → projects → memberships → updates → buzz → tag-assignments — emitting one gitsheets commit per entity type (7 total). Each commit uses the pseudonymous "Code for Philly API" identity per specs/behaviors/storage.md, with Action/Source-Dump/Run-At trailers per specs/behaviors/legacy-id-mapping.md. Idempotence: before any pass, walks the data-repo's git tree to collect existing legacyId → (id, slug) maps for the five sheets with legacyId, plus composite-path sets for memberships + tag-assignments (which have no legacyId field). Re-running against the same dump on an already-imported repo produces zero new files and zero commits. Private store: PrivateProfile records flow through PrivateStoreTx; LegacyPasswordCredential records (one-shot migration data, never written by the runtime API) are seeded via direct flush onto BasePrivateStore's internal legacyPasswords map — keeping the runtime interface narrow. CLI (`npm run -w apps/api script:import-laddr`) accepts --sql, --data-repo, --private-store, --dry-run, --verbose, --limit. Tests cover: dry-run report shape, full-import end-to-end with PII audit (no email patterns or bcrypt hashes anywhere in the public tree), tag namespace splitting, stage normalization, composite-path shapes, per-project ProjectUpdate.number sequencing, second-run no-op, and --limit truncation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced May 16, 2026
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
npm run -w apps/api script:import-laddr) reads a laddr mysqldump and writes records to both the public gitsheets repo and the private filesystem storelegacyId(and on composite paths for memberships + tag-assignments which have nolegacyIdfield) — re-running against the same dump produces zero new commitsPerson.slackSamlNameIdfrom laddrUsername, splits tag handles liketopic.transit, normalizes stage values, slugifies invalid slugs with dedupe suffixesCode for Philly API <api@users.noreply.codeforphilly.org>withAction: import.laddr,Source-Dump,Run-Attrailers perspecs/behaviors/legacy-id-mapping.mdapps/api/scripts/fixtures/laddr-fixture.sql+ a Vitest suite cover the validation criteria inplans/laddr-import.mdImplements plan: laddr-import.
Test plan
--limit=1truncates per-table to 1 imported--dry-runproduces JSON report with no writesPerson.slackSamlNameIdpopulated correctly for every Person; matches their slugPerson.email+LegacyPasswordCredential.passwordHashland in the private store, not the public repo (regex audit: zero hits)topic.transitsplit correctly intonamespace='topic', slug='transit'tag_items.ContextClass→taggableTypemapping correctmember_checkins) skipped without errornpm run lint,type-check,test,buildall green