fix(genesis-writer): emit email_owner_user_id so encrypted emails index - #490
Merged
Conversation
All 4,218 encrypted emails and their 9,282 access grants are lost in a
migration today. encryptedEmailHandler takes the owner from the metadata
key email_owner_user_id rather than from the transaction's user_id, and
encryptedEmailMetadata had exactly three fields -- encrypted_email,
access_grants, created_at -- so every EncryptedEmail/AddEmail transaction
was rejected with "missing required field: email_owner_user_id". They do
reach em_rejected, but as thousands of anonymous rejections among
millions of transactions, which is why this went unnoticed.
access_grants gets the same treatment for the same reason: the handler
rejects the transaction when the key is absent and cannot tell "absent"
from "no grants", so `omitempty` is dropped and a nil slice is serialized
as []. Every email in the production snapshot has at least one grant, so
this is latent rather than active, but it is the identical failure mode.
The step now joins users for the owner's wallet and signs as the owner,
matching every other entity writer. The email handlers do not call
ValidateSigner, so this is not what unblocks them -- it is what keeps the
migration's own convention intact, so the rows are attributable and stay
valid if that check is ever added. The join also means an email whose
owner is not a current user with a wallet is skipped rather than emitted
unattributable; on a production clone all 4,218 qualify, so nothing is
dropped.
The orphan-grant step had a second, latent mismatch: it emitted
EmailAccess/Create with the grant fields at the top level, while
emailAccessHandler is registered for EmailAccess/Update and reads the
same {email_owner_user_id, access_grants} shape as the create path. Those
transactions matched no handler at all, so unlike the emails above they
would not even have surfaced in em_rejected. It is corrected here rather
than left, since it is the same file and the same feature. It has no
effect on the snapshot: every email_access row there has an
encrypted_emails parent, so the step selects nothing.
The test drives the writer step against a source snapshot and replays
what it emits through the real handler, which is the only level at which
this was visible -- the handler-side email tests all passed while no
transaction reached them. Reverting the metadata field alone fails it
with "indexing the emitted encrypted email: missing required field:
email_owner_user_id".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
All 4,218 encrypted emails and their 9,282 access grants are lost in a migration today.
encryptedEmailHandlertakes the owner from the metadata keyemail_owner_user_id, not from the transaction'suser_id:encryptedEmailMetadatahad exactly three fields —encrypted_email,access_grants,created_at— so everyEncryptedEmail/AddEmailtransaction was rejected. These do reachem_rejected, but as thousands of anonymous rejections among millions of transactions, which is why it went unnoticed.Every required key, not just the one
Reading the handler end to end turns up a second key of the same class.
access_grantswas taggedomitempty, andparseAccessGrantsrejects the transaction when the key is absent:The handler cannot tell "absent" from "no grants", so
omitemptyis dropped and a nil slice is serialized as[]rather than omitted (or asnull, which fails the subsequent[]anyassertion with "access_grants must be a list"). Every email in the snapshot has at least one grant, so this one is latent — but it is the identical failure mode, one row away from firing.The per-grant keys (
receiving_user_id,grantor_user_id,encrypted_key) were already emitted correctly, and the snapshot has no NULL or empty values in any of them.Signer
Neither email handler calls
ValidateSigner, and there is no email validation at the chain layer or migration override, so the signer is not what was blocking these. The step still joinsusersand signs as the owner, matching every other entity writer and the convention stated inmigration.go— the rows should be attributable, and should stay valid if that check is ever added.The join also means an email whose owner is not a current user with a wallet is skipped rather than emitted unattributable. Verified against the production snapshot: all 4,218 owners are current users with a non-empty wallet, so nothing is dropped.
EmailAccess
The audit's flag is a genuine mismatch, and a worse one. The orphan-grant step emitted
EmailAccess/Createwith the grant fields at the top level, whileemailAccessHandleris registered forEmailAccess/Updateand reads the same{email_owner_user_id, access_grants}shape as the create path. Wrong action and wrong shape: those transactions matched no handler at all, so unlike the emails above they would not even have surfaced inem_rejected.Fixed here rather than deferred — same file, same feature. It has no effect on the snapshot: every
email_accessrow there has anencrypted_emailsparent, so the step'sNOT EXISTSfilter selects zero rows.One caveat worth recording:
emailAccessHandleralso requires the grantor to already hold access to the email, which an orphan's first grant cannot satisfy — there is no parent row to have granted from. Correcting the shape does not change that, and weakening the handler would be the wrong fix. What it does buy is that if orphan grants ever appear they reach the handler and are accepted or rejected on their merits, instead of vanishing to a routing mismatch. The code comment says so.Snapshot verification
Against
audius_discovery_2026_08_07:encrypted_emailsemail_accessemail_accessrows (no parent email)encrypted_keyor idsTest
TestWriteEncryptedEmails_IndexesWithGrantsdrives the writer step against a source snapshot and replays what it emits through the real handler — the only level at which this was visible. The handler-side email tests inpkg/etlall pass today while no transaction ever reaches them.Reverting just the metadata field fails it with the production symptom:
Reverting the file wholesale also fails the wallet-signer and skip assertions, and
TestWriteEmailAccess_EmitsHandlerShapefails withemitted 0 EmailAccess/Update transactions, want 1.With the fix, both pass, alongside the existing writer and ETL suites:
Note on #488
#488 adds
.UTC()to timestamp formatting in this file. BothFormat(time.RFC3339)calls here already carry.UTC(), so the two changes agree on those lines.🤖 Generated with Claude Code