refactor: shift private dataset sharing to separate private collections - #9513
Open
rasswanth-s wants to merge 5 commits into
Open
rasswanth-s wants to merge 5 commits into
rasswanth-s wants to merge 5 commits into
Conversation
…g encryption for files
rasswanth-s
changed the title
A private dataset can hold nested files (an Orbax checkpoint is a directory of shards), but collections were flat: the upload took only top-level files and Drive file names cannot contain "/". The enclave therefore received gemma3_model without its checkpoint directory and setup_model failed with "Checkpoint ... not found". - transport: a file's path relative to the collection root is the upload key; "/" is encoded into the flat Drive name and decoded on listing/download, so flat names are unchanged and nested files land at the same relative path. - rds: private files are collected recursively (was iterdir on the top level). - router: temp paths for encrypted copies get their parent directories. - test: the end-to-end share now includes a nested file and asserts it arrives. - docs: the two design documents for this change. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Changes
Testing
Asana task
PR naming convention
Your PR title must follow this format or merging will be blocked:
Example titles:
feat: add retry logic to job approvalfix: handle timeout in notification senderdocs: update syft-bg READMEchore: bump dependenciesrefactor: split init flow into helperstest: add criteria validation testsci: add release train workflowAllowed types:
featfeat: add DS rejection emailsfixfix: handle empty peer listdocsdocs: update syft-bg READMEchorechore: bump dependenciesrefactorrefactor: split init flowtesttest: add approval criteria testscici: add release train workflowperfperf: cache Drive API responsesbuildbuild: pin syft-bg>=0.2.0Just edit the PR title to fix any errors — the check re-runs automatically.
Auto-labeling
Labels are applied automatically — you don't need to add them manually:
feat:addsfeature,fix:addsbugfix)packages/syft-bg/addspkg:syft-bg)These labels are used to auto-generate categorized release notes.