-
-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Slightly Amend Zarr Encoding Specification Doc #8749 #11013
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
eshort0401
wants to merge
7
commits into
pydata:main
Choose a base branch
from
eshort0401:amend-zarr-docs
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+14
−11
Open
Changes from 6 commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
631fc64
amend zarr encoding specification doc GH8749
eshort0401 86c9251
additional edits to zarr encoding spec
eshort0401 a54cfa8
minor doc edits for consistency
eshort0401 7bca7cb
Merge remote-tracking branch 'upstream/main' into amend-zarr-docs
eshort0401 ff5502e
amend whats-new.rst
eshort0401 1ae0da3
add pull number to whats-new.rst
eshort0401 0b067d9
fix incorrect usage of 'groups'
eshort0401 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -27,7 +27,7 @@ NetCDF data model in Zarr. | |
| Dimension Encoding in Zarr Formats | ||
| ----------------------------------- | ||
|
|
||
| Xarray encodes array dimensions differently depending on the Zarr format version: | ||
| Xarray encodes/decodes array dimensions differently depending on the Zarr format version: | ||
|
|
||
| **Zarr V2 Format:** | ||
| Xarray uses a special Zarr array attribute: ``_ARRAY_DIMENSIONS``. The value of this | ||
|
|
@@ -43,9 +43,9 @@ When accessing arrays with zarr-python, this information is available in the arr | |
| metadata but not in the attributes dictionary. | ||
|
|
||
| When reading a Zarr group, Xarray looks for dimension information in the appropriate | ||
| location based on the format version, raising an error if it can't be found. The | ||
| location based on the inferred format version, raising an error if it can't be found. The | ||
| dimension information is used to define the variable dimension names and then | ||
| (for Zarr V2) removed from the attributes dictionary returned to the user. | ||
| (for Zarr V2) is removed from the attributes dictionary returned to the user. | ||
|
|
||
| CF Conventions | ||
| -------------- | ||
|
|
@@ -59,17 +59,14 @@ used to describe metadata in NetCDF and Zarr. | |
| Compatibility and Reading | ||
| ------------------------- | ||
|
|
||
| Because of these encoding choices, Xarray cannot read arbitrary Zarr arrays, but only | ||
| Zarr data with valid dimension metadata. Xarray supports: | ||
| Because of these encoding choices, Xarray cannot read arbitrary Zarr groups, but only | ||
| Zarr groups with valid dimension metadata. Xarray supports: | ||
|
|
||
| - Zarr V2 arrays with ``_ARRAY_DIMENSIONS`` attributes | ||
| - Zarr V3 arrays with ``dimension_names`` metadata | ||
| - `NCZarr <https://docs.unidata.ucar.edu/nug/current/nczarr_head.html>`_ format | ||
| (dimension names are defined in the ``.zarray`` file) | ||
| 1. Zarr V3 groups with ``dimension_names`` metadata | ||
| 2. Zarr V2 groups with ``_ARRAY_DIMENSIONS`` attributes | ||
|
||
| 3. `NCZarr <https://docs.unidata.ucar.edu/nug/current/nczarr_head.html>`_ format (dimension names are defined in the ``dimrefs`` field in the custom ``.zarray`` file) | ||
|
|
||
| After decoding the dimension information and assigning the variable dimensions, | ||
| Xarray proceeds to [optionally] decode each variable using its standard CF decoding | ||
| machinery used for NetCDF data. | ||
| Xarray checks each of these three conventions, in the order given above, when looking for dimension name metadata. Note that while Xarray can read NCZarr groups, it currently does not write NCZarr groups. After decoding the dimension information and assigning the variable dimensions, Xarray proceeds to [optionally] decode each variable using its standard CF decoding machinery used for NetCDF data. | ||
|
|
||
| Finally, it's worth noting that Xarray writes (and attempts to read) | ||
| "consolidated metadata" by default (the ``.zmetadata`` file), which is another | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would say "Zarr groups containing arrays with valid dimension metadata" here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed (0b067d9) - thanks @shoyer!