Repository navigation
feat: add AES GCM cipher and AES GCM Stream (AGS1) files - #1030
Open
shangxinli wants to merge 1 commit into
Open
shangxinli wants to merge 1 commit into
shangxinli wants to merge 1 commit into
Conversation
4 tasks done
Add the crypto primitives of table encryption to the bundle library, built when ICEBERG_ENCRYPTION is ON: - AesGcmCipher (OpenSSL EVP): AES-GCM with a random 12-byte nonce and the nonce | ciphertext | tag layout of Java's Ciphers. - AesGcmInputFile / AesGcmOutputFile: AES GCM Stream files (format/gcm-stream-spec.md), used for manifests, manifest lists, Avro data files and Puffin files. 1 MiB plaintext blocks, block AAD = file AAD prefix followed by the block index. The reader is seekable, takes the encrypted length from trusted key metadata, and verifies every block. The writer reports the plaintext Position() and the encrypted StoredLength(). aes_gcm_stream.h is the first public header that needs ICEBERG_ENCRYPTION while sharing a directory with unconditional ones, so the public-header generator grows a per-header override and the C++20 compatibility test a matching ICEBERG_PUBLIC_HEADERS_ENCRYPTION define. Tests decrypt AGS1 streams and KEK-encrypted key metadata produced by Java, and cover multi-block round trips, random access and tampering.
shangxinli
force-pushed
the
enc/04-aes-gcm-stream
branch
from
October 16, 2026 19:09
341ee25 to
046e105
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Step 4 of #988. Adds the crypto primitives of table encryption to the bundle library, built when
ICEBERG_ENCRYPTIONis ON. No behavior change for existing builds: everything here is new, and the default (ICEBERG_ENCRYPTION=OFF) compiles none of it.AesGcmCipher(OpenSSL EVP): AES-GCM with a random 12-byte nonce, in thenonce | ciphertext | taglayout of Java'sCiphers. Key lengths 16/24/32, AAD optional, every OpenSSL return code checked includingEVP_DecryptFinal_ex, so a failed tag check is always an error rather than unauthenticated plaintext.AesGcmInputFile/AesGcmOutputFile: AES GCM Stream files performat/gcm-stream-spec.md, the format used for manifests, manifest lists, Avro data files and Puffin files. 1 MiB plaintext blocks, headerAGS1+ little-endian block size, block AAD = file AAD prefix followed by the block index as a little-endian int32, and only the last block may be partial. The reader is seekable and takes the encrypted length from the key metadata rather than the file, so truncation is detected; the writer reports the plaintextPosition()and the encryptedStoredLength().aes_gcm_stream.his the first public header that requiresICEBERG_ENCRYPTIONwhile sharing a directory with unconditional ones (encrypted_key.h,standard_key_metadata.h).ci/scripts/generate_public_headers.pyonly classifies whole directories, so it grows a small per-header override table, andcpp20_compatibility_testgains a matchingICEBERG_PUBLIC_HEADERS_ENCRYPTIONdefine.Java parity
Ported against
core/src/main/java/org/apache/iceberg/encryption/{Ciphers,AesGcmInputStream,AesGcmOutputStream,AesGcmInputFile,AesGcmOutputFile}.java. Two places where the Java semantics are not obvious from the C++ base classes and are deliberately matched:AesGcmOutputStream::Position()keeps reporting the plaintext length afterClose(). Closing flushes the final block, which resets the in-block position, so the value is captured first — Java does the same with itsfinalPositionfield.StandardKeyMetadatarecords the stored length for every encrypted file because Java needs it for AGS1 reads; that part landed in #991.Test plan
crypto_testpasses. The tests decrypt AGS1 streams of 0, 1 and 100 bytes and a KEK-encrypted manifest-list key metadata record, all produced by Iceberg Java, so a wire-format divergence fails rather than round-tripping against itself. Following feat: add standard encryption key metadata #991, the fixtures are inlined byte arrays rather than checked-in binaries.block-1,block,block+1,2*block+17) with write chunks that straddle blocks, plus random access across a boundary.cpp20_compatibility_testpasses withICEBERG_ENCRYPTION=ON.pre-commitclean, including the generated public-header check.Verified that the
Position()-after-Close()assertion fails without its fix (reports1048576for a 100-byte stream).