Repository navigation
feat: add KeyManagementClient, KmsRegistry and InMemoryKms - #1031
Open
shangxinli wants to merge 2 commits into
Open
shangxinli wants to merge 2 commits into
shangxinli wants to merge 2 commits into
Conversation
Add the crypto primitives of table encryption to the bundle library, built when ICEBERG_ENCRYPTION is ON: - AesGcmCipher (OpenSSL EVP): AES-GCM with a random 12-byte nonce and the nonce | ciphertext | tag layout of Java's Ciphers. - AesGcmInputFile / AesGcmOutputFile: AES GCM Stream files (format/gcm-stream-spec.md), used for manifests, manifest lists, Avro data files and Puffin files. 1 MiB plaintext blocks, block AAD = file AAD prefix followed by the block index. The reader is seekable, takes the encrypted length from trusted key metadata, and verifies every block. The writer reports the plaintext Position() and the encrypted StoredLength(). aes_gcm_stream.h is the first public header that needs ICEBERG_ENCRYPTION while sharing a directory with unconditional ones, so the public-header generator grows a per-header override and the C++20 compatibility test a matching ICEBERG_PUBLIC_HEADERS_ENCRYPTION define. Tests decrypt AGS1 streams and KEK-encrypted key metadata produced by Java, and cover multi-block round trips, random access and tampering.
- KeyManagementClient (core): wraps and unwraps keys with master keys held in a KMS, mirroring Java's interface, including optional key generation. - KmsRegistry (core): creates the catalog's KMS client from the same catalog properties as Java, `encryption.kms-type` (aws, gcp, azure) or `encryption.kms-impl` (a registered name). As in Java, a property that is present but empty is a misconfiguration rather than "no KMS". Clients are initialized with the catalog properties. - InMemoryKms (bundle): master keys in memory, for tests and local use. Wire compatible with Java's MemoryMockKMS, so it unwraps keys written by Java tests. Master keys can be given as catalog properties (encryption.in-memory-kms.key.<id> = Base64).
shangxinli
force-pushed
the
enc/05-kms
branch
from
October 16, 2026 19:09
4852f34 to
78d5a09
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Step 5 of #988. Adds the KMS client interface and the registry that creates one from catalog properties, plus an in-memory client for tests and local development.
KeyManagementClient(core): wraps and unwraps secret keys with a master key held in a KMS, mirroring Java's interface including the optional in-KMS key generation path. The core library stays free of crypto dependencies — this is an interface only.KmsRegistry(core): resolves the catalog's KMS client from the same property names as Java,encryption.kms-type(aws,gcp,azure) orencryption.kms-impl(a name registered withKmsRegistry::Register). Built-in clients register themselves when their library is linked; applications register custom ones before creating catalogs. Returns a null client when neither property is set, so an unencrypted catalog is not an error.InMemoryKms(bundle,ICEBERG_ENCRYPTION): master keys held in memory, wire compatible with Java'sMemoryMockKMS— AES-GCM wrap under the master key with no AAD. Master keys can also come from catalog properties asencryption.in-memory-kms.key.<id>= Base64.Stacked on #1030 (step 4), which
InMemoryKmsuses for the AES-GCM primitives; the first commit here is that PR. Review only the second commit, or wait for #1030 to land.Java parity
Ported against
KeyManagementClient.javaandEncryptionUtil.createKmsClient, withMemoryMockKMS/UnitestKMSas the reference for the in-memory client. Deliberate differences, both flagged in the code:encryption.kms-implto a class name and instantiates it reflectively. C++ has no reflection, so the registry maps a name to a factory; the property keys and their meaning are unchanged.aliyun. Support Iceberg table encryption (Java-compatible) #988 scopes the built-in clients to AWS, GCP and Azure, soaliyunis not built in here; the doc comment says to register one and select it throughencryption.kms-impl. The type list is otherwise identical.One behavior is matched on purpose rather than "improved": a property that is present but empty counts as set, as in Java, so a misconfigured catalog fails instead of silently loading with no KMS.
Test plan
encryption_testpasses: registry loads a custom implementation and initializes it with the catalog properties, resolves a built-in type case-insensitively, and rejects setting both type and impl, an unsupported type, an unregistered name, and a present-but-empty value.crypto_testpasses:InMemoryKmsunwraps a key encryption key that Iceberg Java'sUnitestKMSwrapped under itskeyAmaster key, which is what makes the Java wire compatibility claim testable rather than a C++ round trip. The same works throughKmsRegistrywith a Base64 master key supplied as a catalog property.cpp20_compatibility_testpasses.key_management_client.his core and unconditional;in_memory_kms.his bundle plus encryption, so it joinsaes_gcm_stream.hbehindICEBERG_PUBLIC_HEADERS_ENCRYPTION.pre-commitclean, including the generated public-header check.