Skip to content

feat: add KeyManagementClient, KmsRegistry and InMemoryKms - #1031

Open
shangxinli wants to merge 2 commits into
apache:mainfrom
shangxinli:enc/05-kms
Open

shangxinli wants to merge 2 commits into
apache:mainfrom
shangxinli:enc/05-kms

Conversation

@shangxinli

Copy link
Copy Markdown
Contributor

Summary

Step 5 of #988. Adds the KMS client interface and the registry that creates one from catalog properties, plus an in-memory client for tests and local development.

  • KeyManagementClient (core): wraps and unwraps secret keys with a master key held in a KMS, mirroring Java's interface including the optional in-KMS key generation path. The core library stays free of crypto dependencies — this is an interface only.
  • KmsRegistry (core): resolves the catalog's KMS client from the same property names as Java, encryption.kms-type (aws, gcp, azure) or encryption.kms-impl (a name registered with KmsRegistry::Register). Built-in clients register themselves when their library is linked; applications register custom ones before creating catalogs. Returns a null client when neither property is set, so an unencrypted catalog is not an error.
  • InMemoryKms (bundle, ICEBERG_ENCRYPTION): master keys held in memory, wire compatible with Java's MemoryMockKMS — AES-GCM wrap under the master key with no AAD. Master keys can also come from catalog properties as encryption.in-memory-kms.key.<id> = Base64.

Stacked on #1030 (step 4), which InMemoryKms uses for the AES-GCM primitives; the first commit here is that PR. Review only the second commit, or wait for #1030 to land.

Java parity

Ported against KeyManagementClient.java and EncryptionUtil.createKmsClient, with MemoryMockKMS / UnitestKMS as the reference for the in-memory client. Deliberate differences, both flagged in the code:

  • Java resolves encryption.kms-impl to a class name and instantiates it reflectively. C++ has no reflection, so the registry maps a name to a factory; the property keys and their meaning are unchanged.
  • Java accepts a fourth type, aliyun. Support Iceberg table encryption (Java-compatible) #988 scopes the built-in clients to AWS, GCP and Azure, so aliyun is not built in here; the doc comment says to register one and select it through encryption.kms-impl. The type list is otherwise identical.

One behavior is matched on purpose rather than "improved": a property that is present but empty counts as set, as in Java, so a misconfigured catalog fails instead of silently loading with no KMS.

Test plan

  • encryption_test passes: registry loads a custom implementation and initializes it with the catalog properties, resolves a built-in type case-insensitively, and rejects setting both type and impl, an unsupported type, an unregistered name, and a present-but-empty value.
  • crypto_test passes: InMemoryKms unwraps a key encryption key that Iceberg Java's UnitestKMS wrapped under its keyA master key, which is what makes the Java wire compatibility claim testable rather than a C++ round trip. The same works through KmsRegistry with a Base64 master key supplied as a catalog property.
  • cpp20_compatibility_test passes. key_management_client.h is core and unconditional; in_memory_kms.h is bundle plus encryption, so it joins aes_gcm_stream.h behind ICEBERG_PUBLIC_HEADERS_ENCRYPTION.
  • pre-commit clean, including the generated public-header check.

shangxinli and others added 2 commits October 16, 2026 12:08
Add the crypto primitives of table encryption to the bundle library, built
when ICEBERG_ENCRYPTION is ON:

- AesGcmCipher (OpenSSL EVP): AES-GCM with a random 12-byte nonce and the
  nonce | ciphertext | tag layout of Java's Ciphers.
- AesGcmInputFile / AesGcmOutputFile: AES GCM Stream files
  (format/gcm-stream-spec.md), used for manifests, manifest lists, Avro data
  files and Puffin files. 1 MiB plaintext blocks, block AAD = file AAD prefix
  followed by the block index. The reader is seekable, takes the encrypted
  length from trusted key metadata, and verifies every block. The writer
  reports the plaintext Position() and the encrypted StoredLength().

aes_gcm_stream.h is the first public header that needs ICEBERG_ENCRYPTION
while sharing a directory with unconditional ones, so the public-header
generator grows a per-header override and the C++20 compatibility test a
matching ICEBERG_PUBLIC_HEADERS_ENCRYPTION define.

Tests decrypt AGS1 streams and KEK-encrypted key metadata produced by Java,
and cover multi-block round trips, random access and tampering.
- KeyManagementClient (core): wraps and unwraps keys with master keys held in
  a KMS, mirroring Java's interface, including optional key generation.
- KmsRegistry (core): creates the catalog's KMS client from the same catalog
  properties as Java, `encryption.kms-type` (aws, gcp, azure) or
  `encryption.kms-impl` (a registered name). As in Java, a property that is
  present but empty is a misconfiguration rather than "no KMS". Clients are
  initialized with the catalog properties.
- InMemoryKms (bundle): master keys in memory, for tests and local use. Wire
  compatible with Java's MemoryMockKMS, so it unwraps keys written by Java
  tests. Master keys can be given as catalog properties
  (encryption.in-memory-kms.key.<id> = Base64).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant