[Bug] Iceberg metadata committer fails with NullPointerException

[novakov-alexey]

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

1.3.1

Compute Engine

Flink 2.1.1

Minimal reproduce step

1. Start any Iceberg-REST compatible catalog.
2. Create Paimon Hadoop catalog with S3-based warehouse. Then, create a Paimon table inside the Paimon catalog adding required metadata properties for Iceberg Metadata functionality:

  'metadata.iceberg.storage' = 'rest-catalog',
  'metadata.iceberg.rest.uri' = 'http://localhost:55807/', -- specify your URL
  'metadata.iceberg.rest.warehouse' = 'rck_warehouse', -- specify your warehouse name
  'metadata.iceberg.rest.clients' = '1'

3. Launch Flink job to INSERT data to this table.

What doesn't meet your expectations?

1.Flink could not create a table in the Iceberg REST Catalog. However, some traces of the operation do appear on the S3 location, such as metadata JSON file.
2. Flink Job is not able to finish successfully, it fails on Writer operator, trying to commit table creation to Iceberg REST catalog endpoint, but gets:

java.lang.RuntimeException: java.lang.RuntimeException: Fail to create table or get table: default.my_orders
	at org.apache.paimon.iceberg.IcebergRestMetadataCommitter.commitMetadata(IcebergRestMetadataCommitter.java:123)
	at org.apache.paimon.iceberg.IcebergCommitCallback.createMetadataWithBase(IcebergCommitCallback.java:666)
	at org.apache.paimon.iceberg.IcebergCommitCallback.createMetadata(IcebergCommitCallback.java:281)
	at org.apache.paimon.iceberg.IcebergCommitCallback.call(IcebergCommitCallback.java:229)
	at org.apache.paimon.operation.FileStoreCommitImpl.lambda$tryCommitOnce$16(FileStoreCommitImpl.java:1215)
	at java.base/java.util.ArrayList.forEach(Unknown Source)
	at org.apache.paimon.operation.FileStoreCommitImpl.tryCommitOnce(FileStoreCommitImpl.java:1213)
	at org.apache.paimon.operation.FileStoreCommitImpl.tryCommit(FileStoreCommitImpl.java:840)
	at org.apache.paimon.operation.FileStoreCommitImpl.commit(FileStoreCommitImpl.java:362)
	at org.apache.paimon.table.sink.TableCommitImpl.commitMultiple(TableCommitImpl.java:229)
	at org.apache.paimon.flink.sink.StoreCommitter.commit(StoreCommitter.java:111)
	at org.apache.paimon.flink.sink.CommitterOperator.commitUpToCheckpoint(CommitterOperator.java:215)
	at org.apache.paimon.flink.sink.CommitterOperator.notifyCheckpointComplete(CommitterOperator.java:192)
	at org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.notifyCheckpointComplete(StreamOperatorWrapper.java:104)
	at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.notifyCheckpointComplete(RegularOperatorChain.java:145)
	at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.notifyCheckpoint(SubtaskCheckpointCoordinatorImpl.java:479)
	at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.notifyCheckpointComplete(SubtaskCheckpointCoordinatorImpl.java:412)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.notifyCheckpointComplete(StreamTask.java:1578)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointCompleteAsync$20(StreamTask.java:1519)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointOperation$23(StreamTask.java:1558)
	at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
	at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:118)
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:415)
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsNonBlocking(MailboxProcessor.java:400)
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:362)
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:980)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:917)
	at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:963)
	at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:942)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:756)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:568)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.RuntimeException: Fail to create table or get table: default.my_orders
	at org.apache.paimon.iceberg.IcebergRestMetadataCommitter.commitMetadataImpl(IcebergRestMetadataCommitter.java:167)
	at org.apache.paimon.iceberg.IcebergRestMetadataCommitter.commitMetadata(IcebergRestMetadataCommitter.java:121)
	... 32 more
Caused by: java.lang.NullPointerException
	at org.apache.paimon.iceberg.IcebergRestMetadataCommitter.checkBase(IcebergRestMetadataCommitter.java:355)
	at org.apache.paimon.iceberg.IcebergRestMetadataCommitter.commitMetadataImpl(IcebergRestMetadataCommitter.java:154)
	... 33 more

Anything else?

I used Nessie REST Catalog of version 0.106.0

Are you willing to submit a PR?

• I'm willing to submit a PR!

[astamur]

I have same bug on the same set up (Paimon 1.3.1, Flink 2.1.2, Nessie 0.106.1)

[novakov-alexey]

So sad, it means the entire functionality does not work according to official documentation.

[astamur]

@novakov-alexey Looks like it was some temporary issue (maybe some stale data between environment restarts etc.).
Finally I had another issue with Paimon and Nessie (REST mode) and it was fixed already: projectnessie/nessie#11972

Related to the problem in the current ticket, you can use my small setup and configs I used there: https://github.com/astamur/paimon-nessie-iceberg-validation-issue

[JacobSMoller]

@novakov-alexey after looking at this a bit more with @muttcg it does indeed look like the iceberg rest code will never really work - at least only if you add the iceberg rest settings at the very beginning of your tables lifecycle and then dont change the schema - simply adding fields should be okay, but altering table options or dropping fields paimons schema and icebergs schema will drift, because iceberg reuses schemas

Because paimons rest integration already is sort of hacky by having to add an empty schema to avoid some field id problems that Iceberg has an opinion about so they go from id: 0 -> 1 , 1->2, n->n+1 and then paimon does and update to get them back to 0, 1, 2 etc.

after that paimon tracks the schema of iceberg rest as schema n+1, but if you do changes like alter table set options, paimon writes a new schema, which also triggers a new schema in iceberg - but in iceberg the paimon table options have no meaning and the actual fields haven't changed - iceberg sees this and deduplicates the schemas staying on n, but now paimon is on n+1 so when it next checks the rest catalog the n+1 schema trick no longer works.

I think it is solvable but the whole iceberg commiter needs an overhaul as far as I can tell.

@muttcg correct me if im totally of track :)

https://polaris.apache.org/blog/2026/01/12/mapping-legacy-and-heterogeneous-datalakes-in-apache-polaris/ we are using Polaris and they have a notifications API, where you can send iceberg metadata and treat it as an external catalog so Polaris can only read it - but at least for our usecase that also makes sense because Paimon owns and writes the data.

I dont know if nessie has anything similar

But in its current state i dont think it will ever really work as intended with rest catalog

[astamur]

@JacobSMoller Based on my investigation here projectnessie/nessie#11972, Iceberg compatibility mode works well with Tabular rest catalog. Because it doesn't shift fields indexes. But Nessie did.

[JacobSMoller]

@astamur Alright interesting they dont do that - the field shifting and deduplication of schemas seems to come from iceberg core have you tried adding a couple of fields and then deleting one of them again - to see if tabular rest catalog also doesn't do the deduplication of schemas ?

That being said i still think it could be fixed on Paimons end if it actually wrote metadata that adhered to the rules that iceberg seems to adhere to

[novakov-alexey]

In the meantime, I tried Paimon + Gravitino Iceberg Rest Server. It works for tables without watermark definitions in their DDL, i.e. i can read Iceberg tables by Iceberg tools, which are populated by Paimon Sink table. However, tables with watermark definition fail with similar error I posted above.
@astamur thanks for the link to Nessie Bug-Fix, I will try with newer Nessie version.

Side problem here is that Iceberg+Flink integration does not support Watermarks in DDL yet (at least today at 1.10 version). That means Paimon can't sync metadata (create correct schema) on its own. Perhaps some interception of the such case to remove watermark from DDL before Paimon commits metadata could help. But this would be already too much to support such pipeline in production.