HDFS-17179. DFSInputStream should report CorruptMetaHeaderException a… - #8310
HDFS-17179. DFSInputStream should report CorruptMetaHeaderException a…#8310deepujain wants to merge 2 commits into
Conversation
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
3d0177c to
412f3c5
Compare
|
Rebased on trunk. Adjusted Local tests: |
a6f6354 to
ef0809a
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
CI is green on the current head, and this one still has not had a human review. @Hexiaoqiao @ZanderXu, when you have a moment, could you please take a look? Happy to make any follow-up changes. |
|
CI is still green on the current head, and this one is still waiting on human review. @hfutatzhanghb @Hexiaoqiao, when you have a moment, could you please take a look? Happy to make any follow-up changes. |
|
CI is still green on the current head, and this one is still waiting on HDFS review. @hfutatzhanghb @Hexiaoqiao, when you have a moment, could you please take a look? Happy to make any follow-up changes. |
1 similar comment
|
CI is still green on the current head, and this one is still waiting on HDFS review. @hfutatzhanghb @Hexiaoqiao, when you have a moment, could you please take a look? Happy to make any follow-up changes. |
Sorry for responding too late. Will reivew it quickly |
|
@deepujain Can you fix the checksyle warning? Others LGTM. |
|
Removed the redundant same-package import in |
|
Rebased this branch onto current |
5e6ae2c to
38cdba4
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
No code changes here. I reran the Build workflow to refresh the red |
|
Pushed an empty |
|
💔 -1 overall
This message was automatically generated. |
|
Pushed another empty |
|
💔 -1 overall
This message was automatically generated. |
|
No code changes here. I reran the fork-side |
1a62b51 to
dacc840
Compare
|
No code changes here. I reran the current head again on July 12, 2026 to refresh the stale red GitHub Actions lane. The rerun still failed in the shared If the HDFS-side change itself looks good, @hfutatzhanghb @Hexiaoqiao, could you please take a look when you have a moment? Happy to make any follow-up changes. |
|
No code changes here. The HDFS-side patch and Yetus signal on the current head are unchanged, and this one is still waiting on human review as of July 20, 2026. @hfutatzhanghb @Hexiaoqiao, when you have a moment, could you please take a look? Happy to make any follow-up changes. |
|
No code changes here. I reran the existing fork-side |
|
No code changes here. I reran the fork-side |
|
Hi @deepujain . LGTM. Failed checks is not related to this PR. We can first fix the failed unit tests. @zhtttylz Please also cc . Thanks a lot. |
dacc840 to
f45fe0a
Compare
|
Thanks, @hfutatzhanghb. I rebased this branch onto current Fresh local validation on the rebased head: Result: No code changes beyond the rebase in this update. @zhtttylz, please take a look when you have a moment. Happy to make any follow-up changes. |
|
💔 -1 overall
This message was automatically generated. |
|
No code changes here. The July 22, 2026 rerun on the rebased head finished with GitHub That still looks outside this DFSInputStream corrupt-metadata patch, which only touches the corrupt-metadata handling path plus If the patch itself looks good, @zhtttylz, please take a look when you have a moment. Happy to make any follow-up changes. |
|
No code changes here. The July 22, 2026 rebase and focused local |
f45fe0a to
31b71fd
Compare
|
Rebased this branch onto current apache/trunk on August 4, 2026 and force-pushed the refreshed single commit. No source changes beyond the rebase in this update. The last focused local validation still recorded on this branch is |
|
No code changes here since the August 4, 2026 rebase onto current On the refreshed fork-side run, the failing GitHub Build lanes are stopping in the The earlier focused local validation on this branch remains the latest direct patch signal: |
|
No code changes in this update. I reran the fork-side GitHub Build workflow on the current head so the prior Build failure can get a fresh signal; the branch content itself is unchanged from the latest rebase and validation pass. |
|
No code changes in this update. The August 5 rerun progressed past the earlier image/container setup failure and most shards on the current head passed; the only remaining red GitHub Build lane is now |
|
No code changes here. I reran |
|
No code changes here. On the August 6, 2026 rerun, the fork-side Build progressed through image setup/build and most shards passed; the only red shard was |
3 participants
Summary
When a block read fails with
CorruptMetaHeaderException(e.g. corrupt or too-short block meta file on the DataNode), DFSInputStream did not add the block to the corrupted set or report it to the NameNode, unlikeChecksumException. Corrupt blocks were therefore not invalidated or re-replicated. This change treatsCorruptMetaHeaderExceptionthe same asChecksumExceptionin the client read path so the block is reported to the NameNode and can be re-replicated.Change
CorruptMetaHeaderException. InreadBuffer()(block read loop), add acatch (CorruptMetaHeaderException)that logs, adds the block tocorruptedBlocks, and setsretryCurrentNode = false, mirroring the existingChecksumExceptionhandling. InactualGetFromOneDataNode()(fetchBlockByteRange), add acatch (CorruptMetaHeaderException)that logs, adds the block tocorruptedBlocks, marks the datanode dead, and throws anIOException, mirroring theChecksumExceptionhandling.testReportCorruptMetaHeaderToNameNode(): create a file, corrupt the block meta file with an invalid 7-byte header, read from the client (expect IOException), then wait until the NameNode’s corrupt block count is 1 (HDFS-17179).JIRA
Fixes HDFS-17179