[Question]Full The full nodes of Tron have been stuck at block 80424111

[letmountain]

Tron's entire node has been stuck at 80424111, and after the snapshot update, it has been stuck here. The following are the logs

============ Peer stats: all 9, active 9, passive 0

Peer /8.217.163.67:18888
connect time: 217s [91ms]
last know block num: 0 [217s]
needSyncFromPeer:false
needSyncFromUs:false
syncToFetchSize:0
syncToFetchSizePeekNum:-1
syncBlockRequestedSize:0
remainNum:0
syncChainRequested:0
inactiveSeconds:2
blockInProcess:0

Peer /60.244.109.20:18888
connect time: 26s [130ms]
last know block num: 52812275 [24s]
needSyncFromPeer:false
needSyncFromUs:true
syncToFetchSize:0
syncToFetchSizePeekNum:-1
syncBlockRequestedSize:0
remainNum:27609925
syncChainRequested:0
inactiveSeconds:1
blockInProcess:0

Peer /13.212.39.197:18888
connect time: 2s [156ms]
last know block num: 80424111 [2s]
needSyncFromPeer:true
needSyncFromUs:false
syncToFetchSize:3988
syncToFetchSizePeekNum:80424112
syncBlockRequestedSize:100
remainNum:4708051
syncChainRequested:0
inactiveSeconds:1
blockInProcess:0

Peer /5.223.78.3:18888
connect time: 159s [180ms]
last know block num: 0 [159s]
needSyncFromPeer:false
needSyncFromUs:false
syncToFetchSize:0
syncToFetchSizePeekNum:-1
syncBlockRequestedSize:0
remainNum:0
syncChainRequested:0
inactiveSeconds:159
blockInProcess:0

Peer /87.251.64.114:18888
connect time: 0s [478ms]
last know block num: 0 [0s]
needSyncFromPeer:false
needSyncFromUs:true
syncToFetchSize:0
syncToFetchSizePeekNum:-1
syncBlockRequestedSize:0
remainNum:0
syncChainRequested:0
inactiveSeconds:0
blockInProcess:0

Peer /72.56.98.166:11002
connect time: 4s [624ms]
last know block num: 0 [4s]
needSyncFromPeer:false
needSyncFromUs:true
syncToFetchSize:0
syncToFetchSizePeekNum:-1
syncBlockRequestedSize:0
remainNum:7087094
syncChainRequested:0
inactiveSeconds:2
blockInProcess:0

Peer /157.180.122.128:18888
connect time: 1s [625ms]
last know block num: 0 [0s]
needSyncFromPeer:true
needSyncFromUs:true
syncToFetchSize:0
syncToFetchSizePeekNum:-1
syncBlockRequestedSize:0
remainNum:0
syncChainRequested:0
inactiveSeconds:0
blockInProcess:0

Peer /168.119.30.61:18888
connect time: 4s [754ms]
last know block num: 0 [4s]
needSyncFromPeer:true
needSyncFromUs:false
syncToFetchSize:0
syncToFetchSizePeekNum:-1
syncBlockRequestedSize:0
remainNum:0
syncChainRequested:4
inactiveSeconds:4
blockInProcess:0

Peer /66.163.117.34:18888
connect time: 170s [1550ms]
last know block num: 0 [168s]
needSyncFromPeer:false
needSyncFromUs:false
syncToFetchSize:0
syncToFetchSizePeekNum:-1
syncBlockRequestedSize:0
remainNum:0
syncChainRequested:0
inactiveSeconds:168
blockInProcess:0

05:09:29.378 INFO [peerClient-24] net Receive message from channel: /135.181.229.171:18888, HelloMessage networkId: 11111, version: 1, code: 1
05:09:29.378 INFO [peerClient-24] net Add peer /135.181.229.171:18888, total channels: 10
05:09:29.378 INFO [peerClient-24] net Handshake failed /135.181.229.171:18888, code: 1, reason: TOO_MANY_PEERS, networkId: 11111, version: 1
05:09:29.378 INFO [peerClient-24] net Try to close channel: /135.181.229.171:18888, reason: TOO_MANY_PEERS
05:09:29.378 INFO [peerClient-24] net Close channel:/135.181.229.171:18888
05:09:29.378 INFO [peerClient-24] net Peer stats: channels 9, activePeers 9, active 9, passive 0
05:09:29.380 INFO [connPool] net Connect to peer /37.27.110.7:18888
05:09:29.387 INFO [peerClient-22] net Receive message from channel: /178.128.133.243:18888, HelloMessage networkId: 11111, version: 1, code: 0
05:09:29.387 INFO [peerClient-22] net Add peer /178.128.133.243:18888, total channels: 10
05:09:29.387 INFO [peerClient-22] net Peer stats: channels 10, activePeers 10, active 10, passive 0
05:09:29.387 INFO [peerClient-22] net Send peer /178.128.133.243:18888 message type: P2P_HELLO
from: /103.101.207.204:18888
timestamp: 1786079368591
headBlockId: Num:80424099,ID:0000000004cb2ca31e81f008a1f1d7f8355aab38584c955a65a7d6c153a4d7c5
nodeType: 0
lowestBlockNum: 0
codeVersion:4.8.2.1

05:09:29.388 INFO [peerClient-22] net Receive message from peer: /178.128.133.243:18888, type: P2P_HELLO
from: /178.128.133.243:18888
timestamp: 1786079368927
headBlockId: Num:85136150,ID:0000000005131316318a3142bec243aa884d1c213e019f3481628b9da246a05c
nodeType: 1
lowestBlockNum: 85006827
codeVersion:4.8.1

05:09:29.388 INFO [peerClient-22] net Peer /178.128.133.243:18888 miss block, lowestBlockNum:85006827, headBlockNum:80424099
05:09:29.388 INFO [peerClient-22] net Send peer /178.128.133.243:18888 message type: P2P_DISCONNECT
reason: LIGHT_NODE_SYNC_FAIL
05:09:29.388 INFO [peerClient-22] net Close channel:/178.128.133.243:18888
05:09:29.388 INFO [peerClient-22] net Peer stats: channels 9, activePeers 9, active 9, passive 0
05:09:29.389 INFO [connPool] net Connect to peer /3.216.116.227:18888
05:09:29.450 INFO [peerClient-25] net Receive message from channel: /34.107.3.177:18888, HelloMessage networkId: 11111, version: 1, code: 1
05:09:29.450 INFO [peerClient-25] net Add peer /34.107.3.177:18888, total channels: 10
05:09:29.451 INFO [peerClient-25] net Handshake failed /34.107.3.177:18888, code: 1, reason: TOO_MANY_PEERS, networkId: 11111, version: 1
05:09:29.451 INFO [peerClient-25] net Try to close channel: /34.107.3.177:18888, reason: TOO_MANY_PEERS
05:09:29.451 INFO [peerClient-25] net Close channel:/34.107.3.177:18888
05:09:29.451 INFO [peerClient-25] net Peer stats: channels 9, activePeers 9, active 9, passive 0
05:09:29.452 INFO [connPool] net Connect to peer /65.108.108.100:18888
05:09:29.544 WARN [peerClient-2] net Connect to peer /5.223.60.36:18888 fail, cause:connection timed out after 2000 ms: /5.223.60.36:18888
05:09:29.544 INFO [peerClient-2] net Close channel:null
05:09:29.544 WARN [peerClient-2] net Notify Disconnect peer has no address.
05:09:29.545 INFO [connPool] net Connect to peer /95.217.62.144:18888
05:09:29.635 INFO [peerClient-4] net Receive message from channel: /91.227.33.205:18888, HelloMessage networkId: 11111, version: 1, code: 1
05:09:29.635 INFO [peerClient-4] net Add peer /91.227.33.205:18888, total channels: 10
05:09:29.635 INFO [peerClient-4] net Handshake failed /91.227.33.205:18888, code: 1, reason: TOO_MANY_PEERS, networkId: 11111, version: 1
05:09:29.635 INFO [peerClient-4] net Try to close channel: /91.227.33.205:18888, reason: TOO_MANY_PEERS
05:09:29.635 INFO [peerClient-4] net Close channel:/91.227.33.205:18888
05:09:29.635 INFO [peerClient-4] net Peer stats: channels 9, activePeers 9, active 9, passive 0
05:09:29.644 INFO [peerClient-29] net Send message to channel /3.216.116.227:18888, HelloMessage networkId: 11111, version: 1, code: 0
05:09:29.729 INFO [peerClient-23] net Send message to channel /139.180.184.220:18888, HelloMessage networkId: 11111, version: 1, code: 0
05:09:29.753 INFO [peerClient-30] net Send message to channel /65.108.108.100:18888, HelloMessage networkId: 11111, version: 1, code: 0
05:09:29.760 INFO [peerClient-17] net Receive message from peer: /87.251.64.114:18888, type: SYNC_BLOCK_CHAIN
size: 5, start block: Num:3622875,ID:00000000003747db0f60eb2ffa84f33c370ce7935b0989707b64093fe35bb813, end block Num:3622894,ID:00000000003747eee62d8c5ebdc0b0f770a246b18226ff654f798154a3a7df54
05:09:29.765 INFO [peerClient-17] net Send peer /87.251.64.114:18888 message type: BLOCK_CHAIN_INVENTORY
size: 2001, first blockId: Num:3622894,ID:00000000003747eee62d8c5ebdc0b0f770a246b18226ff654f798154a3a7df54, end blockId: Num:3624894,ID:0000000000374fbef92ec1504312d355a1fdbafb7a5551e81ff902d7c5591977, remain_num: 76799205
05:09:29.809 INFO [peerClient-23] net Receive message from channel: /139.180.184.220:18888, HelloMessage networkId: 11111, version: 1, code: 1
05:09:29.810 INFO [peerClient-23] net Add peer /139.180.184.220:18888, total channels: 10
05:09:29.810 INFO [peerClient-23] net Handshake failed /139.180.184.220:18888, code: 1, reason: TOO_MANY_PEERS, networkId: 11111, version: 1
05:09:29.810 INFO [peerClient-23] net Try to close channel: /139.180.184.220:18888, reason: TOO_MANY_PEERS
05:09:29.810 INFO [peerClient-23] net Close channel:/139.180.184.220:18888
05:09:29.810 INFO [peerClient-23] net Peer stats: channels 9, activePeers 9, active 9, passive 0
05:09:29.845 INFO [peerClient-20] net Receive message from channel: /5.178.101.91:18888, HelloMessage networkId: 11111, version: 1, code: 0
05:09:29.845 INFO [peerClient-20] net Add peer /5.178.101.91:18888, total channels: 10
05:09:29.845 INFO [peerClient-20] net Peer stats: channels 10, activePeers 10, active 10, passive 0
05:09:29.845 INFO [peerClient-20] net Send peer /5.178.101.91:18888 message type: P2P_HELLO
from: /103.101.207.204:18888
timestamp: 1786079368568
headBlockId: Num:80424099,ID:0000000004cb2ca31e81f008a1f1d7f8355aab38584c955a65a7d6c153a4d7c5
nodeType: 0
lowestBlockNum: 0
codeVersion:4.8.2.1

05:09:29.850 INFO [peerClient-31] net Send message to channel /95.217.62.144:18888, HelloMessage networkId: 11111, version: 1, code: 0
05:09:29.901 INFO [peerClient-29] net Receive message from channel: /3.216.116.227:18888, HelloMessage networkId: 11111, version: 1, code: 1
05:09:29.901 INFO [peerClient-29] net Add peer /3.216.116.227:18888, total channels: 11
05:09:29.901 INFO [peerClient-29] net Handshake failed /3.216.116.227:18888, code: 1, reason: TOO_MANY_PEERS, networkId: 11111, version: 1
05:09:29.901 INFO [peerClient-29] net Try to close channel: /3.216.116.227:18888, reason: TOO_MANY_PEERS
05:09:29.901 INFO [peerClient-29] net Close channel:/3.216.116.227:18888
05:09:29.901 INFO [peerClient-29] net Peer stats: channels 10, activePeers 10, active 10, passive 0
05:09:30.011 INFO [peerClient-18] net Receive message from peer: /13.212.39.197:18888, type: BLOCK
Num:80424114,ID:0000000004cb2cb25a4ff95ae9f297c83f44402f1e8e2c49073d780f56940ee4, trx size: 372

05:09:30.015 INFO [peerClient-14] net Close channel:/157.180.122.128:18888
05:09:30.015 INFO [peerClient-14] net Peer stats: channels 9, activePeers 9, active 9, passive 0
05:09:30.055 INFO [peerClient-30] net Receive message from channel: /65.108.108.100:18888, HelloMessage networkId: 11111, version: 1, code: 1
05:09:30.055 INFO [peerClient-30] net Add peer /65.108.108.100:18888, total channels: 10
05:09:30.055 INFO [peerClient-30] net Handshake failed /65.108.108.100:18888, code: 1, reason: TOO_MANY_PEERS, networkId: 11111, version: 1
05:09:30.055 INFO [peerClient-30] net Try to close channel: /65.108.108.100:18888, reason: TOO_MANY_PEERS
05:09:30.055 INFO [peerClient-30] net Close channel:/65.108.108.100:18888
05:09:30.055 INFO [peerClient-30] net Peer stats: channels 9, activePeers 9, active 9, passive 0
05:09:30.083 INFO [peerClient-20] net Close channel:/5.178.101.91:18888
05:09:30.083 INFO [peerClient-20] net Peer stats: channels 8, activePeers 8, active 8, passive 0
05:09:30.157 INFO [peerClient-26] net Receive message from peer: /168.119.30.61:18888, type: BLOCK_CHAIN_INVENTORY
size: 2001, first blockId: Num:80424099,ID:0000000004cb2ca31e81f008a1f1d7f8355aab38584c955a65a7d6c153a4d7c5, end blockId: Num:80426099,ID:0000000004cb3473fddd8783fabf079992016cbcb753e76f69c0a4669f820c96, remain_num: 1199248
05:09:30.164 INFO [peerClient-26] net Block Num:80424111,ID:0000000004cb2cafbca043327a611a3db6f73c4308dcdacd6aefe501fbc7ec17 from /168.119.30.61 is processed
05:09:30.164 INFO [peerClient-26] DB HeadNumber: 80424099, syncBeginNumber: 80424099, solidBlockNumber: 80424081.
05:09:30.165 INFO [peerClient-26] net Get block chain summary, low: 80424099, highNoFork: 80424111, high: 80424111, realHigh: 80426099
05:09:30.165 INFO [peerClient-26] net Send peer /168.119.30.61:18888 message type: SYNC_BLOCK_CHAIN
size: 11, start block: Num:80424099,ID:0000000004cb2ca31e81f008a1f1d7f8355aab38584c955a65a7d6c153a4d7c5, end block Num:80426099,ID:0000000004cb3473fddd8783fabf079992016cbcb753e76f69c0a4669f820c96
05:09:30.303 INFO [peerClient-26] net Send message to channel /34.87.86.135:18888, HelloMessage networkId: 11111, version: 1, code: 0
05:09:30.379 INFO [peerClient-26] net Receive message from channel: /34.87.86.135:18888, HelloMessage networkId: 11111, version: 1, code: 1
05:09:30.379 INFO [peerClient-26] net Add peer /34.87.86.135:18888, total channels: 9
05:09:30.379 INFO [peerClient-26] net Handshake failed /34.87.86.135:18888, code: 1, reason: TOO_MANY_PEERS, networkId: 11111, version: 1
05:09:30.379 INFO [peerClient-26] net Try to close channel: /34.87.86.135:18888, reason: TOO_MANY_PEERS
05:09:30.379 INFO [peerClient-26] net Close channel:/34.87.86.135:18888
05:09:30.379 INFO [peerClient-26] net Peer stats: channels 8, activePeers 8, active 8, passive 0
05:09:30.459 INFO [peerClient-27] net Send message to channel /188.40.83.153:18888, HelloMessage networkId: 11111, version: 1, code: 0
05:09:30.480 INFO [peerClient-17] net Receive message from peer: /87.251.64.114:18888, type: SYNC_BLOCK_CHAIN
size: 11, start block: Num:3622875,ID:00000000003747db0f60eb2ffa84f33c370ce7935b0989707b64093fe35bb813, end block Num:3624894,ID:0000000000374fbef92ec1504312d355a1fdbafb7a5551e81ff902d7c5591977
05:09:30.486 INFO [peerClient-17] net Send peer /87.251.64.114:18888 message type: BLOCK_CHAIN_INVENTORY
size: 2001, first blockId: Num:3624894,ID:0000000000374fbef92ec1504312d355a1fdbafb7a5551e81ff902d7c5591977, end blockId: Num:3626894,ID:000000000037578e39ab91ee851e4746255426775cb5e939a1649d4bd33743b2, remain_num: 76797205
05:09:30.631 INFO [peerClient-28] net Send message to channel /37.27.110.7:18888, HelloMessage networkId: 11111, version: 1, code: 0
05:09:30.737 INFO [peerClient-27] net Receive message from channel: /188.40.83.153:18888, HelloMessage networkId: 11111, version: 1, code: 0
05:09:30.738 INFO [peerClient-27] net Add peer /188.40.83.153:18888, total channels: 9
05:09:30.738 INFO [peerClient-27] net Peer stats: channels 9, activePeers 9, active 9, passive 0
05:09:30.738 INFO [peerClient-

[abn2357]

Based on the available logs, while it has already received block 80424114, blocks 80424112 and 80424113 have not yet been processed in sequence.This is why the chain is stuck at block 80424111. We initially suspect that the peer responsible for providing them encountered a timeout, disconnection, or another issue.
However, the current log is short. Therefore, we cannot determine which peer was requested to provide blocks 80424112 and 80424113, nor can we confirm the root cause.
Please provide the complete, unfiltered logs/tron.log covering 04:30:00–05:30:00 for further investigation.

[letmountain]

@abn2357

tron.log.zip

[halibobo1205]

@letmountain According to the logs, the block database already contains blocks 80424100–80424111, while properties.latestBlockHeaderNumber remains at 80424099. The 80424111 shown in the log is the highest block found in the block store, not the node's current head, which is still 80424099.

Because of this inconsistency, synchronization proceeds to block 80424112, but the block cannot be linked to the in-memory chain because its parent (80424111) is not present in KhaosDB. As a result, the node remains stuck at block 80424099.

Did the machine previously experience an unexpected shutdown, such as a power outage, host crash, or forced reboot?

If so, this is consistent with the database being left in a logically inconsistent state after an ungraceful machine shutdown. A normal restart will not safely repair the existing database. We recommend stopping the node and replacing the entire database with a fresh, internally consistent snapshot. Changing the checkpoint configuration will not repair the current database.

To improve recovery reliability and reduce the risk of cross-database inconsistency after future ungraceful machine shutdowns, we recommend enabling checkpoint V2 with synchronous checkpoint writes:

storage {
  checkpoint.version = 2
  checkpoint.sync = true
}

Please note that checkpoint.sync = true applies only to checkpoint V2 writes and may introduce some additional disk I/O latency, but it provides stronger checkpoint durability during unexpected machine shutdowns.

[letmountain]

@abn2357 It was a power outage, thank you. Then I'll unzip the snapshot and sync it again

[waynercheung]

@letmountain Thanks for confirming. The power outage explains it: LevelDB writes are not synchronous by default, so each store can be left at a different durable point, which is how the block store ended up ahead of properties.

To restore:

1. Download a fresh Mainnet Fullnode LevelDB snapshot – do not reuse the local archive, and make sure the engine matches storage.db.engine. (A Lite Fullnode snapshot is much smaller if you do not need full history.)
2. Verify it with its .md5sum file.
3. Stop the node completely, then extract into a clean, empty directory – never on top of the existing database.

Check df -h first: the archive, the new database, and the old one are each around 3 TB. The old output directory is not a usable rollback (it is stuck at 80424099 by the same deadlock), so save logs/tron.log and config.conf, then free it if you need the space.

To reduce the risk after future ungraceful shutdowns, set storage.checkpoint.version = 2 before starting the restored database. Treat this as a migration: once V2 checkpoint directories exist, reverting to V1 requires removing them on a stopped node. It is preventive only and will not repair the current database.

Please let us know once the node is syncing so we can close this issue.