[Question] Questions regarding the recovery process for simulated production environment failures

[cui2022]

Problem Type (问题类型)

None

Before submit

• 我已经确认现有的 Issues 与 FAQ 中没有相同 / 重复问题 (I have confirmed and searched that there are no similar problems in the historical issue and documents)

Environment (环境信息)

• Server Version: 1.7.0
• Backend: Hstore
• OS: CentOS

Your Question (问题描述)

I pulled the latest source code and built a deployment package. Then I deployed a cluster on my servers, including 5 Store nodes, 3 PD nodes, and 3 Server nodes, with the replica number configured as 3. The deployment was done directly by extracting the packages, without using Docker.

I am currently simulating possible failures that may occur in a production environment and trying to determine a reasonable recovery procedure. The failure scenario I have simulated so far is as follows:

I disconnected the data disk of the second Store node while keeping its system disk running normally (the Store process is running on the system disk). I found that the Store node was still shown as UP, but queries involving data stored on the second Store node would fail.

This behavior is acceptable from my perspective because it indicates that the heartbeat mechanism only checks whether the Store process is alive and does not verify the integrity or availability of the underlying data. After that, I stopped the second Store process manually. The status of the second Store node changed from UP → OFFLINE, and cluster queries returned to normal.

I then checked the replica status. The shards that were originally leaders on the second Store node were switched to followers, and the corresponding shard status became OFFLINE. At this point, I believe the behavior is expected and there are no issues.

However, the next scenario I would like to simulate is a case where the data on the second Store node is permanently lost and cannot be recovered. In this situation, I expect to remove the offline second Store node from the current metadata and then restart this node as a new Store node (equivalent to replacing the failed node with a fresh node).

However, I am not sure what the correct procedure is to achieve this.

I have tried several approaches, but I was unable to make the replicas migrate from the failed second Store node to other available Store nodes.

For example, I tried calling the previously mentioned API:

/v1/store/{storeId}

and setting the second Store node status to Tombstone.

After changing the status to Tombstone, I can confirm through:

/v1/stores

that the Store status has changed to Tombstone.

However, when checking:

/v1/partitions

I found that the shards are still assigned to the Tombstone Store node, and no shard migration or replica recovery has occurred.

Therefore, I would like to know:

1. What is the recommended recovery workflow when a Store node's data is permanently lost?
2. How should an offline/tombstone Store node be removed from PD metadata?
3. How can the replicas on the failed Store node be automatically migrated to healthy Store nodes?
4. If replacing the failed Store node with a new empty node, what is the correct procedure to trigger replica rebuilding?

Thank you very much for your help.

[dosubot]

Hi @cui2022, thanks for the detailed scenario — this is exactly the kind of production-readiness testing that surfaces important operational questions. I dug into the PD codebase to answer each of your questions.

Setting a Store to Tombstone does trigger shard migration, but there are timing factors. When you POST /v1/store/{storeId} with Tombstone state, PD internally calls storeTurnoff() which removes the Store's shards from each shard group and then calls reallocShards() to redistribute replicas to healthy active Stores [1]. However, migration isn't instantaneous — there are two automatic mechanisms that complete the process:

1. Replica repair patrol (patrolPartitions()) runs every 60 seconds on the PD leader. It detects tombstoned Stores and repairs shard groups whose replica count is below the configured value [2].
2. Cluster-wide load balancing (balancePartitionShard()) runs with a 30-minute delay after a Store is removed, redistributing shards evenly across remaining active Stores [3].

If you checked /v1/partitions immediately after setting Tombstone, the migration may not have completed yet. Wait a few minutes and check again — the shard groups should update as the patrol task runs.

One prerequisite: PD validates that removing the Store won't drop the active Store count below minStoreCount. If it would, the Tombstone transition fails with LESS_ACTIVE_STORE [4]. Check the PD response and logs to confirm it actually accepted the state change.

Recommended recovery workflow for permanent data loss:

1. Set the failed Store to Tombstone:

   POST /v1/store/{storeId}
   Body: {"storeState": "Tombstone"}
   

2. Monitor migration progress — poll these endpoints over the next few minutes:

   GET /v1/partitions
   GET /v1/shardGroups
   

   Verify that shards previously on the failed Store are reassigned to other Stores.

3. (Optional) Trigger leader rebalancing:

   GET /v1/balanceLeaders
   

   This is the only manual rebalancing endpoint currently exposed [5].

4. Reinitialize the physical node as a new Store. A tombstoned Store ID is permanently decommissioned — it cannot re-register. You must wipe the Store's data directory and restart it so it registers with a fresh Store ID [6].

Important notes:

• store.max-down-time is not implemented in runtime code. Despite being documented and configurable (default 86400s / 24h), no PD code actually uses this timeout to trigger automatic recovery [7] [8]. Replica migration is only triggered by explicit Store state changes (OFFLINE → Tombstone), not by elapsed downtime. This is a known gap between documentation and implementation.

• There is no REST API for manual partition/shard migration. The internal methods movePartitionsShard() and reallocPartitionShards() exist in PartitionService [9] but are not exposed via REST endpoints. Migration happens automatically through Store state changes.

• Alternative: use the Exiting state for more graceful decommissioning. Setting a Store to Exiting triggers the same shard reallocation, but the Store remains in that state until it reports zero partitions via heartbeat, at which point PD automatically transitions it to Tombstone [10]. This is better suited for planned maintenance.

If you're still seeing shards assigned to the Tombstone Store after several minutes, please share the PD leader logs and the exact response from POST /v1/store/{storeId} — it's possible the transition was rejected due to a minStoreCount constraint or the PD node you're hitting isn't the leader.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[cui2022]

I provided the source code of pd-service to Claude Code and explained my issue. Claude Code analyzed the code and believes that there may be a bug in the replica migration logic when a Store node enters the Tombstone state. The issue may cause the old Store information in the PD metadata to not be updated correctly.

I am not sure whether this analysis can provide any useful information, but I hope it can help with the investigation.

Below is the analysis result from Claude Code:

---

Complete Analysis: Replica Migration Failure During Offline → Tombstone Transition

Initial Conditions

Assume the cluster state is:

partition shardCount = 3
Active Stores:

Store1 (Up)

Store2 (Up)
Store3:

Previously Offline, about to be changed to Tombstone
ShardGroup 0:

[

Store1 (Leader),

Store2 (Follower),

Store3 (Follower)

]

Note:

When Store3 was changed to Offline, updateStore() only called removeActiveStore() (line 306), but did not call storeTurnoff(). Therefore, the ShardGroup stored in RocksDB still contains Store3's shard information.

---

Phase 1: REST API Entry

User sends:

POST /v1/store/3

{

"storeState": "Tombstone"

}

StoreAPI.java:90-97

Metapb.Store lastStore = pdRestService.getStore(storeId);

// Store3, current state = Offline

Metapb.Store newStore = ... setState(Tombstone);
pdRestService.updateStore(newStore);

Then the request enters the core update logic.

---

Phase 2: updateStore()

updateStore() changes:

lastStore.state = Offline

store.state     = Tombstone

State validation

Because:

store.getState() == Tombstone

is true.

However:

lastStore.getState() == Up

is false because the previous state was already Offline.

Therefore, the minStoreCount validation is skipped and the update succeeds.

---

Phase 3: Tombstone Processing

The Store metadata is updated:

storeInfoMeta.updateStore(store);

The state transition is detected:

Offline -> Tombstone

Then:

storeInfoMeta.removeActiveStore(store);

storeTurnoff(store);

is executed.

---

Phase 4: First Replica Migration Attempt

Inside:

storeTurnoff(Store3)

The code finds:

ShardGroup:

[

Store1,

Store2,

Store3

]

Then it removes Store3:

[

Store1,

Store2

]

and calls:

reallocShards(filteredShardGroup)

---

Potential Bug 1: reallocShards() Early Return

Inside:

reallocShards()

Active stores:

[

Store1,

Store2

]

Configured shard count:

3

Then:

shardCount = Math.min(3, activeStores.size());

becomes:

shardCount = 2

The input shard list is already:

[

Store1,

Store2

]

Therefore:

shardCount == shards.size()

and execution reaches:

else {

return shards;

}

The function returns immediately.

As a result:

• storeInfoMeta.updateShardGroup() is not called

• fireChangeShard() is not called

• RocksDB still contains:

[

Store1,

Store2,

Store3

]

The migration attempt effectively does nothing.

---

Phase 5: Store Status Listener Triggered

After:

Offline -> Tombstone

the listener is triggered:

PartitionService.storeOffline(store);

This starts another migration attempt.

---

Phase 6: Second Migration Attempt

storeOffline() reads the ShardGroup from RocksDB:

[

Store1,

Store2,

Store3

]

Because the previous migration did not update metadata.

Then it calls:

reallocShards(shardGroup);

---

Potential Bug 2: Wrong Replica Removal

Now:

activeStores = [

Store1,

Store2

]

shards = [

Store1 Leader,

Store2 Follower,

Store3 Follower

]

Again:

shardCount = Math.min(3,2)

so:

shardCount = 2

The system needs to remove one replica.

The current logic:

if (iterator.next().getRole() != Leader) {

iterator.remove();

}

only checks whether the shard is a Leader.

It does not check:

• whether the Store is Tombstone

• whether the Store is Offline

• whether the Store is healthy

Because the iteration order is:

Store1 Leader

Store2 Follower

Store3 Follower

The result becomes:

Store1 Leader  (kept)

Store2 Follower (removed)
Store3 Tombstone (kept)

Therefore:

• A healthy Store2 is removed.

• The failed Tombstone Store3 remains in the replica list.

The metadata is then updated incorrectly:

[

Store1,

Store3

]

and a conf change is sent:

fireChangeShard([Store1, Store3])

which still includes the unavailable Store3.

---

Root Cause Summary

There appear to be two potential bugs:

Bug	Location	Problem
Bug 1	reallocShards()	When the filtered shard list size equals the calculated shard count, the function returns directly without updating PD metadata. The removed Tombstone store remains in RocksDB.
Bug 2	reallocShards()	Replica removal only avoids removing the Leader. It does not consider Store state, so a healthy follower can be removed while a Tombstone store remains.

---

Trigger Condition

The issue happens when:

activeStoreCount <= configuredShardCount

Because:

Math.min(configuredShardCount, activeStoreCount)

reduces the target replica count.

When the filtered shard list size equals this reduced value, the early return path is triggered.

---

[dosubot]

@cui2022, this is a very thorough analysis. I verified it against the actual source code, and both bugs appear to be real.

Bug 1 — reallocShards() early return without metadata update: Confirmed.

storeTurnoff() correctly filters out the tombstoned Store's shards before calling reallocShards() [1]. However, reallocShards() calculates shardCount = Math.min(configuredShardCount, activeStores.size()) and when the filtered shard list size already equals this value, it hits the else { return shards; } path at line 545 [2]. This returns without calling updateShardGroup() or fireChangeShard() [3], so the old ShardGroup (still containing the tombstoned Store) remains in RocksDB.

In your scenario with 5 Stores and shardCount=3, this would trigger when only 2 active Stores remain in a shard group after filtering — Math.min(3, activeStores) could reduce to 2, matching the filtered list size. With 4 remaining active Stores this specific early-return wouldn't trigger (since Math.min(3,4)=3 > 2), but the bug is latent for any cluster where activeStores.size() <= configuredShardCount.

Bug 2 — Replica removal ignores Store state: Confirmed.

The storeOffline() listener in PartitionService is triggered on the Tombstone state change [4] and calls shardOffline() → reallocPartitionShards(), which reads the ShardGroup from metadata without filtering out the tombstoned Store [5] [6]. When reallocShards() needs to reduce replicas, it iterates shards and removes the first non-Leader it finds [7]:

if (iterator.next().getRole() != Metapb.ShardRole.Leader) {
    iterator.remove();
    subCount--;
}

This checks only ShardRole, not Store state — so a healthy follower can be removed while the tombstoned follower is kept, exactly as your analysis describes.

Additional detail on the dual invocation: Both storeTurnoff() (called directly in updateStore() at line 313) and storeOffline() (called via listener at line 319) run for a Tombstone transition [8]. Because Bug 1 causes storeTurnoff() to skip the metadata update, the subsequent storeOffline() call reads stale metadata and hits Bug 2.

Regarding your questions:

1. Is this consistent with expected behavior? — No, the intent of the code is clearly to migrate replicas away from a tombstoned Store. The early return and state-unaware removal are bugs.

2. Known issue? — I did not find an existing issue or prior discussion tracking these specific bugs.

3. Workaround — There is no exposed REST API to manually trigger shard migration. The only rebalancing endpoint is GET /v1/balanceLeaders, which only rebalances leaders, not replicas [9]. The internal methods movePartitionsShard() and reallocPartitionShards() in PartitionService could fix the shard groups, but they are not exposed via REST [10]. PD's TaskScheduleService.patrolPartitions() runs every 60 seconds and attempts to repair shard groups with incorrect replica counts [11], but it may also be affected by the same reallocShards() bugs.

4. Manual repair — Currently no API supports this directly. A potential manual workaround would be to restart the tombstoned Store node with a fresh identity (new data directory) so it registers as a new active Store, which might allow the patrol task to add it as a new replica — but this doesn't clean up the stale shard group entries.

This looks like a legitimate bug worth filing a fix for. The reallocShards() method would need two changes: (1) the else branch should still update metadata if the input ShardGroup differs from what's stored, and (2) the replica removal logic should prefer removing shards on unhealthy/tombstoned Stores before removing healthy followers.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.