【TiDB 使用环境】生产环境
【TiDB 版本】V7.5.5
【部署方式】机器部署
问题：目前发现TIDB集群中有两个TIKV节点region 非常不均衡。有两个TIKV节点的leader 非常失调，新加入一个TIKV 可以正常均衡leader 。但是有两个TIKV完全不反应。是什么问题？请大佬协助
pd-ctl store
Starting component ctl: /root/.tiup/components/ctl/v7.5.5/ctl pd -u http://10.40.67.167:13160 store
{
“count”: 10,
“stores”: [
{
“store”: {
“id”: 179486,
“address”: “10.40.67.182:36160”,
“labels”: [
{
“key”: “engine”,
“value”: “tiflash”
}
],
“version”: “v7.5.5”,
“peer_address”: “10.40.67.182:37160”,
“status_address”: “10.40.67.182:38160”,
“git_hash”: “3cad810ee920091cabccc706e74e4c2a55bed954”,
“start_timestamp”: 1753687074,
“deploy_path”: “/work/tidb-oltp-160-v7.5.5/deploy/tiflash-34160/bin/tiflash”,
“last_heartbeat”: 1789648850381094634,
“state_name”: “Up”
},
“status”: {
“capacity”: “8.725TiB”,
“available”: “4.136TiB”,
“used_size”: “323MiB”,
“leader_count”: 0,
“leader_weight”: 1,
“leader_score”: 0,
“leader_size”: 0,
“region_count”: 19,
“region_weight”: 1,
“region_score”: 1427,
“region_size”: 1427,
“learner_count”: 19,
“slow_score”: 1,
“slow_trend”: {
“cause_value”: 250011.38333333333,
“cause_rate”: 0,
“result_value”: 0,
“result_rate”: 0
},
“start_ts”: “2025-07-28T15:17:54+08:00”,
“last_heartbeat_ts”: “2026-09-17T20:40:50.381094634+08:00”,
“uptime”: “9989h22m56.381094634s”
}
},
{
“store”: {
“id”: 179515,
“address”: “10.40.64.247:17160”,
“version”: “7.5.5”,
“peer_address”: “10.40.64.247:17160”,
“status_address”: “10.40.64.247:18160”,
“git_hash”: “7a8c4c7c003b7d658075172cdc09622c922eb60d”,
“start_timestamp”: 1784546462,
“deploy_path”: “/work/tidb-oltp-160-v7.5.5/deploy/tikv-17160/bin”,
“last_heartbeat”: 1789648854830962455,
“state_name”: “Up”
},
“status”: {
“capacity”: “5.197TiB”,
“available”: “2.423TiB”,
“used_size”: “679GiB”,
“leader_count”: 11004,
“leader_weight”: 1,
“leader_score”: 11004,
“leader_size”: 722437,
“region_count”: 45795,
“region_weight”: 1,
“region_score”: 2986001,
“region_size”: 2986001,
“slow_score”: 1,
“slow_trend”: {
“cause_value”: 500000,
“cause_rate”: 0,
“result_value”: 1263.5,
“result_rate”: 0
},
“start_ts”: “2026-07-20T19:21:02+08:00”,
“last_heartbeat_ts”: “2026-09-17T20:40:54.830962455+08:00”,
“uptime”: “1417h19m52.830962455s”
}
},
{
“store”: {
“id”: 179516,
“address”: “10.40.64.245:17160”,
“version”: “7.5.5”,
“peer_address”: “10.40.64.245:17160”,
“status_address”: “10.40.64.245:18160”,
“git_hash”: “7a8c4c7c003b7d658075172cdc09622c922eb60d”,
“start_timestamp”: 1753687762,
“deploy_path”: “/work/tidb-oltp-160-v7.5.5/deploy/tikv-17160/bin”,
“last_heartbeat”: 1789648856878573907,
“state_name”: “Up”
},
“status”: {
“capacity”: “5.197TiB”,
“available”: “1.297TiB”,
“used_size”: “2.818GiB”,
“leader_count”: 36,
“leader_weight”: 1,
“leader_score”: 36,
“leader_size”: 313,
“region_count”: 58,
“region_weight”: 1,
“region_score”: 540706707.6491807,
“region_size”: 619,
“slow_score”: 1,
“slow_trend”: {
“cause_value”: 500000,
“cause_rate”: 0,
“result_value”: 237,
“result_rate”: 0
},
“start_ts”: “2025-07-28T15:29:22+08:00”,
“last_heartbeat_ts”: “2026-09-17T20:40:56.878573907+08:00”,
“uptime”: “9989h11m34.878573907s”
}
},
{
“store”: {
“id”: 1377505,
“address”: “10.40.67.156:17160”,
“version”: “7.5.5”,
“peer_address”: “10.40.67.156:17160”,
“status_address”: “10.40.67.156:18160”,
“git_hash”: “7a8c4c7c003b7d658075172cdc09622c922eb60d”,
“start_timestamp”: 1789640275,
“deploy_path”: “/work/tidb-oltp-160-v7.5.5/deploy/tikv-17160/bin”,
“last_heartbeat”: 1789648856249533173,
“state_name”: “Up”
},
“status”: {
“capacity”: “5.236TiB”,
“available”: “4.721TiB”,
“used_size”: “27.33GiB”,
“leader_count”: 1837,
“leader_weight”: 1,
“leader_score”: 1837,
“leader_size”: 119008,
“region_count”: 1841,
“region_weight”: 1,
“region_score”: 119340,
“region_size”: 119340,
“slow_score”: 1,
“slow_trend”: {
“cause_value”: 500000,
“cause_rate”: 0,
“result_value”: 300,
“result_rate”: 0
},
“start_ts”: “2026-09-17T18:17:55+08:00”,
“last_heartbeat_ts”: “2026-09-17T20:40:56.249533173+08:00”,
“uptime”: “2h23m1.249533173s”
}
},
{
“store”: {
“id”: 179485,
“address”: “10.40.67.220:36160”,
“labels”: [
{
“key”: “engine”,
“value”: “tiflash”
}
],
“version”: “v7.5.5”,
“peer_address”: “10.40.67.220:37160”,
“status_address”: “10.40.67.220:38160”,
“git_hash”: “3cad810ee920091cabccc706e74e4c2a55bed954”,
“start_timestamp”: 1753687067,
“deploy_path”: “/work/tidb-oltp-160-v7.5.5/deploy/tiflash-34160/bin/tiflash”,
“last_heartbeat”: 1789648850955215269,
“state_name”: “Up”
},
“status”: {
“capacity”: “8.725TiB”,
“available”: “4.346TiB”,
“used_size”: “250.2MiB”,
“leader_count”: 0,
“leader_weight”: 1,
“leader_score”: 0,
“leader_size”: 0,
“region_count”: 13,
“region_weight”: 1,
“region_score”: 1108,
“region_size”: 1108,
“learner_count”: 13,
“slow_score”: 1,
“slow_trend”: {
“cause_value”: 250012.16666666666,
“cause_rate”: 0,
“result_value”: 0,
“result_rate”: 0
},
“start_ts”: “2025-07-28T15:17:47+08:00”,
“last_heartbeat_ts”: “2026-09-17T20:40:50.955215269+08:00”,
“uptime”: “9989h23m3.955215269s”
}
},
{
“store”: {
“id”: 179517,
“address”: “10.40.65.183:17160”,
“version”: “7.5.5”,
“peer_address”: “10.40.65.183:17160”,
“status_address”: “10.40.65.183:18160”,
“git_hash”: “7a8c4c7c003b7d658075172cdc09622c922eb60d”,
“start_timestamp”: 1753687762,
“deploy_path”: “/work/tidb-oltp-160-v7.5.5/deploy/tikv-17160/bin”,
“last_heartbeat”: 1789648856312677159,
“state_name”: “Up”
},
“status”: {
“capacity”: “5.197TiB”,
“available”: “1.545TiB”,
“used_size”: “364.9GiB”,
“leader_count”: 11014,
“leader_weight”: 1,
“leader_score”: 11014,
“leader_size”: 718714,
“region_count”: 22816,
“region_weight”: 1,
“region_score”: 29920505.684189677,
“region_size”: 1488781,
“slow_score”: 1,
“slow_trend”: {
“cause_value”: 500000,
“cause_rate”: 0,
“result_value”: 1257,
“result_rate”: 0
},
“start_ts”: “2025-07-28T15:29:22+08:00”,
“last_heartbeat_ts”: “2026-09-17T20:40:56.312677159+08:00”,
“uptime”: “9989h11m34.312677159s”
}
},
{
“store”: {
“id”: 179518,
“address”: “10.40.64.249:17160”,
“version”: “7.5.5”,
“peer_address”: “10.40.64.249:17160”,
“status_address”: “10.40.64.249:18160”,
“git_hash”: “7a8c4c7c003b7d658075172cdc09622c922eb60d”,
“start_timestamp”: 1753687762,
“deploy_path”: “/work/tidb-oltp-160-v7.5.5/deploy/tikv-17160/bin”,
“last_heartbeat”: 1789648853500838082,
“state_name”: “Up”
},
“status”: {
“capacity”: “5.197TiB”,
“available”: “1.3TiB”,
“used_size”: “9.897GiB”,
“leader_count”: 32,
“leader_weight”: 1,
“leader_score”: 32,
“leader_size”: 613,
“region_count”: 38,
“region_weight”: 1,
“region_score”: 534272319.2504023,
“region_size”: 875,
“slow_score”: 1,
“slow_trend”: {
“cause_value”: 500000,
“cause_rate”: 0,
“result_value”: 239,
“result_rate”: 0
},
“start_ts”: “2025-07-28T15:29:22+08:00”,
“last_heartbeat_ts”: “2026-09-17T20:40:53.500838082+08:00”,
“uptime”: “9989h11m31.500838082s”
}
},
{
“store”: {
“id”: 694014,
“address”: “10.40.65.103:17160”,
“version”: “7.5.5”,
“peer_address”: “10.40.65.103:17160”,
“status_address”: “10.40.65.103:18160”,
“git_hash”: “7a8c4c7c003b7d658075172cdc09622c922eb60d”,
“start_timestamp”: 1780303994,
“deploy_path”: “/work/tidb-oltp-160-v7.5.5/deploy/tikv-17160/bin”,
“last_heartbeat”: 1789648856948386559,
“state_name”: “Up”
},
“status”: {
“capacity”: “3.489TiB”,
“available”: “1.042TiB”,
“used_size”: “566.2GiB”,
“leader_count”: 11004,
“leader_weight”: 1,
“leader_score”: 11004,
“leader_size”: 715308,
“region_count”: 37886,
“region_weight”: 1,
“region_score”: 15559326.877673388,
“region_size”: 2469522,
“slow_score”: 1,
“slow_trend”: {
“cause_value”: 500000,
“cause_rate”: 0,
“result_value”: 868,
“result_rate”: 0
},
“start_ts”: “2026-06-01T16:53:14+08:00”,
“last_heartbeat_ts”: “2026-09-17T20:40:56.948386559+08:00”,
“uptime”: “2595h47m42.948386559s”
}
},
{
“store”: {
“id”: 179487,
“address”: “10.40.66.221:36160”,
“labels”: [
{
“key”: “engine”,
“value”: “tiflash”
}
],
“version”: “v7.5.5”,
“peer_address”: “10.40.66.221:37160”,
“status_address”: “10.40.66.221:38160”,
“git_hash”: “3cad810ee920091cabccc706e74e4c2a55bed954”,
“start_timestamp”: 1753687081,
“deploy_path”: “/work/tidb-oltp-160-v7.5.5/deploy/tiflash-34160/bin/tiflash”,
“last_heartbeat”: 1789648853627891028,
“state_name”: “Up”
},
“status”: {
“capacity”: “8.658TiB”,
“available”: “3.824TiB”,
“used_size”: “89.25MiB”,
“leader_count”: 0,
“leader_weight”: 1,
“leader_score”: 0,
“leader_size”: 0,
“region_count”: 8,
“region_weight”: 1,
“region_score”: 427,
“region_size”: 427,
“learner_count”: 8,
“slow_score”: 1,
“slow_trend”: {
“cause_value”: 250011.625,
“cause_rate”: 0,
“result_value”: 0,
“result_rate”: 0
},
“start_ts”: “2025-07-28T15:18:01+08:00”,
“last_heartbeat_ts”: “2026-09-17T20:40:53.627891028+08:00”,
“uptime”: “9989h22m52.627891028s”
}
},
{
“store”: {
“id”: 179514,
“address”: “10.40.66.218:17160”,
“version”: “7.5.5”,
“peer_address”: “10.40.66.218:17160”,
“status_address”: “10.40.66.218:18160”,
“git_hash”: “7a8c4c7c003b7d658075172cdc09622c922eb60d”,
“start_timestamp”: 1753687762,
“deploy_path”: “/work/tidb-oltp-160-v7.5.5/deploy/tikv-17160/bin”,
“last_heartbeat”: 1789648859510887238,
“state_name”: “Up”
},
“status”: {
“capacity”: “5.197TiB”,
“available”: “1.56TiB”,
“used_size”: “445.7GiB”,
“leader_count”: 11006,
“leader_weight”: 1,
“leader_score”: 11006,
“leader_size”: 717002,
“region_count”: 29365,
“region_weight”: 1,
“region_score”: 1915047,
“region_size”: 1915047,
“slow_score”: 1,
“slow_trend”: {
“cause_value”: 500000,
“cause_rate”: 0,
“result_value”: 1669,
“result_rate”: 0
},
“start_ts”: “2025-07-28T15:29:22+08:00”,
“last_heartbeat_ts”: “2026-09-17T20:40:59.510887238+08:00”,
“uptime”: “9989h11m37.510887238s”
}
}
]
}
先看下这两个节点的 store 状态是否正常：pd-ctl store 查看对应 store 的 state_name 和 leader_count/region_count，重点确认是不是处于 Offline 或有 limit 限制。
常见原因有几种：
- 节点磁盘使用率超过
high-space-ratio（默认 0.8），PD 会停止往该节点调度，检查下df -h和 PD 配置。
节点都是正常的，没有offline和limit限制，奇怪就是在我磁盘使用率74%，没有触发低空间惩罚。但是这两个节点的得分很高。远超其他节点。
应该就是30%的问题，想验证的话，将245的可用空间释放到可用至少在30%+，看看是否会均衡。
AI了一下，感觉很有道理，得分的计算方式待验证。
只有磁盘可用空间低于 20%（使用率超 80%）时 PD 才不往该节点调度
楼主解决问题了没，也想知道为什么不平衡
region-score-formula-version 得分是V1还是V2的？两种计算得分的方式不同。
另外，这两个store是新扩的，还是历史的，region个数是由多到少？还是其它？
V2的算法，历史的，region原本也不知道是多少，目前一个月都是三十多个region。不过看了源码 ，通过源码V2得分计算 确实接近5亿得分，但是high-space-ratio这个参数设置的0.8，原本以为是超过这个参数才会出现得分惩罚，目前看是更具avg_available来触发得分惩罚。
这个参数只是用于限制是否会往store进行调度，region score是用来判断是否迁出。应该是available来触发惩罚的。
不过，AI算了下，说你的异常得分属于V1 的方式。。。
此话题已在最后回复的 7 天后被自动关闭。不再允许新回复。