TiDB集群抖动,TiKV出现报错Already many pending tasks in the read queue, task is rejected

TiDB集群的duration突然升高,导致业务抖动

【TiDB 使用环境】生产环境
【TiDB 版本】7.5.6
【部署方式】云上部署(什么云)/机器部署
【操作系统/CPU 架构/芯片详情】
【机器部署详情】CPU大小/内存大小/磁盘大小
【集群数据量】
【集群节点数】
【问题复现路径】做过哪些操作出现的问题
【遇到的问题:问题现象及影响】
【资源配置】进入到 TiDB Dashboard -集群信息 (Cluster Info) -主机(Hosts) 截图此页面
【复制黏贴 ERROR 报错的日志】

[2026/02/24 20:32:01.576 +08:00] [WARN] [read_pool.rs:315] ["Already many pending tasks in the read queue, task is rejected"] [busy_err
="reason: \"estimated wait time exceeds threshold\" estimated_wait_ms: 3480"] [busy_threshold=1s] [thread_id=10]
[2026/02/24 20:32:01.576 +08:00] [WARN] [read_pool.rs:315] ["Already many pending tasks in the read queue, task is rejected"] [busy_err
="reason: \"estimated wait time exceeds threshold\" estimated_wait_ms: 3480"] [busy_threshold=1s] [thread_id=7]
[2026/02/24 20:32:01.576 +08:00] [WARN] [read_pool.rs:315] ["Already many pending tasks in the read queue, task is rejected"] [busy_err
="reason: \"estimated wait time exceeds threshold\" estimated_wait_ms: 3480"] [busy_threshold=1s] [thread_id=7]
[2026/02/24 20:32:01.576 +08:00] [WARN] [read_pool.rs:315] ["Already many pending tasks in the read queue, task is rejected"] [busy_err
="reason: \"estimated wait time exceeds threshold\" estimated_wait_ms: 3480"] [busy_threshold=1s] [thread_id=8]
[2026/02/24 20:32:01.576 +08:00] [WARN] [read_pool.rs:315] ["Already many pending tasks in the read queue, task is rejected"] [busy_err
="reason: \"estimated wait time exceeds threshold\" estimated_wait_ms: 3480"] [busy_threshold=1s] [thread_id=8]
[2026/02/24 20:32:01.576 +08:00] [WARN] [read_pool.rs:315] ["Already many pending tasks in the read queue, task is rejected"] [busy_err
="reason: \"estimated wait time exceeds threshold\" estimated_wait_ms: 3480"] [busy_threshold=1s] [thread_id=7]
[2026/02/24 20:32:01.576 +08:00] [WARN] [read_pool.rs:315] ["Already many pending tasks in the read queue, task is rejected"] [busy_err
="reason: \"estimated wait time exceeds threshold\" estimated_wait_ms: 3480"] [busy_threshold=1s] [thread_id=9]
[2026/02/24 20:32:01.576 +08:00] [WARN] [read_pool.rs:315] ["Already many pending tasks in the read queue, task is rejected"] [busy_err
="reason: \"estimated wait time exceeds threshold\" estimated_wait_ms: 3480"] [busy_threshold=1s] [thread_id=9]
[2026/02/24 20:32:01.576 +08:00] [WARN] [read_pool.rs:315] ["Already many pending tasks in the read queue, task is rejected"] [busy_err
="reason: \"estimated wait time exceeds threshold\" estimated_wait_ms: 3480"] [busy_threshold=1s] [thread_id=10]
[2026/02/24 20:32:01.576 +08:00] [WARN] [read_pool.rs:315] ["Already many pending tasks in the read queue, task is rejected"] [busy_err
="reason: \"estimated wait time exceeds threshold\" estimated_wait_ms: 3480"] [busy_threshold=1s] [thread_id=10]
[2026/02/24 20:32:01.576 +08:00] [WARN] [read_pool.rs:315] ["Already many pending tasks in the read queue, task is rejected"] [busy_err
="reason: \"estimated wait time exceeds threshold\" estimated_wait_ms: 3480"] [busy_threshold=1s] [thread_id=11]
[2026/02/24 20:32:01.576 +08:00] [WARN] [read_pool.rs:315] ["Already many pending tasks in the read queue, task is rejected"] [busy_err
="reason: \"estimated wait time exceeds threshold\" estimated_wait_ms: 3480"] [busy_threshold=1s] [thread_id=8]
[2026/02/24 20:32:01.576 +08:00] [WARN] [read_pool.rs:315] ["Already many pending tasks in the read queue, task is rejected"] [busy_err="reason: \"estimated wait time exceeds threshold\" estimated_wait_ms: 3480"] [busy_threshold=1s] [thread_id=7]
[2026/02/24 20:32:01.576 +08:00] [WARN] [read_pool.rs:315] ["Already many pending tasks in the read queue, task is rejected"] [busy_err="reason: \"estimated wait time exceeds threshold\" estimated_wait_ms: 3480"] [busy_threshold=1s] [thread_id=9]
[2026/02/24 20:32:01.576 +08:00] [WARN] [read_pool.rs:315] ["Already many pending tasks in the read queue, task is rejected"] [busy_err="reason: \"estimated wait time exceeds threshold\" estimated_wait_ms: 3480"] [busy_threshold=1s] [thread_id=9]
[2026/02/24 20:32:01.576 +08:00] [WARN] [write.rs:855] ["[store 3686910660] async write too slow, write_kv: 0s, write_raft: 6.610306337s, send: 0.000038747s, callback: 0s thread: sync-writer"] [takes=6610] [thread_id=142]
[2026/02/24 20:32:01.576 +08:00] [WARN] [store.rs:1162] ["[store 3686910660] handle 16 pending peers include 16 ready, 8 entries, 8 messages and 0 snapshots"] [takes=6610] [thread_id=142]
[2026/02/24 20:32:01.576 +08:00] [WARN] [pd.rs:2015] ["scheduling store_heartbeat timeout, force report store slow score to pd."] [store_id=3686910660] [thread_id=27]
[2026/02/24 20:32:01.576 +08:00] [WARN] [read_pool.rs:315] ["Already many pending tasks in the read queue, task is rejected"] [busy_err="reason: \"estimated wait time exceeds threshold\" estimated_wait_ms: 3480"] [busy_threshold=1s] [thread_id=10]
[2026/02/24 20:32:01.576 +08:00] [WARN] [read_pool.rs:315] ["Already many pending tasks in the read queue, task is rejected"] [busy_err="reason: \"estimated wait time exceeds threshold\" estimated_wait_ms: 3480"] [busy_threshold=1s] [thread_id=8]
[2026/02/24 20:32:01.576 +08:00] [WARN] [read_pool.rs:315] ["Already many pending tasks in the read queue, task is rejected"] [busy_err="reason: \"estimated wait time exceeds threshold\" estimated_wait_ms: 3480"] [busy_threshold=1s] [thread_id=11]
[2026/02/24 20:32:01.576 +08:00] [WARN] [read_pool.rs:315] ["Already many pending tasks in the read queue, task is rejected"] [busy_err="reason: \"estimated wait time exceeds threshold\" estimated_wait_ms: 3480"] [busy_threshold=1s] [thread_id=10]
[2026/02/24 20:32:01.576 +08:00] [WARN] [read_pool.rs:315] ["Already many pending tasks in the read queue, task is rejected"] [busy_err="reason: \"estimated wait time exceeds threshold\" estimated_wait_ms: 3480"] [busy_threshold=1s] [thread_id=11]
[2026/02/24 20:32:01.576 +08:00] [WARN] [read_pool.rs:315] ["Already many pending tasks in the read queue, task is rejected"] [busy_err="reason: \"estimated wait time exceeds threshold\" estimated_wait_ms: 3480"] [busy_threshold=1s] [thread_id=11]
[2026/02/24 20:32:01.576 +08:00] [WARN] [read_pool.rs:315] ["Already many pending tasks in the read queue, task is rejected"] [busy_err="reason: \"estimated wait time exceeds threshold\" estimated_wait_ms: 3480"] [busy_threshold=1s] [thread_id=9]

【其他附件:截图/日志/监控】

这个过程 做过什么操作没有,看看 top sql 页面

数据库的一种自我保护措施,处理速度跟不上请求速度,导致任务过多队列资源用完了。看看集群dashboard面板上是否有过多慢sql或者是否出现了高并发,以及系统资源cpu和内存使用情况

处理速度跟不上了,资源紧张吗

如果是有慢SQL楼主应该是已经发现了,综合下来还是怀疑云平台抖动导致的,可以排查下当时的磁盘IO响应时间是否变化?

应该是当时请求变多,有个TiKV节点apply log 积压导致被限流,这个集群是从旧版本升级上来的 max-pending-peer-count 配置还是 16,新版我看已经变成64了,先调整这个参数了

1 个赞

TiKV 的读队列积压严重,预估等待时间超过阈值(3.48 秒),导致新的读任务被直接拒绝,进而引发集群抖动和业务延迟升高。

提供的信息有点少,不好判断

此话题已在最后回复的 7 天后被自动关闭。不再允许新回复。