tikv 疑似遇到BUG了，在恢复数据后无法重启

🌞 建议反馈 产品缺陷
, ,
1

背景：搭建1套新集群新集群版本是8.5.4,，用br恢复数据到新集群后，新集群tikv 无法重启，启动报错

[2026/07/16 00:00:29.198 +08:00] [FATAL] [lib.rs:481] ["failed to get timestamp from PD: Other(\"[components/pd_client/src/tso.rs:100]: Timestamp channel is dropped\")"] [backtrace="   0: tikv_util::set_panic_hook::{{closure}}\n             at /workspace/source/tikv/components/tikv_util/src/lib.rs:480:18\n   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call\n             at /root/.rustup/toolchains/nightly-2023-12-28-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:2029:9\n      std::panicking::rust_panic_with_hook\n             at /root/.rustup/toolchains/nightly-2023-12-28-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:783:13\n   2: std::panicking::begin_panic_handler::{{closure}}\n             at /root/.rustup/toolchains/nightly-2023-12-28-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:657:13\n   3: std::sys_common::backtrace::__rust_end_short_backtrace\n             at /root/.rustup/toolchains/nightly-2023-12-28-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:171:18\n   4: rust_begin_unwind\n             at /root/.rustup/toolchains/nightly-2023-12-28-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:645:5\n   5: core::panicking::panic_fmt\n             at /root/.rustup/toolchains/nightly-2023-12-28-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panicking.rs:72:14\n   6: core::result::unwrap_failed\n             at /root/.rustup/toolchains/nightly-2023-12-28-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/result.rs:1649:5\n   7: core::result::Result<T,E>::expect\n             at /root/.rustup/toolchains/nightly-2023-12-28-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/result.rs:1030:23\n      server::server::TikvServer<ER,F>::init\n             at /workspace/source/tikv/components/server/src/server.rs:413:25\n      server::server::run_impl\n             at /workspace/source/tikv/components/server/src/server.rs:168:20\n   8: server::server::run_tikv\n             at /workspace/source/tikv/components/server/src/server.rs:240:5\n      tikv_server::main\n             at /workspace/source/tikv/cmd/tikv-server/src/main.rs:249:31\n   9: core::ops::function::FnOnce::call_once\n             at /root/.rustup/toolchains/nightly-2023-12-28-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:250:5\n      std::sys_common::backtrace::__rust_begin_short_backtrace\n             at /root/.rustup/toolchains/nightly-2023-12-28-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:155:18\n  10: main\n  11: __libc_start_main\n  12: <unknown>\n"] [location=components/server/src/server.rs:413] [thread_name=main] [thread_id=1]

开了一下Github有类似的BUG
https://github.com/tikv/tikv/issues/18109?utm_source=chatgpt.com

image
image1193×458 41.2 KB

这个触发条件是什么啊，有哪位老师遇到过
这应该算是BUG了

搭建集群之前更改过时区，把utc时间改为了上海时区，是搭建之前，搭建之后没有改过

2

tikv 多大的盘，导入了多少数据量
是 SSD 的盘么

3

对比一下前后 2 套 TiDB 集群的资源配置差异，另外重点关注一下 PD 这里面的磁盘类型以及读写速度，先确认一下是不是因为 BR 恢复数据量太大，PD 节点的磁盘瓶颈导致的。可以先看看 PD 和 tikv 的日志对比一下？ PD 和 TiKV 日志都要看看？

4

PD 目前是什么状态

5

tikv 是2.5T的盘 这个是云服务器，aws 云服务器
军哥 帮忙内部发个帖子问下，这个BUG感觉有点严重啊，我这台服务器初始时区是UTC的，搭建集群之前，是搭建之前改为了上海时区，应该和这个没关系吧

6

pd 可以启动，pd 里面没有报错

7

辛苦提供下完整的 tikv panic 以及前后日志。
我让 AI 先分析下。

你 br 恢复，br 版本是 v8.5.4 的么？日志也上传一份呗。

8

军哥，这个不是tikv painc，是重启的时候报错了
tikv.log 就是直接显示这个报错，没有上下文了

[2026/07/16 00:00:23.581 +08:00] [INFO] [util.rs:809] ["connected to PD member"] [endpoints=http://192.168.170.92:2379] [thread_id=16]
[2026/07/16 00:00:23.582 +08:00] [INFO] [util.rs:248] ["heartbeat sender and receiver are stale, refreshing ..."] [thread_id=16]
[2026/07/16 00:00:23.582 +08:00] [INFO] [util.rs:261] ["buckets sender and receiver are stale, refreshing ..."] [thread_id=16]
[2026/07/16 00:00:23.582 +08:00] [INFO] [util.rs:280] ["acquire_token_buckets sender and receiver are stale, refreshing ..."] [thread_id=16]
[2026/07/16 00:00:23.582 +08:00] [INFO] [util.rs:303] ["update pd client"] [via=] [leader=http://192.168.170.92:2379] [prev_via=] [prev_leader=http://192.168.170.92:2379] [thread_id=16]
[2026/07/16 00:00:23.582 +08:00] [INFO] [util.rs:435] ["trying to update PD client done"] [spend=2.613368ms] [thread_id=16]
[2026/07/16 00:00:24.282 +08:00] [INFO] [resource_group.rs:151] ["add resource group"] [ru=2147483647] [name=default] [thread_id=20]
[2026/07/16 00:00:24.283 +08:00] [INFO] [service.rs:193] ["load controller config"] [config="RequestUnitConfig { read_base_cost: 0.125, read_cost_per_byte: 1.52587890625e-5, write_base_cost: 1.0, write_cost_per_byte: 0.0009765625, read_cpu_ms_cost: 0.3333333333333333 }"] [thread_id=21]
[2026/07/16 00:00:24.283 +08:00] [INFO] [service.rs:70] ["pd meta client creating watch stream."] [rev=21305] [path=resource_group/settings] [thread_id=20]
[2026/07/16 00:00:29.198 +08:00] [FATAL] [lib.rs:481] ["failed to get timestamp from PD: Other(\"[components/pd_client/src/tso.rs:100]: Timestamp channel is dropped\")"] [backtrace="   0: tikv_util::set_panic_hook::{{closure}}\n             at /workspace/source/tikv/components/tikv_util/src/lib.rs:480:18\n   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call\n             at /root/.rustup/toolchains/nightly-2023-12-28-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:2029:9\n      std::panicking::rust_panic_with_hook\n             at /root/.rustup/toolchains/nightly-2023-12-28-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:783:13\n   2: std::panicking::begin_panic_handler::{{closure}}\n             at /root/.rustup/toolchains/nightly-2023-12-28-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:657:13\n   3: std::sys_common::backtrace::__rust_end_short_backtrace\n             at /root/.rustup/toolchains/nightly-2023-12-28-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:171:18\n   4: rust_begin_unwind\n             at /root/.rustup/toolchains/nightly-2023-12-28-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:645:5\n   5: core::panicking::panic_fmt\n             at /root/.rustup/toolchains/nightly-2023-12-28-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panicking.rs:72:14\n   6: core::result::unwrap_failed\n             at /root/.rustup/toolchains/nightly-2023-12-28-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/result.rs:1649:5\n   7: core::result::Result<T,E>::expect\n             at /root/.rustup/toolchains/nightly-2023-12-28-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/result.rs:1030:23\n      server::server::TikvServer<ER,F>::init\n             at /workspace/source/tikv/components/server/src/server.rs:413:25\n      server::server::run_impl\n             at /workspace/source/tikv/components/server/src/server.rs:168:20\n   8: server::server::run_tikv\n             at /workspace/source/tikv/components/server/src/server.rs:240:5\n      tikv_server::main\n             at /workspace/source/tikv/cmd/tikv-server/src/main.rs:249:31\n   9: core::ops::function::FnOnce::call_once\n             at /root/.rustup/toolchains/nightly-2023-12-28-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:250:5\n      std::sys_common::backtrace::__rust_begin_short_backtrace\n             at /root/.rustup/toolchains/nightly-2023-12-28-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:155:18\n  10: main\n  11: __libc_start_main\n  12: <unknown>\n"] [location=components/server/src/server.rs:413] [thread_name=main] [thread_id=1]
9

br 恢复有没有报错

10

重启还是搞不定么？
我这边内部 AI 分析了下，历史上遇到的问题可能是 pd 这边有时钟跳转或者当时卡了下。

pd leader 的日志发一下。

11

如果是这个问题的话，应该还没有修：

image
image1920×719 39.7 KB

12

AI 综合分析

最终结论 + 对客回复

核心结论

tikv#18109 已知 bug，TiKV v8.5.4 受影响，目前无 fix。

TiKV 启动 init 阶段调用 block_on(pd_client.get_tso()).expect(...) 获取首次 TSO，该调用无重试机制。底层 tso-worker 线程因 TSO gRPC stream 异常退出后，所有 pending 的 oneshot 通道被 drop，报 “Timestamp channel is dropped” → 直接 FATAL panic。

该问题已被客户独立上报为 tikv#19845（2026-07-16），cross-ref 到 #18109。

触发条件

BR 全量恢复后 PD 处于 region meta 密集写入窗口，TSO gRPC stream 易被瞬时 reset，触发此 bug。jepsen 随机测试可在无任何外部操作下复现，时区变更与本 bug 无因果。

判断依据

  1. tikv#18109 open、无 fix、无 backport、may-affects-8.5 标签
  2. 客户已上报 tikv#19845，stack 完全一致（server.rs:413 / tso.rs:100 / lib.rs:481）
  3. 源码确认 init 阶段 TSO 调用无重试（expect() 直接 panic）
  4. #18109 jepsen 可稳定复现，无需时区变更等外部操作

建议对客回复

您遇到的问题对应 tikv/tikv#18109（已知 bug，截至当前仍 open，无 fix）。同问题您已独立上报为 tikv#19845，社区会跟进处理。

根因：TiKV 启动阶段首次获取 TSO 时，底层 TSO gRPC stream 因异常断开，但 init 阶段无重试机制，直接触发 FATAL panic。

触发场景：BR 全量恢复后 PD 处于 region 元数据密集加载窗口，TSO stream 易被瞬时 reset，属于该 bug 的典型触发条件。

临时恢复：确认 PD 健康后重新启动 TiKV 即可（等 PD 渡过 meta 加载窗口后通常能成功）。若反复失败，建议在 tikv#19845 下跟进社区修复进度。

1 个赞
13

好吧，谢谢军哥

14

看看 tikv 机器 dmesg