tikv不断重启,查询某个 Raft 日志索引的 term 时没有找到对应日志,代码直接 unwrap(),触发 Rust panic

一个好的问题描述有利于社区小伙伴更快帮你定位到问题,高效解决你的问题

【TiDB 使用环境】生产环境
【TiDB 版本】5.7.25-TiDB-v6.5.0
【部署方式】k8s部署
【操作系统/CPU 架构/芯片详情】
【机器部署详情】CPU96核/内存64GB/磁盘4T
【集群数据量】
【集群节点数】3节点集群
【问题复现路径】
【遇到的问题:问题现象及影响】
有两个tikv节点不断重启,报错如下:
[2026/10/09 09:33:28.321 +00:00] [FATAL] [lib.rs:495] [“called Option::unwrap() on a None value”] [backtrace=" 0: tikv_util::set_panic_hook::{{closure}}\n at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tikv_util/src/lib.rs:494:18\n 1: <alloc::boxed::Box<F,A> as core::ops::function::Fn>::call\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:2032:9\n std::panicking::rust_panic_with_hook\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:692:13\n 2: std::panicking::begin_panic_handler::{{closure}}\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:577:13\n 3: std::sys_common::backtrace::__rust_end_short_backtrace\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:137:18\n 4: rust_begin_unwind\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:575:5\n 5: core::panicking::panic_fmt\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panicking.rs:65:14\n 6: core::panicking::panic\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panicking.rs:114:5\n 7: raftstore::store::entry_storage::EntryStorage<EK,ER>::term\n 8: <raftstore::store::peer_storage::PeerStorage<EK,ER> as raft::storage::Storage>::term\n at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/raftstore/src/store/peer_storage.rs:273:9\n raft::raft_log::RaftLog::term\n at rust/git/checkouts/raft-rs-42b8049ef2e3af07/2357cb2/src/raft_log.rs:131:18\n 9: raft::raft_log::RaftL
og::find_conflict_by_term\n at rust/git/checkouts/raft-rs-42b8049ef2e3af07/2357cb2/src/raft_log.rs:224:19\n 10: raft::raft::Raft::handle_append_response\n at rust/git/checkouts/raft-rs-42b8049ef2e3af07/2357cb2/src/raft.rs:1694:32\n raft::raft::Raft::step_leader\n at rust/git/checkouts/raft-rs-42b8049ef2e3af07/2357cb2/src/raft.rs:2133:17\n raft::raft::Raft::step\n at rust/git/checkouts/raft-rs-42b8049ef2e3af07/2357cb2/src/raft.rs:1498:38\n 11: raft::raw_node::RawNode::step\n at rust/git/checkouts/raft-rs-42b8049ef2e3af07/2357cb2/src/raw_node.rs:408:20\n 12: raftstore::store::peer::Peer<EK,ER>::step\n at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/raftstore/src/store/peer.rs:1915:9\n 13: raftstore::store::fsm::peer::PeerFsmDelegate<EK,ER,T>::on_raft_message\n at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/raftstore/src/store/fsm/peer.rs:2538:13\n raftstore::store::fsm::peer::PeerFsmDelegate<EK,ER,T>::handle_msgs\n at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/raftstore/src/store/fsm/peer.rs:613:37\n 14: <raftstore::store::fsm::store::RaftPoller<EK,ER,T> as batch_system::batch::PollHandler<raftstore::store::fsm::peer::PeerFsm<EK,ER>,raftstore::store::fsm::store::StoreFsm>>::handle_normal\n at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/raftstore/src/store/fsm/store.rs:942:9\n 15: batch_system::batch::Poller<N,C,Handler>::poll\n at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/batch-system/src/batch.rs:472:27\n 16: batch_system::batch::BatchSystem<N,C>::start_poller::{{closure}}\n at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/batch-system/src/batch.rs:600:17\n <std::thread::Builder as tikv_util::sys::thread
::StdThreadBuildWrapper>::spawn_wrapper::{{closure}}\n at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tikv_util/src/sys/thread.rs:415:23\n std::sys_common::backtrace::rust_begin_short_backtrace\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:121:18\n 17: std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:551:17\n <core::panic::unwind_safe::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:271:9\n std::panicking::try::do_call\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:483:40\n std::panicking::try\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:447:19\n std::panic::catch_unwind\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:137:14\n std::thread::Builder::spawn_unchecked::{{closure}}\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:550:30\n core::ops::function::FnOnce::call_once{{vtable.shim}}\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:513:5\n 18: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:2000:9\n <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once\n at rust/t
oolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:2000:9\n std::sys::unix::thread::thread::new::thread_start\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:108:17\n 19: \n 20: clone\n"] [location=/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/raftstore/src/store/entry_storage.rs:952] [thread_name=raftstore-1005-2]
【资源配置】进入到 TiDB Dashboard -集群信息 (Cluster Info) -主机(Hosts) 截图此页面
【复制黏贴 ERROR 报错的日志】
【其他附件:截图/日志/监控】

raft日志term对不上,多半engine损坏,坏副本用unsafe-recovery踢掉

这个 panic 是本地 raft log 出问题了——leader 处理 append response 的时候按 index 去查本地的 term,没查到(entry 已经被截断或者损坏了),unwrap 直接把 TiKV 打挂。

先别让 k8s 一直无脑重启,把 panic 日志和 TiKV 的 data 目录留个档。多数副本还在、集群还能读写的话,就走无损那条路:坏掉的 store 先用 pd-ctl store delete 下线,扩一台新 TiKV 进来,等 region 补齐了再换盘,别直接删 data 目录重建。要是多数副本已经不可用、集群都起不来,才上 Online Unsafe Recovery,这个是有损恢复,一定要一次性把所有要丢弃的 store 都列全,参考官方文档 Online Unsafe Recovery 使用文档 。

v6.5.0 有点老了,恢复完建议排期升到 v6.5 最新的补丁版本。生产环境又涉及数据风险,建议同时提个工单让官方支持同学一起看下。方便的话把完整 panic 日志、还有 Dashboard - 集群信息 - 主机 的截图补上~

是 raft-rs 在 term(idx) 里没查到 entry 就 unwrap,说明 raft log 的 truncated 和 last index 对不上,k8s 上多半是 PVC 复用或者掉电写坏了 raft engine。别让它空转重启:保住一个健康副本,坏的两个 store 清数据目录重新加入,让 PD 补副本。6.5.0 太老,新版本这里已经改成返回 Error 了。

生产环境适合部署在K8S中吗?我们之前做过关系型库的K8S部署性能压测,IO性能比物理机直接部署差太多了

一个是量大管饱,一个是性能目标,场景不一样的,运维的难度也不一样,有取舍了

1 个赞