tikv故障启动不了 无法注册pd 有没有办法恢复数据

GangShen
Tue, 16 Mar 2021 03:01:14 +0000

PD 一开始是什么时间点宕机无法使用的？
pd-recovery 操作是什么时间点操作的？

目前看到最新的 TiKV 无法启动的报错是

[2021/03/16 10:14:48.123 +08:00] [FATAL] [lib.rs:481] ["entries[6:3766] is unavailable from storage, raft_id: 99011, region_id: 99009"] [backtrace="stack backtrace:\
   0: tikv_util::set_panic_hook::{{closure}}\
             at components/tikv_util/src/lib.rs:480\
   1: std::panicking::rust_panic_with_hook\
             at src/libstd/panicking.rs:475\
   2: rust_begin_unwind\
             at src/libstd/panicking.rs:375\
   3: std::panicking::begin_panic_fmt\
             at src/libstd/panicking.rs:326\
   4: raft::raft_log::RaftLog<T>::slice\
             at /home/jenkins/agent/workspace/build_tikv_multi_branch_v4.0.0/tikv/<::std::macros::panic macros>:9\
   5: raft::raft_log::RaftLog<T>::next_entries_since\
             at /rust/git/checkouts/raft-rs-841f8a6db665c5c0/b5f5830/src/raft_log.rs:362\
      raft::raw_node::Ready::new\
             at /rust/git/checkouts/raft-rs-841f8a6db665c5c0/b5f5830/src/raw_node.rs:129\
      raft::raw_node::RawNode<T>::ready_since\
             at /rust/git/checkouts/raft-rs-841f8a6db665c5c0/b5f5830/src/raw_node.rs:346\
      raftstore::store::peer::Peer::handle_raft_ready_append\
             at /home/jenkins/agent/workspace/build_tikv_multi_branch_v4.0.0/tikv/components/raftstore/src/store/peer.rs:1215\
   6: raftstore::store::fsm::peer::PeerFsmDelegate<T,C>::collect_ready\
             at /home/jenkins/agent/workspace/build_tikv_multi_branch_v4.0.0/tikv/components/raftstore/src/store/fsm/peer.rs:628\
      <raftstore::store::fsm::store::RaftPoller<T,C> as batch_system::batch::PollHandler<raftstore::store::fsm::peer::PeerFsm<engine_rocks::engine::RocksEngine>,raftstore::store::fsm::store::StoreFsm>>::handle_normal\
             at /home/jenkins/agent/workspace/build_tikv_multi_branch_v4.0.0/tikv/components/raftstore/src/store/fsm/store.rs:704\
   7: batch_system::batch::Poller<N,C,Handler>::poll\
             at /home/jenkins/agent/workspace/build_tikv_multi_branch_v4.0.0/tikv/components/batch-system/src/batch.rs:324\
   8: batch_system::batch::BatchSystem<N,C>::spawn::{{closure}}\
             at /home/jenkins/agent/workspace/build_tikv_multi_branch_v4.0.0/tikv/components/batch-system/src/batch.rs:398\
      std::sys_common::backtrace::__rust_begin_short_backtrace\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/sys_common/backtrace.rs:136\
   9: std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/thread/mod.rs:469\
      <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/panic.rs:318\
      std::panicking::try::do_call\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/panicking.rs:292\
      std::panicking::try\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8//src/libpanic_unwind/lib.rs:78\
      std::panic::catch_unwind\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/panic.rs:394\
      std::thread::Builder::spawn_unchecked::{{closure}}\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/thread/mod.rs:468\
      core::ops::function::FnOnce::call_once{{vtable.shim}}\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libcore/ops/function.rs:232\
10: <alloc::boxed::Box<F> as core::ops::function::FnOnce>::call_once\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/liballoc/boxed.rs:1022\
11: <alloc::boxed::Box<F> as core::ops::function::FnOnce>::call_once\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/liballoc/boxed.rs:1022\
      std::sys_common::thread::start_thread\
             at src/libstd/sys_common/thread.rs:13\
      std::sys::unix::thread::Thread::new::thread_start\
             at src/libstd/sys/unix/thread.rs:80\
  12: start_thread\
  13: __clone\
"] [location=/rust/git/checkouts/raft-rs-841f8a6db665c5c0/b5f5830/src/raft_log.rs:459] [thread_name=raftstore-5-0]

这个错误可以参考这个 FAQ 恢复一下

[FAQ] TiKV 启动报错 FATAL [region xxx] xxx entries[xx:xxx] is unavailable from storage TiDB 常见 FAQ

[问题澄清] 前提： 此案例为 2 副本 TiKV 启动报错 FATAL [region xxx] xxx entries[xx:xxx] is unavailable from storage ，如下： [image] [原因分析] 由于 sync-log 设置为 false，导致region同步时丢失数据 [解决方案] 拓扑介绍： store_221 down, store…

阅读完整话题