tikv 重启后一直 panic 无法稳定

小老板努力变强 · 2026 年7 月 1 日 10:06

一个好的问题描述有利于社区小伙伴更快帮你定位到问题，高效解决你的问题

【TiDB 使用环境】生产环境
【TiDB 版本】7.5.4
【部署方式】机器部署
【操作系统/CPU 架构/芯片详情】
【机器部署详情】CPU大小/内存大小/磁盘大小
【集群数据量】180T
【集群节点数】40+
【问题复现路径】
【遇到的问题：问题现象及影响】
tikv 一直重启，日志显示进行了一段时间的 raft 之后，然后直接 panic
【资源配置】进入到 TiDB Dashboard -集群信息 (Cluster Info) -主机(Hosts) 截图此页面
【复制黏贴 ERROR 报错的日志】
[2026/07/01 16:57:30.093 +08:00] [INFO] [raft.rs:1127] [“became follower at term 287”] [term=287] [raft_id=2425640462] [region_id=490446032] [thread_id=282]
[2026/07/01 16:57:30.112 +08:00] [FATAL] [lib.rs:510] [“range end index 66 out of range for slice of length 64”] [backtrace=" 0: tikv_util::set_panic_hook::{{closure}}\n at /workspace/source/tikv/components/tikv_util/src/lib.rs:509:18\n 1: <alloc::boxed::Box<F,A> as core::ops::function::Fn>::call\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:2032:9\n std::panicking::rust_panic_with_hook\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:692:13\n 2: std::panicking::begin_panic_handler::{{closure}}\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:579:13\n 3: std::sys_common::backtrace::rust_end_short_backtrace\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:137:18\n 4: rust_begin_unwind\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:575:5\n 5: core::panicking::panic_fmt\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panicking.rs:65:14\n 6: core::slice::index::slice_end_index_len_fail_rt\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/slice/index.rs:76:5\n 7: core::slice::index::slice_end_index_len_fail\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/slice/index.rs:69:9\n 8: <core::ops::range::Range as core::slice::index::SliceIndex<[T]>>::index\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/slice/index.rs:408:13\n core::slice::index::<impl core::ops::index::Index for [T]>::index\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/slice/index.rs:18:9\n tidb_query_expr::impl_like::like\n at /workspace/source/tikv/components/tidb_query_expr/src/impl_like.rs:50:42\n 9: <tidb_query_expr::types::function::Arg<Arg0,tidb_query_expr::types::function::Arg<Arg1,tidb_query_expr::types::function::Arg<Arg2_,tidb_query_expr::types::function::Null>>> as tidb_query_expr::impl_like::Like_Fn<C,CS>>::eval\n at /workspace/source/tikv/components/tidb_query_expr/src/impl_like.rs:7:1\n <tidb_query_expr::impl_like::Like_Evaluator<C,CS> as tidb_query_expr::types::function::Evaluator>::eval\n at /workspace/source/tikv/components/tidb_query_expr/src/impl_like.rs:7:1\n <tidb_query_expr::types::function::ArgConstructor<A,E> as tidb_query_expr::types::function::Evaluator>::eval\n at /workspace/source/tikv/components/tidb_query_expr/src/types/function.rs:261:17\n <tidb_query_expr::types::function::ArgConstructor<A,E> as tidb_query_expr::types::function::Evaluator>::eval\n at /workspace/source/tikv/components/tidb_query_expr/src/types/function.rs:245:17\n <tidb_query_expr::types::function::ArgConstructor<A,E> as tidb_query_expr::types::function::Evaluator>::eval\n at /workspace/source/tikv/components/tidb_query_expr/src/types/function.rs:245:17\n 10: tidb_query_expr::impl_like::like_fn_meta::run\n at /workspace/source/tikv/components/tidb_query_expr/src/impl_like.rs:7:1\n 11: tidb_query_expr::types::expr_eval::::eval_decoded\n at /workspace/source/tikv/components/tidb_query_expr/src/types/expr_eval.rs:264:31\n 12: tidb_query_expr::types::expr_eval::::eval\n at /workspace/source/tikv/components/tidb_query_expr/src/types/expr_eval.rs:174:9\n tidb_query_executors::selection_executor::BatchSelectionExecutor::handle_src_result\n at /workspace/source/tikv/components/tidb_query_executors/src/selection_executor.rs:82:19\n <tidb_query_executors::selection_executor::BatchSelectionExecutor as tidb_query_executors::interface::BatchExecutor>::next_batch::{{closure}}\n at /workspace/source/tikv/components/tidb_query_executors/src/selection_executor.rs:182:25\n <core::future::from_generator::GenFuture as core::future::future::Future>::poll\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/mod.rs:91:19\n 13: <core::pin::Pin

as core::future::future::Future>::poll\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/future.rs:124:9\n <tidb_query_common::execute_stats::WithSummaryCollector<C,T> as tidb_query_executors::interface::BatchExecutor>::next_batch::{{closure}}\n at /workspace/source/tikv/components/tidb_query_executors/src/interface.rs:114:54\n <core::future::from_generator::GenFuture as core::future::future::Future>::poll\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/mod.rs:91:19\n 14: <core::pin::Pin

as core::future::future::Future>::poll\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/future.rs:124:9\n <alloc::boxed::Box as tidb_query_executors::interface::BatchExecutor>::next_batch::{{closure}}\n at /workspace/source/tikv/components/tidb_query_executors/src/interface.rs:82:39\n <core::future::from_generator::GenFuture as core::future::future::Future>::poll\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/mod.rs:91:19\n 15: <core::pin::Pin

as core::future::future::Future>::poll\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/future.rs:124:9\n tidb_query_executors::BatchExecutorsRunner::internal_handle_request::{{closure}}\n at /workspace/source/tikv/components/tidb_query_executors/src/runner.rs:653:71\n <core::future::from_generator::GenFuture as core::future::future::Future>::poll\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/mod.rs:91:19\n 16: <tikv_util::quota_limiter::CpuObserveFuture as core::future::future::Future>::poll\n at /workspace/source/tikv/components/tikv_util/src/quota_limiter.rs:177:19\n tidb_query_executors::BatchExecutorsRunner::handle_request::{{closure}}\n at /workspace/source/tikv/components/tidb_query_executors/src/runner.rs:517:21\n <core::future::from_generator::GenFuture as core::future::future::Future>::poll\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/mod.rs:91:19\n <tikv::coprocessor::dag::BatchDagHandler as tikv::coprocessor::RequestHandler>::handle_request::{{closure}}\n at /workspace/source/tikv/src/coprocessor/dag/mod.rs:124:50\n <core::future::from_generator::GenFuture as core::future::future::Future>::poll\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/mod.rs:91:19\n 17: <core::pin::Pin

as core::future::future::Future>::poll\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/future.rs:124:9\n <tikv::coprocessor::interceptors::deadline::DeadlineChecker as core::future::future::Future>::poll\n at /workspace/source/tikv/src/coprocessor/interceptors/deadline.rs:37:9\n <tikv::coprocessor::interceptors::tracker::Tracker<F,E> as core::future::future::Future>::poll\n at /workspace/source/tikv/src/coprocessor/interceptors/tracker.rs:54:19\n <tikv::coprocessor::interceptors::concurrency_limiter::ConcurrencyLimiter<PF,F> as core::future::future::Future>::poll\n at /workspace/source/tikv/src/coprocessor/interceptors/concurrency_limiter.rs:112:15\n tikv::coprocessor::endpoint::Endpoint::handle_unary_request_impl::{{closure}}\n at /workspace/source/tikv/src/coprocessor/endpoint.rs:475:86\n <core::future::from_generator::GenFuture as core::future::future::Future>::poll\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/mod.rs:91:19\n <resource_metering::InTags as core::future::future::Future>::poll\n at /workspace/source/tikv/components/resource_metering/src/lib.rs:273:9\n <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll\n at /workspace/.cargo/registry/src/mirrors.tuna.tsinghua.edu.cn-df7c3c540f42cdbd/futures-util-0.3.15/src/future/future/map.rs:55:37\n <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll\n at /workspace/.cargo/registry/src/mirrors.tuna.tsinghua.edu.cn-df7c3c540f42cdbd/futures-util-0.3.15/src/lib.rs:93:13\n <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll\n at /workspace/.cargo/registry/src/mirrors.tuna.tsinghua.edu.cn-df7c3c540f42cdbd/futures-util-0.3.15/src/future/future/map.rs:55:37\n <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll\n at /workspace/.cargo/registry/src/mirrors.tuna.tsinghua.edu.cn-df7c3c540f42cdbd/futures-util-0.3.15/src/lib.rs:93:13\n 18: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll\n at /workspace/.cargo/registry/src/mirrors.tuna.tsinghua.edu.cn-df7c3c540f42cdbd/futures-util-0.3.15/src/future/future/map.rs:55:37\n <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll\n at /workspace/.cargo/registry/src/mirrors.tuna.tsinghua.edu.cn-df7c3c540f42cdbd/futures-util-0.3.15/src/lib.rs:93:13\n <resource_control::future::ControlledFuture as core::future::future::Future>::poll\n at /workspace/source/tikv/components/resource_control/src/future.rs:48:19\n 19: resource_control::future::with_resource_limiter::{{closure}}\n at /workspace/source/tikv/components/resource_control/src/future.rs:240:10\n <core::future::from_generator::GenFuture as core::future::future::Future>::poll\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/mod.rs:91:19\n <tracker::tls::TrackedFuture as core::future::future::Future>::poll::{{closure}}\n at /workspace/source/tikv/components/tracker/src/tls.rs:64:23\n std::local::LocalKey::try_with\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:446:16\n std::local::LocalKey::with\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:422:9\n <tracker::tls::TrackedFuture as core::future::future::Future>::poll\n at /workspace/source/tikv/components/tracker/src/tls.rs:62:9\n yatp::task::future::RawTask::poll\n at /workspace/.cargo/git/checkouts/yatp-e704b73c3ee279b6/5572a78/src/task/future.rs:59:9\n 20: yatp::task::future::TaskCell::poll\n at /workspace/.cargo/git/checkouts/yatp-e704b73c3ee279b6/5572a78/src/task/future.rs:103:9\n <yatp::task::future::Runner as yatp::pool::Runner>::handle\n at /workspace/.cargo/git/checkouts/yatp-e704b73c3ee279b6/5572a78/src/task/future.rs:387:20\n 21: <tikv_util::yatp_pool::YatpPoolRunner as yatp::pool::Runner>::handle\n at /workspace/source/tikv/components/tikv_util/src/yatp_pool/mod.rs:199:24\n <yatp::queue::multilevel::TrackedRunner as yatp::pool::Runner>::handle\n at /workspace/.cargo/git/checkouts/yatp-e704b73c3ee279b6/5572a78/src/queue/multilevel.rs:285:19\n 22: yatp::pool::worker::WorkerThread<T,R>::run\n at /workspace/.cargo/git/checkouts/yatp-e704b73c3ee279b6/5572a78/src/pool/worker.rs:48:13\n yatp::pool::builder::LazyBuilder::build::{{closure}}\n at /workspace/.cargo/git/checkouts/yatp-e704b73c3ee279b6/5572a78/src/pool/builder.rs:114:25\n std::sys_common::backtrace::rust_begin_short_backtrace\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:121:18\n 23: std::Builder::spawn_unchecked::{{closure}}::{{closure}}\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:551:17\n <core::panic::unwind_safe::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:271:9\n std::panicking::try::do_call\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:483:40\n std::panicking::try\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:447:19\n std::panic::catch_unwind\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:137:14\n std::Builder::spawn_unchecked::{{closure}}\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:550:30\n core::ops::function::FnOnce::call_once{{vtable.shim}}\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:513:5\n 24: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:2000:9\n <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:2000:9\n std::sys::unix::thread_start\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:108:17\n 25: start_thread\n 26: clone\n"] [location=components/tidb_query_expr/src/impl_like.rs:50] [thread_name=unified-read-pool-9] [thread_id=50]
【其他附件：截图/日志/监控】

菩提老祖 · 2026 年7 月 2 日 00:40

业务执行 WHERE 字符串字段 LIKE 'xxx' 模糊查询
TiDB 优化器把过滤逻辑下推到 TiKV，用批量向量化引擎提升查询性能
TiKV 在执行 LIKE 模式匹配时，内部预分配 64 字节固定缓冲区处理字符串
遇到特殊字符串（超长、多字节中文、异常转义符、脏编码数据），代码计算出的截取结束位置是 66，超过缓冲区最大长度 64
触发切片越界 → Rust 抛出 panic → TiKV 进程直接退出崩溃重启
所属 Region 刚刚切为 Follower，查询调度到该 TiKV 节点，刚好复现该 BUG
建议：临时关闭向量化引擎，是否会复现，验证bug
升级修复后的版本

克里克里克 · 2026 年7 月 2 日 00:58

AI反馈就是like 模糊查询下推到tikv触发的bug。
可以看看找到该SQL先加入黑名单，确认是否生效，如果确认问题要么就是更改SQL，要么就是升级版本了。

异乡的大人 · 2026 年7 月 2 日 01:19

使用豆包分析了一下，根治的方法是升级到v7.5.7 及以上，该版本合入了多项协处理器表达式边界检查修复，覆盖了多字节字符下的 LIKE 计算越界场景；

异乡的大人 · 2026 年7 月 2 日 01:19

长期规避建议

对前缀模糊匹配（%xxx）类的 LIKE 查询，本身无法利用索引，下推收益有限，可主动通过 SQL 改写禁止下推；
业务侧避免传入超长、包含特殊多字节字符的 LIKE 模式串，减少边界场景触发概率

wbslxw · 2026 年7 月 2 日 08:27

对前缀匹配 LIKE 'xxx%'：建立普通索引，改写避免全下推；
对后缀 / 全模糊 LIKE '%xxx'：这类无法走索引，主动在 TiDB 层过滤，禁止下推 TiKV；
限制传入超长、含异常多字节 / 脏编码的匹配字符串。

a大力啊 · 2026 年7 月 3 日 01:37

每天学习一下子

TiDBer_bawpguhQ · 2026 年7 月 3 日 08:23

先看 tikv.log 的 panic 栈，确认是磁盘/文件损坏还是 region 异常，必要时下线该 store 让 PD 补副本。

kang · 2026 年7 月 4 日 01:16

TiKV崩溃先检查TiKV日志的panic信息，确认是OOM、磁盘坏道还是bug触发。

小老板努力变强 · 2026 年7 月 4 日 04:52

差不多也是用 AI 去看了下是这类原因，不过还没找到具体引发崩溃的 SQL

WalterWj · 2026 年7 月 7 日 02:40

AI 解析：

社区问题分析报告：TiKV LIKE 表达式 Panic

问题链接: tikv 重启后一直 panic 无法稳定

1. 问题定性

TiKV 在 coprocessor 执行下推的 LIKE 表达式时 panic，调用链：

impl_like::like (impl_like.rs:50)
  → Like_Fn::eval → BatchSelectionExecutor::handle_src_result
    → coprocessor::endpoint::Endpoint::handle_unary_request_impl

Panic 原因：range end index 66 out of range for slice of length 64

2. 根因

impl_like.rs 中 CS::decode_one() 返回的字符偏移量超出剩余 slice 长度。

LIKE 函数在匹配循环中调用 CS::decode_one(&pattern[px..]) 获取字符及字节偏移 poff，然后执行 &pattern[px..px + poff]。当 BIT 列类型的原始字节被 collation decoder 误解码为 UTF-8 时，decode_one 返回的偏移量超出 slice 边界，触发 index out of range。

已知触发条件：

对 BIT(N) 列（N ≥ 30，约 4 字节）使用 LIKE / NOT LIKE 操作
典型 SQL 模式：WHERE c1 NOT LIKE c1、UPDATE ... WHERE bit_col LIKE ...
LIKE 表达式被 下推到 TiKV coprocessor 执行

3. 上游 Issue（均未修复）

Issue	描述	状态
tikv/tikv#19010	TiKV panic in `impl_like::LikeFn::eval`，SQLancer 发现	OPEN，无 fix PR
pingcap/tidb#64318	BIT(N) NOT LIKE 触发 panic，含最小复现	OPEN，无 fix PR

最小复现（v8.5.3）：

CREATE TABLE t (c1 BIT(30));
INSERT INTO t VALUES (b'100001001101010001000010001111');
UPDATE t SET c1 = b'111001111010010110110010001011' WHERE c1 NOT LIKE c1;
-- [FATAL] range end index 2 out of range for slice of length 1

4. 版本影响

v7.5.4（客户版本）：impl_like.rs 代码存在（2019 年起），BIT 列 + LIKE 下推路径可触发
v7.5.6 release notes 提到一个相关修复 converting data from BIT type to CHAR type might cause TiKV panics（#56494），但根因不同
v8.5.3 确认复现，v8.5.0 未复现 — bug 在 v8.5.0~v8.5.3 之间引入/暴露
该 bug 目前在所有版本均未修复（上游 issue 仍 OPEN）

5. Workaround（推荐）

方案	操作	效果
P0：避免 BIT 列 LIKE	将 BIT 列转为 `CAST(c1 AS CHAR)` 或 `CAST(c1 AS UNSIGNED)` 再做比较	完全规避
P0：改写 SQL	`WHERE c1 NOT LIKE c1` → `WHERE c1 <> c1`（语义不同需验证）	避免下推
P1：强制 TiDB 侧计算	`WHERE CAST(bit_col AS CHAR) LIKE '%pattern%'`	CAST 阻止 TiKV 侧下推

对客户的建议：

排查业务中是否有对 BIT 列使用 LIKE 的 SQL（慢查询日志 + general log）
找到后改写为 CAST 或其他等价写法
这是 FATAL 级别 panic，每次触发都会重启 TiKV，需要尽快定位并改写相关 SQL

6. 对客回复建议

您好，经分析，TiKV panic 发生在 coprocessor 执行 LIKE 表达式时（impl_like.rs:50）。已知根因：当 LIKE 表达式作用于 BIT 类型列 时，TiKV 的 collation 解码器会误读字节偏移，导致 slice 越界 panic。这是上游已知 bug（tikv#19010、tidb#64318），目前尚未修复。

临时解决方案：请排查业务 SQL 中是否有对 BIT 列使用 LIKE / NOT LIKE 的语句，将其改写为 CAST(bit_col AS CHAR) LIKE ... 或使用其他等价比较方式。找到并改写相关 SQL 后，TiKV 即可恢复稳定。