TiDB 8.5.6出现tikv崩溃问题

一个好的问题描述有利于社区小伙伴更快帮你定位到问题,高效解决你的问题

【TiDB 使用环境】生产环境 /测试环境
【TiDB 版本】8.5.6
【部署方式】机器部署
【操作系统/CPU 架构/芯片详情】
【机器部署详情】CPU大小/内存大小/磁盘大小
【集群数据量】
【集群节点数】
【问题复现路径】做过哪些操作出现的问题
【遇到的问题:问题现象及影响】构建了一个主备集群,其中主集群是7.5.2,备集群是8.5.6版本,然后通过ticdc进行数据增量同步,目前是备集群在执行analyze的时候出现region is not available。每次analyze的时候触发 部分tikv节点重启。看到日志为[2026/09/16 15:11:20.408 +08:00] [FATAL] [lib.rs:481] [“index out of bounds: the len is 183969 but the index is 183969”] [backtrace=" 0: tikv_util::set_panic_hook::{{closure}}\n at /workspace/source/tikv/components/tikv_util/src/lib.rs:480:18\n 1: <alloc::boxed::Box<F,A> as core::ops::function::Fn>::call\n at /root/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:2029:9\n std::panicking::rust_panic_with_hook\n at /root/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:783:13\n 2: std::panicking::begin_panic_handler::{{closure}}\n at /root/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:657:13\n 3: std::sys_common::backtrace::__rust_end_short_backtrace\n at /root/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:171:18\n 4: rust_begin_unwind\n at /root/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:645:5\n 5: core::panicking::panic_fmt\n at /root/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panicking.rs:72:14\n 6: core::panicking::panic_bounds_check\n at /root/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panicking.rs:208:5\n 7: <tidb_query_datatype::codec::collation::collator::utf8mb4_uca::data_0900::Unicode0900 as tidb_query_datatype::codec::collation::collator::utf8mb4_uca::UnicodeVersion>::char_weight\n at /workspace/source/tikv/components/tidb_query_datatype/src/codec/collation/collator/utf8mb4_uca/data_0900.rs:25:17\n 8: <tidb_query_datatype::codec::collation::collator::utf8mb4_uca::CollatorUca as tidb_query_datatype::codec::collation::Collator>::char_weight\n at /workspace/source/tikv/components/tidb_query_datatype/src/codec/collation/collator/utf8mb4_uca/mod.rs:36:9\n <tidb_query_datatype::codec::collation::collator::utf8mb4_uca::CollatorUca as tidb_query_datatype::codec::collation::Collator>::write_sort_key\n at /workspace/source/tikv/components/tidb_query_datatype/src/codec/collation/collator/utf8mb4_uca/mod.rs:48:38\n 9: tikv::coprocessor::statistics::analyze::RowSampleBuilder<S,F>::collect_column_stats::{{closure}}\n at /workspace/source/tikv/src/coprocessor/statistics/analyze.rs:129:29\n 10: tikv::coprocessor::statistics::analyze_context::AnalyzeContext<S,F>::handle_full_sampling::{{closure}}\n at /workspace/source/tikv/src/coprocessor/statistics/analyze_context.rs:123:57\n <tikv::coprocessor::statistics::analyze_context::AnalyzeContext<S,F> as tikv::coprocessor::RequestHandler>::handle_request::{{closure}}\n at /workspace/source/tikv/src/coprocessor/statistics/analyze_context.rs:286:78\n
【资源配置】进入到 TiDB Dashboard -集群信息 (Cluster Info) -主机(Hosts) 截图此页面
【复制黏贴 ERROR 报错的日志】[2026/09/16 15:11:20.408 +08:00] [FATAL] [lib.rs:481] [“index out of bounds: the len is 183969 but the index is 183969”] [backtrace=" 0: tikv_util::set_panic_hook::{{closure}}\n at /workspace/source/tikv/components/tikv_util/src/lib.rs:480:18\n 1: <alloc::boxed::Box<F,A> as core::ops::function::Fn>::call\n at /root/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:2029:9\n std::panicking::rust_panic_with_hook\n at /root/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:783:13\n 2: std::panicking::begin_panic_handler::{{closure}}\n at /root/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:657:13\n 3: std::sys_common::backtrace::__rust_end_short_backtrace\n at /root/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:171:18\n 4: rust_begin_unwind\n at /root/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:645:5\n 5: core::panicking::panic_fmt\n at /root/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panicking.rs:72:14\n 6: core::panicking::panic_bounds_check\n at /root/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panicking.rs:208:5\n 7: <tidb_query_datatype::codec::collation::collator::utf8mb4_uca::data_0900::Unicode0900 as tidb_query_datatype::codec::collation::collator::utf8mb4_uca::UnicodeVersion>::char_weight\n at /workspace/source/tikv/components/tidb_query_datatype/src/codec/collation/collator/utf8mb4_uca/data_0900.rs:25:17\n 8: <tidb_query_datatype::codec::collation::collator::utf8mb4_uca::CollatorUca as tidb_query_datatype::codec::collation::Collator>::char_weight\n at /workspace/source/tikv/components/tidb_query_datatype/src/codec/collation/collator/utf8mb4_uca/mod.rs:36:9\n <tidb_query_datatype::codec::collation::collator::utf8mb4_uca::CollatorUca as tidb_query_datatype::codec::collation::Collator>::write_sort_key\n at /workspace/source/tikv/components/tidb_query_datatype/src/codec/collation/collator/utf8mb4_uca/mod.rs:48:38\n 9: tikv::coprocessor::statistics::analyze::RowSampleBuilder<S,F>::collect_column_stats::{{closure}}\n at /workspace/source/tikv/src/coprocessor/statistics/analyze.rs:129:29\n 10: tikv::coprocessor::statistics::analyze_context::AnalyzeContext<S,F>::handle_full_sampling::{{closure}}\n at /workspace/source/tikv/src/coprocessor/statistics/analyze_context.rs:123:57\n <tikv::coprocessor::statistics::analyze_context::AnalyzeContext<S,F> as tikv::coprocessor::RequestHandler>::handle_request::{{closure}}\n at /workspace/source/tikv/src/coprocessor/statistics/analyze_context.rs:286:78\n
【其他附件:截图/日志/监控】

1 个赞

原表结构中 prompt text COLLATE utf8mb4_0900_ai_ci DEFAULT NULL,
reply text COLLATE utf8mb4_0900_ai_ci DEFAULT NULL,排序集合为utf8mb4_0900_ai_ci ,analyze会 报错
image
,修改排序集合后通过analyze:


image

跳过text列收集是否会导致问题?如果没问题就别收集text字段的统计信息了。

主备版本差异问题吗

这个 FATAL 是 TiKV 内部 panic(index out of bounds),基本可以确定是 8.5.6 的 bug,不是配置问题。建议按下面几步处理:

  1. 先确认版本:select * from information_schema.cluster_info; 看 TiKV 具体 build 版本,8.5.6 是较新版本,建议升级到 8.5 最新 patch(8.5.7+)或 8.5.x 的稳定小版本,这类 panic 通常在新 patch 已修。

密密麻麻的看的脑仁疼。

主集群7.5.2版本低没问题,备集群8.5.6反而有问题

两个版本对 utf8mb4_0900_ai_ci 的 “实现深度” 不同:7.5.2 大概率是 “部分支持、实际降级处理”(analyze 根本不算 0900 权重,所以不崩);8.5.6 是 “完整支持、真正用上了 UCA 0900 权重表”,于是把实现里的越界 bug 暴露出来了。

难道是出bug了

lib.rs:481 只是 panic hook,越界在哪要看后面几帧。翻 tikv.log 里完整 backtrace,确认是不是 collation 那个栈。临时先只对非 text 列 analyze 绕过去。

text列00900排序集analyze会panic,改排序集

看起来像是8.5.6 的 TiKV 存在 collation/ANALYZE 路径的 Bug