我部署了一个10台机器的集群5.0.2的版本,最近发现tikv-server机器32G内存一直95%以上,打算升级一下数据库,没有快照,直接把数据库升崩了,退也退不了。。

集群情况
[tidb@tslink01 tidb-zjy]$ tiup cluster display tidb-zjy
Cluster type: tidb
Cluster name: tidb-zjy
Cluster version: v5.0.3
Deploy user: tidb
SSH type: builtin
Dashboard URL: http://10.2.29.16:2379/dashboard
Dashboard URLs: http://10.2.29.16:2379/dashboard
Grafana URL: http://10.2.29.4:3000
ID Role Host Ports OS/Arch Status Data Dir Deploy Dir


10.2.29.4:9093 alertmanager 10.2.29.4 9093/9094 linux/x86_64 Down /data/tidb-data/alertmanager-9093 /data/tidb-deploy/alertmanager-9093
10.2.29.4:3000 grafana 10.2.29.4 3000 linux/x86_64 Down - /data/tidb-deploy/grafana-3000
10.2.29.15:2379 pd 10.2.29.15 2379/2380 linux/x86_64 Up /data/tidb-data/pd-2379 /data/tidb-deploy/pd-2379
10.2.29.16:2379 pd 10.2.29.16 2379/2380 linux/x86_64 Up|UI /data/tidb-data/pd-2379 /data/tidb-deploy/pd-2379
10.2.29.17:2379 pd 10.2.29.17 2379/2380 linux/x86_64 Up|L /data/tidb-data/pd-2379 /data/tidb-deploy/pd-2379
10.2.29.4:9090 prometheus 10.2.29.4 9090/9115/9100 linux/x86_64 Down /data/tidb-data/prometheus-9090 /data/tidb-deploy/prometheus-9090
10.2.29.11:4000 tidb 10.2.29.11 4000/10080 linux/x86_64 Down - /data/tidb-deploy/tidb-4000
10.2.29.12:4000 tidb 10.2.29.12 4000/10080 linux/x86_64 Down - /data/tidb-deploy/tidb-4000
10.2.29.13:4000 tidb 10.2.29.13 4000/10080 linux/x86_64 Down - /data/tidb-deploy/tidb-4000
10.2.29.21:20160 tikv 10.2.29.21 20160/20180 linux/x86_64 Down /data/tidb-data/tikv-20160 /data/tidb-deploy/tikv-20160
10.2.29.22:20161 tikv 10.2.29.22 20161/20181 linux/x86_64 Down /data/tidb-data/tikv-20161 /data/tidb-deploy/tikv-20161
10.2.29.23:20160 tikv 10.2.29.23 20160/20180 linux/x86_64 Down /data/tidb-data/tikv-20160 /data/tidb-deploy/tikv-20160

【部署方式】私有云本地部署
【操作系统/CPU 架构/芯片详情】 centos7.9 x86
【机器部署详情】
TiDB(接入语法协议服务器) tidb01 CentOS Linux7.8 16核、32G 512GB 10.2.29.11
tidb02 CentOS Linux7.8 16核、32G 512GB 10.2.29.12
tidb03 CentOS Linux7.8 16核、32G 512GB 10.2.29.13
PD(原数据管理服务器) pd01/tiflash CentOS Linux7.8 16核、32G 512GB 10.2.29.15
pd02/tiflash CentOS Linux7.8 16核、32G 512GB 10.2.29.16
pd03 CentOS Linux7.8 16核、32G 512GB 10.2.29.17
TiKV(存储服务) tikv01 CentOS Linux7.8 16核、32G 512GB 10.2.29.21
tikv02 CentOS Linux7.8 16核、32G 512GB 10.2.29.22
tikv03 CentOS Linux7.8 16核、32G 512GB 10.2.29.23
Monitoring & Grafana(监控) “Monitoring & Grafana
monitoring&grafana” CentOS Linux7.8 8核、16G 512GB 10.2.29.4
【集群数据量】
【集群节点数】
【问题复现路径】发现tikv-server三台机的内存一直95%以上。想升级一下版本。我最开始执行升级到8.5的版本,发现升完,启不来了,想回退也不行,现在数据库崩了。哎。
启动tikv-server报错
bin/pd-server: /lib64/libc.so.6: version `GLIBC_2.28’ not found (required by bin/pd-server)
发现是版本问题,后来又升到8.5.1,
[2026/07/28 20:21:29.288 +08:00] [FATAL] [lib.rs:479] [“failed to load_latest_options "Invalid argument: Could not find option: : enable_multi_thread_write"”] [backtrace=" 0: tikv_util::set_panic_hook::{{closure}}\n at /workspace/source/tikv/components/tikv_util/src/lib.rs:478:18\n 1: <alloc::boxed::Box<F,A> as core::ops::function::Fn>::call\n at /root/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:2029:9\n std::panicking::rust_panic_with_hook\n at /root/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:783:13\n 2: std::panicking::begin_panic_handler::{{closure}}\n at /root/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:657:13\n 3: std::sys_common::backtrace::__rust_end_short_backtrace\n at /root/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:171:18\n 4: rust_begin_unwind\n at /root/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:645:5\n 5: core::panicking::panic_fmt\n at /root/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panicking.rs:72:14\n 6: engine_rocks::util::new_engine_opt::{{closure}}\n at /workspace/source/tikv/components/engine_rocks/src/util.rs:82:33\n core::result::Result<T,E>::unwrap_or_else\n at /root/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/result.rs:1426:23\n engine_rocks::util::new_engine_opt\n at /workspace/source/tikv/components/engine_rocks/src/util.rs:82:14\n 7: <raft_log_engine::engine::RaftLogEngine as server::common::ConfiguredRaftEngine>::build\n at /workspace/source/tikv/components/server/src/common.rs:833:26\n 8: server::server::TikvServer<CER,F>::init_raw_engines\n at /workspace/source/tikv/components/server/src/server.rs:1677:46\n server::server::run_impl\n at /workspace/source/tikv/components/server/src/server.rs:177:53\n 9: server::server::run_tikv\n at /workspace/source/tikv/components/server/src/server.rs:237:5\n tikv_server::main\n at /workspace/source/tikv/cmd/tikv-server/src/main.rs:249:31\n 10: core::ops::function::FnOnce::call_once\n at /root/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:250:5\n std::sys_common::backtrace::__rust_begin_short_backtrace\n at /root/.rustup/toolchains/nightly-2023-12-28-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:155:18\n 11: main\n 12: __libc_start_main\n 13: \n"] [location=components/engine_rocks/src/util.rs:82] [thread_name=main] [thread_id=1]
启动失败提示如下:

  • [ Serial ] - StartCluster
    Starting component pd
    Starting instance 10.2.29.15:2379
    Starting instance 10.2.29.17:2379
    Starting instance 10.2.29.16:2379
    Start instance 10.2.29.15:2379 success
    Start instance 10.2.29.17:2379 success
    Start instance 10.2.29.16:2379 success
    Starting component tikv
    Starting instance 10.2.29.22:20161
    Starting instance 10.2.29.21:20160
    Starting instance 10.2.29.23:20160

Error: failed to start tikv: failed to start: 10.2.29.22 tikv-20161.service, please check the instance’s log(/data/tidb-deploy/tikv-20161/log) for more detail.: timed out waiting for port 20161 to be started after 2m0s
后面就废了。

版本兼容性问题,应该是版本5中的集群上配置了参数“enable_multi_thread_write”,但是高版本已经遗弃了, 修改 TiKV 配置文件 (tikv.toml )后再重启试试

学习了

tikv.toml里只有几行注释内容,什么都没有。

我想退回去,发现不管怎么样tikv已经启不来了:
[2026/07/30 01:20:40.865 +08:00] [INFO] [config.rs:891] [“data dir”] [mount_fs=“FsInfo { tp: "ext4", opts: "rw,noatime,nodelalloc,data=ordered", mnt_dir: "/data", fsname: "/dev/mapper/datavg-datalv" }”] [data_path=/data/tidb-data/tikv-20160/raft]
[2026/07/30 01:20:40.865 +08:00] [WARN] [config.rs:894] [“not on SSD device”] [data_path=/data/tidb-data/tikv-20160/raft]
[2026/07/30 01:20:40.865 +08:00] [INFO] [server.rs:338] [“using config”] [config=“{"log-level":"info","log-file":"","log-format":"text","log-rotation-timespan":"0s","log-rotation-size":"300MiB","slow-log-file":"","slow-log-threshold":"1s","panic-when-unexpected-key-or-data":false,"abort-on-panic":false,"memory-usage-limit":25190290773,"memory-usage-high-water":0.9,"log":{"level":"info","format":"text","enable-timestamp":true,"file":{"filename":"/data/tidb-deploy/tikv-20160/log/tikv.log","max-size":300,"max-days":0,"max-backups":0}},"quota":{"foreground-cpu-time":0,"foreground-write-bandwidth":"0KiB","foreground-read-bandwidth":"0KiB","max-delay-duration":"500ms"},"readpool":{"unified":{"min-thread-count":1,"max-thread-count":12,"stack-size":"10MiB","max-tasks-per-worker":2000},"storage":{"use-unified-pool":true,"high-concurrency":8,"normal-concurrency":8,"low-concurrency":8,"max-tasks-per-worker-high":2000,"max-tasks-per-worker-normal":2000,"max-tasks-per-worker-low":2000,"stack-size":"10MiB"},"coprocessor":{"use-unified-pool":true,"high-concurrency":12,"normal-concurrency":12,"low-concurrency":12,"max-tasks-per-worker-high":2000,"max-tasks-per-worker-normal":2000,"max-tasks-per-worker-low":2000,"stack-size":"10MiB"}},"server":{"addr":"0.0.0.0:20160","advertise-addr":"10.2.29.21:20160","status-addr":"0.0.0.0:20180","advertise-status-addr":"10.2.29.21:20180","status-thread-pool-size":1,"max-grpc-send-msg-len":10485760,"raft-client-grpc-send-msg-buffer":524288,"raft-client-queue-size":8192,"raft-msg-max-batch-size":128,"grpc-compression-type":"none","grpc-concurrency":5,"grpc-concurrent-stream":1024,"grpc-raft-conn-num":1,"grpc-memory-pool-quota":9223372036854775807,"grpc-stream-initial-window-size":"2MiB","grpc-keepalive-time":"10s","grpc-keepalive-timeout":"3s","concurrent-send-snap-limit":32,"concurrent-recv-snap-limit":32,"end-point-recursion-limit":1000,"end-point-stream-channel-size":8,"end-point-batch-row-limit":64,"end-point-stream-batch-row-limit":128,"end-point-enable-batch-if-possible":true,"end-point-request-max-handle-duration":"1m","end-point-max-concurrency":16,"end-point-perf-level":0,"snap-max-write-bytes-per-sec":"100MiB","snap-max-total-size":"0KiB","stats-concurrency":1,"heavy-load-threshold":75,"heavy-load-wait-duration":null,"enable-request-batch":true,"background-thread-count":2,"end-point-slow-log-threshold":"1s","forward-max-connections-per-address":4,"reject-messages-on-memory-ratio":0.2,"labels":{}},"storage":{"data-dir":"/data/tidb-data/tikv-20160","gc-ratio-threshold":1.1,"max-key-size":8192,"scheduler-concurrency":524288,"scheduler-worker-pool-size":8,"scheduler-pending-write-threshold":"100MiB","reserve-space":"5GiB","enable-async-apply-prewrite":false,"api-version":1,"enable-ttl":false,"background-error-recovery-window":"1h","ttl-check-poll-interval":"12h","flow-control":{"enable":true,"soft-pending-compaction-bytes-limit":"192GiB","hard-pending-compaction-bytes-limit":"1TiB","memtables-threshold":5,"l0-files-threshold":20},"block-cache":{"shared":true,"capacity":"14414MiB","num-shard-bits":6,"strict-capacity-limit":false,"high-pri-pool-ratio":0.8,"memory-allocator":"nodump"},"io-rate-limit":{"max-bytes-per-sec":"0KiB","mode":"write-only","strict":false,"foreground-read-priority":"high","foreground-write-priority":"high","flush-priority":"high","level-zero-compaction-priority":"medium","compaction-priority":"low","replication-priority":"high","load-balance-priority":"high","gc-priority":"high","import-priority":"medium","export-priority":"medium","other-priority":"high"}},"pd":{"endpoints":["10.2.29.15:2379","10.2.29.16:2379","10.2.29.17:2379"],"retry-interval":"300ms","retry-max-count":9223372036854775807,"retry-log-every":10,"update-interval":"10m","enable-forwarding":false},"metric":{"job":"tikv"},"raftstore":{"prevote":true,"raftdb-path":"/data/tidb-data/tikv-20160/raft","capacity":"0KiB","raft-base-tick-interval":"1s","raft-heartbeat-ticks":2,"raft-election-timeout-ticks":10,"raft-min-election-timeout-ticks":10,"raft-max-election-timeout-ticks":20,"raft-max-size-per-msg":"1MiB","raft-max-inflight-msgs":256,"raft-entry-max-size":"8MiB","raft-log-compact-sync-interval":"2s","raft-log-gc-tick-interval":"3s","raft-log-gc-threshold":50,"raft-log-gc-count-limit":73728,"raft-log-gc-size-limit":"72MiB","raft-log-reserve-max-ticks":6,"raft-engine-purge-interval":"10s","raft-entry-cache-life-time":"30s","split-region-check-tick-interval":"10s","region-split-check-diff":"6MiB","region-compact-check-interval":"5m","region-compact-check-step":100,"region-compact-min-tombstones":10000,"region-compact-tombstones-percent":30,"pd-heartbeat-tick-interval":"1m","pd-store-heartbeat-tick-interval":"10s","snap-mgr-gc-tick-interval":"1m","snap-gc-timeout":"4h","lock-cf-compact-interval":"10m","lock-cf-compact-bytes-threshold":"256MiB","notify-capacity":40960,"messages-per-tick":4096,"max-peer-down-duration":"10m","max-leader-missing-duration":"2h","abnormal-leader-missing-duration":"10m","peer-stale-state-check-interval":"5m","leader-transfer-max-log-lag":128,"snap-apply-batch-size":"10MiB","consistency-check-interval":"0s","report-region-flow-interval":"1m","raft-store-max-leader-lease":"9s","check-leader-lease-interval":"2s250ms","renew-leader-lease-advance-duration":"2s250ms","right-derive-when-split":true,"merge-max-log-gap":10,"merge-check-tick-interval":"2s","use-delete-range":false,"snap-generator-pool-size":2,"cleanup-import-sst-interval":"10m","local-read-batch-size":1024,"apply-max-batch-size":256,"apply-pool-size":2,"apply-reschedule-duration":"5s","apply-low-priority-pool-size":1,"store-max-batch-size":256,"store-pool-size":2,"store-reschedule-duration":"5s","store-low-priority-pool-size":0,"store-io-pool-size":0,"store-io-notify-capacity":40960,"future-poll-size":1,"hibernate-regions":true,"dev-assert":false,"apply-yield-duration":"500ms","perf-level":0,"evict-cache-on-memory-ratio":0.0,"cmd-batch":true,"cmd-batch-concurrent-ready-max-count":1,"raft-write-size-limit":"1MiB","waterfall-metrics":true,"io-reschedule-concurrent-max-count":4,"io-reschedule-hotpot-duration":"5s","inspect-interval":"500ms","report-min-resolved-ts-interval":"0s","reactive-memory-lock-tick-interval":"2s","reactive-memory-lock-timeout-tick":5,"report-region-buckets-tick-interval":"10s","max-snapshot-file-raw-size":"100MiB"},"coprocessor":{"split-region-on-table":false,"batch-split-limit":10,"region-max-size":"144MiB","region-split-size":"96MiB","region-max-keys":1440000,"region-split-keys":960000,"consistency-check-method":"mvcc","enable-region-bucket":false,"region-bucket-size":"96MiB","region-size-threshold-for-approximate":"1440MiB","prefer-approximate-bucket":true,"region-bucket-merge-size-ratio":0.33},"coprocessor-v2":{"coprocessor-plugin-directory":null},"rocksdb":{"info-log-level":"info","wal-recovery-mode":2,"wal-dir":"","wal-ttl-seconds":0,"wal-size-limit":"0KiB","max-total-wal-size":"4GiB","max-background-jobs":9,"max-background-flushes":3,"max-manifest-file-size":"128MiB","create-if-missing":true,"max-open-files":40960,"enable-statistics":true,"stats-dump-period":"10m","compaction-readahead-size":"0KiB","info-log-max-size":"1GiB","info-log-roll-time":"0s","info-log-keep-log-file-num":10,"info-log-dir":"","rate-bytes-per-sec":"10GiB","rate-limiter-refill-period":"100ms","rate-limiter-mode":2,"rate-limiter-auto-tuned":true,"bytes-per-sync":"1MiB","wal-bytes-per-sync":"512KiB","max-sub-compactions":3,"writable-file-max-buffer-size":"1MiB","use-direct-io-for-flush-and-compaction":false,"enable-pipelined-write":false,"enable-multi-batch-write":true,"enable-unordered-write":false,"defaultcf":{"block-size":"64KiB","block-cache-size":"8190MiB","disable-block-cache":false,"cache-index-and-filter-blocks":true,"pin-l0-filter-and-index-blocks":true,"use-bloom-filter":true,"optimize-filters-for-hits":true,"whole-key-filtering":true,"bloom-filter-bits-per-key":10,"block-based-bloom-filter":false,"read-amp-bytes-per-bit":0,"compression-per-level":["no","no","lz4","lz4","lz4","zstd","zstd"],"write-buffer-size":"128MiB","max-write-buffer-number":5,"min-write-buffer-number-to-merge":1,"max-bytes-for-level-base":"512MiB","target-file-size-base":"8MiB","level0-file-num-compaction-trigger":4,"level0-slowdown-writes-trigger":20,"level0-stop-writes-trigger":36,"max-compaction-bytes":"2GiB","compaction-pri":3,"dynamic-level-bytes":true,"num-levels":7,"max-bytes-for-level-multiplier":10,"compaction-style":0,"disable-auto-compactions":false,"disable-write-stall":true,"soft-pending-compaction-bytes-limit":"192GiB","hard-pending-compaction-bytes-limit":"1TiB","force-consistency-checks":false,"prop-size-index-distance":4194304,"prop-keys-index-distance":40960,"enable-doubly-skiplist":true,"enable-compaction-guard":true,"compaction-guard-min-output-file-size":"8MiB","compaction-guard-max-output-file-size":"128MiB","bottommost-level-compression":"zstd","bottommost-zstd-compression-dict-size":0,"bottommost-zstd-compression-sample-size":0,"titan":{"min-blob-size":"1KiB","blob-file-compression":"lz4","blob-cache-size":"0KiB","min-gc-batch-size":"16MiB","max-gc-batch-size":"64MiB","discardable-ratio":0.5,"sample-ratio":0.1,"merge-small-file-threshold":"8MiB","blob-run-mode":"normal","level-merge":false,"range-merge":true,"max-sorted-runs":20,"gc-merge-rewrite":false}},"writecf":{"block-size":"64KiB","block-cache-size":"4914MiB","disable-block-cache":false,"cache-index-and-filter-blocks":true,"pin-l0-filter-and-index-blocks":true,"use-bloom-filter":true,"optimize-filters-for-hits":false,"whole-key-filtering":false,"bloom-filter-bits-per-key":10,"block-based-bloom-filter":false,"read-amp-bytes-per-bit":0,"compression-per-level":["no","no","lz4","lz4","lz4","zstd","zstd"],"write-buffer-size":"128MiB","max-write-buffer-number":5,"min-write-buffer-number-to-merge":1,"max-bytes-for-level-base":"512MiB","target-file-size-base":"8MiB","level0-file-num-compaction-trigger":4,"level0-slowdown-writes-trigger":20,"level0-stop-writes-trigger":36,"max-compaction-bytes":"2GiB","compaction-pri":3,"dynamic-level-bytes":true,"num-levels":7,"max-bytes-for-level-multiplier":10,"compaction-style":0,"disable-auto-compactions":false,"disable-write-stall":true,"soft-pending-compaction-bytes-limit":"192GiB","hard-pending-compaction-bytes-limit":"1TiB","force-consistency-checks":false,"prop-size-index-distance":4194304,"prop-keys-index-distance":40960,"enable-doubly-skiplist":true,"enable-compaction-guard":true,"compaction-guard-min-output-file-size":"8MiB","compaction-guard-max-output-file-size":"128MiB","bottommost-level-compression":"zstd","bottommost-zstd-compression-dict-size":0,"bottommost-zstd-compression-sample-size":0,"titan":{"min-blob-size":"1KiB","blob-file-compression":"lz4","blob-cache-size":"0KiB","min-gc-batch-size":"16MiB","max-gc-batch-size":"64MiB","discardable-ratio":0.5,"sample-ratio":0.1,"merge-small-file-threshold":"8MiB","blob-run-mode":"read-only","level-merge":false,"range-merge":true,"max-sorted-runs":20,"gc-merge-rewrite":false}},"lockcf":{"block-size":"16KiB","block-cache-size":"655MiB","disable-block-cache":false,"cache-index-and-filter-blocks":true,"pin-l0-filter-and-index-blocks":true,"use-bloom-filter":true,"optimize-filters-for-hits":false,"whole-key-filtering":true,"bloom-filter-bits-per-key":10,"block-based-bloom-filter":false,"read-amp-bytes-per-bit":0,"compression-per-level":["no","no","no","no","no","no","no"],"write-buffer-size":"32MiB","max-write-buffer-number":5,"min-write-buffer-number-to-merge":1,"max-bytes-for-level-base":"128MiB","target-file-size-base":"8MiB","level0-file-num-compaction-trigger":1,"level0-slowdown-writes-trigger":20,"level0-stop-writes-trigger":36,"max-compaction-bytes":"2GiB","compaction-pri":0,"dynamic-level-bytes":true,"num-levels":7,"max-bytes-for-level-multiplier":10,"compaction-style":0,"disable-auto-compactions":false,"disable-write-stall":true,"soft-pending-compaction-bytes-limit":"192GiB","hard-pending-compaction-bytes-limit":"1TiB","force-consistency-checks":false,"prop-size-index-distance":4194304,"prop-keys-index-distance":40960,"enable-doubly-skiplist":true,"enable-compaction-guard":false,"compaction-guard-min-output-file-size":"8MiB","compaction-guard-max-output-file-size":"128MiB","bottommost-level-compression":"disable","bottommost-zstd-compression-dict-size":0,"bottommost-zstd-compression-sample-size":0,"titan":{"min-blob-size":"1KiB","blob-file-compression":"lz4","blob-cache-size":"0KiB","min-gc-batch-size":"16MiB","max-gc-batch-size":"64MiB","discardable-ratio":0.5,"sample-ratio":0.1,"merge-small-file-threshold":"8MiB","blob-run-mode":"read-only","level-merge":false,"range-merge":true,"max-sorted-runs":20,"gc-merge-rewrite":false}},"raftcf":{"block-size":"16KiB","block-cache-size":"128MiB","disable-block-cache":false,"cache-index-and-filter-blocks":true,"pin-l0-filter-and-index-blocks":true,"use-bloom-filter":true,"optimize-filters-for-hits":true,"whole-key-filtering":true,"bloom-filter-bits-per-key":10,"block-based-bloom-filter":false,"read-amp-bytes-per-bit":0,"compression-per-level":["no","no","no","no","no","no","no"],"write-buffer-size":"128MiB","max-write-buffer-number":5,"min-write-buffer-number-to-merge":1,"max-bytes-for-level-base":"128MiB","target-file-size-base":"8MiB","level0-file-num-compaction-trigger":1,"level0-slowdown-writes-trigger":20,"level0-stop-writes-trigger":36,"max-compaction-bytes":"2GiB","compaction-pri":0,"dynamic-level-bytes":true,"num-levels":7,"max-bytes-for-level-multiplier":10,"compaction-style":0,"disable-auto-compactions":false,"disable-write-stall":true,"soft-pending-compaction-bytes-limit":"192GiB","hard-pending-compaction-bytes-limit":"1TiB","force-consistency-checks":false,"prop-size-index-distance":4194304,"prop-keys-index-distance":40960,"enable-doubly-skiplist":true,"enable-compaction-guard":false,"compaction-guard-min-output-file-size":"8MiB","compaction-guard-max-output-file-size":"128MiB","bottommost-level-compression":"disable","bottommost-zstd-compression-dict-size":0,"bottommost-zstd-compression-sample-size":0,"titan":{"min-blob-size":"1KiB","blob-file-compression":"lz4","blob-cache-size":"0KiB","min-gc-batch-size":"16MiB","max-gc-batch-size":"64MiB","discardable-ratio":0.5,"sample-ratio":0.1,"merge-small-file-threshold":"8MiB","blob-run-mode":"read-only","level-merge":false,"range-merge":true,"max-sorted-runs":20,"gc-merge-rewrite":false}},"titan":{"enabled":false,"dirname":"","disable-gc":false,"max-background-gc":4,"purge-obsolete-files-period":"10s"}},"raftdb":{"wal-recovery-mode":2,"wal-dir":"","wal-ttl-seconds":0,"wal-size-limit":"0KiB","max-total-wal-size":"4GiB","max-background-jobs":4,"max-background-flushes":1,"max-manifest-file-size":"20MiB","create-if-missing":true,"max-open-files":40960,"enable-statistics":true,"stats-dump-period":"10m","compaction-readahead-size":"0KiB","info-log-max-size":"1GiB","info-log-roll-time":"0s","info-log-keep-log-file-num":10,"info-log-dir":"","info-log-level":"info","max-sub-compactions":2,"writable-file-max-buffer-size":"1MiB","use-direct-io-for-flush-and-compaction":false,"enable-pipelined-write":true,"enable-unordered-write":false,"allow-concurrent-memtable-write":true,"bytes-per-sync":"1MiB","wal-bytes-per-sync":"512KiB","defaultcf":{"block-size":"64KiB","block-cache-size":"655MiB","disable-block-cache":false,"cache-index-and-filter-blocks":true,"pin-l0-filter-and-index-blocks":true,"use-bloom-filter":false,"optimize-filters-for-hits":true,"whole-key-filtering":true,"bloom-filter-bits-per-key":10,"block-based-bloom-filter":false,"read-amp-bytes-per-bit":0,"compression-per-level":["no","no","lz4","lz4","lz4","zstd","zstd"],"write-buffer-size":"128MiB","max-write-buffer-number":5,"min-write-buffer-number-to-merge":1,"max-bytes-for-level-base":"512MiB","target-file-size-base":"8MiB","level0-file-num-compaction-trigger":4,"level0-slowdown-writes-trigger":10000,"level0-stop-writes-trigger":10000,"max-compaction-bytes":"2GiB","compaction-pri":0,"dynamic-level-bytes":true,"num-levels":7,"max-bytes-for-level-multiplier":10,"compaction-style":0,"disable-auto-compactions":false,"disable-write-stall":false,"soft-pending-compaction-bytes-limit":"0KiB","hard-pending-compaction-bytes-limit":"0KiB","force-consistency-checks":false,"prop-size-index-distance":4194304,"prop-keys-index-distance":40960,"enable-doubly-skiplist":true,"enable-compaction-guard":false,"compaction-guard-min-output-file-size":"8MiB","compaction-guard-max-output-file-size":"128MiB","bottommost-level-compression":"disable","bottommost-zstd-compression-dict-size":0,"bottommost-zstd-compression-sample-size":0,"titan":{"min-blob-size":"1KiB","blob-file-compression":"lz4","blob-cache-size":"0KiB","min-gc-batch-size":"16MiB","max-gc-batch-size":"64MiB","discardable-ratio":0.5,"sample-ratio":0.1,"merge-small-file-threshold":"8MiB","blob-run-mode":"normal","level-merge":false,"range-merge":true,"max-sorted-runs":20,"gc-merge-rewrite":false}},"titan":{"enabled":false,"dirname":"","disable-gc":false,"max-background-gc":4,"purge-obsolete-files-period":"10s"}},"raft-engine":{"enable":true,"dir":"/data/tidb-data/tikv-20160/raft-engine","recovery-mode":"tolerate-corrupted-tail-records","recovery-read-block-size":"16KiB","recovery-threads":4,"batch-compression-threshold":"8KiB","bytes-per-sync":"4MiB","target-file-size":"128MiB","purge-threshold":"10GiB","purge-rewrite-threshold":"1GiB","purge-rewrite-garbage-ratio":0.6,"memory-limit":"5032119KiB"},"security":{"ca-path":"","cert-path":"","key-path":"","cert-allowed-cn":,"redact-info-log":null,"encryption":{"data-encryption-method":"plaintext","data-key-rotation-period":"7d","enable-file-dictionary-log":true,"file-dictionary-rewrite-threshold":1000000,"master-key":{"type":"plaintext"},"previous-master-key":{"type":"plaintext"}}},"import":{"num-threads":8,"stream-channel-window":128,"import-mode-timeout":"10m"},"backup":{"num-threads":8,"batch-size":8,"sst-max-size":"144MiB","enable-auto-tune":true,"auto-tune-remain-threads":3,"auto-tune-refresh-interval":"1m","io-thread-size":2,"s3-multi-part-size":"5MiB","hadoop":{"home":"","linux-user":""}},"log-backup":{"max-flush-interval":"5m","num-threads":8,"io-threads":2,"enable":false,"temp-path":"/data/tidb-data/tikv-20160/log-backup-tmp","temp-file-size-limit-per-task":"128MiB","initial-scan-pending-memory-quota":"512MiB"},"pessimistic-txn":{"wait-for-lock-timeout":"1s","wake-up-delay-duration":"20ms","pipelined":true,"in-memory":true},"gc":{"ratio-threshold":1.1,"batch-keys":512,"max-write-bytes-per-sec":"0KiB","enable-compaction-filter":true,"compaction-filter-skip-version-check":false},"split":{"qps-threshold":3000,"split-balance-score":0.25,"split-contained-score":0.5,"detect-times":10,"sample-num":20,"sample-threshold":100,"byte-threshold":31457280},"cdc":{"min-ts-interval":"1s","hibernate-regions-compatible":true,"incremental-scan-threads":4,"incremental-scan-concurrency":6,"incremental-scan-speed-limit":"128MiB","incremental-scan-ts-filter-ratio":0.2,"sink-memory-quota":"512MiB","old-value-cache-memory-quota":"512MiB"},"resolved-ts":{"enable":true,"advance-ts-interval":"1s","scan-lock-pool-size":2},"resource-metering":{"receiver-address":"","report-receiver-interval":"1m","max-resource-groups":100,"precision":"1s"},"causal-ts":{"renew-interval":"100ms","renew-batch-min-size":100}}”]
[2026/07/30 01:20:40.901 +08:00] [INFO] [mod.rs:124] [“encryption: none of key dictionary and file dictionary are found.”]
[2026/07/30 01:20:40.901 +08:00] [INFO] [mod.rs:485] [“encryption is disabled.”]
[2026/07/30 01:20:40.902 +08:00] [INFO] [config.rs:1532] [“Removing directory”] [path=/data/tidb-data/tikv-20160/raft-engine]
[2026/07/30 01:20:40.904 +08:00] [INFO] [pipe_builder.rs:96] [“Create raft log directory: /data/tidb-data/tikv-20160/raft-engine”]
[2026/07/30 01:20:40.906 +08:00] [INFO] [engine.rs:91] [“Recovering raft logs takes 2.548263ms”]
[2026/07/30 01:20:44.334 +08:00] [FATAL] [lib.rs:491] [“couldn’t find the OPTIONS file”] [backtrace=" 0: tikv_util::set_panic_hook::{{closure}}\n at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tikv_util/src/lib.rs:490:18\n 1: std::panicking::rust_panic_with_hook\n at /rust/toolchains/nightly-2022-02-14-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:702:17\n 2: std::panicking::begin_panic::{{closure}}\n at /rust/toolchains/nightly-2022-02-14-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:617:9\n 3: std::sys_common::backtrace::__rust_end_short_backtrace\n at /rust/toolchains/nightly-2022-02-14-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:138:18\n 4: std::panicking::begin_panic\n at /rust/toolchains/nightly-2022-02-14-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:616:12\n 5: engine_rocks::raw_util::new_engine_opt::{{closure}}\n at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/engine_rocks/src/raw_util.rs:121:32\n core::option::Option::unwrap_or_else\n at /rust/toolchains/nightly-2022-02-14-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/option.rs:802:21\n engine_rocks::raw_util::new_engine_opt\n at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/engine_rocks/src/raw_util.rs:119:24\n 6: <raft_log_engine::engine::RaftLogEngine as server::server::ConfiguredRaftEngine>::build\n at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/server/src/server.rs:1442:26\n server::server::TiKvServer::init_raw_engines\n at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/server/src/server.rs:1468:27\n 7: server::server::run_impl\n at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/server/src/server.rs:129:35\n server::server::run_tikv\n at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/server/src/server.rs:163:5\n 8: tikv_server::main\n at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/cmd/tikv-server/src/main.rs:189:5\n 9: core::ops::function::FnOnce::call_once\n at /rust/toolchains/nightly-2022-02-14-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227:5\n std::sys_common::backtrace::__rust_begin_short_backtrace\n at /rust/toolchains/nightly-2022-02-14-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:122:18\n 10: main\n 11: __libc_start_main\n 12: \n"] [location=components/engine_rocks/src/raw_util.rs:121] [thread_name=main]
[root@tikv01 lo

只能从每日备份恢复了,先把数据导出来,换去pg了,使用要求太高了。

第一次失败是因为 CentOS 7 的 glibc 版本太低(2.17),无法运行 TiDB 8.x 要求的环境;后面强行升级后,TiKV 读取原 5.0.3 的 RocksDB 配置时发现 8.5.1 不支持旧参数 enable_multi_thread_write,导致 TiKV 启动失败。由于 TiKV 是数据存储层,三台 TiKV 都起不来,所以整个集群不可用。当前不要继续升级或删除数据,应先恢复 TiKV/TiDB/PD 到 5.0.3 版本,保证数据目录不变,先把集群恢复,再规划按官方跨版本路径升级。

但是我现在有bin文件都退回5.0.3,启动就说options文件找不到。

我现在pd改为了5.0.3
-rwxr-xr-x 1 tidb tidb 96764600 Jun 28 2021 pd-server
[tidb@pd01 bin]$ ./pd-server --version
Release Version: v5.0.3
Edition: Community
Git Commit Hash: 9fff28d370fa79daead70bf1929748d9a72a09c2
Git Branch: heads/refs/tags/v5.0.3
UTC Build Time: 2021-06-28 10:05:08
[tidb@pd01 bin]$

再把tikv-server使用6.1.0版本。开启了
[
raft-engine
]
enable = true
现在启动tikv-server报错集群id都不一样了:
[2026/07/30 14:29:56.840 +08:00] [INFO] [mod.rs:124] [“encryption: none of key dictionary and file dictionary are found.”]
[2026/07/30 14:29:56.840 +08:00] [INFO] [mod.rs:485] [“encryption is disabled.”]
[2026/07/30 14:29:56.842 +08:00] [INFO] [config.rs:1532] [“Removing directory”] [path=/data/tidb-data/tikv-20160/raft-engine]
[2026/07/30 14:29:56.844 +08:00] [INFO] [pipe_builder.rs:96] [“Create raft log directory: /data/tidb-data/tikv-20160/raft-engine”]
[2026/07/30 14:29:56.847 +08:00] [INFO] [engine.rs:91] [“Recovering raft logs takes 2.188514ms”]
[2026/07/30 14:29:57.425 +08:00] [INFO] [raft_engine_switch.rs:36] [“Start to scan raft log from RocksEngine and dump into RaftLogEngine”]
[2026/07/30 14:30:10.831 +08:00] [INFO] [raft_engine_switch.rs:55] [“Scanned all region id and waiting for dump”]
[2026/07/30 14:30:10.838 +08:00] [INFO] [raft_engine_switch.rs:60] [“Finished dump, total regions: 299173; Total bytes: 22903990; Consumed time: 13.413629004s”]
[2026/07/30 14:30:10.838 +08:00] [INFO] [config.rs:1532] [“Removing directory”] [path=/data/tidb-data/tikv-20160/raft]
[2026/07/30 14:30:12.000 +08:00] [INFO] [config.rs:1537] [“Removing file”] [path=/data/tidb-data/tikv-20160/MIGRATING-RAFT]
[2026/07/30 14:30:26.823 +08:00] [INFO] [mod.rs:259] [“Storage started.”]
[2026/07/30 14:30:26.828 +08:00] [FATAL] [server.rs:810] [“failed to bootstrap node id: "[src/server/node.rs:269]: cluster ID mismatch, local 6896729876697786249 != remote 7667765693746552485, you are trying to connect to another cluster, please reconnect to the correct PD"”]

数据库的系统变量呢,有该参数吗

看标题忍不住笑了,不该笑。
tidb不能回退吧。 有备份吗

官方最佳实践也不推荐扩大版本升级吧

这种升级没备份确实危险。你现在版本是v5.0.3,集群部分组件Down了,建议先恢复集群再考虑升级。

试试用tiup cluster restart强制重启所有节点,如果不行,检查下PD和TiKV进程日志,看看具体报错:
tail -100 /data/tidb-deploy/tikv-*/log/tikv.log

如果PD挂了,可以手动拉起PD并尝试恢复元数据。

  1. tiup 回退版本至 v5.0.3
  2. 删除所有 tikv.toml 内 enable_multi_thread_write 配置项
  3. 写入优化后的内存管控 tikv 配置
  4. 顺序启动 tikv、tidb、监控,验证业务连通
  5. 观察 TiKV 内存占用 24 小时确认回落稳定
  6. 若后续仍需新版本,规划操作系统替换或是止步 6.5 版本升级