集群故障恢复后数据丢失

TiDBer_CjNBNP1L · 2025 年6 月 16 日 23:41

【TiDB 使用环境】生产环境 /测试/ Poc
【TiDB 版本】8.1.1
【操作系统】openEuler 24.03
【部署方式】云上部署（什么云）/机器部署（什么机器配置、什么硬盘）
【集群数据量】自有物理机私有部署 72核96G1.8T硬盘*2
【集群节点数】2
【问题复现路径】做过哪些操作出现的问题
正常运行中宕机，发现组件状态变成down。
重启时无法启动，pd提示启动成功，但实际没能启动。
后来重启过服务器。
参照文档【 PD Recover 使用文档】，使用pd-convery修复后启动集群。
发现存在问题。
【遇到的问题：问题现象及影响】
无法添加/修改账户权限。
数据显示不完整。count结果和实际显示的数据条数不一致。
数据存在丢失。
【资源配置】进入到 TiDB Dashboard -集群信息 (Cluster Info) -主机(Hosts) 截图此页面
【复制黏贴 ERROR 报错的日志】
【其他附件：截图/日志/监控】

terry0219 · 2025 年6 月 17 日 01:39

可以把相关错误日志发出来看看

Miracle · 2025 年6 月 17 日 02:13

集群有几个pd几个kv？

lllzd · 2025 年6 月 17 日 09:09

请贴出日志，让大家一起看看分析一下。

小龙虾爱大龙虾 · 2025 年6 月 18 日 01:43

2 个节点的集群，不是标准部署，应该不是生产环境吧，重建吧

asmile · 2025 年6 月 18 日 01:45

没日志，没法看

TiDBer_CjNBNP1L · 2025 年6 月 18 日 06:39

[2025/06/16 20:06:20.750 +08:00] [INFO] [trace.go:152] [“trace[1341699209] linearizableReadLoop”] [detail=“{readStateIndex:20424070; appliedIndex:20424070; }”] [duration=371.398576ms] [start=2025/06/16 20:06:20.285 +08:00] [end=2025/06/16 20:06:20.656 +08:00] [steps=“["trace[1341699209] ‘read index received’ (duration: 309.93787ms)","trace[1341699209] ‘applied index is now lower than readState.Index’ (duration: 32.622237ms)"]”]
[2025/06/16 20:06:21.484 +08:00] [INFO] [trace.go:152] [“trace[1149556088] linearizableReadLoop”] [detail=“{readStateIndex:20424070; appliedIndex:20424070; }”] [duration=436.869085ms] [start=2025/06/16 20:06:21.013 +08:00] [end=2025/06/16 20:06:21.450 +08:00] [steps=“["trace[1149556088] ‘read index received’ (duration: 416.060875ms)","trace[1149556088] ‘applied index is now lower than readState.Index’ (duration: 18.641434ms)"]”]
[2025/06/16 20:06:22.146 +08:00] [INFO] [trace.go:152] [“trace[1419105571] linearizableReadLoop”] [detail=“{readStateIndex:20424070; appliedIndex:20424070; }”] [duration=291.807223ms] [start=2025/06/16 20:06:21.853 +08:00] [end=2025/06/16 20:06:22.144 +08:00] [steps=“["trace[1419105571] ‘read index received’ (duration: 288.856258ms)","trace[1419105571] ‘applied index is now lower than readState.Index’ (duration: 2.947921ms)"]”]
[2025/06/16 20:06:22.147 +08:00] [WARN] [util.go:167] [“apply request took too long”] [took=1.101536979s] [expected-duration=100ms] [prefix=“read-only range “] [request=“key:"health" “] [response=“range_response_count:0 size:7”]
[2025/06/16 20:06:22.147 +08:00] [WARN] [util.go:167] [“apply request took too long”] [took=1.101534847s] [expected-duration=100ms] [prefix=“read-only range “] [request=“key:"health" “] [response=“range_response_count:0 size:7”]
[2025/06/16 20:06:22.150 +08:00] [INFO] [trace.go:152] [“trace[2138823441] range”] [detail=”{range_begin:health; range_end:; response_count:0; response_revision:20415977; }”] [duration=1.111262901s] [start=2025/06/16 20:06:21.039 +08:00] [end=2025/06/16 20:06:22.150 +08:00] [steps=”["trace[2138823441] ‘agreement among raft nodes before linearized reading’ (duration: 1.106628844s)"]”]
[2025/06/16 20:06:22.150 +08:00] [INFO] [trace.go:152] [“trace[1612245232] range”] [detail=”{range_begin:health; range_end:; response_count:0; response_revision:20415977; }”] [duration=1.111250489s] [start=2025/06/16 20:06:21.039 +08:00] [end=2025/06/16 20:06:22.150 +08:00] [steps=”["trace[1612245232] ‘agreement among raft nodes before linearized reading’ (duration: 1.106634383s)"]”]
[2025/06/16 20:06:45.428 +08:00] [INFO] [grpc_service.go:99] [“watch request”] [key=resource_group/settings] [range-end=resource_group/settingt] [start-revision=17058772]
[2025/06/16 20:06:45.428 +08:00] [INFO] [grpc_service.go:99] [“watch request”] [key=resource_group/controller] [range-end=resource_group/controlles] [start-revision=17058778]
[2025/06/16 20:06:45.428 +08:00] [INFO] [grpc_service.go:99] [“watch request”] [key=resource_group/controller] [range-end=resource_group/controlles] [start-revision=17058772]
[2025/06/16 20:06:45.428 +08:00] [INFO] [grpc_service.go:99] [“watch request”] [key=resource_group/settings] [range-end=resource_group/settingt] [start-revision=17058778]
[2025/06/16 20:06:51.525 +08:00] [INFO] [grpc_service.go:2173] [“update service GC safe point”] [service-id=ticdc-default-11035117345847807324] [expire-at=4044809431] [safepoint=1060299874335981567]
[2025/06/16 20:07:15.515 +08:00] [INFO] [grpc_service.go:99] [“watch request”] [key=resource_group/settings] [range-end=resource_group/settingt] [start-revision=17058778]
[2025/06/16 20:07:15.515 +08:00] [INFO] [grpc_service.go:99] [“watch request”] [key=resource_group/controller] [range-end=resource_group/controlles] [start-revision=17058772]
[2025/06/16 20:07:15.515 +08:00] [INFO] [grpc_service.go:99] [“watch request”] [key=resource_group/settings] [range-end=resource_group/settingt] [start-revision=17058772]
[2025/06/16 20:07:15.515 +08:00] [INFO] [grpc_service.go:99] [“watch request”] [key=resource_group/controller] [range-end=resource_group/controlles] [start-revision=17058778]
[2025/06/16 20:07:21.683 +08:00] [INFO] [trace.go:152] [“trace[482489639] linearizableReadLoop”] [detail=“{readStateIndex:20424087; appliedIndex:20424087; }”] [duration=302.12552ms] [start=2025/06/16 20:07:21.355 +08:00] [end=2025/06/16 20:07:21.657 +08:00] [steps=“["trace[482489639] ‘read index received’ (duration: 226.646133ms)","trace[482489639] ‘applied index is now lower than readState.Index’ (duration: 63.383429ms)"]”]
[2025/06/16 20:07:22.604 +08:00] [INFO] [trace.go:152] [“trace[686981697] linearizableReadLoop”] [detail=“{readStateIndex:20424087; appliedIndex:20424087; }”] [duration=453.190276ms] [start=2025/06/16 20:07:22.131 +08:00] [end=2025/06/16 20:07:22.584 +08:00] [steps=“["trace[686981697] ‘read index received’ (duration: 453.123028ms)","trace[686981697] ‘applied index is now lower than readState.Index’ (duration: 63.77µs)"]”]
[2025/06/16 20:07:23.335 +08:00] [WARN] [v3_server.go:887] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=4769501714904180845] [retry-timeout=500ms]
[2025/06/16 20:07:24.085 +08:00] [WARN] [util.go:167] [“apply request took too long”] [took=1.470740179s] [expected-duration=100ms] [prefix=“read-only range “] [request=“key:"/pd/7403334201117082689/config" “] [response=“range_response_count:1 size:4922”]
[2025/06/16 20:07:24.085 +08:00] [WARN] [util.go:167] [“apply request took too long”] [took=1.636959544s] [expected-duration=100ms] [prefix=“read-only range “] [request=“key:"health" “] [response=“range_response_count:0 size:7”]
[2025/06/16 20:07:23.888 +08:00] [INFO] [trace.go:152] [“trace[1784715042] linearizableReadLoop”] [detail=”{readStateIndex:20424087; appliedIndex:20424087; }”] [duration=858.840762ms] [start=2025/06/16 20:07:22.806 +08:00] [end=2025/06/16 20:07:23.665 +08:00] [steps=”["trace[1784715042] ‘read index received’ (duration: 851.648675ms)","trace[1784715042] ‘applied index is now lower than readState.Index’ (duration: 3.718427ms)"]”]
[2025/06/16 20:07:24.140 +08:00] [INFO] [trace.go:152] [“trace[2091386993] range”] [detail=”{range_begin:health; range_end:; response_count:0; response_revision:20415994; }”] [duration=1.838110904s] [start=2025/06/16 20:07:22.296 +08:00] [end=2025/06/16 20:07:24.134 +08:00] [steps=”["trace[2091386993] ‘agreement among raft nodes before linearized reading’ (duration: 1.368258016s)","trace[2091386993] ‘range keys from in-memory index tree’ (duration: 134.074996ms)","trace[2091386993] ‘filter and sort the key-value pairs’ (duration: 47.852053ms)"]”]
[2025/06/16 20:07:24.142 +08:00] [WARN] [v3_server.go:856] [“ignored out-of-date read index response; local node read indexes queueing up and waiting to be in sync with leader”] [sent-request-id=4769501714904180846] [received-request-id=4769501714904180845]
[2025/06/16 20:07:24.142 +08:00] [INFO] [trace.go:152] [“trace[381049577] range”] [detail=“{range_begin:/pd/7403334201117082689/config; range_end:; response_count:1; response_revision:20415994; }”] [duration=1.528961641s] [start=2025/06/16 20:07:22.605 +08:00] [end=2025/06/16 20:07:24.134 +08:00] [steps=“["trace[381049577] ‘agreement among raft nodes before linearized reading’ (duration: 1.059213388s)","trace[381049577] ‘range keys from in-memory index tree’ (duration: 137.461364ms)","trace[381049577] ‘range keys from bolt db’ (duration: 157.878668ms)"]”]
[2025/06/16 20:07:24.144 +08:00] [WARN] [util.go:167] [“apply request took too long”] [took=291.209976ms] [expected-duration=100ms] [prefix=“read-only range “] [request=“key:"health" “] [response=“range_response_count:0 size:7”]
[2025/06/16 20:07:24.144 +08:00] [WARN] [util.go:167] [“apply request took too long”] [took=291.208865ms] [expected-duration=100ms] [prefix=“read-only range “] [request=“key:"health" “] [response=“range_response_count:0 size:7”]
[2025/06/16 20:07:24.144 +08:00] [INFO] [trace.go:152] [“trace[1080787956] range”] [detail=”{range_begin:health; range_end:; response_count:0; response_revision:20415994; }”] [duration=307.706496ms] [start=2025/06/16 20:07:23.836 +08:00] [end=2025/06/16 20:07:24.144 +08:00] [steps=”["trace[1080787956] ‘agreement among raft nodes before linearized reading’ (duration: 306.300588ms)"]”]
[2025/06/16 20:07:24.144 +08:00] [INFO] [trace.go:152] [“trace[1902671673] range”] [detail=”{range_begin:health; range_end:; response_count:0; response_revision:20415994; }”] [duration=307.713696ms] [start=2025/06/16 20:07:23.836 +08:00] [end=2025/06/16 20:07:24.144 +08:00] [steps=”["trace[1902671673] ‘agreement among raft nodes before linearized reading’ (duration: 306.287225ms)"]”]
[2025/06/16 20:07:45.562 +08:00] [INFO] [grpc_service.go:99] [“watch request”] [key=resource_group/controller] [range-end=resource_group/controlles] [start-revision=17058778]
[2025/06/16 20:07:45.562 +08:00] [INFO] [grpc_service.go:99] [“watch request”] [key=resource_group/controller] [range-end=resource_group/controlles] [start-revision=17058772]
[2025/06/16 20:07:45.562 +08:00] [INFO] [grpc_service.go:99] [“watch request”] [key=resource_group/settings] [range-end=resource_group/settingt] [start-revision=17058778]
[2025/06/16 20:07:45.562 +08:00] [INFO] [grpc_service.go:99] [“watch request”] [key=resource_group/settings] [range-end=resource_group/settingt] [start-revision=17058772]
[2025/06/16 20:07:51.480 +08:00] [INFO] [grpc_service.go:2173] [“update service GC safe point”] [service-id=ticdc-default-11035117345847807324] [expire-at=4044809431] [safepoint=1060299874346729471]

TiDBer_CjNBNP1L · 2025 年6 月 18 日 06:42

[2025/06/16 20:02:29.091 +08:00] [INFO] [util.rs:639] [“connecting to PD endpoint”] [endpoints=http://192.168.100.13:2379] [thread_id=12]
[2025/06/16 20:06:21.455 +08:00] [WARN] [peer.rs:717] [“[region 3751] 3752 handle 2 peer messages ("Tick", 2)”] [takes=114] [thread_id=368]
[2025/06/16 20:06:21.628 +08:00] [INFO] [trend.rs:292] [“history window flipping: enter”] [increasing_rate=0] [flip_margin_error=1500] [delta=4231.843088125897] [name=L1] [thread_id=32]
[2025/06/16 20:06:21.906 +08:00] [INFO] [trend.rs:292] [“history window flipping: enter”] [increasing_rate=0] [flip_margin_error=1000] [delta=1121.5444661010697] [name=L2] [thread_id=32]
[2025/06/16 20:06:22.144 +08:00] [WARN] [peer.rs:717] [“[region 45] 46 handle 2 peer messages ("RaftMessage", 1), ("Tick", 1)”] [takes=130] [thread_id=367]
[2025/06/16 20:06:22.147 +08:00] [WARN] [peer.rs:717] [“[region 3763] 3764 handle 3 peer messages ("Tick", 3)”] [takes=114] [thread_id=368]
[2025/06/16 20:06:22.155 +08:00] [WARN] [store.rs:1192] [“[store 1] handle 19 pending peers include 18 ready, 0 entries, 11 messages and 0 snapshots”] [takes=1153] [thread_id=368]
[2025/06/16 20:06:22.155 +08:00] [WARN] [store.rs:1192] [“[store 1] handle 15 pending peers include 15 ready, 0 entries, 0 messages and 0 snapshots”] [takes=1722] [thread_id=367]
[2025/06/16 20:06:22.654 +08:00] [INFO] [trend.rs:273] [“history window flipping: end”] [flipping_duration=1] [increasing_rate=0] [time_based_multiple=0.00029993703107215343] [flip_margin_error=500] [delta=12747.370161211526] [name=L1] [thread_id=32]
[2025/06/16 20:06:22.657 +08:00] [INFO] [trend.rs:273] [“history window flipping: end”] [flipping_duration=1] [increasing_rate=0] [time_based_multiple=0.00029993703107215343] [flip_margin_error=500] [delta=3208.9129727249674] [name=L2] [thread_id=32]
[2025/06/16 20:07:21.666 +08:00] [WARN] [peer.rs:717] [“[region 3778] 3779 handle 1 peer messages ("Tick", 1)”] [takes=120] [thread_id=368]
[2025/06/16 20:07:22.319 +08:00] [WARN] [peer.rs:717] [“[region 3691] 3692 handle 2 peer messages ("Tick", 2)”] [takes=108] [thread_id=367]
[2025/06/16 20:07:22.567 +08:00] [WARN] [peer.rs:717] [“[region 3751] 3752 handle 3 peer messages ("Tick", 3)”] [takes=116] [thread_id=367]
[2025/06/16 20:07:22.783 +08:00] [WARN] [peer.rs:717] [“[region 3703] 3704 handle 4 peer messages ("Tick", 4)”] [takes=120] [thread_id=368]
[2025/06/16 20:07:22.961 +08:00] [WARN] [peer.rs:717] [“[region 3733] 3734 handle 3 peer messages ("Tick", 3)”] [takes=120] [thread_id=367]
[2025/06/16 20:07:23.321 +08:00] [WARN] [peer.rs:717] [“[region 87] 88 handle 3 peer messages ("Tick", 3)”] [takes=138] [thread_id=368]
[2025/06/16 20:07:23.521 +08:00] [WARN] [peer.rs:717] [“[region 3667] 3668 handle 4 peer messages ("Tick", 3), ("CasualMessage", 1)”] [takes=275] [thread_id=367]
[2025/06/16 20:07:23.823 +08:00] [WARN] [peer.rs:717] [“[region 119] 120 handle 3 peer messages ("Tick", 3)”] [takes=219] [thread_id=368]
[2025/06/16 20:07:24.098 +08:00] [WARN] [peer.rs:717] [“[region 111] 112 handle 4 peer messages ("Tick", 3), ("CasualMessage", 1)”] [takes=360] [thread_id=367]
[2025/06/16 20:07:24.102 +08:00] [WARN] [peer.rs:717] [“[region 3736] 3737 handle 3 peer messages ("Tick", 3)”] [takes=170] [thread_id=368]
[2025/06/16 20:07:24.130 +08:00] [WARN] [peer.rs:717] [“[region 3715] 3716 handle 2 peer messages ("RaftMessage", 1), ("Tick", 1)”] [takes=100] [thread_id=367]
[2025/06/16 20:07:24.142 +08:00] [WARN] [store.rs:1192] [“[store 1] handle 17 pending peers include 17 ready, 0 entries, 15 messages and 0 snapshots”] [takes=2701] [thread_id=368]
[2025/06/16 20:07:24.152 +08:00] [WARN] [store.rs:1192] [“[store 1] handle 26 pending peers include 26 ready, 0 entries, 0 messages and 0 snapshots”] [takes=2365] [thread_id=367]
[2025/06/16 20:07:24.162 +08:00] [INFO] [tracker.rs:275] [slow-query] [perf_stats.internal_delete_skipped_count=0] [perf_stats.internal_key_skipped_count=10] [perf_stats.block_read_byte=0] [perf_stats.block_read_count=0] [perf_stats.block_cache_hit_count=4] [scan.range.first=“Some(start: 7480000000000002185F698000000000000003040000000000047F29 end: 7480000000000002185F698000000000000003040000000000047F2A)”] [scan.ranges=1] [scan.total=11] [scan.processed_size=460] [scan.processed=10] [scan.is_desc=false] [tag=index] [table_id=536] [txn_start_ts=1060299874334455827] [total_suspend_time=207.385156ms] [total_process_time=1.202794501s] [handler_build_time=253.659182ms] [wait_time.snapshot=104.670602ms] [wait_time.schedule=12.973963ms] [wait_time=117.644565ms] [total_lifetime=1.7866527s] [remote_host=ipv4:192.168.100.13:45104] [region_id=4229] [query_digest=57b004fdebfe930dfc5a8815e6bcc64fb57ffec86d729703cfcdd8ea235ef692] [session_alias=] [connection_id=1816562710] [thread_id=74]
[2025/06/16 20:07:29.031 +08:00] [INFO] [compact.rs:337] [“compact range finished”] [compact_options=“ManualCompactionOptions { exclusive_manual: false, max_subcompactions: 1, bottommost_level_force: false }”] [time_takes=3.720566ms] [cf=default] [range_end=7A7480000000000001FFE800000000000000F8] [range_start=7A7480000000000001FF9100000000000000F8] [thread_id=345]
[2025/06/16 20:07:29.938 +08:00] [INFO] [compact.rs:337] [“compact range finished”] [compact_options=“ManualCompactionOptions { exclusive_manual: false, max_subcompactions: 1, bottommost_level_force: false }”] [time_takes=906.54484ms] [cf=write] [range_end=7A7480000000000001FFE800000000000000F8] [range_start=7A7480000000000001FF9100000000000000F8] [thread_id=345]
[2025/06/16 20:08:21.497 +08:00] [WARN] [peer.rs:717] [“[region 4235] 4236 handle 2 peer messages ("Tick", 2)”] [takes=103] [thread_id=368]
[2025/06/16 20:08:21.840 +08:00] [WARN] [peer.rs:717] [“[region 3664] 3665 handle 2 peer messages ("Tick", 2)”] [takes=205] [thread_id=367]
[2025/06/16 20:08:21.948 +08:00] [WARN] [peer.rs:717] [“[region 4193] 4194 handle 3 peer messages ("Tick", 3)”] [takes=188] [thread_id=368]
[2025/06/16 20:08:21.963 +08:00] [WARN] [store.rs:1192] [“[store 1] handle 5 pending peers include 5 ready, 0 entries, 4 messages and 0 snapshots”] [takes=1097] [thread_id=368]
[2025/06/16 20:08:21.969 +08:00] [WARN] [store.rs:1192] [“[store 1] handle 20 pending peers include 17 ready, 0 entries, 12 messages and 0 snapshots”] [takes=1058] [thread_id=367]
[2025/06/16 20:08:24.897 +08:00] [WARN] [peer.rs:717] [“[region 9013] 9014 handle 2 peer messages ("Tick", 2)”] [takes=164] [thread_id=368]
[2025/06/16 20:08:24.925 +08:00] [WARN] [peer.rs:717] [“[region 3739] 3740 handle 2 peer messages ("Tick", 2)”] [takes=160] [thread_id=367]
[2025/06/16 20:08:25.258 +08:00] [WARN] [peer.rs:717] [“[region 3703] 3704 handle 1 peer messages ("Tick", 1)”] [takes=158] [thread_id=367]
[2025/06/16 20:08:26.808 +08:00] [WARN] [peer.rs:717] [“[region 4307] 4308 handle 3 peer messages ("RaftMessage", 1), ("Tick", 2)”] [takes=176] [thread_id=368]
[2025/06/16 20:08:27.618 +08:00] [WARN] [peer.rs:717] [“[region 61] 62 handle 4 peer messages ("RaftMessage", 2), ("Tick", 2)”] [takes=186] [thread_id=368]
[2025/06/16 20:08:27.986 +08:00] [WARN] [peer.rs:717] [“[region 4001] 4002 handle 4 peer messages ("Tick", 4)”] [takes=128] [thread_id=367]
[2025/06/16 20:08:28.345 +08:00] [WARN] [peer.rs:717] [“[region 111] 112 handle 3 peer messages ("Tick", 3)”] [takes=200] [thread_id=368]
[2025/06/16 20:08:28.753 +08:00] [WARN] [peer.rs:717] [“[region 95] 96 handle 3 peer messages ("Tick", 3)”] [takes=341] [thread_id=367]
[2025/06/16 20:08:28.816 +08:00] [WARN] [peer.rs:717] [“[region 105] 106 handle 5 peer messages ("RaftMessage", 1), ("Tick", 4)”] [takes=263] [thread_id=368]
[2025/06/16 20:08:28.826 +08:00] [WARN] [peer.rs:717] [“[region 3763] 3764 handle 3 peer messages ("Tick", 3)”] [takes=170] [thread_id=367]
[2025/06/16 20:08:29.319 +08:00] [WARN] [peer.rs:717] [“[region 4235] 4236 handle 4 peer messages ("RaftMessage", 1), ("Tick", 3)”] [takes=332] [thread_id=367]
[2025/06/16 20:08:29.735 +08:00] [WARN] [peer.rs:717] [“[region 3640] 3641 handle 4 peer messages ("Tick", 4)”] [takes=159] [thread_id=368]
[2025/06/16 20:08:30.086 +08:00] [WARN] [peer.rs:717] [“[region 3736] 3737 handle 6 peer messages ("RaftMessage", 1), ("Tick", 4), ("CasualMessage", 1)”] [takes=155] [thread_id=367]
[2025/06/16 20:08:30.376 +08:00] [WARN] [store.rs:1192] [“[store 1] handle 8 pending peers include 5 ready, 0 entries, 5 messages and 0 snapshots”] [takes=5172] [thread_id=367]
[2025/06/16 20:08:30.902 +08:00] [WARN] [store.rs:904] [“[store 1] handle 27 store messages ("Tick", 1), ("UpdateReplicationMode", 1), ("LatencyInspect", 25)”] [takes=115] [thread_id=367]
[2025/06/16 20:08:31.414 +08:00] [WARN] [peer.rs:717] [“[region 9292] 9293 handle 2 peer messages ("RaftMessage", 1), ("Tick", 1)”] [takes=159] [thread_id=368]
[2025/06/16 20:08:31.496 +08:00] [WARN] [peer.rs:717] [“[region 4397] 4398 handle 3 peer messages ("Tick", 3)”] [takes=398] [thread_id=367]
[2025/06/16 20:08:31.498 +08:00] [WARN] [peer.rs:717] [“[region 3649] 3650 handle 2 peer messages ("RaftMessage", 1), ("Tick", 1)”] [takes=223] [thread_id=368]
[2025/06/16 20:08:31.498 +08:00] [WARN] [peer.rs:717] [“[region 3703] 3704 handle 3 peer messages ("RaftMessage", 1), ("Tick", 2)”] [takes=182] [thread_id=367]
[2025/06/16 20:08:31.498 +08:00] [WARN] [peer.rs:717] [“[region 45] 46 handle 4 peer messages ("RaftMessage", 1), ("Tick", 2), ("SignificantMsg", 1)”] [takes=282] [thread_id=368]
[2025/06/16 20:08:31.528 +08:00] [INFO] [tracker.rs:275] [slow-query] [perf_stats.internal_delete_skipped_count=0] [perf_stats.internal_key_skipped_count=183] [perf_stats.block_read_byte=0] [perf_stats.block_read_count=0] [perf_stats.block_cache_hit_count=4] [scan.range.first=“Some(start: 7480000000000000185F698000000000000001038000000000000218 end: 7480000000000000185F698000000000000001038000000000000219)”] [scan.ranges=1] [scan.total=184] [scan.processed_size=3782] [scan.processed=61] [scan.is_desc=false] [tag=index] [table_id=24] [txn_start_ts=1060299874334469401] [total_suspend_time=53.389157ms] [total_process_time=3.31761379s] [handler_build_time=214.423025ms] [wait_time.snapshot=601.749373ms] [wait_time.schedule=28.400366ms] [wait_time=630.149739ms] [total_lifetime=4.221320364s] [remote_host=ipv4:192.168.100.13:45090] [region_id=47] [query_digest=1772d6feeaef69bc4752bc98819036e7f725f5e752a8756f269ddcdfe147b24a] [session_alias=] [connection_id=0] [thread_id=86]
[2025/06/16 20:08:31.546 +08:00] [WARN] [store.rs:1192] [“[store 1] handle 105 pending peers include 102 ready, 0 entries, 22 messages and 0 snapshots”] [takes=1324] [thread_id=367]
[2025/06/16 20:08:31.549 +08:00] [WARN] [store.rs:1192] [“[store 1] handle 50 pending peers include 46 ready, 0 entries, 0 messages and 0 snapshots”] [takes=6569] [thread_id=368]
[2025/06/16 20:08:31.607 +08:00] [WARN] [scheduler.rs:1214] [“[region 47] scheduler handle command: commit, ts: 1060299874334469372, details: SchedulerDetails { tracker: TrackerToken { shard_id: 63, seq: 1892384, key: 1 }, stat: Statistics { lock: CfStatistics { processed_keys: 0, get: 52, next: 0, prev: 0, seek: 0, seek_for_prev: 0, over_seek_bound: 0, flow_stats: FlowStatistics { read_keys: 0, read_bytes: 0 }, next_tombstone: 0, prev_tombstone: 0, seek_tombstone: 0, seek_for_prev_tombstone: 0, raw_value_tombstone: 0 }, write: CfStatistics { processed_keys: 0, get: 0, next: 0, prev: 0, seek: 0, seek_for_prev: 0, over_seek_bound: 0, flow_stats: FlowStatistics { read_keys: 0, read_bytes: 0 }, next_tombstone: 0, prev_tombstone: 0, seek_tombstone: 0, seek_for_prev_tombstone: 0, raw_value_tombstone: 0 }, data: CfStatistics { processed_keys: 0, get: 0, next: 0, prev: 0, seek: 0, seek_for_prev: 0, over_seek_bound: 0, flow_stats: FlowStatistics { read_keys: 0, read_bytes: 0 }, next_tombstone: 0, prev_tombstone: 0, seek_tombstone: 0, seek_for_prev_tombstone: 0, raw_value_tombstone: 0 }, processed_size: 0, load_data_hint: LoadDataHintStatistics { last_write_over_seek_bound: 0 } }, start_process_instant: Monotonic(Timespec { sec: 5937755, nsec: 543539583 }), cmd_process_nanos: 2119093957, quota_limit_delay_nanos: 0, flow_control_nanos: 0, async_write_nanos: 4491000000 }”] [takes=6657] [thread_id=113]
[2025/06/16 20:08:32.824 +08:00] [WARN] [peer.rs:717] [“[region 9] 10 handle 2 peer messages ("Tick", 2)”] [takes=173] [thread_id=368]
[2025/06/16 20:08:32.942 +08:00] [WARN] [peer.rs:717] [“[region 4217] 4218 handle 2 peer messages ("Tick", 2)”] [takes=152] [thread_id=367]
[2025/06/16 20:08:33.134 +08:00] [WARN] [peer.rs:717] [“[region 4235] 4236 handle 2 peer messages ("Tick", 2)”] [takes=104] [thread_id=368]
[2025/06/16 20:08:35.933 +08:00] [WARN] [store.rs:1192] [“[store 1] handle 12 pending peers include 12 ready, 0 entries, 0 messages and 0 snapshots”] [takes=2919] [thread_id=367]
[2025/06/16 20:08:35.953 +08:00] [INFO] [tracker.rs:275] [slow-query] [perf_stats.internal_delete_skipped_count=0] [perf_stats.internal_key_skipped_count=52] [perf_stats.block_read_byte=0] [perf_stats.block_read_count=0] [perf_stats.block_cache_hit_count=4] [scan.range.first=“Some(start: 7480000000000000185F698000000000000001038000000000000514 end: 7480000000000000185F698000000000000001038000000000000515)”] [scan.ranges=1] [scan.total=53] [scan.processed_size=1612] [scan.processed=26] [scan.is_desc=false] [tag=index] [table_id=24] [txn_start_ts=1060299874334469641] [total_suspend_time=68.917906ms] [total_process_time=3.409848863s] [handler_build_time=122.642243ms] [wait_time.snapshot=21.112182ms] [wait_time.schedule=4.149328ms] [wait_time=25.26151ms] [total_lifetime=3.629254305s] [remote_host=ipv4:192.168.100.14:34284] [region_id=47] [query_digest=1772d6feeaef69bc4752bc98819036e7f725f5e752a8756f269ddcdfe147b24a] [session_alias=] [connection_id=0] [thread_id=68]
[2025/06/16 20:08:35.955 +08:00] [WARN] [store.rs:1192] [“[store 1] handle 26 pending peers include 26 ready, 1 entries, 16 messages and 0 snapshots”] [takes=3549] [thread_id=368]
[2025/06/16 20:08:36.701 +08:00] [WARN] [peer.rs:717] [“[region 41] 42 handle 1 peer messages ("RaftMessage", 1)”] [takes=116] [thread_id=368]
[2025/06/16 20:08:37.343 +08:00] [WARN] [peer.rs:717] [“[region 4001] 4002 handle 1 peer messages ("Tick", 1)”] [takes=143] [thread_id=368]
[2025/06/16 20:08:37.671 +08:00] [WARN] [peer.rs:717] [“[region 3820] 3821 handle 1 peer messages ("Tick", 1)”] [takes=111] [thread_id=367]
[2025/06/16 20:08:37.909 +08:00] [WARN] [peer.rs:717] [“[region 4331] 4332 handle 1 peer messages ("Tick", 1)”] [takes=108] [thread_id=368]
[2025/06/16 20:08:38.381 +08:00] [WARN] [peer.rs:717] [“[region 3646] 3647 handle 1 peer messages ("Tick", 1)”] [takes=107] [thread_id=367]
[2025/06/16 20:08:38.894 +08:00] [WARN] [peer.rs:717] [“[region 41] 42 handle 3 peer messages ("Tick", 1), ("SignificantMsg", 1), ("Persisted", 1)”] [takes=135] [thread_id=367]
[2025/06/16 20:08:39.367 +08:00] [WARN] [peer.rs:717] [“[region 3745] 3746 handle 2 peer messages ("Tick", 2)”] [takes=158] [thread_id=368]
[2025/06/16 20:08:39.876 +08:00] [WARN] [peer.rs:717] [“[region 4232] 4233 handle 2 peer messages ("Tick", 2)”] [takes=338] [thread_id=367]
[2025/06/16 20:08:40.219 +08:00] [WARN] [peer.rs:717] [“[region 85] 86 handle 2 peer messages ("RaftMessage", 1), ("Tick", 1)”] [takes=240] [thread_id=368]
[2025/06/16 20:08:40.564 +08:00] [WARN] [peer.rs:717] [“[region 3664] 3665 handle 3 peer messages ("Tick", 3)”] [takes=389] [thread_id=368]
[2025/06/16 20:08:40.981 +08:00] [WARN] [peer.rs:717] [“[region 69] 70 handle 3 peer messages ("Tick", 3)”] [takes=349] [thread_id=367]
[2025/06/16 20:08:41.491 +08:00] [WARN] [peer.rs:717] [“[region 3751] 3752 handle 3 peer messages ("Tick", 3)”] [takes=361] [thread_id=368]
[2025/06/16 20:08:41.719 +08:00] [WARN] [peer.rs:717] [“[region 3718] 3719 handle 3 peer messages ("Tick", 3)”] [takes=361] [thread_id=367]
[2025/06/16 20:08:41.952 +08:00] [WARN] [peer.rs:717] [“[region 3733] 3734 handle 3 peer messages ("Tick", 3)”] [takes=362] [thread_id=368]
[2025/06/16 20:08:42.008 +08:00] [WARN] [peer.rs:717] [“[region 39] 40 handle 3 peer messages ("Tick", 3)”] [takes=408] [thread_id=367]
[2025/06/16 20:08:42.305 +08:00] [WARN] [store.rs:1192] [“[store 1] handle 8 pending peers include 7 ready, 0 entries, 10 messages and 0 snapshots”] [takes=4399] [thread_id=367]
[2025/06/16 20:08:42.761 +08:00] [WARN] [peer.rs:717] [“[region 9] 10 handle 3 peer messages ("Tick", 3)”] [takes=153] [thread_id=368]
[2025/06/16 20:08:43.667 +08:00] [WARN] [peer.rs:717] [“[region 3577] 3578 handle 2 peer messages ("RaftMessage", 1), ("Tick", 1)”] [takes=119] [thread_id=367]
[2025/06/16 20:08:44.108 +08:00] [WARN] [advance.rs:350] [“check leader rpc costs too long, to_store: 4”] [takes=3358] [thread_id=379]
[2025/06/16 20:08:44.708 +08:00] [WARN] [peer.rs:717] [“[region 4397] 4398 handle 4 peer messages ("Tick", 4)”] [takes=505] [thread_id=368]
[2025/06/16 20:08:44.785 +08:00] [WARN] [peer.rs:717] [“[region 61] 62 handle 3 peer messages ("RaftCommand", 1), ("Tick", 2)”] [takes=479] [thread_id=367]
[2025/06/16 20:08:45.299 +08:00] [WARN] [peer.rs:717] [“[region 3646] 3647 handle 4 peer messages ("Tick", 4)”] [takes=236] [thread_id=367]
[2025/06/16 20:08:45.792 +08:00] [WARN] [peer.rs:717] [“[region 4358] 4359 handle 4 peer messages ("Tick", 4)”] [takes=283] [thread_id=368]
[2025/06/16 20:08:46.104 +08:00] [WARN] [peer.rs:717] [“[region 4199] 4200 handle 2 peer messages ("RaftMessage", 1), ("Tick", 1)”] [takes=214] [thread_id=367]
[2025/06/16 20:08:46.390 +08:00] [WARN] [peer.rs:717] [“[region 9007] 9008 handle 3 peer messages ("Tick", 3)”] [takes=203] [thread_id=368]
[2025/06/16 20:08:46.662 +08:00] [WARN] [peer.rs:717] [“[region 3778] 3779 handle 4 peer messages ("Tick", 4)”] [takes=204] [thread_id=367]
[2025/06/16 20:08:47.051 +08:00] [WARN] [peer.rs:717] [“[region 3748] 3749 handle 3 peer messages ("Tick", 3)”] [takes=264] [thread_id=368]
[2025/06/16 20:08:47.714 +08:00] [WARN] [peer.rs:717] [“[region 3667] 3668 handle 4 peer messages ("Tick", 4)”] [takes=478] [thread_id=367]
[2025/06/16 20:08:48.316 +08:00] [WARN] [peer.rs:717] [“[region 4307] 4308 handle 3 peer messages ("Tick", 3)”] [takes=383] [thread_id=368]
[2025/06/16 20:08:48.733 +08:00] [WARN] [scheduler.rs:1214] [“[region 187] scheduler handle command: key_mvcc, ts: 0, details: SchedulerDetails { tracker: TrackerToken { shard_id: 35, seq: 1892387, key: 4 }, stat: Statistics { lock: CfStatistics { processed_keys: 0, get: 1, next: 0, prev: 0, seek: 0, seek_for_prev: 0, over_seek_bound: 0, flow_stats: FlowStatistics { read_keys: 0, read_bytes: 0 }, next_tombstone: 0, prev_tombstone: 0, seek_tombstone: 0, seek_for_prev_tombstone: 0, raw_value_tombstone: 0 }, write: CfStatistics { processed_keys: 0, get: 0, next: 1, prev: 0, seek: 1, seek_for_prev: 0, over_seek_bound: 0, flow_stats: FlowStatistics { read_keys: 1, read_bytes: 203 }, next_tombstone: 0, prev_tombstone: 0, seek_tombstone: 0, seek_for_prev_tombstone: 0, raw_value_tombstone: 0 }, data: CfStatistics { processed_keys: 0, get: 0, next: 0, prev: 0, seek: 1, seek_for_prev: 0, over_seek_bound: 0, flow_stats: FlowStatistics { read_keys: 1, read_bytes: 53 }, next_tombstone: 0, prev_tombstone: 0, seek_tombstone: 0, seek_for_prev_tombstone: 0, raw_value_tombstone: 0 }, processed_size: 0, load_data_hint: LoadDataHintStatistics { last_write_over_seek_bound: 0 } }, start_process_instant: Monotonic(Timespec { sec: 5937773, nsec: 864916237 }), cmd_process_nanos: 0, quota_limit_delay_nanos: 0, flow_control_nanos: 0, async_write_nanos: 0 }”] [takes=2073] [thread_id=117]
[2025/06/16 20:08:49.172 +08:00] [WARN] [peer.rs:717] [“[region 111] 112 handle 5 peer messages ("Tick", 4), ("CasualMessage", 1)”] [takes=563] [thread_id=367]
[2025/06/16 20:08:49.530 +08:00] [WARN] [peer.rs:717] [“[region 187] 188 handle 6 peer messages ("Tick", 5), ("CasualMessage", 1)”] [takes=257] [thread_id=368]

TiDBer_CjNBNP1L · 2025 年6 月 18 日 06:43

[2025/06/16 19:21:59.910 +08:00] [INFO] [index.go:215] [“compact tree index”] [revision=20411775]
[2025/06/16 19:21:59.924 +08:00] [INFO] [kvstore_compaction.go:58] [“finished scheduled compaction”] [compact-revision=20411775] [took=13.344236ms] [current-db-size-bytes=5509120] [current-db-size=“5.5 MB”] [current-db-size-in-use-bytes=737280] [current-db-size-in-use=“737 kB”]
[2025/06/16 20:06:20.208 +08:00] [INFO] [trace.go:152] [“trace[1859353720] linearizableReadLoop”] [detail=“{readStateIndex:20424070; appliedIndex:20424070; }”] [duration=132.570655ms] [start=2025/06/16 20:06:20.076 +08:00] [end=2025/06/16 20:06:20.208 +08:00] [steps=“["trace[1859353720] ‘read index received’ (duration: 132.566188ms)","trace[1859353720] ‘applied index is now lower than readState.Index’ (duration: 3.405µs)"]”]
[2025/06/16 20:06:20.209 +08:00] [WARN] [util.go:167] [“apply request took too long”] [took=132.736359ms] [expected-duration=100ms] [prefix=“read-only range “] [request=“key:"health" “] [response=“range_response_count:0 size:7”]
[2025/06/16 20:06:20.209 +08:00] [INFO] [trace.go:152] [“trace[1845240014] range”] [detail=”{range_begin:health; range_end:; response_count:0; response_revision:20415977; }”] [duration=132.871792ms] [start=2025/06/16 20:06:20.076 +08:00] [end=2025/06/16 20:06:20.209 +08:00] [steps=”["trace[1845240014] ‘agreement among raft nodes before linearized reading’ (duration: 132.708727ms)"]”]
[2025/06/16 20:06:21.347 +08:00] [WARN] [v3_server.go:887] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=13862244731413608561] [retry-timeout=500ms]
[2025/06/16 20:06:21.480 +08:00] [INFO] [trace.go:152] [“trace[1530525527] linearizableReadLoop”] [detail=“{readStateIndex:20424070; appliedIndex:20424070; }”] [duration=632.829911ms] [start=2025/06/16 20:06:20.847 +08:00] [end=2025/06/16 20:06:21.480 +08:00] [steps=“["trace[1530525527] ‘read index received’ (duration: 632.8238ms)","trace[1530525527] ‘applied index is now lower than readState.Index’ (duration: 4.991µs)"]”]
[2025/06/16 20:06:21.480 +08:00] [WARN] [util.go:167] [“apply request took too long”] [took=632.962062ms] [expected-duration=100ms] [prefix=“read-only range “] [request=“key:"health" “] [response=“range_response_count:0 size:7”]
[2025/06/16 20:06:21.480 +08:00] [INFO] [trace.go:152] [“trace[1379216432] range”] [detail=”{range_begin:health; range_end:; response_count:0; response_revision:20415977; }”] [duration=633.065278ms] [start=2025/06/16 20:06:20.847 +08:00] [end=2025/06/16 20:06:21.480 +08:00] [steps=”["trace[1379216432] ‘agreement among raft nodes before linearized reading’ (duration: 632.934432ms)"]”]
[2025/06/16 20:06:21.981 +08:00] [WARN] [v3_server.go:887] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=13862244731413608562] [retry-timeout=500ms]
[2025/06/16 20:06:22.129 +08:00] [WARN] [v3_server.go:856] [“ignored out-of-date read index response; local node read indexes queueing up and waiting to be in sync with leader”] [sent-request-id=13862244731413608562] [received-request-id=13862244731413608561]
[2025/06/16 20:06:22.129 +08:00] [INFO] [trace.go:152] [“trace[1500389343] linearizableReadLoop”] [detail=“{readStateIndex:20424070; appliedIndex:20424070; }”] [duration=648.930697ms] [start=2025/06/16 20:06:21.480 +08:00] [end=2025/06/16 20:06:22.129 +08:00] [steps=“["trace[1500389343] ‘read index received’ (duration: 648.924534ms)","trace[1500389343] ‘applied index is now lower than readState.Index’ (duration: 5.275µs)"]”]
[2025/06/16 20:06:22.129 +08:00] [WARN] [util.go:167] [“apply request took too long”] [took=1.053286507s] [expected-duration=100ms] [prefix=“read-only range “] [request=“key:"health" “] [response=“range_response_count:0 size:7”]
[2025/06/16 20:06:22.129 +08:00] [WARN] [util.go:167] [“apply request took too long”] [took=1.281922506s] [expected-duration=100ms] [prefix=“read-only range “] [request=“key:"health" “] [response=“range_response_count:0 size:7”]
[2025/06/16 20:06:22.129 +08:00] [INFO] [trace.go:152] [“trace[369652972] range”] [detail=”{range_begin:health; range_end:; response_count:0; response_revision:20415977; }”] [duration=1.053373531s] [start=2025/06/16 20:06:21.076 +08:00] [end=2025/06/16 20:06:22.129 +08:00] [steps=”["trace[369652972] ‘agreement among raft nodes before linearized reading’ (duration: 1.053256872s)"]”]
[2025/06/16 20:06:22.129 +08:00] [INFO] [trace.go:152] [“trace[1240824059] range”] [detail=”{range_begin:health; range_end:; response_count:0; response_revision:20415977; }”] [duration=1.281981382s] [start=2025/06/16 20:06:20.847 +08:00] [end=2025/06/16 20:06:22.129 +08:00] [steps=”["trace[1240824059] ‘agreement among raft nodes before linearized reading’ (duration: 1.281901899s)"]”]
[2025/06/16 20:06:22.129 +08:00] [WARN] [util.go:167] [“apply request took too long”] [took=1.260363469s] [expected-duration=100ms] [prefix=“read-only range “] [request=“key:"/pd/7403334201117082689/config" “] [response=“range_response_count:1 size:4922”]
[2025/06/16 20:06:22.129 +08:00] [INFO] [trace.go:152] [“trace[196856402] range”] [detail=”{range_begin:/pd/7403334201117082689/config; range_end:; response_count:1; response_revision:20415977; }”] [duration=1.260468763s] [start=2025/06/16 20:06:20.869 +08:00] [end=2025/06/16 20:06:22.129 +08:00] [steps=”["trace[196856402] ‘agreement among raft nodes before linearized reading’ (duration: 1.260297747s)"]”]
[2025/06/16 20:06:22.130 +08:00] [WARN] [etcdutil.go:153] [“kv gets too slow”] [request-key=/pd/7403334201117082689/config] [cost=1.261082935s]
[2025/06/16 20:06:22.138 +08:00] [INFO] [health_checker.go:299] [“evicted etcd endpoint found”] [endpoint=http://192.168.100.14:2379] [source=election-etcd-client]
[2025/06/16 20:06:22.138 +08:00] [INFO] [health_checker.go:137] [“update endpoints”] [num-change=2->1] [last-endpoints=“[http://192.168.100.13:2379,http://192.168.100.14:2379]”] [endpoints=“[http://192.168.100.13:2379]”] [source=election-etcd-client]
[2025/06/16 20:06:30.849 +08:00] [INFO] [health_checker.go:309] [“evicted etcd endpoint picked again”] [picked-count-threshold=3] [picked-count=1] [endpoint=http://192.168.100.14:2379] [source=election-etcd-client]
[2025/06/16 20:06:40.859 +08:00] [INFO] [health_checker.go:309] [“evicted etcd endpoint picked again”] [picked-count-threshold=3] [picked-count=2] [endpoint=http://192.168.100.14:2379] [source=election-etcd-client]
[2025/06/16 20:06:50.854 +08:00] [INFO] [health_checker.go:309] [“evicted etcd endpoint picked again”] [picked-count-threshold=3] [picked-count=3] [endpoint=http://192.168.100.14:2379] [source=election-etcd-client]
[2025/06/16 20:06:50.854 +08:00] [INFO] [health_checker.go:329] [“add evicted etcd endpoint back”] [picked-count-threshold=3] [picked-count=3] [endpoint=http://192.168.100.14:2379] [source=election-etcd-client]
[2025/06/16 20:06:50.855 +08:00] [INFO] [health_checker.go:137] [“update endpoints”] [num-change=1->2] [last-endpoints=“[http://192.168.100.13:2379]”] [endpoints=“[http://192.168.100.13:2379,http://192.168.100.14:2379]”] [source=election-etcd-client]
[2025/06/16 20:07:23.325 +08:00] [WARN] [v3_server.go:887] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=13862244731413608751] [retry-timeout=500ms]
[2025/06/16 20:07:23.826 +08:00] [WARN] [v3_server.go:887] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=13862244731413608751] [retry-timeout=500ms]
[2025/06/16 20:07:24.080 +08:00] [INFO] [trace.go:152] [“trace[299536561] linearizableReadLoop”] [detail=“{readStateIndex:20424087; appliedIndex:20424087; }”] [duration=1.256290127s] [start=2025/06/16 20:07:22.824 +08:00] [end=2025/06/16 20:07:24.080 +08:00] [steps=“["trace[299536561] ‘read index received’ (duration: 1.256284621s)","trace[299536561] ‘applied index is now lower than readState.Index’ (duration: 4.343µs)"]”]
[2025/06/16 20:07:24.081 +08:00] [WARN] [util.go:167] [“apply request took too long”] [took=1.256441153s] [expected-duration=100ms] [prefix=“read-only range “] [request=“key:"health" “] [response=“range_response_count:0 size:7”]
[2025/06/16 20:07:24.081 +08:00] [INFO] [trace.go:152] [“trace[239811636] range”] [detail=”{range_begin:health; range_end:; response_count:0; response_revision:20415994; }”] [duration=1.256536047s] [start=2025/06/16 20:07:22.824 +08:00] [end=2025/06/16 20:07:24.081 +08:00] [steps=”["trace[239811636] ‘agreement among raft nodes before linearized reading’ (duration: 1.256400441s)"]”]
[2025/06/16 20:07:24.082 +08:00] [WARN] [util.go:167] [“apply request took too long”] [took=1.257415674s] [expected-duration=100ms] [prefix=“read-only range “] [request=“key:"health" “] [response=“range_response_count:0 size:7”]
[2025/06/16 20:07:24.082 +08:00] [INFO] [trace.go:152] [“trace[55202091] range”] [detail=”{range_begin:health; range_end:; response_count:0; response_revision:20415994; }”] [duration=1.257481485s] [start=2025/06/16 20:07:22.824 +08:00] [end=2025/06/16 20:07:24.082 +08:00] [steps=”["trace[55202091] ‘agreement among raft nodes before linearized reading’ (duration: 1.257401076s)"]”]
[2025/06/16 20:07:24.129 +08:00] [WARN] [etcdutil.go:153] [“kv gets too slow”] [request-key=/pd/7403334201117082689/config] [cost=2.261172539s]
[2025/06/16 20:08:21.348 +08:00] [WARN] [v3_server.go:887] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=13862244731413608919] [retry-timeout=500ms]
[2025/06/16 20:08:21.568 +08:00] [INFO] [trace.go:152] [“trace[607853683] linearizableReadLoop”] [detail=“{readStateIndex:20424104; appliedIndex:20424104; }”] [duration=720.740913ms] [start=2025/06/16 20:08:20.847 +08:00] [end=2025/06/16 20:08:21.568 +08:00] [steps=“["trace[607853683] ‘read index received’ (duration: 720.735751ms)","trace[607853683] ‘applied index is now lower than readState.Index’ (duration: 4.086µs)"]”]
[2025/06/16 20:08:21.568 +08:00] [WARN] [util.go:167] [“apply request took too long”] [took=720.895782ms] [expected-duration=100ms] [prefix=“read-only range “] [request=“key:"health" “] [response=“range_response_count:0 size:7”]
[2025/06/16 20:08:21.568 +08:00] [INFO] [trace.go:152] [“trace[2139504722] range”] [detail=”{range_begin:health; range_end:; response_count:0; response_revision:20416011; }”] [duration=721.021398ms] [start=2025/06/16 20:08:20.847 +08:00] [end=2025/06/16 20:08:21.568 +08:00] [steps=”["trace[2139504722] ‘agreement among raft nodes before linearized reading’ (duration: 720.862194ms)"]”]
[2025/06/16 20:08:21.789 +08:00] [WARN] [v3_server.go:856] [“ignored out-of-date read index response; local node read indexes queueing up and waiting to be in sync with leader”] [sent-request-id=13862244731413608920] [received-request-id=13862244731413608919]
[2025/06/16 20:08:21.940 +08:00] [INFO] [trace.go:152] [“trace[1760844116] linearizableReadLoop”] [detail=“{readStateIndex:20424104; appliedIndex:20424104; }”] [duration=371.561408ms] [start=2025/06/16 20:08:21.568 +08:00] [end=2025/06/16 20:08:21.940 +08:00] [steps=“["trace[1760844116] ‘read index received’ (duration: 371.555771ms)","trace[1760844116] ‘applied index is now lower than readState.Index’ (duration: 4.718µs)"]”]
[2025/06/16 20:08:21.940 +08:00] [WARN] [util.go:167] [“apply request took too long”] [took=1.092513523s] [expected-duration=100ms] [prefix=“read-only range “] [request=“key:"health" “] [response=“range_response_count:0 size:7”]
[2025/06/16 20:08:21.940 +08:00] [INFO] [trace.go:152] [“trace[1310390733] range”] [detail=”{range_begin:health; range_end:; response_count:0; response_revision:20416011; }”] [duration=1.092571476s] [start=2025/06/16 20:08:20.847 +08:00] [end=2025/06/16 20:08:21.940 +08:00] [steps=”["trace[1310390733] ‘agreement among raft nodes before linearized reading’ (duration: 1.092489827s)"]”]
[2025/06/16 20:08:21.954 +08:00] [INFO] [health_checker.go:299] [“evicted etcd endpoint found”] [endpoint=http://192.168.100.14:2379] [source=election-etcd-client]
[2025/06/16 20:08:21.954 +08:00] [INFO] [health_checker.go:137] [“update endpoints”] [num-change=2->1] [last-endpoints=“[http://192.168.100.13:2379,http://192.168.100.14:2379]”] [endpoints=“[http://192.168.100.13:2379]”] [source=election-etcd-client]
[2025/06/16 20:08:21.955 +08:00] [WARN] [etcdutil.go:153] [“kv gets too slow”] [request-key=/pd/7403334201117082689/config] [cost=1.085863535s]
[2025/06/16 20:08:22.528 +08:00] [INFO] [trace.go:152] [“trace[1273308651] linearizableReadLoop”] [detail=“{readStateIndex:20424104; appliedIndex:20424104; }”] [duration=452.999736ms] [start=2025/06/16 20:08:22.075 +08:00] [end=2025/06/16 20:08:22.528 +08:00] [steps=“["trace[1273308651] ‘read index received’ (duration: 452.99325ms)","trace[1273308651] ‘applied index is now lower than readState.Index’ (duration: 4.925µs)"]”]
[2025/06/16 20:08:22.529 +08:00] [WARN] [util.go:167] [“apply request took too long”] [took=453.139244ms] [expected-duration=100ms] [prefix=“read-only range “] [request=“key:"health" “] [response=“range_response_count:0 size:7”]
[2025/06/16 20:08:22.529 +08:00] [INFO] [trace.go:152] [“trace[324714432] range”] [detail=”{range_begin:health; range_end:; response_count:0; response_revision:20416011; }”] [duration=453.269796ms] [start=2025/06/16 20:08:22.075 +08:00] [end=2025/06/16 20:08:22.529 +08:00] [steps=”["trace[324714432] ‘agreement among raft nodes before linearized reading’ (duration: 453.100487ms)"]”]
[2025/06/16 20:08:23.249 +08:00] [INFO] [trace.go:152] [“trace[892039986] linearizableReadLoop”] [detail=“{readStateIndex:20424104; appliedIndex:20424104; }”] [duration=172.977763ms] [start=2025/06/16 20:08:23.076 +08:00] [end=2025/06/16 20:08:23.249 +08:00] [steps=“["trace[892039986] ‘read index received’ (duration: 172.967122ms)","trace[892039986] ‘applied index is now lower than readState.Index’ (duration: 8.583µs)"]”]
[2025/06/16 20:08:23.249 +08:00] [WARN] [util.go:167] [“apply request took too long”] [took=173.093392ms] [expected-duration=100ms] [prefix=“read-only range “] [request=“key:"health" “] [response=“range_response_count:0 size:7”]
[2025/06/16 20:08:23.249 +08:00] [INFO] [trace.go:152] [“trace[450284657] range”] [detail=”{range_begin:health; range_end:; response_count:0; response_revision:20416011; }”] [duration=173.189809ms] [start=2025/06/16 20:08:23.076 +08:00] [end=2025/06/16 20:08:23.249 +08:00] [steps=”["trace[450284657] ‘agreement among raft nodes before linearized reading’ (duration: 173.06984ms)"]”]
[2025/06/16 20:08:23.887 +08:00] [WARN] [etcdutil.go:153] [“kv gets too slow”] [request-key=/pd/7403334201117082689/config] [cost=1.01831353s]
[2025/06/16 20:08:24.901 +08:00] [INFO] [trace.go:152] [“trace[1792392318] linearizableReadLoop”] [detail=“{readStateIndex:20424104; appliedIndex:20424104; }”] [duration=353.588427ms] [start=2025/06/16 20:08:24.547 +08:00] [end=2025/06/16 20:08:24.901 +08:00] [steps=“["trace[1792392318] ‘read index received’ (duration: 353.57832ms)","trace[1792392318] ‘applied index is now lower than readState.Index’ (duration: 4.514µs)"]”]
[2025/06/16 20:08:24.901 +08:00] [WARN] [util.go:167] [“apply request took too long”] [took=353.777595ms] [expected-duration=100ms] [prefix=“read-only range “] [request=“key:"health" “] [response=“range_response_count:0 size:7”]
[2025/06/16 20:08:24.901 +08:00] [INFO] [trace.go:152] [“trace[36253279] range”] [detail=”{range_begin:health; range_end:; response_count:0; response_revision:20416011; }”] [duration=353.867629ms] [start=2025/06/16 20:08:24.547 +08:00] [end=2025/06/16 20:08:24.901 +08:00] [steps=”["trace[36253279] ‘agreement among raft nodes before linearized reading’ (duration: 353.745336ms)"]”]
[2025/06/16 20:08:25.422 +08:00] [WARN] [v3_server.go:887] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=13862244731413608931] [retry-timeout=500ms]
[2025/06/16 20:08:25.499 +08:00] [INFO] [trace.go:152] [“trace[1387589283] linearizableReadLoop”] [detail=“{readStateIndex:20424104; appliedIndex:20424104; }”] [duration=578.306964ms] [start=2025/06/16 20:08:24.921 +08:00] [end=2025/06/16 20:08:25.499 +08:00] [steps=“["trace[1387589283] ‘read index received’ (duration: 578.300139ms)","trace[1387589283] ‘applied index is now lower than readState.Index’ (duration: 5.363µs)"]”]
[2025/06/16 20:08:25.499 +08:00] [WARN] [util.go:167] [“apply request took too long”] [took=578.484173ms] [expected-duration=100ms] [prefix=“read-only range “] [request=“key:"/topology/tidb/" range_end:"/topology/tidb0" “] [response=“range_response_count:4 size:709”]
[2025/06/16 20:08:25.499 +08:00] [INFO] [trace.go:152] [“trace[296449963] range”] [detail=”{range_begin:/topology/tidb/; range_end:/topology/tidb0; response_count:4; response_revision:20416011; }”] [duration=578.567438ms] [start=2025/06/16 20:08:24.921 +08:00] [end=2025/06/16 20:08:25.499 +08:00] [steps=”["trace[296449963] ‘agreement among raft nodes before linearized reading’ (duration: 578.416142ms)"]”]
[2025/06/16 20:08:25.742 +08:00] [WARN] [v3_server.go:856] [“ignored out-of-date read index response; local node read indexes queueing up and waiting to be in sync with leader”] [sent-request-id=13862244731413608932] [received-request-id=13862244731413608931]
[2025/06/16 20:08:25.999 +08:00] [WARN] [v3_server.go:887] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=13862244731413608932] [retry-timeout=500ms]
[2025/06/16 20:08:26.239 +08:00] [INFO] [trace.go:152] [“trace[1472358481] linearizableReadLoop”] [detail=“{readStateIndex:20424104; appliedIndex:20424104; }”] [duration=739.930921ms] [start=2025/06/16 20:08:25.499 +08:00] [end=2025/06/16 20:08:26.239 +08:00] [steps=“["trace[1472358481] ‘read index received’ (duration: 739.926317ms)","trace[1472358481] ‘applied index is now lower than readState.Index’ (duration: 3.454µs)"]”]
[2025/06/16 20:08:26.239 +08:00] [WARN] [util.go:167] [“apply request took too long”] [took=1.163220785s] [expected-duration=100ms] [prefix="read-only range "] [request="key:"health" "] [response=“range_response_count:0 size:7”]

TiDBer_CjNBNP1L · 2025 年6 月 18 日 06:43

[2025/06/16 19:02:42.545 +08:00] [INFO] [util.rs:639] [“connecting to PD endpoint”] [endpoints=http://192.168.100.13:2379] [thread_id=12]
[2025/06/16 19:12:42.548 +08:00] [INFO] [util.rs:639] [“connecting to PD endpoint”] [endpoints=http://192.168.100.13:2379] [thread_id=12]
[2025/06/16 19:22:42.552 +08:00] [INFO] [util.rs:639] [“connecting to PD endpoint”] [endpoints=http://192.168.100.13:2379] [thread_id=12]
[2025/06/16 19:23:30.388 +08:00] [INFO] [compact.rs:337] [“compact range finished”] [compact_options=“ManualCompactionOptions { exclusive_manual: false, max_subcompactions: 1, bottommost_level_force: false }”] [time_takes=4.863006663s] [cf=default] [range_end=7A7480000000000000FFC600000000000000F8] [range_start=7A6D00000000000000F8] [thread_id=345]
[2025/06/16 19:23:37.742 +08:00] [INFO] [compact.rs:337] [“compact range finished”] [compact_options=“ManualCompactionOptions { exclusive_manual: false, max_subcompactions: 1, bottommost_level_force: false }”] [time_takes=7.353958088s] [cf=write] [range_end=7A7480000000000000FFC600000000000000F8] [range_start=7A6D00000000000000F8] [thread_id=345]
[2025/06/16 19:28:25.528 +08:00] [INFO] [compact.rs:337] [“compact range finished”] [compact_options=“ManualCompactionOptions { exclusive_manual: false, max_subcompactions: 1, bottommost_level_force: false }”] [time_takes=52.965µs] [cf=default] [range_end=7A7480000000000001FF9100000000000000F8] [range_start=7A7480000000000000FFC600000000000000F8] [thread_id=345]
[2025/06/16 19:28:25.529 +08:00] [INFO] [compact.rs:337] [“compact range finished”] [compact_options=“ManualCompactionOptions { exclusive_manual: false, max_subcompactions: 1, bottommost_level_force: false }”] [time_takes=15.848µs] [cf=write] [range_end=7A7480000000000001FF9100000000000000F8] [range_start=7A7480000000000000FFC600000000000000F8] [thread_id=345]
[2025/06/16 19:32:42.555 +08:00] [INFO] [util.rs:639] [“connecting to PD endpoint”] [endpoints=http://192.168.100.13:2379] [thread_id=12]
[2025/06/16 19:33:25.533 +08:00] [INFO] [compact.rs:337] [“compact range finished”] [compact_options=“ManualCompactionOptions { exclusive_manual: false, max_subcompactions: 1, bottommost_level_force: false }”] [time_takes=57.972µs] [cf=default] [range_end=7A7480000000000001FFE800000000000000F8] [range_start=7A7480000000000001FF9100000000000000F8] [thread_id=345]
[2025/06/16 19:33:26.343 +08:00] [INFO] [compact.rs:337] [“compact range finished”] [compact_options=“ManualCompactionOptions { exclusive_manual: false, max_subcompactions: 1, bottommost_level_force: false }”] [time_takes=810.332127ms] [cf=write] [range_end=7A7480000000000001FFE800000000000000F8] [range_start=7A7480000000000001FF9100000000000000F8] [thread_id=345]
[2025/06/16 19:42:42.559 +08:00] [INFO] [util.rs:639] [“connecting to PD endpoint”] [endpoints=http://192.168.100.13:2379] [thread_id=12]
[2025/06/16 19:52:42.562 +08:00] [INFO] [util.rs:639] [“connecting to PD endpoint”] [endpoints=http://192.168.100.13:2379] [thread_id=12]
[2025/06/16 19:58:25.552 +08:00] [INFO] [compact.rs:337] [“compact range finished”] [compact_options=“ManualCompactionOptions { exclusive_manual: false, max_subcompactions: 1, bottommost_level_force: false }”] [time_takes=205.611µs] [cf=default] [range_end=7A7480000000000000FFC600000000000000F8] [range_start=7A6D00000000000000F8] [thread_id=345]
[2025/06/16 19:58:32.910 +08:00] [INFO] [compact.rs:337] [“compact range finished”] [compact_options=“ManualCompactionOptions { exclusive_manual: false, max_subcompactions: 1, bottommost_level_force: false }”] [time_takes=7.357293187s] [cf=write] [range_end=7A7480000000000000FFC600000000000000F8] [range_start=7A6D00000000000000F8] [thread_id=345]
[2025/06/16 20:02:42.565 +08:00] [INFO] [util.rs:639] [“connecting to PD endpoint”] [endpoints=http://192.168.100.13:2379] [thread_id=12]
[2025/06/16 20:03:25.556 +08:00] [INFO] [compact.rs:337] [“compact range finished”] [compact_options=“ManualCompactionOptions { exclusive_manual: false, max_subcompactions: 1, bottommost_level_force: false }”] [time_takes=52.748µs] [cf=default] [range_end=7A7480000000000001FF9100000000000000F8] [range_start=7A7480000000000000FFC600000000000000F8] [thread_id=345]
[2025/06/16 20:03:25.556 +08:00] [INFO] [compact.rs:337] [“compact range finished”] [compact_options=“ManualCompactionOptions { exclusive_manual: false, max_subcompactions: 1, bottommost_level_force: false }”] [time_takes=18.527µs] [cf=write] [range_end=7A7480000000000001FF9100000000000000F8] [range_start=7A7480000000000000FFC600000000000000F8] [thread_id=345]
[2025/06/16 20:08:25.560 +08:00] [INFO] [compact.rs:337] [“compact range finished”] [compact_options=“ManualCompactionOptions { exclusive_manual: false, max_subcompactions: 1, bottommost_level_force: false }”] [time_takes=62.24µs] [cf=default] [range_end=7A7480000000000001FFE800000000000000F8] [range_start=7A7480000000000001FF9100000000000000F8] [thread_id=345]
[2025/06/16 20:08:26.385 +08:00] [INFO] [compact.rs:337] [“compact range finished”] [compact_options=“ManualCompactionOptions { exclusive_manual: false, max_subcompactions: 1, bottommost_level_force: false }”] [time_takes=825.514116ms] [cf=write] [range_end=7A7480000000000001FFE800000000000000F8] [range_start=7A7480000000000001FF9100000000000000F8] [thread_id=345]
[2025/06/16 20:08:31.159 +08:00] [ERROR] [pd.rs:1341] [“store heartbeat failed”] [err=“Grpc(RpcFailure(RpcStatus { code: 4-DEADLINE_EXCEEDED, message: "Deadline Exceeded", details: }))”] [thread_id=32]
[2025/06/16 20:08:31.369 +08:00] [WARN] [pd.rs:1773] [“report min resolved_ts failed”] [err=“Grpc(RpcFailure(RpcStatus { code: 4-DEADLINE_EXCEEDED, message: "Deadline Exceeded", details: }))”] [thread_id=32]
[2025/06/16 20:08:35.495 +08:00] [WARN] [advance.rs:350] [“check leader rpc costs too long, to_store: 1”] [takes=2571] [thread_id=379]
[2025/06/16 20:08:38.376 +08:00] [WARN] [pd.rs:1773] [“report min resolved_ts failed”] [err=“Grpc(RpcFailure(RpcStatus { code: 4-DEADLINE_EXCEEDED, message: "Deadline Exceeded", details: }))”] [thread_id=32]
[2025/06/16 20:08:40.378 +08:00] [WARN] [pd.rs:1773] [“report min resolved_ts failed”] [err=“Grpc(RpcFailure(RpcStatus { code: 4-DEADLINE_EXCEEDED, message: "Deadline Exceeded", details: }))”] [thread_id=32]
[2025/06/16 20:08:41.160 +08:00] [ERROR] [pd.rs:1341] [“store heartbeat failed”] [err=“Grpc(RpcFailure(RpcStatus { code: 4-DEADLINE_EXCEEDED, message: "Deadline Exceeded", details: }))”] [thread_id=32]
[2025/06/16 20:08:41.379 +08:00] [WARN] [pd.rs:1773] [“report min resolved_ts failed”] [err=“Grpc(RpcFailure(RpcStatus { code: 4-DEADLINE_EXCEEDED, message: "Deadline Exceeded", details: }))”] [thread_id=32]
[2025/06/16 20:08:44.024 +08:00] [ERROR] [util.rs:497] [“request failed, retry”] [err_code=KV:Pd:Grpc] [err=“Grpc(RpcFailure(RpcStatus { code: 4-DEADLINE_EXCEEDED, message: "Deadline Exceeded", details: }))”] [thread_id=373]
[2025/06/16 20:08:44.382 +08:00] [WARN] [pd.rs:1773] [“report min resolved_ts failed”] [err=“Grpc(RpcFailure(RpcStatus { code: 4-DEADLINE_EXCEEDED, message: "Deadline Exceeded", details: }))”] [thread_id=32]
[2025/06/16 20:08:46.025 +08:00] [ERROR] [util.rs:497] [“request failed, retry”] [err_code=KV:Pd:Grpc] [err=“Grpc(RpcFailure(RpcStatus { code: 4-DEADLINE_EXCEEDED, message: "Deadline Exceeded", details: }))”] [thread_id=373]
[2025/06/16 20:08:47.386 +08:00] [WARN] [pd.rs:1773] [“report min resolved_ts failed”] [err=“Grpc(RpcFailure(RpcStatus { code: 4-DEADLINE_EXCEEDED, message: "Deadline Exceeded", details: }))”] [thread_id=32]
[2025/06/16 20:08:48.026 +08:00] [ERROR] [util.rs:497] [“request failed, retry”] [err_code=KV:Pd:Grpc] [err=“Grpc(RpcFailure(RpcStatus { code: 4-DEADLINE_EXCEEDED, message: "Deadline Exceeded", details: }))”] [thread_id=373]
[2025/06/16 20:08:48.026 +08:00] [INFO] [util.rs:639] [“connecting to PD endpoint”] [endpoints=http://192.168.100.13:2379] [thread_id=373]
[2025/06/16 20:08:48.219 +08:00] [INFO] [] [“ipv4:192.168.100.14:50960: Keepalive watchdog fired. Closing transport.”] [thread_id=428]
[2025/06/16 20:08:48.219 +08:00] [INFO] [kv.rs:1067] [“kv rpc failed”] [err=RemoteStopped] [request=batch_commands] [thread_id=8]
[2025/06/16 20:08:48.239 +08:00] [INFO] [] [“ipv4:192.168.100.14:20160: Keepalive watchdog fired. Closing transport.”] [thread_id=429]
[2025/06/16 20:08:48.240 +08:00] [ERROR] [raft_client.rs:585] [“connection aborted”] [addr=192.168.100.14:20160] [receiver_err=“Some(RpcFailure(RpcStatus { code: 14-UNAVAILABLE, message: "keepalive watchdog timeout", details: }))”] [sink_error=“Some(RpcFinished(Some(RpcStatus { code: 14-UNAVAILABLE, message: "keepalive watchdog timeout", details: })))”] [store_id=1] [thread_id=7]
[2025/06/16 20:08:48.240 +08:00] [ERROR] [raft_client.rs:904] [“connection abort”] [addr=192.168.100.14:20160] [store_id=1] [thread_id=329]
[2025/06/16 20:08:48.240 +08:00] [INFO] [store.rs:3267] [“broadcasting unreachable”] [unreachable_store_id=1] [store_id=4] [thread_id=367]
[2025/06/16 20:08:48.869 +08:00] [WARN] [raft.rs:2041] [“stepped down to follower since quorum is not active”] [raft_id=4285] [region_id=4283] [thread_id=368]
[2025/06/16 20:08:48.869 +08:00] [INFO] [raft.rs:1151] [“became follower at term 11”] [term=11] [raft_id=4285] [region_id=4283] [thread_id=368]
[2025/06/16 20:08:48.869 +08:00] [INFO] [endpoint.rs:748] [“deregister observe region”] [observe_id=ObserveId(92)] [region_id=4283] [store_id=Some(4)] [thread_id=326]
[2025/06/16 20:08:49.105 +08:00] [INFO] [util.rs:639] [“connecting to PD endpoint”] [endpoints=http://192.168.100.14:2379] [thread_id=373]
[2025/06/16 20:08:49.949 +08:00] [INFO] [raft_client.rs:830] [“resolve store address ok”] [addr=192.168.100.14:20160] [store_id=1] [thread_id=329]
[2025/06/16 20:08:49.949 +08:00] [INFO] [raft_client.rs:699] [“server: new connection with tikv endpoint”] [store_id=1] [addr=192.168.100.14:20160] [thread_id=329]
[2025/06/16 20:08:51.072 +08:00] [INFO] [util.rs:808] [“connected to PD member”] [endpoints=http://192.168.100.14:2379] [thread_id=373]
[2025/06/16 20:08:51.072 +08:00] [WARN] [pd.rs:1773] [“report min resolved_ts failed”] [err=“Grpc(RpcFailure(RpcStatus { code: 14-UNAVAILABLE, message: "not leader", details: }))”] [thread_id=32]
[2025/06/16 20:08:51.072 +08:00] [ERROR] [pd.rs:1341] [“store heartbeat failed”] [err=“Grpc(RpcFailure(RpcStatus { code: 14-UNAVAILABLE, message: "not leader", details: }))”] [thread_id=32]
[2025/06/16 20:08:51.073 +08:00] [WARN] [pd.rs:1773] [“report min resolved_ts failed”] [err=“Grpc(RpcFailure(RpcStatus { code: 14-UNAVAILABLE, message: "not leader", details: }))”] [thread_id=32]
[2025/06/16 20:08:51.073 +08:00] [INFO] [util.rs:248] [“heartbeat sender and receiver are stale, refreshing …”] [thread_id=373]
[2025/06/16 20:08:51.073 +08:00] [INFO] [util.rs:261] [“buckets sender and receiver are stale, refreshing …”] [thread_id=373]
[2025/06/16 20:08:51.073 +08:00] [INFO] [client.rs:652] [“cancel region heartbeat sender”] [thread_id=7]
[2025/06/16 20:08:51.073 +08:00] [INFO] [util.rs:280] [“acquire_token_buckets sender and receiver are stale, refreshing …”] [thread_id=373]
[2025/06/16 20:08:51.074 +08:00] [INFO] [tso.rs:162] [“TSO worker terminated”] [receiver_cause=None] [sender_cause=None] [thread_id=424]
[2025/06/16 20:08:51.074 +08:00] [INFO] [client.rs:153] [“TSO stream is closed, reconnect to PD”] [thread_id=12]
[2025/06/16 20:08:51.074 +08:00] [WARN] [raft.rs:2041] [“stepped down to follower since quorum is not active”] [raft_id=4201] [region_id=4199] [thread_id=368]
[2025/06/16 20:08:51.074 +08:00] [INFO] [raft.rs:1151] [“became follower at term 10”] [term=10] [raft_id=4201] [region_id=4199] [thread_id=368]
[2025/06/16 20:08:51.074 +08:00] [INFO] [endpoint.rs:748] [“deregister observe region”] [observe_id=ObserveId(89)] [region_id=4199] [store_id=Some(4)] [thread_id=326]
[2025/06/16 20:08:51.074 +08:00] [INFO] [util.rs:303] [“update pd client”] [via=] [leader=http://192.168.100.14:2379] [prev_via=] [prev_leader=http://192.168.100.14:2379] [thread_id=373]
[2025/06/16 20:08:51.074 +08:00] [INFO] [util.rs:435] [“trying to update PD client done”] [spend=3.04888035s] [thread_id=373]
[2025/06/16 20:08:51.075 +08:00] [INFO] [util.rs:639] [“connecting to PD endpoint”] [endpoints=http://192.168.100.13:2379] [thread_id=12]
[2025/06/16 20:08:51.077 +08:00] [INFO] [util.rs:639] [“connecting to PD endpoint”] [endpoints=http://192.168.100.14:2379] [thread_id=12]
[2025/06/16 20:08:51.082 +08:00] [INFO] [util.rs:808] [“connected to PD member”] [endpoints=http://192.168.100.14:2379] [thread_id=12]
[2025/06/16 20:08:51.082 +08:00] [INFO] [util.rs:248] [“heartbeat sender and receiver are stale, refreshing …”] [thread_id=12]
[2025/06/16 20:08:51.082 +08:00] [INFO] [util.rs:261] [“buckets sender and receiver are stale, refreshing …”] [thread_id=12]
[2025/06/16 20:08:51.083 +08:00] [INFO] [util.rs:280] [“acquire_token_buckets sender and receiver are stale, refreshing …”] [thread_id=12]
[2025/06/16 20:08:51.084 +08:00] [INFO] [util.rs:303] [“update pd client”] [via=] [leader=http://192.168.100.14:2379] [prev_via=] [prev_leader=http://192.168.100.14:2379] [thread_id=12]
[2025/06/16 20:08:51.084 +08:00] [INFO] [util.rs:435] [“trying to update PD client done”] [spend=9.935648ms] [thread_id=12]
[2025/06/16 20:08:51.085 +08:00] [INFO] [tso.rs:162] [“TSO worker terminated”] [receiver_cause=None] [sender_cause=None] [thread_id=430]
[2025/06/16 20:08:51.096 +08:00] [INFO] [raft_client.rs:873] [“connection established”] [try_count=1] [msg_count=29] [cost=2.856168653s] [addr=192.168.100.14:20160] [store_id=1] [thread_id=329]
[2025/06/16 20:08:51.096 +08:00] [ERROR] [util.rs:497] [“request failed, retry”] [err_code=KV:Pd:Grpc] [err=“Grpc(RpcFailure(RpcStatus { code: 14-UNAVAILABLE, message: "not leader", details: }))”] [thread_id=373]
[2025/06/16 20:08:51.097 +08:00] [ERROR] [util.rs:497] [“request failed, retry”] [err_code=KV:Pd:Grpc] [err=“Grpc(RpcFailure(RpcStatus { code: 14-UNAVAILABLE, message: "not leader", details: }))”] [thread_id=373]
[2025/06/16 20:08:51.097 +08:00] [ERROR] [util.rs:497] [“request failed, retry”] [err_code=KV:Pd:Grpc] [err=“Grpc(RpcFailure(RpcStatus { code: 14-UNAVAILABLE, message: "not leader", details: }))”] [thread_id=373]
[2025/06/16 20:08:51.243 +08:00] [INFO] [raft.rs:1556] [“starting a new election”] [term=11] [raft_id=5006] [region_id=5004] [thread_id=368]
[2025/06/16 20:08:51.243 +08:00] [INFO] [raft.rs:1201] [“became pre-candidate at term 11”] [term=11] [raft_id=5006] [region_id=5004] [thread_id=368]
[2025/06/16 20:08:51.243 +08:00] [INFO] [raft.rs:1321] [“broadcasting vote request”] [to=“[5005]”] [log_index=12] [log_term=11] [term=11] [type=MsgRequestPreVote] [raft_id=5006] [region_id=5004] [thread_id=368]
[2025/06/16 20:08:51.243 +08:00] [INFO] [raft.rs:1556] [“starting a new election”] [term=11] [raft_id=3606] [region_id=3604] [thread_id=367]

TiDBer_CjNBNP1L · 2025 年6 月 18 日 06:53

cchouqiang · 2025 年6 月 20 日 08:48

一台机器部署了这么多组件，把内存都占满了

zhanggame1 · 2025 年6 月 20 日 08:54

2台怎么部署的，什么架构，很好奇