tidb组件重启报错:sync: unlock of unlocked mutex

一个好的问题描述有利于社区小伙伴更快帮你定位到问题,高效解决你的问题

【TiDB 使用环境】生产环境
【TiDB 版本】v7.5.0
【部署方式】自建机房K8S部署
【操作系统/CPU 架构/芯片详情】

目前使用k8s部署了一个3pd,3tidb,3kv节点的集群,如下图

但是tidb组件频繁重启,日志内容如下:

root@pu20k8sp101039:~# kubectl logs -n prod tidb-ims-tidb-0 -c tidb --previous | egrep -i “fatal|panic|signal” -n | tail -n 50
8859:fatal error: sync: unlock of unlocked mutex
8862:sync.fatal({0x59bbf1d?, 0x30?})
8863: /usr/local/go/src/runtime/panic.go:1061 +0x18
11528:github.com/pingcap/tidb/pkg/util/signal.SetupSignalHandler.func2()
11529: /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/util/signal/signal_posix.go:53 +0x37
11530:created by github.com/pingcap/tidb/pkg/util/signal.SetupSignalHandler in goroutine 1
11531: /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/util/signal/signal_posix.go:52 +0x186
11834:os/signal.signal_recv()
11836:os/signal.loop()
11837: /usr/local/go/src/os/signal/signal_unix.go:23 +0x13
11838:created by os/signal.Notify.func1.1 in goroutine 1
11839: /usr/local/go/src/os/signal/signal.go:151 +0x1f
11842:github.com/pingcap/tidb/pkg/util/signal.SetupSignalHandler.func1()
11843: /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/util/signal/signal_posix.go:37 +0x51
11844:created by github.com/pingcap/tidb/pkg/util/signal.SetupSignalHandler in goroutine 1
11845: /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/util/signal/signal_posix.go:34 +0xa5

Containers:
slowlog:
Container ID: containerd://11f2be6771b93bc7e2163242c6b03428f454131f79920dfd2209ab5d16a3e70e
Image: registry-harbor.yafex.cn/base/alpine:3.16.0
Image ID: registry-harbor.yafex.cn/base/alpine@sha256:4ff3ca91275773af45cb4b0834e12b7eb47d1c18f770a0b151381cd227f4c253
Port:
Host Port:
Command:
sh
-c
touch /var/log/tidb/slowlog; tail -n0 -F /var/log/tidb/slowlog;
State: Running
Started: Thu, 11 Jun 2026 15:31:49 +0800
Ready: True
Restart Count: 0
Environment:
Mounts:
/var/log/tidb from slowlog (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kn25r (ro)
tidb:
Container ID: containerd://a600a3d5502197fc7ad83496f2d6fffaae599ca923cde50172578e81e0f102e1
Image: registry-harbor.yafex.cn/base/pingcap/tidb:v7.5.0
Image ID: registry-harbor.yafex.cn/base/pingcap/tidb@sha256:0c9de57434eafa090a11cb510e0748bb39c39837538f15b348a7fc968b128243
Ports: 4000/TCP, 10080/TCP
Host Ports: 0/TCP, 0/TCP
Command:
/bin/sh
/usr/local/bin/tidb_start_script.sh
State: Running
Started: Tue, 07 Jul 2026 17:11:40 +0800
Last State: Terminated
** Reason: Error**
** Exit Code: 2**
Started: Tue, 07 Jul 2026 13:50:22 +0800
Finished: Tue, 07 Jul 2026 17:11:38 +0800
Ready: True
Restart Count: 69
Limits:
cpu: 12
memory: 50Gi
Requests:
cpu: 2
memory: 6Gi
Readiness: tcp-socket :4000 delay=10s timeout=1s period=10s #success=1 #failure=3
Environment:
CLUSTER_NAME: tidb-ims
TZ: Asia/Shanghai
BINLOG_ENABLED: false
SLOW_LOG_FILE: /var/log/tidb/slowlog
POD_NAME: tidb-ims-tidb-0 (v1:metadata.name)
NAMESPACE: prod (v1:metadata.namespace)
HEADLESS_SERVICE_NAME: tidb-ims-tidb-peer
Mounts:
/etc/podinfo from annotations (ro)
/etc/tidb from config (ro)
/usr/local/bin from startup-script (ro)
/var/log/tidb from slowlog (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kn25r (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
annotations:
Type: DownwardAPI (a volume populated by information about the pod)
Items:
metadata.annotations → annotations
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: tidb-ims-tidb-3836383
Optional: false
startup-script:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: tidb-ims-tidb-3836383
Optional: false
slowlog:
Type: EmptyDir (a temporary directory that shares a pod’s lifetime)
Medium:
SizeLimit:
kube-api-access-kn25r:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: tidb-ims-db=enabled
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:

看了日志,也没有OOM的相关错误,TIDB组件对应的NODE节点服务器上也没有相关的重启问题,倒是看到:fatal error: sync: unlock of unlocked mutex
这种错误,这是这个版本的bug吗,还是说应用程序再使用锁相关出现了问题?

这看起来是TiDB进程内部goroutine并发操作互斥锁时出现问题。建议按以下步骤排查:

  1. 先检查TiDB Pod资源限制是否足够,特别是CPU。

OOM是记录在tidb.log里的.

  • 查看 tidb.log,可发现如下日志条目:
    • OOM 相关的 Alarm:[WARN] [memory_usage_alarm.go:139] ["tidb-server has the risk of OOM because of memory usage exceeds alarm ratio. Running SQLs and heap profile will be recorded in record path"]。关于该日志的详细说明,请参考 memory-usage-alarm-ratio
    • 重启相关的日志条目:[INFO] [printer.go:33] ["Welcome to TiDB."]
  1. 自定义 shell 启动脚本未透传 SIGTERM(无trap转发),会造成 tidb-server 收到双重终止信号
  2. 激进就绪探针(10s 探测周期、3 次失败即杀容器)会持续高频下发 SIGTERM,大幅提升并发信号竞争概率;
  3. Burstable QoS 容器 CPU 调度抢占,goroutine 时序抖动,更容易复现锁非法释放 panicTiDB。 这也是该 Bug 在 K8s 部署场景远高于物理机 / 虚拟机 TiUP 部署的核心原因。

资源是足够的,CPU使用率也低,内存的限制也是够的

没有看到相关的OOM的日志:

那这个BUG在后续哪个版本有解决吗?

这是 Go runtime 的 fatal error,通常由并发 bug 触发(如重复 unlock 或 signal handler 中不当操作)。v7.5.0 已较老,建议升级到该大版本最新 patch,同时检查是否有 goroutine 泄漏导致内存压力触发竞态。

旧版本 binlog/pump 逻辑缺陷,并发下锁释放逻辑错乱;

小版本是多少,可能是版本bug