救命呀，OOM报错，导致节点异常无法正常连接，还有其他需要排查的吗

[一只大芒果 (Ti D Ber L B8j9nt J)]

一个好的问题描述有利于社区小伙伴更快帮你定位到问题，高效解决你的问题

【TiDB 使用环境】测试环境
【TiDB 版本】8.5.4
3PD/TiDB同服务器部署各16DB,3TiKV各32GB,1TiFlash32G

这样的配置低吗？目前PD平均使用率48%，TiKV平均使用率70%

报错
9001:PD server timeout: start timestamp may fall behind safe point
排查：
– 1
2026-06-24 13:25:33 (UTC+08:00)TiDB 1.6.1.1:4000[memoryusagealarm.go:215] [“tidb-server has the risk of OOM because of memory usage exceeds alarm ratio. Running SQLs and heap profile will be recorded in record path”] [“is tidb_server_memory_limit set”=true] [tidb_server_memory_limit=13193022640] [“tidb-server memory usage”=13106027264] [memory-usage-alarm-ratio=0.7] [“record path”=/app/tidb-deploy/tidb-4000/log/oom_record]
2026-06-24 13:25:34 (UTC+08:00)TiDB 1.6.1.1:4000[memoryusagealarm.go:215] [“tidb-server has the risk of OOM because of memory usage exceeds alarm ratio. Running SQLs and heap profile will be recorded in record path”] [“is tidb_server_memory_limit set”=true] [tidb_server_memory_limit=13193022640] [“tidb-server memory usage”=9699310160] [memory-usage-alarm-ratio=0.7] [“record path”=/app/tidb-deploy/tidb-4000/log/oom_record]
2026-06-24 13:25:54 (UTC+08:00)TiDB 1.6.1.1:4000[memoryusagealarm.go:215] [“tidb-server has the risk of OOM because of memory usage grows too fast. Running SQLs and heap profile will be recorded in record path”] [“is tidb_server_memory_limit set”=true] [tidb_server_memory_limit=13193022640] [“tidb-server memory usage”=11025140336] [memory-usage-alarm-ratio=0.7] [“record path”=/app/tidb-deploy/tidb-4000/log/oom_record]
2026-06-24 13:26:13 (UTC+08:00)TiDB 1.6.1.1:4000[memoryusagealarm.go:215] [“tidb-server has the risk of OOM because of memory usage grows too fast. Running SQLs and heap profile will be recorded in record path”] [“is tidb_server_memory_limit set”=true] [tidb_server_memory_limit=13193022640] [“tidb-server memory usage”=12354325072] [memory-usage-alarm-ratio=0.7] [“record path”=/app/tidb-deploy/tidb-4000/log/oom_record]
2026-06-24 13:27:13 (UTC+08:00)TiDB 1.6.1.1:4000[memoryusagealarm.go:215] [“tidb-server has the risk of OOM because of memory usage exceeds alarm ratio. Running SQLs and heap profile will be recorded in record path”] [“is tidb_server_memory_limit set”=true] [tidb_server_memory_limit=13193022640] [“tidb-server memory usage”=12239751192] [memory-usage-alarm-ratio=0.7] [“record path”=/app/tidb-deploy/tidb-4000/log/oom_record]
2026-06-24 13:29:32 (UTC+08:00)TiDB 1.6.1.1:4000[memoryusagealarm.go:215] [“tidb-server has the risk of OOM because of memory usage exceeds alarm ratio. Running SQLs and heap profile will be recorded in record path”] [“is tidb_server_memory_limit set”=true] [tidb_server_memory_limit=13193022640] [“tidb-server memory usage”=12974118272] [memory-usage-alarm-ratio=0.7] [“record path”=/app/tidb-deploy/tidb-4000/log/oom_record]
2026-06-24 13:30:32 (UTC+08:00)TiDB 1.6.1.1:4000[memoryusagealarm.go:215] [“tidb-server has the risk of OOM because of memory usage exceeds alarm ratio. Running SQLs and heap profile will be recorded in record path”] [“is tidb_server_memory_limit set”=true] [tidb_server_memory_limit=13193022640] [“tidb-server memory usage”=10634219480] [memory-usage-alarm-ratio=0.7] [“record path”=/app/tidb-deploy/tidb-4000/log/oom_record]
2026-06-24 13:31:04 (UTC+08:00)TiDB 1.6.1.1:4000[memoryusagealarm.go:215] [“tidb-server has the risk of OOM because of memory usage grows too fast. Running SQLs and heap profile will be recorded in record path”] [“is tidb_server_memory_limit set”=true] [tidb_server_memory_limit=13193022640] [“tidb-server memory usage”=11957534720] [memory-usage-alarm-ratio=0.7] [“record path”=/app/tidb-deploy/tidb-4000/log/oom_record]
2026-06-24 13:31:27 (UTC+08:00)TiDB 1.6.1.1:4000[memoryusagealarm.go:215] [“tidb-server has the risk of OOM because of memory usage grows too fast. Running SQLs and heap profile will be recorded in record path”] [“is tidb_server_memory_limit set”=true] [tidb_server_memory_limit=13193022640] [“tidb-server memory usage”=13282437952] [memory-usage-alarm-ratio=0.7] [“record path”=/app/tidb-deploy/tidb-4000/log/oom_record]
2026-06-24 13:41:34 (UTC+08:00)TiDB 1.6.1.1:4000[memoryusagealarm.go:215] [“tidb-server has the risk of OOM because of memory usage exceeds alarm ratio. Running SQLs and heap profile will be recorded in record path”] [“is tidb_server_memory_limit set”=true] [tidb_server_memory_limit=13193022640] [“tidb-server memory usage”=10239130768] [memory-usage-alarm-ratio=0.7] [“record path”=/app/tidb-deploy/tidb-4000/log/oom_record]
2026-06-24 13:41:46 (UTC+08:00)TiDB 1.6.1.1:4000[memoryusagealarm.go:215] [“tidb-server has the risk of OOM because of memory usage grows too fast. Running SQLs and heap profile will be recorded in record path”] [“is tidb_server_memory_limit set”=true] [tidb_server_memory_limit=13193022640] [“tidb-server memory usage”=11567812400] [memory-usage-alarm-ratio=0.7] [“record path”=/app/tidb-deploy/tidb-4000/log/oom_record]
2026-06-24 13:41:58 (UTC+08:00)TiDB 1.6.1.1:4000[memoryusagealarm.go:215] [“tidb-server has the risk of OOM because of memory usage grows too fast. Running SQLs and heap profile will be recorded in record path”] [“is tidb_server_memory_limit set”=true] [tidb_server_memory_limit=13193022640] [“tidb-server memory usage”=12919506536] [memory-usage-alarm-ratio=0.7] [“record path”=/app/tidb-deploy/tidb-4000/log/oom_record]
2026-06-24 13:45:11 (UTC+08:00)TiDB 1.6.1.1:4000[memoryusagealarm.go:215] [“tidb-server has the risk of OOM because of memory usage exceeds alarm ratio. Running SQLs and heap profile will be recorded in record path”] [“is tidb_server_memory_limit set”=true] [tidb_server_memory_limit=13193022640] [“tidb-server memory usage”=13136988544] [memory-usage-alarm-ratio=0.7] [“record path”=/app/tidb-deploy/tidb-4000/log/oom_record]

—2
/app/tidb-deploy/tidb-4000/log/oom_record/record2026-06-24T13:46:11+08:00
[root@fnzx-tidb-tidb-pd-1 record2026-06-24T13:46:11+08:00]# go tool pprof -text heap | head -100
File: tidb-server
Build ID: bf748c24ff79b131391fc0b018c548f4022caaca
Type: inuse_space
Time: 2026-06-24 13:46:11 CST
Showing nodes accounting for 9397.72MB, 96.85% of 9703.20MB total
Dropped 884 nodes (cum <= 48.52MB)
flat flat% sum% cum cum%
6820.44MB 70.29% 70.29% 6820.44MB 70.29% github.com/pingcap/tidb/pkg/server/internal.(*PacketIO).readOnePacket
996.52MB 10.27% 80.56% 997.38MB 10.28% github.com/pingcap/tidb/pkg/store/copr.(*copIteratorWorker).handleCopCache
766.59MB 7.90% 88.46% 766.59MB 7.90% github.com/tikv/client-go/v2/internal/unionstore/arena.(*MemdbArena).enlarge (inline)
358.56MB 3.70% 92.16% 358.56MB 3.70% github.com/pingcap/tidb/pkg/session.(*txnManager).recordEvent
221MB 2.28% 94.43% 221MB 2.28% github.com/tikv/client-go/v2/txnkv/transaction.(*KVTxn).DoneAggressiveLocking
99MB 1.02% 95.45% 99MB 1.02% github.com/tikv/client-go/v2/txnkv/transaction.(*KVTxn).lockKeys
76.53MB 0.79% 96.24% 76.53MB 0.79% reflect.New
54.58MB 0.56% 96.81% 54.58MB 0.56% google.golang.org/grpc.nopBufferPool.Get
4.50MB 0.046% 96.85% 103.53MB 1.07% github.com/pingcap/tidb/pkg/meta.(*Mutator).GetTable
0 0% 96.85% 98.03MB 1.01% encoding/json.(*decodeState).array
0 0% 96.85% 99.03MB 1.02% encoding/json.(*decodeState).object
0 0% 96.85% 99.03MB 1.02% encoding/json.(*decodeState).unmarshal
0 0% 96.85% 99.03MB 1.02% encoding/json.(*decodeState).value
0 0% 96.85% 99.03MB 1.02% encoding/json.Unmarshal
0 0% 96.85% 76.53MB 0.79% encoding/json.indirect
0 0% 96.85% 55.58MB 0.57% github.com/pingcap/kvproto/pkg/tikvpb.(*tikvBatchCommandsClient).Recv
0 0% 96.85% 141.15MB 1.45% github.com/pingcap/tidb/pkg/distsql.(*selectResult).Next
0 0% 96.85% 139.07MB 1.43% github.com/pingcap/tidb/pkg/distsql.(*selectResult).fetchResp
0 0% 96.85% 126.79MB 1.31% github.com/pingcap/tidb/pkg/domain.(*Domain).Reload
0 0% 96.85% 126.29MB 1.30% github.com/pingcap/tidb/pkg/domain.(*Domain).Start.func2
0 0% 96.85% 126.79MB 1.31% github.com/pingcap/tidb/pkg/domain.(*Domain).loadInfoSchema
0 0% 96.85% 126.29MB 1.30% github.com/pingcap/tidb/pkg/domain.(*Domain).loadSchemaInLoop
0 0% 96.85% 126.29MB 1.30% github.com/pingcap/tidb/pkg/domain.(*Domain).tryLoadSchemaDiffs
0 0% 96.85% 1145.10MB 11.80% github.com/pingcap/tidb/pkg/executor.(*ExecStmt).Exec
0 0% 96.85% 1141.57MB 11.76% github.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelay
0 0% 96.85% 821.57MB 8.47% github.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelayExecutor
0 0% 96.85% 1125.59MB 11.60% github.com/pingcap/tidb/pkg/executor.(*ExecStmt).handlePessimisticDML
0 0% 96.85% 221MB 2.28% github.com/pingcap/tidb/pkg/executor.(*ExecStmt).handlePessimisticDML.func2
0 0% 96.85% 904.15MB 9.32% github.com/pingcap/tidb/pkg/executor.(*ExecStmt).next
0 0% 96.85% 818.87MB 8.44% github.com/pingcap/tidb/pkg/executor.(*InsertExec).Next
0 0% 96.85% 767.09MB 7.91% github.com/pingcap/tidb/pkg/executor.(*InsertExec).exec
0 0% 96.85% 767.09MB 7.91% github.com/pingcap/tidb/pkg/executor.(*InsertValues).addRecordWithAutoIDHint
0 0% 96.85% 138.08MB 1.42% github.com/pingcap/tidb/pkg/executor.(*TableReaderExecutor).Next
0 0% 96.85% 82.58MB 0.85% github.com/pingcap/tidb/pkg/executor.(*recordSet).Next
0 0% 96.85% 138.08MB 1.42% github.com/pingcap/tidb/pkg/executor.(*tableResultHandler).nextChunk
0 0% 96.85% 805.59MB 8.30% github.com/pingcap/tidb/pkg/executor.insertRows
0 0% 96.85% 964.43MB 9.94% github.com/pingcap/tidb/pkg/executor/internal/exec.Next
0 0% 96.85% 63.54MB 0.65% github.com/pingcap/tidb/pkg/executor/sortexec.(*TopNExec).Next
0 0% 96.85% 63.54MB 0.65% github.com/pingcap/tidb/pkg/executor/sortexec.(*TopNExec).fetchChunks
0 0% 96.85% 63.54MB 0.65% github.com/pingcap/tidb/pkg/executor/sortexec.(*TopNExec).loadChunksUntilTotalLimit
0 0% 96.85% 126.29MB 1.30% github.com/pingcap/tidb/pkg/infoschema.(*Builder).ApplyDiff
0 0% 96.85% 125.29MB 1.29% github.com/pingcap/tidb/pkg/infoschema.(*Builder).applyTableUpdateV2
0 0% 96.85% 124.78MB 1.29% github.com/pingcap/tidb/pkg/infoschema.applyCreateTable
0 0% 96.85% 97.28MB 1.00% github.com/pingcap/tidb/pkg/infoschema.applyDefaultAction
0 0% 96.85% 125.29MB 1.29% github.com/pingcap/tidb/pkg/infoschema.applyTableUpdate
0 0% 96.85% 8434.19MB 86.92% github.com/pingcap/tidb/pkg/server.(*Server).onConn
0 0% 96.85% 1510.64MB 15.57% github.com/pingcap/tidb/pkg/server.(*TiDBContext).ExecuteStmt
0 0% 96.85% 8433.19MB 86.91% github.com/pingcap/tidb/pkg/server.(*clientConn).Run
0 0% 96.85% 1612.75MB 16.62% github.com/pingcap/tidb/pkg/server.(*clientConn).dispatch
0 0% 96.85% 1612.75MB 16.62% github.com/pingcap/tidb/pkg/server.(*clientConn).handleQuery
0 0% 96.85% 1601.74MB 16.51% github.com/pingcap/tidb/pkg/server.(*clientConn).handleStmt
0 0% 96.85% 6820.44MB 70.29% github.com/pingcap/tidb/pkg/server.(*clientConn).readPacket
0 0% 96.85% 83.09MB 0.86% github.com/pingcap/tidb/pkg/server.(*clientConn).writeChunks
0 0% 96.85% 83.09MB 0.86% github.com/pingcap/tidb/pkg/server.(*clientConn).writeResultSet
0 0% 96.85% 6820.44MB 70.29% github.com/pingcap/tidb/pkg/server/internal.(*PacketIO).ReadPacket
0 0% 96.85% 82.58MB 0.85% github.com/pingcap/tidb/pkg/server/internal/resultset.(*tidbResultSet).Next
0 0% 96.85% 221MB 2.28% github.com/pingcap/tidb/pkg/session.(*LazyTxn).DoneFairLocking
0 0% 96.85% 99MB 1.02% github.com/pingcap/tidb/pkg/session.(*LazyTxn).LockKeys
0 0% 96.85% 99MB 1.02% github.com/pingcap/tidb/pkg/session.(*LazyTxn).LockKeysFunc
0 0% 96.85% 1518.69MB 15.65% github.com/pingcap/tidb/pkg/session.(*session).ExecuteStmt
0 0% 96.85% 221MB 2.28% github.com/pingcap/tidb/pkg/session.(*txnManager).OnPessimisticStmtEnd
0 0% 96.85% 358.56MB 3.70% github.com/pingcap/tidb/pkg/session.(*txnManager).OnStmtEnd
0 0% 96.85% 1147.60MB 11.83% github.com/pingcap/tidb/pkg/session.runStmt
0 0% 96.85% 221MB 2.28% github.com/pingcap/tidb/pkg/sessiontxn/isolation.(*basePessimisticTxnContextProvider).OnPessimisticStmtEnd
0 0% 96.85% 138.57MB 1.43% github.com/pingcap/tidb/pkg/store/copr.(*copIterator).Next
0 0% 96.85% 828.30MB 8.54% github.com/pingcap/tidb/pkg/store/copr.(*copIteratorWorker).handleCopPagingResult
0 0% 96.85% 997.38MB 10.28% github.com/pingcap/tidb/pkg/store/copr.(*copIteratorWorker).handleCopResponse
0 0% 96.85% 870.64MB 8.97% github.com/pingcap/tidb/pkg/store/copr.(*copIteratorWorker).handleTask
0 0% 96.85% 1009.20MB 10.40% github.com/pingcap/tidb/pkg/store/copr.(*copIteratorWorker).handleTaskOnce
0 0% 96.85% 870.64MB 8.97% github.com/pingcap/tidb/pkg/store/copr.(*copIteratorWorker).run
0 0% 96.85% 138.57MB 1.43% github.com/pingcap/tidb/pkg/store/copr.(*liteCopIteratorWorker).liteSendReq
0 0% 96.85% 766.59MB 7.90% github.com/pingcap/tidb/pkg/store/driver/txn.(*memBuffer).SetWithFlags
0 0% 96.85% 221MB 2.28% github.com/pingcap/tidb/pkg/store/driver/txn.(*tikvTxn).DoneFairLocking
0 0% 96.85% 99MB 1.02% github.com/pingcap/tidb/pkg/store/driver/txn.(*tikvTxn).LockKeysFunc
0 0% 96.85% 767.09MB 7.91% github.com/pingcap/tidb/pkg/table/tables.(*TableCommon).AddRecord
0 0% 96.85% 767.09MB 7.91% github.com/pingcap/tidb/pkg/table/tables.(*TableCommon).addRecord
0 0% 96.85% 766.59MB 7.90% github.com/pingcap/tidb/pkg/table/tblctx.(*EncodeRowBuffer).WriteMemBufferEncoded
0 0% 96.85% 157.88MB 1.63% github.com/pingcap/tidb/pkg/util.(*WaitGroupEnhancedWrapper).Run.func1
0 0% 96.85% 55.58MB 0.57% github.com/tikv/client-go/v2/internal/client.(*batchCommandsClient).batchRecvLoop
0 0% 96.85% 55.58MB 0.57% github.com/tikv/client-go/v2/internal/client.(*batchCommandsStream).recv
0 0% 96.85% 766.59MB 7.90% github.com/tikv/client-go/v2/internal/unionstore.(*artDBWithContext).SetWithFlags
0 0% 96.85% 766.59MB 7.90% github.com/tikv/client-go/v2/internal/unionstore.(*artDBWithContext).set
0 0% 96.85% 766.59MB 7.90% github.com/tikv/client-go/v2/internal/unionstore/arena.(*MemdbArena).Alloc
0 0% 96.85% 510.60MB 5.26% github.com/tikv/client-go/v2/internal/unionstore/arena.(*MemdbVlog[go.shape.*uint8,go.shape.*uint8]).AppendValue
0 0% 96.85% 766.59MB 7.90% github.com/tikv/client-go/v2/internal/unionstore/art.(*ART).Set
0 0% 96.85% 128MB 1.32% github.com/tikv/client-go/v2/internal/unionstore/art.(*ART).expandLeaf
0 0% 96.85% 221.95MB 2.29% github.com/tikv/client-go/v2/internal/unionstore/art.(*ART).newLeaf (inline)
0 0% 96.85% 255.99MB 2.64% github.com/tikv/client-go/v2/internal/unionstore/art.(*ART).recursiveInsert
0 0% 96.85% 510.60MB 5.26% github.com/tikv/client-go/v2/internal/unionstore/art.(*ART).setValue
0 0% 96.85% 255.99MB 2.64% github.com/tikv/client-go/v2/internal/unionstore/art.(*ART).traverse
0 0% 96.85% 221.95MB 2.29% github.com/tikv/client-go/v2/internal/unionstore/art.(*artAllocator).allocLeaf
0 0% 96.85% 99MB 1.02% github.com/tikv/client-go/v2/txnkv/transaction.(*KVTxn).LockKeysFunc (inline)
0 0% 96.85% 58.16MB 0.6% google.golang.org/grpc.(*clientStream).RecvMsg

-------3
[root@fnzx-tidb-tidb-pd-1 record2026-06-24T13:46:11+08:00]# cat running_sql
The 10 SQLs with the most memory usage for OOM analysis

The 10 SQLs with the most time usage for OOM analysis

—4 查看慢查询

[像风一样的男子]

dashboard上查下慢查询和topsql中是哪些sql资源消耗多，优先优化sql，如果一时半会优化不了只能加资源

[克里克里克 (Ti D Ber H052ej9m)]

302课程有对OOM的诊断，可以参考下。

[一只大芒果 (Ti D Ber L B8j9nt J)]

太棒啦，有地址吗？

[一只大芒果 (Ti D Ber L B8j9nt J)]

有条select * 50多万条，还有两个left join 导致OOM了
OOM报错会导致应用无法连接吗

[克里克里克 (Ti D Ber H052ej9m)]

302-TiDB 高级系统管理

OOM 就查询占用内存高的SQL，限制单条SQL的内存使用。
OOM就导致tidb Server重启了，应用短期内肯定就无法连接了。

[像风一样的男子]

肯定啊 tidb节点oom了就不能对外提供服务了，单条sql内存限制了吗？tidb_mem_quota_query这个参数

[异乡的大人 (Ti D Ber 2 Qs S2z Ws)]

超过了内存限制，放大一些内存呢

[porpoiselxj]

先限制下sql使用的最大内存，保证tidb不OOM，然后再查是哪个sql的问题
select @@tidb_mem_quota_query

[Kongdom]

会的，OOM会导致数据库无法连接。

[一只大芒果 (Ti D Ber L B8j9nt J)]

已经定位问题，明天整理下排查思路和解决办法哦。还望帮忙看下是否是准确的。

[一只大芒果 (Ti D Ber L B8j9nt J)]

@porpoiselxj

select @@tidb_mem_quota_query

太感谢啦，马上去学习

[一只大芒果 (Ti D Ber L B8j9nt J)]

好呀好呀，时间是多久呢，有参数可以查看吗？还是等到TiDB取消，释放内存之后，正常连接呀？

[一只大芒果 (Ti D Ber L B8j9nt J)]

4G，是不是有些大呀？
但是tidb_distsql_scan_concurrency设置的15，导致内存一直飙升。

[随缘天空 (Ti D Ber Ivw R7o Pj)]

oom溢出会导致tidb服务器重启，但是如果集群中是3台tidb节点，不同时出现重启的话，如果有负载代理，业务还是可以连接数据库使用的，不过重启过程中，连接到重启机器上的sql会报错然后负载发到正常的机器上。从你的报错信息和日志来看，oom持续告警了，应该是大事务导致内存持续升高造成的，pd超时的问题重点看下pd服务是否负载过大或者tidb和pd间的网络通讯问题

[wbslxw (Ti D Ber Cl S0j Eng)]

这是大sql引发的TiDB内存溢出，导致的问题

[TiDBer_lPQGLonR]

抽空分享下解决思路啊，学习学习

[Kongdom]

OOM之后会立即结束查询的。一般会立即重启节点，重启之后就能正常连接了。

[TiDBer_bgWIGhji]

跨版本问题源于优化器规则和存储引擎层变更。

[一只大芒果 (Ti D Ber L B8j9nt J)]

好的呀，已经写成博客了哦，等审核过了，把连接贴到帖子里