tiup dmct start-task 报错---此处卡住超过3个小时

tsaiqili · 2020 年12 月 23 日 02:30

提示下游tidb数据源有问题，配置检查了很多遍，手动输入过，依然提示"Please check the database connection and the database config in configuration file"

tsaiqili · 2020 年12 月 23 日 02:31

task.yaml 配置
name: “test”
task-mode: “incremental”
target-database:
host: “172.25.99.188”
port: 4000
user: “root”
password: “”

block-allow-list:
global:
do-dbs: [“usercenter”]

mydumpers:
global:
extra-args: “–consistency none”

mysql-instances:

source-id: "aurora-replica-01"
meta:
  binlog-name: log-bin.000004
  binlog-pos: 110072
block-allow-list:  "global"
mydumper-config-name: "global"

tsaiqili · 2020 年12 月 23 日 02:32

source.yaml

Aurora-1

source-id: “aurora-replica-01”

基于 GTID 进行数据迁移时，需要将该项设置为 true

enable-gtid: false

from:
host: “172.25.99.179”
user: “root”
password: “123456”
port: 3306

tsaiqili · 2020 年12 月 23 日 02:32

navicat mysql 和tidb 都连的通

北京大爷 · 2020 年12 月 23 日 02:44

请反馈下 DM 的版本。
以及绑定 task 的 dm-worker 与 dm-master 的log

tsaiqili · 2020 年12 月 23 日 02:48

dm v2.0.0

master log

[2020/12/23 10:47:39.835 +08:00] [INFO] [server.go:1890] [payload="{HidePasswordObject=“task:\“name: \\\“test\\\”\\ task-mode: \\\“incremental\\\”\\ target-database: \\ host: \\\“172.25.99.188\\\”\\ port: 4000\\ user: \\\“root\\\”\\ password: \\\”******\\\”\\ \\ \\ block-allow-list: \\ global: \\ do-dbs: [\\\“usercenter\\\”]\\ \\ \\ mydumpers: \\ global: \\ extra-args: \\\"–consistency none\\\"\\ \\ mysql-instances:\\ -\\ source-id: \\\“aurora-replica-01\\\”\\ meta: \\ binlog-name: log-bin.000004\\ binlog-pos: 110072\\ block-allow-list: \\\“global\\\”\\ mydumper-config-name: \\\“global\\\”\\ \\ \" “}”] [request=StartTask]
[2020/12/23 10:47:39.835 +08:00] [INFO] [server.go:1905] [“will forward after a short interval”] [from=dm-172.25.99.188-8261] [to=dm-172.25.99.179-8261] [request=StartTask]

worker log

[2020/12/23 10:47:40.079 +08:00] [INFO] [worker.go:382] [“receive subtask stage change”] [stage="{“expect”:2,“source”:“aurora-replica-01”,“task”:“test”}"] [“is deleted”=false]
[2020/12/23 10:47:40.123 +08:00] [INFO] [server.go:447] [request=QueryStatus] [payload=“name:“test” “]
[2020/12/23 10:47:40.123 +08:00] [INFO] [worker.go:416] [“start to create subtask”] [sourceID=aurora-replica-01] [task=test]
[2020/12/23 10:47:40.124 +08:00] [INFO] [worker.go:219] [“subtask created”] [component=“worker controller”] [config=”{“is-sharding”:false,“shard-mode”:”",“online-ddl-scheme”:"",“case-sensitive”:false,“name”:“test”,“mode”:“incremental”,“ignore-checking-items”:null,“source-id”:“aurora-replica-01”,“server-id”:429536946,“flavor”:“mysql”,“meta-schema”:“dm_meta”,“heartbeat-update-interval”:1,“heartbeat-report-interval”:10,“enable-heartbeat”:false,“meta”:{“BinLogName”:“log-bin.000004”,“BinLogPos”:110072,“BinLogGTID”:""},“Timezone”:"",“relay-dir”:“relay-dir”,“use-relay”:false,“from”:{“host”:“172.25.99.179”,“port”:3306,“user”:“root”,“max-allowed-packet”:null,“session”:null,“security”:null},“to”:{“host”:“172.25.99.188”,“port”:4000,“user”:“root”,“max-allowed-packet”:null,“session”:{“tidb_txn_mode”:“optimistic”},“security”:null},“route-rules”:[],“filter-rules”:[],“mapping-rule”:[],“black-white-list”:null,“block-allow-list”:{“do-tables”:null,“do-dbs”:[“usercenter”],“ignore-tables”:null,“ignore-dbs”:null},“mydumper-path”:"./bin/mydumper",“threads”:4,“chunk-filesize”:“64”,“statement-size”:0,“rows”:0,“where”:"",“skip-tz-utc”:true,“extra-args”:"–consistency none",“pool-size”:16,“dir”:"./dumped_data.test",“meta-file”:"",“worker-count”:16,“batch”:100,“queue-size”:1024,“checkpoint-flush-interval”:30,“max-retry”:0,“auto-fix-gtid”:false,“enable-gtid”:false,“disable-detect”:false,“safe-mode”:false,“enable-ansi-quotes”:false,“log-level”:"",“log-file”:"",“log-format”:"",“log-rotate”:"",“pprof-addr”:"",“status-addr”:"",“config-file”:"",“clean-dump-file”:true,“ansi-quotes”:false}"]
[2020/12/23 10:47:40.124 +08:00] [WARN] [syncer.go:2660] [“use system default time location”] [task=test] [unit=“binlog replication”]
[2020/12/23 10:47:40.124 +08:00] [INFO] [syncer.go:2662] [“use timezone”] [task=test] [unit=“binlog replication”] [location=Local]
[2020/12/23 10:47:40.125 +08:00] [ERROR] [subtask.go:187] [“fail to initial subtask”] [subtask=test] [error="[code=10001:class=database:scope=upstream:level=high], Message: fail to initial unit Sync of subtask test : database driver error, RawCause: Error 1129: Host ‘172.25.99.188’ is blocked because of many connection errors; unblock with ‘mysqladmin flush-hosts’, Workaround: Please check the database connection and the database config in configuration file."]
[2020/12/23 10:47:41.124 +08:00] [INFO] [server.go:447] [request=QueryStatus] [payload=“name:“test” “]
[2020/12/23 10:47:42.612 +08:00] [WARN] [task_checker.go:320] [“backoff skip auto resume task”] [component=“task checker”] [task=test] [latestResumeTime=2020/12/23 10:47:42.612 +08:00] [duration=1s]
[2020/12/23 10:47:47.612 +08:00] [INFO] [worker.go:268] [“auto_resume sub task”] [component=“worker controller”] [task=test]
[2020/12/23 10:47:47.612 +08:00] [WARN] [syncer.go:2660] [“use system default time location”] [task=test] [unit=“binlog replication”]
[2020/12/23 10:47:47.612 +08:00] [INFO] [syncer.go:2662] [“use timezone”] [task=test] [unit=“binlog replication”] [location=Local]
[2020/12/23 10:47:47.614 +08:00] [ERROR] [subtask.go:187] [“fail to initial subtask”] [subtask=test] [error=”[code=10001:class=database:scope=upstream:level=high], Message: fail to initial unit Sync of subtask test : database driver error, RawCause: Error 1129: Host ‘172.25.99.188’ is blocked because of many connection errors; unblock with ‘mysqladmin flush-hosts’, Workaround: Please check the database connection and the database config in configuration file.”]
[2020/12/23 10:47:47.614 +08:00] [INFO] [task_checker.go:328] [“dispatch auto resume task”] [component=“task checker”] [task=test]
[2020/12/23 10:47:52.613 +08:00] [INFO] [worker.go:268] [“auto_resume sub task”] [component=“worker controller”] [task=test]
[2020/12/23 10:47:52.613 +08:00] [WARN] [syncer.go:2660] [“use system default time location”] [task=test] [unit=“binlog replication”]
[2020/12/23 10:47:52.613 +08:00] [INFO] [syncer.go:2662] [“use timezone”] [task=test] [unit=“binlog replication”] [location=Local]
[2020/12/23 10:47:52.614 +08:00] [ERROR] [subtask.go:187] [“fail to initial subtask”] [subtask=test] [error="[code=10001:class=database:scope=upstream:level=high], Message: fail to initial unit Sync of subtask test : database driver error, RawCause: Error 1129: Host ‘172.25.99.188’ is blocked because of many connection errors; unblock with ‘mysqladmin flush-hosts’, Workaround: Please check the database connection and the database config in configuration file."]
[2020/12/23 10:47:52.615 +08:00] [INFO] [task_checker.go:328] [“dispatch auto resume task”] [component=“task checker”] [task=test]
[2020/12/23 10:47:57.612 +08:00] [INFO] [worker.go:268] [“auto_resume sub task”] [component=“worker controller”] [task=test]
[2020/12/23 10:47:57.612 +08:00] [WARN] [syncer.go:2660] [“use system default time location”] [task=test] [unit=“binlog replication”]
[2020/12/23 10:47:57.612 +08:00] [INFO] [syncer.go:2662] [“use timezone”] [task=test] [unit=“binlog replication”] [location=Local]
[2020/12/23 10:47:57.613 +08:00] [ERROR] [subtask.go:187] [“fail to initial subtask”] [subtask=test] [error="[code=10001:class=database:scope=upstream:level=high], Message: fail to initial unit Sync of subtask test : database driver error, RawCause: Error 1129: Host ‘172.25.99.188’ is blocked because of many connection errors; unblock with ‘mysqladmin flush-hosts’, Workaround: Please check the database connection and the database config in configuration file."]
[2020/12/23 10:47:57.613 +08:00] [INFO] [task_checker.go:328] [“dispatch auto resume task”] [component=“task checker”] [task=test]
[2020/12/23 10:48:02.612 +08:00] [INFO] [worker.go:268] [“auto_resume sub task”] [component=“worker controller”] [task=test]
[2020/12/23 10:48:02.612 +08:00] [WARN] [syncer.go:2660] [“use system default time location”] [task=test] [unit=“binlog replication”]
[2020/12/23 10:48:02.613 +08:00] [INFO] [syncer.go:2662] [“use timezone”] [task=test] [unit=“binlog replication”] [location=Local]
[2020/12/23 10:48:02.613 +08:00] [ERROR] [subtask.go:187] [“fail to initial subtask”] [subtask=test] [error="[code=10001:class=database:scope=upstream:level=high], Message: fail to initial unit Sync of subtask test : database driver error, RawCause: Error 1129: Host ‘172.25.99.188’ is blocked because of many connection errors; unblock with ‘mysqladmin flush-hosts’, Workaround: Please check the database connection and the database config in configuration file."]

tsaiqili · 2020 年12 月 23 日 03:01

北京大爷 · 2020 年12 月 23 日 03:23

确认下 dm -worker 服务器是否可以正确访问 172.25.99.188:4000

tsaiqili · 2020 年12 月 23 日 03:26

怎么判断呢，
本地电脑通过navicat ，访问虚拟机中tidb，是可以访问,172.25.99.188:4000的，全量数据已经通过dumper 和lighting 导入进去

三台虚拟机的防火墙都关闭了

tsaiqili · 2020 年12 月 23 日 03:33

scope 提示的是内部（internal）
level 提示是的（high）

tsaiqili · 2020 年12 月 23 日 04:49

mysql 连接错误导致，已经解决了，不过，提示信息引导的方向有点不对

小王同学 · 2020 年12 月 23 日 06:12

mysql 迁移数据到tidb dmctl 启动报错这个帖子吧？同一个问题辛苦只提一次就好了。多谢配合。

tsaiqili · 2020 年12 月 23 日 07:44

收到

tsaiqili · 2020 年12 月 23 日 09:21

下午尝试了下增量同步数据，
mysql 写入50万数据，
tidb 同步了9万数据左右，出现了三次
Host ‘172.25.99.188’ is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts这个错误，每一次都停止任务，进行 flush hosts，再次启动任务才能继续同步数据

tsaiqili · 2020 年12 月 24 日 02:08

去mysql 官网查询了下，原因，大致意思是说，允许主机中断连接的次数，如果超过了，就会阻塞连接。

问题是，为什么会这么高频率的出现中断连接

tsaiqili · 2020 年12 月 24 日 02:23

把 max_connect_error 调大到10000，再次测试，当sum_connect_errors 大于235左右的时候，自动清零了。

小王同学 · 2020 年12 月 29 日 11:07

你好，DM已经发了 2.0.1，可以先升级下，另外确认下网络是千兆还是万兆，建议升级下，如果还有问题，辛苦提供下 DM-worker 的完整日志，多谢。