TiCDC发送到kafka的消息分区参数不生效

乾坤大挪移 · 2026 年4 月 13 日 02:19

CAI001:

TiDB 使用环境】测试
启动命令中设置的分区数量是3，但配置文件中动态生成的topic的分区数量并没有按照命令行上的分区数量设置。
./cdc cli changefeed create --server=http://`````````:8300 --sink-uri=“kafka://`````````:9092/topic_ticdc_saleslink_uc_employee?protocol=canal-json&kafka-version=2.6.2&partition-num=3&max-message-bytes=67108864&replication-factor=3” --changefeed-id=“data-syc-two-way-saleslink-uc-employee” --config=/data/tidb/ticdc-changefeed/changefeed-data-syc-saleslink_uc_employee.conf

[consistent]
level = "none"

[filter]
rules = ['saleslink_uc_employee.emp_employee']

[[filter.event-filters]]
matcher = ['saleslink_uc_employee.emp_employee']

[mounter]
worker-num = 16

# Sink 配置
[sink]
protocol = "debezium"
dispatchers = [
  {matcher = ['saleslink_uc_employee.emp_employee'], topic = "topic_ticdc_{schema}_{table}", partition = "index-value"}
]

结果默认topic是3分区topic_ticdc_saleslink_uc_employee，配置内表相关的topic用的是默认12分区

[root@saleslink-uat-ticdc-01 bin]# ./kafka-topics.sh --describe --bootstrap-server *******:9092 --topic topic_ticdc_saleslink_uc_employee
Topic: topic_ticdc_saleslink_uc_employee        PartitionCount: 3       ReplicationFactor: 1    Configs: 
        Topic: topic_ticdc_saleslink_uc_employee        Partition: 0    Leader: 103     Replicas: 103   Isr: 103        Elr: N/A        LastKnownElr: N/A
        Topic: topic_ticdc_saleslink_uc_employee        Partition: 1    Leader: 101     Replicas: 101   Isr: 101        Elr: N/A        LastKnownElr: N/A
        Topic: topic_ticdc_saleslink_uc_employee        Partition: 2    Leader: 102     Replicas: 102   Isr: 102        Elr: N/A        LastKnownElr: N/A

[root@saleslink-uat-ticdc-01 bin]# ./kafka-topics.sh --describe --bootstrap-server *******:9092 --topic topic_ticdc_saleslink_uc_employee_emp_employee
Topic: topic_ticdc_saleslink_uc_employee_emp_employee   PartitionCount: 12      ReplicationFactor: 1    Configs: 
        Topic: topic_ticdc_saleslink_uc_employee_emp_employee   Partition: 0    Leader: 101     Replicas: 101   Isr: 101        Elr: N/A        LastKnownElr: N/A
        Topic: topic_ticdc_saleslink_uc_employee_emp_employee   Partition: 1    Leader: 102     Replicas: 102   Isr: 102        Elr: N/A        LastKnownElr: N/A
        Topic: topic_ticdc_saleslink_uc_employee_emp_employee   Partition: 2    Leader: 103     Replicas: 103   Isr: 103        Elr: N/A        LastKnownElr: N/A
        Topic: topic_ticdc_saleslink_uc_employee_emp_employee   Partition: 3    Leader: 101     Replicas: 101   Isr: 101        Elr: N/A        LastKnownElr: N/A
        Topic: topic_ticdc_saleslink_uc_employee_emp_employee   Partition: 4    Leader: 102     Replicas: 102   Isr: 102        Elr: N/A        LastKnownElr: N/A
        Topic: topic_ticdc_saleslink_uc_employee_emp_employee   Partition: 5    Leader: 103     Replicas: 103   Isr: 103        Elr: N/A        LastKnownElr: N/A
        Topic: topic_ticdc_saleslink_uc_employee_emp_employee   Partition: 6    Leader: 101     Replicas: 101   Isr: 101        Elr: N/A        LastKnownElr: N/A
        Topic: topic_ticdc_saleslink_uc_employee_emp_employee   Partition: 7    Leader: 102     Replicas: 102   Isr: 102        Elr: N/A        LastKnownElr: N/A
        Topic: topic_ticdc_saleslink_uc_employee_emp_employee   Partition: 8    Leader: 103     Replicas: 103   Isr: 103        Elr: N/A        LastKnownElr: N/A
        Topic: topic_ticdc_saleslink_uc_employee_emp_employee   Partition: 9    Leader: 101     Replicas: 101   Isr: 101        Elr: N/A        LastKnownElr: N/A
        Topic: topic_ticdc_saleslink_uc_employee_emp_employee   Partition: 10   Leader: 102     Replicas: 102   Isr: 102        Elr: N/A        LastKnownElr: N/A
        Topic: topic_ticdc_saleslink_uc_employee_emp_employee   Partition: 11   Leader: 103     Replicas: 103   Isr: 103        Elr: N/A        LastKnownElr: N/A

partition-num=3 只对“默认 topic”生效
你配置的 topic_ticdc_{schema}_{table} 是 动态 topic ，不会使用这个参数
动态 topic 的分区数 = Kafka broker 默认分区