kafka 集群在线扩容 – 21运维
通知: .-...

kafka 集群在线扩容

kafka&zookeeper 21运维 1443浏览

生产环境中,一般环境都是从简单到慢慢变的稍微复杂,比如单机到集群。 kafka也是一样,这里记录下kafka集群扩容。
1,搭建kafka节点环境
搭建kafka环境,直接参考《Kafka zookeeper 集群生产环境安装》即可。这里主要修改的是zookeeper的链接信息,集群里边所有kafka 节点(broker)的zookeeper都要信息一致。
比如:

zookeeper.connect=10.1.12.85:2181,10.1.12.90:2181,10.1.12.94:2181,10.1.12.95:2181,10.1.12.93:2181/kafkagroup

这样zookeeper 集群才会对kafka 节点进行一个有效的调度,比如副本和partition分配等。

2,验证集群是否添加成功kafka节点。
到zookeeper的bin目录使用客户端工具 ./zkCli.sh 用命令查看brokers下的ids。如果看到kafka节点id的信息,说明kafka添加到了集群。

[zk: localhost:2181(CONNECTED) 2] ls /kafkagroup/brokers/ids
[1, 2, 3, 4, 5]

3,创建一个topic

[[email protected] bin]# ./kafka-topics.sh --create --zookeeper 10.1.12.85/kafkagroup --replication-factor 1 --partitions 1 --topic 21yunwei
Created topic 21yunwei.

后边我们以21yunwei这个topic为测试。

列出所有kafka 的topic:

[[email protected] bin]# ./kafka-topics.sh  --list  --zookeeper 10.1.12.85:2181/kafkagroup         
21yunwei
__consumer_offsets
aaa
bbb
ccc
ddd

查看具体21yunwei这个topic的详细信息:

[[email protected] bin]# ./kafka-topics.sh  --describe  --zookeeper 10.1.12.85:2181/kafkagroup  21yunwei
Topic:21yunwei  PartitionCount:1        ReplicationFactor:1     Configs:
        Topic: 21yunwei Partition: 0    Leader: 5       Replicas: 5     Isr: 5

4,修改topic的partition,并进行查看

[[email protected] bin]# ./kafka-topics.sh   --zookeeper 10.1.12.85:2181/kafkagroup   --alter --topic 21yunwei --partitions 3
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!
[[email protected] bin]# ./kafka-topics.sh  --describe  --zookeeper 10.1.12.85:2181/kafkagroup  21yunwei                     
Topic:21yunwei  PartitionCount:3        ReplicationFactor:1     Configs:
        Topic: 21yunwei Partition: 0    Leader: 5       Replicas: 5     Isr: 5
        Topic: 21yunwei Partition: 1    Leader: 1       Replicas: 1     Isr: 1
        Topic: 21yunwei Partition: 2    Leader: 2       Replicas: 2     Isr: 2

这里将partitions由默认的1改成了3,后边的describe 可以看到有三个partition了,其中每个partiton的leader都可以看到(注意:这里副本信息还是依旧在自己的机器节点上,没有集群效果,只是partition有了)。

5,修改replicas。
通过创建一个json文件来定义:

在当前目录创建一个json文件
increasers_node.json


加入内容:

{
    "partitions": [
        {
            "topic": "21yunwei",
            "partition": 0,
            "replicas": [
                1,
                2,
                3
            ]
        },
        {
            "topic": "21yunwei",
            "partition": 1,
            "replicas": [
                1,
                2,
                3
            ]
        }
        {
            "topic": "21yunwei",
            "partition": 2,
            "replicas": [
                1,
                2,
                3
            ]
        }
    ]
}

修改replicas:

[[email protected] bin]# ./kafka-reassign-partitions.sh    --zookeeper 10.1.12.85:2181/kafkagroup  --reassignment-json-file  increasers_node.json  --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"21yunwei","partition":0,"replicas":[5],"log_dirs":["any"]},{"topic":"21yunwei","partition":2,"replicas":[2],"log_dirs":["any"]},{"topic":"21yunwei","partition":1,"replicas":[1],"log_dirs":["any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.

重新查看replicas:

[[email protected] bin]# ./kafka-topics.sh  --describe  --zookeeper 10.1.12.85:2181/kafkagroup  21yunwei                                    
Topic:21yunwei  PartitionCount:3        ReplicationFactor:3     Configs:
        Topic: 21yunwei Partition: 0    Leader: 1       Replicas: 1,2,3 Isr: 1,2,3
        Topic: 21yunwei Partition: 1    Leader: 1       Replicas: 1,2,3 Isr: 1,2,3
        Topic: 21yunwei Partition: 2    Leader: 2       Replicas: 1,2,3 Isr: 2,1,3

可以看到partition和replicas都对了。

topic比较多的情况,建议初级上脚本,高级做成服务用接口来实现。

转载请注明:21运维 » kafka 集群在线扩容