Dashuang Du.
Dashuang Du.

study and practice, years of it.

Kafka 安装与部署

2018, Jun 19    

本文档介绍Kafka安装，部署，及简要使用（以ubuntu 16.04为例）

官方文档

简介

Kafka 是由 Linkedin 公司用scala语言开发的一个分布式、分区的、多副本的、多订阅者，基于 zookeeper 协调的分布式日志系统(也可以当做MQ系统)， 常见可以用于 web/nginx 日志、访问日志，消息服务等等，Linkedin 于2010年贡献给了Apache基金会并成为顶级开源项目。

安装及部署

  • install jvm

      sudo apt-get update
  sudo apt-get install default-jdk

  • download the 1.1.0 release and un-tar it.

      wget http://mirror.bit.edu.cn/apache/kafka/1.1.0/kafka_2.11-1.1.0.tgz

      tar -xzf kafka_2.11-1.1.0.tgz
  cd kafka_2.11-1.1.0

  • start zookeeper

      bin/zookeeper-server-start.sh config/zookeeper.properties

  • start kafka server

      bin/kafka-server-start.sh config/server.properties

  • create a topic

      bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

  • show topic list

      bin/kafka-topics.sh --list --zookeeper localhost:2181

  • show topic detail

      bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test

  • send some messages

      > bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
  Hello World!
  Hello Kafka!

  • start a consumer

      > bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
  Hello World!
  Hello Kafka!

  • setting up a multi-broker cluster

    make a config file for each of the brokers

      cp config/server.properties config/server-1.properties
  cp config/server.properties config/server-2.properties

      config/server-1.properties:
      broker.id=1
      listeners=PLAINTEXT://:9093
      log.dir=/tmp/kafka-logs-1
    
  config/server-2.properties:
      broker.id=2
      listeners=PLAINTEXT://:9094
      log.dir=/tmp/kafka-logs-2

    start brokers

      bin/kafka-server-start.sh config/server-1.properties

      bin/kafka-server-start.sh config/server-2.properties

简单使用（以 PHP 为例）

安装 php-rdkafka 扩展

这里使用PHP的php-rdkafka扩展， 在这之前， 首先要安装Kafka的基础库librdkafka

  • install librdkafka

      git clone https://github.com/edenhill/librdkafka
  cd librdkafka
  ./configure
  make
  sudo make install

  • install phpize

      sudo apt install php7.0-dev

  • install php-rdkafka

      wget -c https://github.com/arnaud-lb/php-rdkafka/archive/3.0.5.tar.gz

      tar xvzf 3.0.5.tar.gz
  cd php-rdkafka-3.0.5

      phpize
  ./configure --with-php-config=/usr/bin/php-config --with-rdkafka

      make all -j 5
  sudo make install

  • update php.ini, add following line

      extension=rdkafka.so

在命令行中运行php -r " new RdKafka\Conf();"，如果没有报错，则说明安装成功。

使用实例

for example: 官方实例

kafka-manager 的安装与使用

官方仓库

  • install

      git clone https://github.com/yahoo/kafka-manager.git
  cd kafka-manager

  • config

      kafka-manager.zkhosts="my.zookeeper.host.com:2181,other.zookeeper.host.com:2181"

      ZK_HOSTS="my.zookeeper.host.com:2181"

      application.features=["KMClusterManagerFeature","KMTopicManagerFeature","KMPreferredReplicaElectionFeature","KMReassignPartitionsFeature"]

  • packaging

    wait for a long time.

      ./sbt clean dist

  • start server

      cd target/universal/kafka-manager-1.3.3.17
  bin/kafka-manager

访问http://myhost:9000

# message queue
TOP