技術文章 » ActiveMQ 常見叢集模式

ActiveMQ 常見叢集模式

2023-01-20 18:00:26

Master Slave 架構模式

這種模式是基於檔案共用鎖的高可用叢集，個人理解其實是一種 Failover 模式，可以是一主一從，也可以是一主兩從。

本文使用 Docker 搭建一個主從模式的 ActiveMQ 5.8.0 叢集，一個主節點，兩個從節點。

ActiveMQ 映象的建立可以參考 構建ActiveMQ映象並執行

version: "2.1"
services:
  activemq-A:
    image: cocowool/activemq:5.8.0
    hostname: activemq-A
    expose:
      - "8161"
      - "61613"
    ports:
      - "8161:8161"
      - "61613:61613"
    volumes:
      - ~/Projects/sh-valley/docker-conf/activemq/activemq-5.8.0-A.xml:/usr/local/apache-activemq-5.8.0/conf/activemq.xml
      - ~/Projects/sh-valley/docker-conf/activemq/kahadb:/usr/local/apache-activemq-5.8.0/data/kahadb
    networks:
      - default
  activemq-B:
    image: cocowool/activemq:5.8.0
    hostname: activemq-B
    expose:
      - "8162"
      - "61614"
    ports:
      - "8162:8161"
      - "61614:61613"
    volumes:
      - ~/Projects/sh-valley/docker-conf/activemq/activemq-5.8.0-B.xml:/usr/local/apache-activemq-5.8.0/conf/activemq.xml
      - ~/Projects/sh-valley/docker-conf/activemq/kahadb:/usr/local/apache-activemq-5.8.0/data/kahadb
    networks:
      - default

其實兩個組態檔是一樣的，關鍵是將 kahadb 的目錄利用本地檔案實現在兩個容器見共用。驗證過程記錄。

$ docker-compose up
Attaching to activemq_activemq-B_1, activemq_activemq-A_1
activemq-B_1  | INFO: Using default configuration
activemq-B_1  | (you can configure options in one of these file: /etc/default/activemq /root/.activemqrc)
activemq-B_1  | 
activemq-B_1  | INFO: Invoke the following command to create a configuration file
activemq-B_1  | /usr/local/apache-activemq-5.8.0/bin/activemq setup [ /etc/default/activemq | /root/.activemqrc ]
activemq-B_1  | 
activemq-A_1  | INFO: Using default configuration
activemq-A_1  | (you can configure options in one of these file: /etc/default/activemq /root/.activemqrc)
activemq-A_1  | 
activemq-A_1  | INFO: Invoke the following command to create a configuration file
activemq-B_1  | INFO: Using java '/usr/local/jdk1.8.0_144/bin/java'
activemq-A_1  | /usr/local/apache-activemq-5.8.0/bin/activemq setup [ /etc/default/activemq | /root/.activemqrc ]
activemq-A_1  | 
activemq-B_1  | INFO: Starting in foreground, this is just for debugging purposes (stop process by pressing CTRL+C)
activemq-A_1  | INFO: Using java '/usr/local/jdk1.8.0_144/bin/java'
activemq-A_1  | INFO: Starting in foreground, this is just for debugging purposes (stop process by pressing CTRL+C)
activemq-B_1  | Java Runtime: Oracle Corporation 1.8.0_144 /usr/local/jdk1.8.0_144/jre
activemq-B_1  |   Heap sizes: current=1005056k  free=989327k  max=1005056k
activemq-B_1  |     JVM args: -Xms1G -Xmx1G -Djava.util.logging.config.file=logging.properties -Dcom.sun.management.jmxremote -Djava.io.tmpdir=/usr/local/apache-activemq-5.8.0/tmp -Dactivemq.classpath=/usr/local/apache-activemq-5.8.0/conf; -Dactivemq.home=/usr/local/apache-activemq-5.8.0 -Dactivemq.base=/usr/local/apache-activemq-5.8.0 -Dactivemq.conf=/usr/local/apache-activemq-5.8.0/conf -Dactivemq.data=/usr/local/apache-activemq-5.8.0/data
activemq-B_1  | Extensions classpath:
activemq-B_1  |   [/usr/local/apache-activemq-5.8.0/lib,/usr/local/apache-activemq-5.8.0/lib/camel,/usr/local/apache-activemq-5.8.0/lib/optional,/usr/local/apache-activemq-5.8.0/lib/web,/usr/local/apache-activemq-5.8.0/lib/extra]
activemq-B_1  | ACTIVEMQ_HOME: /usr/local/apache-activemq-5.8.0
activemq-B_1  | ACTIVEMQ_BASE: /usr/local/apache-activemq-5.8.0
activemq-B_1  | ACTIVEMQ_CONF: /usr/local/apache-activemq-5.8.0/conf
activemq-B_1  | ACTIVEMQ_DATA: /usr/local/apache-activemq-5.8.0/data
activemq-A_1  | Java Runtime: Oracle Corporation 1.8.0_144 /usr/local/jdk1.8.0_144/jre
activemq-A_1  |   Heap sizes: current=1005056k  free=989327k  max=1005056k
activemq-A_1  |     JVM args: -Xms1G -Xmx1G -Djava.util.logging.config.file=logging.properties -Dcom.sun.management.jmxremote -Djava.io.tmpdir=/usr/local/apache-activemq-5.8.0/tmp -Dactivemq.classpath=/usr/local/apache-activemq-5.8.0/conf; -Dactivemq.home=/usr/local/apache-activemq-5.8.0 -Dactivemq.base=/usr/local/apache-activemq-5.8.0 -Dactivemq.conf=/usr/local/apache-activemq-5.8.0/conf -Dactivemq.data=/usr/local/apache-activemq-5.8.0/data
activemq-A_1  | Extensions classpath:
activemq-A_1  |   [/usr/local/apache-activemq-5.8.0/lib,/usr/local/apache-activemq-5.8.0/lib/camel,/usr/local/apache-activemq-5.8.0/lib/optional,/usr/local/apache-activemq-5.8.0/lib/web,/usr/local/apache-activemq-5.8.0/lib/extra]
activemq-A_1  | ACTIVEMQ_HOME: /usr/local/apache-activemq-5.8.0
activemq-A_1  | ACTIVEMQ_BASE: /usr/local/apache-activemq-5.8.0
activemq-A_1  | ACTIVEMQ_CONF: /usr/local/apache-activemq-5.8.0/conf
activemq-A_1  | ACTIVEMQ_DATA: /usr/local/apache-activemq-5.8.0/data
activemq-B_1  | Loading message broker from: xbean:activemq.xml
activemq-A_1  | Loading message broker from: xbean:activemq.xml
activemq-B_1  |  INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@68c4039c: startup date [Wed Sep 07 13:16:20 UTC 2022]; root of context hierarchy
activemq-A_1  |  INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@68c4039c: startup date [Wed Sep 07 13:16:20 UTC 2022]; root of context hierarchy
activemq-B_1  |  INFO | PListStore:[/usr/local/apache-activemq-5.8.0/data/localhost/tmp_storage] started
activemq-B_1  |  INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[/usr/local/apache-activemq-5.8.0/data/kahadb]
activemq-A_1  |  INFO | PListStore:[/usr/local/apache-activemq-5.8.0/data/localhost/tmp_storage] started
activemq-A_1  |  INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[/usr/local/apache-activemq-5.8.0/data/kahadb]
activemq-A_1  |  INFO | Database /usr/local/apache-activemq-5.8.0/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/usr/local/apache-activemq-5.8.0/data/kahadb/lock' could not be locked.
activemq-B_1  |  INFO | KahaDB is version 4
activemq-B_1  |  INFO | Recovering from the journal ...
activemq-B_1  |  INFO | Recovery replayed 1 operations from the journal in 0.022 seconds.
activemq-B_1  |  INFO | Apache ActiveMQ 5.8.0 (localhost, ID:activemq-B-41795-1662556582026-0:1) is starting
activemq-B_1  |  INFO | Listening for connections at: tcp://activemq-B:61616?maximumConnections=1000&wireformat.maxFrameSize=104857600
activemq-B_1  |  INFO | Connector openwire Started
activemq-B_1  |  INFO | Listening for connections at: amqp://activemq-B:5672?maximumConnections=1000&wireformat.maxFrameSize=104857600
activemq-B_1  |  INFO | Connector amqp Started
activemq-B_1  |  INFO | Listening for connections at: stomp://activemq-B:61612?transport.closeAsync=false
activemq-B_1  |  INFO | Connector stomp Started
activemq-B_1  |  INFO | Listening for connections at: stomp+nio://activemq-B:61613?transport.closeAsync=false
activemq-B_1  |  INFO | Connector stomp+nio Started
activemq-B_1  |  INFO | Apache ActiveMQ 5.8.0 (localhost, ID:activemq-B-41795-1662556582026-0:1) started
activemq-B_1  |  INFO | For help or more information please see: http://activemq.apache.org
activemq-B_1  | ERROR | Temporary Store limit is 51200 mb, whilst the temporary data directory: /usr/local/apache-activemq-5.8.0/data/localhost/tmp_storage only has 42398 mb of usable space
activemq-B_1  |  INFO | Web console type: embedded
activemq-B_1  |  INFO | ActiveMQ WebConsole initialized.
activemq-B_1  |  INFO | Initializing Spring FrameworkServlet 'dispatcher'
activemq-B_1  |  INFO | jolokia-agent: No access restrictor found at classpath:/jolokia-access.xml, access to all MBeans is allowed
activemq-A_1  |  INFO | Database /usr/local/apache-activemq-5.8.0/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/usr/local/apache-activemq-5.8.0/data/kahadb/lock' could not be locked.

根據啟動的順序不同，A、B 都有可能搶到鎖對外提供服務，未搶到鎖的範例紀錄檔一直會報無法解鎖的錯誤，同時不會對外提供 Web Console 服務。使用 Python 的連線例子，發現 stomp.py 有一個問題不能很好的支援 failover。

Networks of Brokers

這是一種負載均衡的架構，各個 Broker 通過互相連線並共用佇列資訊，實現叢集的負載均衡，即只要消費者連線到 Broker 網路中任意一臺，就可以消費所有的訊息。

這種方案主要關注 ActiveMQ 設定中的兩個引數

  • TransportConnector，傳輸聯結器引數主要控制伺服器端和使用者端之間的通訊方式
  • NetworkConnector，網路聯結器引數主要控制伺服器端和伺服器端之間的通訊

本文構建了這樣一種架構，Broker1 作為與生產者溝通的節點，負責接收訊息以及傳遞消費者反饋給生產者的訊息，Broker2、Broker3作為與消費者溝通的節點，支援與大量的消費者建立連結。這種架構適用於消費者數量非常多，通常是超過1萬個節點的情況。根據實際生產執行的情況，一臺4C16G的虛擬機器器，一個ActiveMQ範例通常能夠承受2000個左右的連結能保持長期穩定執行。

具體的 docker 設定如下，相關的檔案可以從我的 Github 上下載。

version: "2.1"
services:
  activemq-broker1:
    image: cocowool/activemq:5.8.0
    hostname: activemq-broker1
    expose:
      - "8161"
      - "61611"
    ports:
      - "8161:8161"
      - "61611:61612"
    volumes:
      - ~/Projects/sh-valley/docker-conf/activemq-network/activemq-5.8.0-broker1.xml:/usr/local/apache-activemq-5.8.0/conf/activemq.xml
    networks:
      activemq-network:
        ipv4_address: 172.28.1.11
  activemq-broker2:
    image: cocowool/activemq:5.8.0
    hostname: activemq-broker2
    expose:
      - "8162"
      - "61612"
    ports:
      - "8162:8161"
      - "61612:61612"
    volumes:
      - ~/Projects/sh-valley/docker-conf/activemq-network/activemq-5.8.0-broker2.xml:/usr/local/apache-activemq-5.8.0/conf/activemq.xml
    networks:
      activemq-network:
        ipv4_address: 172.28.1.12
  activemq-broker3:
    image: cocowool/activemq:5.8.0
    hostname: activemq-broker3
    expose:
      - "8163"
      - "61613"
    ports:
      - "8163:8161"
      - "61613:61612"
    volumes:
      - ~/Projects/sh-valley/docker-conf/activemq-network/activemq-5.8.0-broker3.xml:/usr/local/apache-activemq-5.8.0/conf/activemq.xml
    networks:
      activemq-network:
        ipv4_address: 172.28.1.13
networks:
  activemq-network:
    ipam:
      config:
      - subnet: 172.28.1.0/24

執行 docker-compose up 命令可以啟動三個 ActiveMQ 範例，然後分別通過 http://localhost:8161\8162\8163 可以檢視三個範例的控制檯，在 Broker2\Broker3 的 Network 頁簽下可以看到到 Broker1 的連結。我們在生產實踐過程中發現過這個連線出問題的情況，此時 ActiveMQ 範例程序並無異常，這時就會影響連線到這個範例的消費者。

要測試這種架構的可用性，可以參考 Python 與 ActiveMQ 互動的一些例子 中的程式碼範例，修改對應的埠地址就能看到效果。

這個架構可以配合 Master Slave 進一步優化，如下圖所示：

生產者連線的範例採用 Master Slave Failover 方式，消費者連線的範例在設定 NetworkConnector 時也採用 Failover 模式，這樣 Broker1 就提供了主從的高可用模式，進一步提高了整個叢集應對風險的能力。

參考資料

  1. ActiveMQ叢集安裝
  2. Persistence vs. Durability in Messaging. Do you know the difference?
  3. Shared File System Master Slave
  4. Networks of Brokers
  5. docker-compose 固定IP
  6. ActiveMQ實現負載均衡+高可用部署方案
  7. ActiveMQ中的NetworkConnector（網路聯結器）詳解