快速搭建高可用Redis Cluster主從模式叢集的填坑之路
2020-10-01 21:00:46
Redis Cluster主從模式
原理
redis cluster 為了保證資料的高可用性，加入了主從模式，一個主節點對應一個或多個從節點，主節點提供資料存取，從節點則是從主節點拉取資料。
備份，當這個主節點掛掉後，就會有這個從節點選取一個來充當主節點，從而保證叢集不會掛掉。
說明
假如叢集有A、B、C三個主節點, 如果這3個節點都沒有加入從節點，如果B掛掉了，我們就無法存取整個叢集了， A和C的slot也無法存取。
所以在叢集建立的時候，一定要為每個主節點都新增了從節點,。比如，像叢集包含主節點A、B、C, 以及從節點A1、B1、C1, 那麼即使B掛掉，系統也可以繼續正確工作 ，B1節點替代了B節點，故Redis叢集將會選擇B1節點，作為新的主節點，叢集將會繼續正確地提供服務 當B重新開啟後，它就會變成B1的從節點。
注意
如果節點B和B1同時掛了，Redis叢集就無法繼續正確地提供服務了。
故最小的redis叢集需要3個主節點（即Redis Cluster叢集至少需要3個master節點，也就是說至少需要6個節點才能構建Redis cluster叢集），一臺機器可執行多個redis範例（一般使用兩臺機器，每臺啟動3個redis範例，即三個主節點，三個從節點）。
很多案例使用單臺伺服器開6個埠，操作差不多，只是設定基本相對簡單點，多臺伺服器更接近生產環境 。
當叢集最開始建立好後，要記住各節點的主從關係（或是建立的時候指定主從關係）；若是其中一臺機器重新啟動，重新啟動後，需重新將其加入到redis cluster叢集中；這就需要將這臺機器上的各節點之前的從節點變為主節點（使用者端執行slaveof no one），然後再根據新的主節點，新增這臺機器的各節點到叢集中，新增後變為從節點。
搭建環境
環境 : CentOS7
redis01 : 172.16.7.11 埠 : 7000、7001、7002
redis02 : 172.16.7.22 埠 : 7003、7004、7005
redis03 : 172.16.7.33 埠 : 7006、7007、7008
關閉SElinux和防火牆
安裝redis
首先選擇一臺伺服器，進行安裝
初始安裝redis01
[root@redis01 ~]# yum install -y gcc g++ make gcc-c++ kernel-devel automake autoconf libtool make wget tcl vim ruby rubygems unzip git
[root@redis01 ~]# wget http://download.redis.io/releases/redis-4.0.1.tar.gz
[root@redis01 ~]# tar xf redis-4.0.1.tar.gz -C /usr/local/src/
[root@redis01 ~]# cd /usr/local/src/redis-4.0.1/
[root@redis01 redis-4.0.1]# make && make test && make PREFIX=/usr/local/redis install
[root@redis01 redis-4.0.1]# mkdir /usr/local/redis/conf
[root@redis01 redis-4.0.1]# cp *.conf /usr/local/redis/conf/
[root@redis01 redis-4.0.1]# cp /usr/local/src/redis-4.0.1/src/redis-trib.rb /usr/local/redis/bin/
[root@redis01 redis-4.0.1]# ln -s /usr/local/redis/bin/* /usr/local/bin/進行作業系統基礎調優設定
[root@redis01 redis-4.0.1]# echo "* - nofile 10240" >> /etc/security/limits.conf
[root@redis01 redis-4.0.1]# echo "net.core.somaxconn = 10240" >> /etc/sysctl.conf
[root@redis01 redis-4.0.1]# echo "vm.overcommit_memory = 1" >> /etc/sysctl.conf
[root@redis01 redis-4.0.1]# sysctl -p
net.core.somaxconn = 10240
vm.overcommit_memory = 1
[root@redis01 redis-4.0.1]# echo never > /sys/kernel/mm/transparent_hugepage/enabled
[root@redis01 redis-4.0.1]# echo never > /sys/kernel/mm/transparent_hugepage/defrag
[root@redis01 redis-4.0.1]# echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.local
[root@redis01 redis-4.0.1]# echo 'echo never > /sys/kernel/mm/transparent_hugepage/defrag' >> /etc/rc.local
[root@redis01 redis-4.0.1]# su -l建立節點
建立叢集節點目錄
[root@redis01 ~]# mkdir /data/redis/redis-cluster -p
[root@redis01 redis-cluster]# cd /data/redis/redis-cluster/
[root@redis01 redis-cluster]# mkdir 7000 7001 7002
執行下面的指令碼，設定各個節點的redis.conf
[root@redis01 redis-cluster]# for i in 0 1 2;
do
echo -e "cluster-enabled yes\n\
bind 0.0.0.0\n\
port 700$i\n\
pidfile /data/redis-cluster/700$i/redis.pid\n\
logfile "/data/redis-cluster/700$i/redis.log"\n\
dir /data/redis-cluster/7000/\n\
tcp-backlog 1024\n\
timeout 0\n\
cluster-node-timeout 10100\n\
tcp-keepalive 0\n\
daemonize yes\n\
loglevel notice\n\
databases 16\n\
stop-writes-on-bgsave-error yes\n\
rdbcompression yes\n\
rdbchecksum yes\n\
dbfilename "dump.rdb"\n\
slave-serve-stale-data yes\n\
slave-read-only yes\n\
repl-diskless-sync no\n\
repl-diskless-sync-delay 5\n\
repl-disable-tcp-nodelay no\n\
slave-priority 100\n\
lazyfree-lazy-eviction no\n\
lazyfree-lazy-expire no\n\
lazyfree-lazy-server-del no\n\
slave-lazy-flush no\n\
appendonly no\n\
appendfilename "appendonly.aof"\n\
appendfsync everysec\n\
no-appendfsync-on-rewrite yes\n\
auto-aof-rewrite-percentage 100\n\
auto-aof-rewrite-min-size 64mb\n\
aof-load-truncated yes\n\
lua-time-limit 5000\n\
slowlog-log-slower-than 10000\n\
slowlog-max-len 128\n\
latency-monitor-threshold 0\n\
notify-keyspace-events ""\n\
hash-max-ziplist-entries 512\n\
hash-max-ziplist-value 64\n\
list-max-ziplist-entries 512\n\
list-max-ziplist-value 64\n\
set-max-intset-entries 512\n\
zset-max-ziplist-entries 128\n\
zset-max-ziplist-value 64\n\
hll-sparse-max-bytes 3000\n\
activerehashing yes\n\
client-output-buffer-limit normal 0 0 0\n\
client-output-buffer-limit slave 256mb 64mb 60\n\
client-output-buffer-limit pubsub 32mb 8mb 60\n\
hz 10\n\
aof-rewrite-incremental-fsync yes" > 700$i/redis.conf;
done檢視各個節點的檔案樹
[root@redis01 redis-cluster]# tree
.
├── 7000
│ └── redis.conf
├── 7001
│ └── redis.conf
└── 7002
└── redis.conf啟動redis01的叢集服務
[root@redis01 redis-cluster]for i in 0 1 2;
do redis-server /data/redis/redis-cluster/700$i/redis.conf
done
68820:C 08 Oct 22:21:14.965 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
68820:C 08 Oct 22:21:14.965 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=68820, just started
68820:C 08 Oct 22:21:14.965 # Configuration loaded
68822:C 08 Oct 22:21:14.972 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
68822:C 08 Oct 22:21:14.972 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=68822, just started
68822:C 08 Oct 22:21:14.972 # Configuration loaded
68824:C 08 Oct 22:21:14.981 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
68824:C 08 Oct 22:21:14.981 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=68824, just started
68824:C 08 Oct 22:21:14.981 # Configuration loadedredis02 伺服器的安裝
redis02 ，redis03不需要完全按上面的步驟，只部分需要複製redis01的安裝解壓目錄
複製redis01的安裝解壓目錄
[root@node02 ~]# scp -r node01:/usr/local/redis /usr/local/
[root@redis02 ~]# scp -r node01:/data/redis/redis-cluster /data/redis/
修改某個伺服器的各個節點的redis.conf檔案
[root@redis02 redis-cluster]# mv 7000 7003
[root@redis02 redis-cluster]# mv 7001 7004
[root@redis02 redis-cluster]# mv 7002 7005
[root@redis02 redis-cluster]# ls
7003 7004 7005
[root@redis02 redis-cluster]# sed -i 's#7000#7003#g' 7003/redis.conf
[root@redis02 redis-cluster]# sed -i 's#7001#7004#g' 7005/redis.conf
[root@redis02 redis-cluster]# sed -i 's#7002#7005#g' 7004/redis.conf做執行指令碼加入bin目錄
[root@node02 ~]# ln -s /usr/local/redis/bin/* /usr/local/bin/
進行作業系統基礎調優設定
如同redis01的作業系統基礎調優設定
啟動redis02的叢集服務
[root@redis02 redis-cluster]# for i in 3 4 5;
do redis-server /data/redis/redis-cluster/700$i/redis.conf
done
41766:C 08 Oct 22:21:42.369 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
41766:C 08 Oct 22:21:42.369 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=41766, just started
41766:C 08 Oct 22:21:42.369 # Configuration loaded
41768:C 08 Oct 22:21:42.375 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
41768:C 08 Oct 22:21:42.375 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=41768, just started
41768:C 08 Oct 22:21:42.375 # Configuration loaded
41770:C 08 Oct 22:21:42.380 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
41770:C 08 Oct 22:21:42.380 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=41770, just started
41770:C 08 Oct 22:21:42.380 # Configuration loadedredis03伺服器的安裝
類比redis02 伺服器的安裝
redis01，redis02，redis03聚合建立叢集
查各Redis節點的啟動狀態
#yum -y install net-tools
[root@redis01 redis-cluster]# netstat -antup|grep redis
tcp 0 0 172.16.7.11:17000 0.0.0.0:* LISTEN 68821/redis-server
tcp 0 0 172.16.7.11:17001 0.0.0.0:* LISTEN 68823/redis-server
tcp 0 0 172.16.7.11:17002 0.0.0.0:* LISTEN 68828/redis-server
tcp 0 0 172.16.7.11:7000 0.0.0.0:* LISTEN 68821/redis-server
tcp 0 0 172.16.7.11:7001 0.0.0.0:* LISTEN 68823/redis-server
tcp 0 0 172.16.7.11:7002 0.0.0.0:* LISTEN 68828/redis-server 安裝ruby
[root@redis01 redis-cluster]# yum -y install rh-ruby23
[root@redis01 redis-cluster]# ruby -v
ruby 2.0.0p648 (2015-12-16) [x86_64-linux]
[root@redis01 redis-cluster]# scl enable rh-ruby23 bash
[root@redis01 redis-cluster]# ruby -v
ruby 2.3.6p384 (2017-12-14 revision 61254) [x86_64-linux]
解決方案；
yum install centos-release-scl-rh
再次安裝 rh-ruby23
安裝ruby的redis擴充套件
[root@redis01 redis-cluster]# gem install redis
Fetching: redis-4.0.2.gem (100%)
Successfully installed redis-4.0.2
Parsing documentation for redis-4.0.2
Installing ri documentation for redis-4.0.2
Done installing documentation for redis after 0 seconds
1 gem installed使用redis自帶的叢集管理工具redis-trib.rb建立叢集
在redis01上執行以下指令碼
[root@redis01 redis-cluster]# redis-trib.rb create --replicas 1 172.16.7.11:7000 172.16.7.11:7001 172.16.7.11:7002 172.16.7.22:7003 172.16.7.22:7004 172.16.7.22:7005 172.16.7.33:7006 172.16.7.33:7007 172.16.7.33:7008
>>> Creating cluster
>>> Performing hash slots allocation on 9 nodes...
Using 4 masters:
172.16.7.11:7000
172.16.7.22:7003
172.16.7.33:7006
172.16.7.11:7001
Adding replica 172.16.7.22:7004 to 172.16.7.11:7000
Adding replica 172.16.7.33:7007 to 172.16.7.22:7003
Adding replica 172.16.7.11:7002 to 172.16.7.33:7006
Adding replica 172.16.7.22:7005 to 172.16.7.11:7001
Adding replica 172.16.7.33:7008 to 172.16.7.11:7000
M: d8138e15d9cb69962abb7457125a7965b735a658 172.16.7.11:7000
slots:0-4095 (4096 slots) master
M: 72f44da597f73c8d23c6d3fb60fa95489eb0168d 172.16.7.11:7001
slots:12288-16383 (4096 slots) master
S: 711af78396c3ba966f751beef745ae10ff053a3f 172.16.7.11:7002
replicates af2e1c600ed2511155a6cad61cabeb095affff86
M: 691428ae7c1728a76520b8f9d46fa6d83930f191 172.16.7.22:7003
slots:4096-8191 (4096 slots) master
S: e752d865374c8ca4149737d81f19ac95cc9f6a44 172.16.7.22:7004
replicates d8138e15d9cb69962abb7457125a7965b735a658
S: ba4ce74ba7b84bde9ea60cd8bf4c5996f1ca01e5 172.16.7.22:7005
replicates 72f44da597f73c8d23c6d3fb60fa95489eb0168d
M: af2e1c600ed2511155a6cad61cabeb095affff86 172.16.7.33:7006
slots:8192-12287 (4096 slots) master
S: b1e201b2acdc0a0740a764978436fa3946325a11 172.16.7.33:7007
replicates 691428ae7c1728a76520b8f9d46fa6d83930f191
S: 6913f44486938c49eed6efa6f18c782df73e6456 172.16.7.33:7008
replicates d8138e15d9cb69962abb7457125a7965b735a658
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join........
>>> Performing Cluster Check (using node 172.16.7.11:7000)
M: d8138e15d9cb69962abb7457125a7965b735a658 172.16.7.11:7000
slots:0-4095 (4096 slots) master
2 additional replica(s)
S: b1e201b2acdc0a0740a764978436fa3946325a11 172.16.7.33:7007
slots: (0 slots) slave
replicates 691428ae7c1728a76520b8f9d46fa6d83930f191
S: e752d865374c8ca4149737d81f19ac95cc9f6a44 172.16.7.22:7004
slots: (0 slots) slave
replicates d8138e15d9cb69962abb7457125a7965b735a658
M: 691428ae7c1728a76520b8f9d46fa6d83930f191 172.16.7.22:7003
slots:4096-8191 (4096 slots) master
1 additional replica(s)
M: af2e1c600ed2511155a6cad61cabeb095affff86 172.16.7.33:7006
slots:8192-12287 (4096 slots) master
1 additional replica(s)
M: 72f44da597f73c8d23c6d3fb60fa95489eb0168d 172.16.7.11:7001
slots:12288-16383 (4096 slots) master
1 additional replica(s)
S: 711af78396c3ba966f751beef745ae10ff053a3f 172.16.7.11:7002
slots: (0 slots) slave
replicates af2e1c600ed2511155a6cad61cabeb095affff86
S: ba4ce74ba7b84bde9ea60cd8bf4c5996f1ca01e5 172.16.7.22:7005
slots: (0 slots) slave
replicates 72f44da597f73c8d23c6d3fb60fa95489eb0168d
S: 6913f44486938c49eed6efa6f18c782df73e6456 172.16.7.33:7008
slots: (0 slots) slave
replicates d8138e15d9cb69962abb7457125a7965b735a658
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.這樣，三個伺服器啟動了9個範例，設定成4主5從,其中一個是一主兩從,其他都是一主一從
叢集驗證
選項 -c 可連線叢集, 因為組態檔中bind改成了ip，所以必須要用 -h 選項，p選項後接埠號
在redis01上的7000埠的範例上set一個key
[root@redis01 redis-cluster]# redis-cli -h 172.16.7.11 -c -p 7000
172.16.7.11:7000> set name haha
-> Redirected to slot [5798] located at 172.16.7.22:7003
OK
172.16.7.22:7003> get name
"haha"
由上可見 , redis01 set name 後,將資料重定向到了redis02的7003埠的範例上
在redis03的7008範例上get一個key
[root@redis01 redis-cluster]# redis-cli -h 172.16.7.33 -c -p 7008
172.16.7.33:7008> get name
-> Redirected to slot [5798] located at 172.16.7.22:7003
"weihan"
172.16.7.22:7003>get name 還是會重定向到7003這個範例上 , 這說明redis cluster叢集已經是可用的了
redis-trib.rb檢查叢集狀態
4個master會分配到4個伺服器上, 而且每個伺服器上必須有一個master，建立叢集的時候，可以指定master，若建立的時候沒指定,是預設分配
[root@redis01 redis-cluster]# redis-trib.rb check 172.16.7.11:7000
>>> Performing Cluster Check (using node 172.16.7.11:7000)
M: d8138e15d9cb69962abb7457125a7965b735a658 172.16.7.11:7000
slots:0-4095 (4096 slots) master
2 additional replica(s)
S: b1e201b2acdc0a0740a764978436fa3946325a11 172.16.7.33:7007
slots: (0 slots) slave
replicates 691428ae7c1728a76520b8f9d46fa6d83930f191
S: e752d865374c8ca4149737d81f19ac95cc9f6a44 172.16.7.22:7004
slots: (0 slots) slave
replicates d8138e15d9cb69962abb7457125a7965b735a658
M: 691428ae7c1728a76520b8f9d46fa6d83930f191 172.16.7.22:7003
slots:4096-8191 (4096 slots) master
1 additional replica(s)
M: af2e1c600ed2511155a6cad61cabeb095affff86 172.16.7.33:7006
slots:8192-12287 (4096 slots) master
1 additional replica(s)
M: 72f44da597f73c8d23c6d3fb60fa95489eb0168d 172.16.7.11:7001
slots:12288-16383 (4096 slots) master
1 additional replica(s)
S: 711af78396c3ba966f751beef745ae10ff053a3f 172.16.7.11:7002
slots: (0 slots) slave
replicates af2e1c600ed2511155a6cad61cabeb095affff86
S: ba4ce74ba7b84bde9ea60cd8bf4c5996f1ca01e5 172.16.7.22:7005
slots: (0 slots) slave
replicates 72f44da597f73c8d23c6d3fb60fa95489eb0168d
S: 6913f44486938c49eed6efa6f18c782df73e6456 172.16.7.33:7008
slots: (0 slots) slave
replicates d8138e15d9cb69962abb7457125a7965b735a658
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.列出叢集節點
[root@redis01 redis-cluster]# redis-cli -h 172.16.7.11 -c -p 7000
172.16.7.11:7000> CLUSTER NODES
b1e201b2acdc0a0740a764978436fa3946325a11 172.16.7.33:7007@17007 slave 691428ae7c1728a76520b8f9d46fa6d83930f191 0 1539016061737 8 connected
e752d865374c8ca4149737d81f19ac95cc9f6a44 172.16.7.22:7004@17004 slave d8138e15d9cb69962abb7457125a7965b735a658 0 1539016062000 5 connected
691428ae7c1728a76520b8f9d46fa6d83930f191 172.16.7.22:7003@17003 master - 0 1539016064655 4 connected 4096-8191
af2e1c600ed2511155a6cad61cabeb095affff86 172.16.7.33:7006@17006 master - 0 1539016062000 7 connected 8192-12287
72f44da597f73c8d23c6d3fb60fa95489eb0168d 172.16.7.11:7001@17001 master - 0 1539016063648 2 connected 12288-16383
711af78396c3ba966f751beef745ae10ff053a3f 172.16.7.11:7002@17002 slave af2e1c600ed2511155a6cad61cabeb095affff86 0 1539016063000 7 connected
d8138e15d9cb69962abb7457125a7965b735a658 172.16.7.11:7000@17000 myself,master - 0 1539016063000 1 connected 0-4095
ba4ce74ba7b84bde9ea60cd8bf4c5996f1ca01e5 172.16.7.22:7005@17005 slave 72f44da597f73c8d23c6d3fb60fa95489eb0168d 0 1539016063000 6 connected
6913f44486938c49eed6efa6f18c782df73e6456 172.16.7.33:7008@17008 slave d8138e15d9cb69962abb7457125a7965b735a658 0 1539016063000 9 connected列印叢集資訊
172.16.7.11:7000> CLUSTER INFO
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:9
cluster_size:4
cluster_current_epoch:9
cluster_my_epoch:1
cluster_stats_messages_ping_sent:1240
cluster_stats_messages_pong_sent:1251
cluster_stats_messages_sent:2491
cluster_stats_messages_ping_received:1243
cluster_stats_messages_pong_received:1240
cluster_stats_messages_meet_received:8
cluster_stats_messages_received:2491備註
若指定redis cluster中的master和slave ，可以用下面的方法
# 新增主庫
[root@redis01 redis-cluster]# redis-trib.rb create 172.16.7.11:7000 172.16.7.22:7003 172.16.7.33:7006
# 新增從庫
[root@redis01 redis-cluster]# redis-trib.rb add-node --slave 172.16.7.22:7001 172.16.7.11:7000 # 把172.16.7.11的7001 作為從庫新增到172.16.7.11的7000