使用 etcdadm 快速、彈性部署 etcd 叢集

Etcd 是一個可靠的分散式鍵值儲存, 常用於分散式系統關鍵資料的儲存；而 etcdadm 是一個用於操作 etcd 叢集的命令列工具，它可以輕鬆建立叢集、向現有叢集新增成員、從現有叢集中刪除成員等操作；其使用方式類似 kubeadm, 即主要操作流程為: 先啟動第一個叢集節點，後續節點直接 join 即可

建議通過 PC 端，存取 運維技術幫 (https://ywjsbang.com) 以獲取更好閱讀體驗，由於精力有限，該文章的後續更新、完善僅限此處，望理解 !!

測試環境

安裝 etcdadm

1、預編譯二進位制安裝

wget https://github.com/kubernetes-sigs/etcdadm/releases/download/v0.1.5/etcdadm-linux-amd64
mv etcdadm-linux-amd64 /usr/local/bin/etcdadm
chmod +x /usr/local/bin/etcdadm
scp /usr/local/bin/etcdadm 192.168.31.{38,39}:/usr/local/bin/

2、各節點系統防火牆放行埠 2379，2380

firewall-cmd --add-port=2379/tcp
firewall-cmd --add-port=2380/tcp

初始化 etcd 節點

1、初始化第一個 etcd 叢集節點

etcdadm init \
--version "3.5.5" \
--init-system "systemd" \
--install-dir "/opt/bin/" \
--certs-dir "/etc/etcd/pki" \
--data-dir "/var/lib/etcd" \
--release-url "https://github.com/etcd-io/etcd/releases/download"

# 主要選項解析
--version      # 指定部署的 etcd 版本
--init-system  # 設定 etcd 程序管理方式，預設 systemd，取值 kubelet 時，則以容器方法執行 etcd 程序
--install-dir  # etcd 二進位制程式安裝目錄

2、etcdadm init 初始化過程解析

# 下載解壓、安裝二進位制檔案 etcd、etcdctl
2022-10-20 14:26:12.781166 I | [install] Artifact not found in cache. Trying to fetch from upstream: https://github.com/etcd-io/etcd/releases/download
INFO[0000] [install] Downloading & installing etcd https://github.com/etcd-io/etcd/releases/download from 3.5.5 to /var/cache/etcdadm/etcd/v3.5.5
INFO[0000] [install] downloading etcd from https://github.com/etcd-io/etcd/releases/download/v3.5.5/etcd-v3.5.5-linux-amd64.tar.gz to /var/cache/etcdadm/etcd/v3.5.5/etcd-v3.5.5-linux-amd64.tar.gz
INFO[0009] [install] extracting etcd archive /var/cache/etcdadm/etcd/v3.5.5/etcd-v3.5.5-linux-amd64.tar.gz to /tmp/etcd641204404
INFO[0009] [install] verifying etcd 3.5.5 is installed in /opt/bin/

# 生成一個自簽名的 CA 證書及私鑰
INFO[0001] [certificates] creating PKI assets
INFO[0001] creating a self signed etcd CA certificate and key files
[certificates] Generated ca certificate and key.
> /etc/etcd/pki/ca.crt
> /etc/etcd/pki/ca.key

# 生成一個 server 證書及私鑰
INFO[0001] creating a new server certificate and key files for etcd
[certificates] Generated server certificate and key.
[certificates] server serving cert is signed for DNS names [c7] and IPs [192.168.31.37 127.0.0.1]
# > /etc/etcd/pki/server.crt
# > /etc/etcd/pki/server.key

# 生成一個 peer 證書及私鑰
INFO[0001] creating a new certificate and key files for etcd peering
[certificates] Generated peer certificate and key.
[certificates] peer serving cert is signed for DNS names [c7] and IPs [192.168.31.37]
# > /etc/etcd/pki/peer.crt
# > /etc/etcd/pki/peer.key

# 生成一個用於 etcdctl 的 client 證書及私鑰
INFO[0001] creating a new client certificate for the etcdctl
[certificates] Generated etcdctl-etcd-client certificate and key.
# > /etc/etcd/pki/etcdctl-etcd-client.crt
# > /etc/etcd/pki/etcdctl-etcd-client.key

# 生成一個用於 k8s apiserver 呼叫 etcd 時的 client 證書及私鑰
INFO[0002] creating a new client certificate for the apiserver calling etcd
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/etcd/pki"
# > /etc/etcd/pki/apiserver-etcd-client.crt
# > /etc/etcd/pki/apiserver-etcd-client.key

# 檢查本地 etcd 端點是否健康
INFO[0003] [health] Checking local etcd endpoint health
INFO[0003] [health] Local etcd endpoint is healthy

# 複製 CA cert/key 到其它 etcd 節點，並在其它 etcd 節點執行 etcdadm join 命令， 將其它 etcd 節點加入叢集
INFO[0003] To add another member to the cluster, copy the CA cert/key to its certificate dir and run:
INFO[0003]  etcdadm join https://192.168.31.37:2379

3、向其它節點分發 CA 根證書及私鑰

ssh [email protected] "mkdir /etc/etcd/pki/"
scp -r /etc/etcd/pki/{ca.crt,ca.key} 192.168.31.38:/etc/etcd/pki/

ssh [email protected] "mkdir /etc/etcd/pki/"
scp -r /etc/etcd/pki/{ca.crt,ca.key} 192.168.31.39:/etc/etcd/pki/

新增 etcd 節點

若當前主機無法下載，可提前將 etcd 二進位制程式包存放在如下路徑: /var/cache/etcdadm/etcd/v3.5.5/etcd-v3.5.5-linux-amd64.tar.gz

1、新增節點 192.168.31.38

etcdadm join https://192.168.31.38:2379 \
--version "3.5.5" \
--init-system "systemd" \
--install-dir "/opt/bin/"   \
--certs-dir  "/etc/etcd/pki" \
--data-dir "/var/lib/etcd"  \
--release-url "https://github.com/etcd-io/etcd/releases/download"

2、新增節點 192.168.31.39

etcdadm join https://192.168.31.38:2379 \
--version "3.5.5" \
--init-system "systemd" \
--install-dir "/opt/bin/"   \
--certs-dir  "/etc/etcd/pki" \
--data-dir "/var/lib/etcd"  \
--release-url "https://github.com/etcd-io/etcd/releases/download"

Etcd Server

1、用於 Etcd Server 的環境變數設定 /etc/etcd/etcd.env

ETCD_NAME=c7

# Initial cluster configuration
ETCD_INITIAL_CLUSTER=c7=https://192.168.31.37:2380
ETCD_INITIAL_CLUSTER_TOKEN=dee8095f
ETCD_INITIAL_CLUSTER_STATE=new

# Peer configuration
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.31.37:2380
ETCD_LISTEN_PEER_URLS=https://192.168.31.37:2380

ETCD_CLIENT_CERT_AUTH=true
ETCD_PEER_CERT_FILE=/etc/etcd/pki/peer.crt
ETCD_PEER_KEY_FILE=/etc/etcd/pki/peer.key
ETCD_PEER_TRUSTED_CA_FILE=/etc/etcd/pki/ca.crt

# Client/server configuration
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.31.37:2379
ETCD_LISTEN_CLIENT_URLS=https://192.168.31.37:2379,https://127.0.0.1:2379

ETCD_PEER_CLIENT_CERT_AUTH=true
ETCD_CERT_FILE=/etc/etcd/pki/server.crt
ETCD_KEY_FILE=/etc/etcd/pki/server.key
ETCD_TRUSTED_CA_FILE=/etc/etcd/pki/ca.crt

# Other
ETCD_DATA_DIR=/var/lib/etcd
ETCD_STRICT_RECONFIG_CHECK=true
GOMAXPROCS=8

# Logging configuration
# Profiling/metrics

2、Etcd Server 啟動指令碼

# cat /etc/systemd/system/etcd.service
[Unit]
Description=etcd
Documentation=https://github.com/coreos/etcd
Conflicts=etcd-member.service
Conflicts=etcd2.service

[Service]
EnvironmentFile=/etc/etcd/etcd.env
ExecStart=/opt/bin/etcd

Type=notify
TimeoutStartSec=0
Restart=on-failure
RestartSec=5s

LimitNOFILE=65536
Nice=-10
IOSchedulingClass=best-effort
IOSchedulingPriority=2
MemoryLow=200M

[Install]
WantedBy=multi-user.target

etcdctl.sh

1、用於 etcdctl 的環境變數設定 /etc/etcd/etcdctl.env

export ETCDCTL_API=3

export ETCDCTL_CACERT=/etc/etcd/pki/ca.crt
export ETCDCTL_CERT=/etc/etcd/pki/etcdctl-etcd-client.crt
export ETCDCTL_KEY=/etc/etcd/pki/etcdctl-etcd-client.key

export ETCDCTL_DIAL_TIMEOUT=3s

2、指令碼 etcdctl.sh 是對 etcdctl 命令的簡單包裝，其用法與 etcdctl 一致

cat /opt/bin/etcdctl.sh
#!/usr/bin/env sh
if ! [ -r "/etc/etcd/etcdctl.env" ]; then
        echo "Unable to read the etcdctl environment file '/etc/etcd/etcdctl.env'. The file must exist, and this wrapper must be run as root."
        exit 1
fi
. "/etc/etcd/etcdctl.env"   # 相當於 source 該環境變陣列態檔
"/opt/bin/etcdctl" "$@"     # $@ 表示指令碼 etcdctl.sh 的命令列引數

管理命令

# 檢視命令列 init 或 join 的幫助資訊
etcdadm init|join --help

# 從 etcd 叢集移除當前節點
etcdadm reset

# 檢視叢集節點成員
/opt/bin/etcdctl.sh member list
# > 19fc11a542653f62, started, c9, https://192.168.31.39:2380, https://192.168.31.39:2379, false
# > 9a246c6786d36273, started, c7, https://192.168.31.37:2380, https://192.168.31.37:2379, false
# > a509d3d8e8aa4911, started, c8, https://192.168.31.38:2380, https://192.168.31.38:2379, false

# 檢視當前節點是否正常
/opt/bin/etcdctl.sh endpoint health
# 127.0.0.1:2379 is healthy: successfully committed proposal: took = 17.112587ms

# 檢視當前節點狀態
/opt/bin/etcdctl.sh endpoint status
# > 127.0.0.1:2379, 9a246c6786d36273, 3.5.5, 20 kB, true, false, 3, 10, 10,

由於筆者時間、視野、認知有限，本文難免出現錯誤、疏漏等問題，期待各位讀者朋友、業界大佬指正交流, 共同進步 !!