技術文章 » etcd — 安裝部署

etcd — 安裝部署

2020-10-05 16:00:14

目錄

文章目錄

服務程序

etcd 目前預設使用 2379 埠提供 HTTP API 服務，2380 埠提供 Peer 通訊（這兩個埠已經被 IANA 官方預留給 etcd），在之前的版本中，可能會分別使用 4001 和 7001，在使用的過程中需要注意這個區別。

雖然 etcd 也支援單點部署，但是在生產環境中推薦叢集方式部署，一般 etcd 節點數會選擇 3、5、7。etcd 會保證所有的節點都會儲存資料，並保證資料的一致性和正確性。

單點部署

因為 etcd 是 Golang 編寫的，安裝只需要下載對應的二進位制檔案，並放到合適的路徑就行。如果在測試環境，啟動一個單點的 etcd 服務，只需要執行 etcd 執行即可。

git clone https://github.com/etcd-io/etcd.git
cd etcd
./build

使用 build 指令碼構建會在當前專案的 bin 目錄生產 etcd 和 etcdctl 可執行程式。etcd 就是 etcd Server，而 etcdctl 主要為 etcd Server 提供指令行操作。

檢視版本：

$ ./bin/etcd --version
etcd Version: 3.5.0-pre
Git SHA: ab4cc3cae
Go Version: go1.14.4
Go OS/Arch: darwin/amd64

$ ./bin/etcdctl version
etcdctl version: 3.5.0-pre
API version: 3.5

啟動 etcd Server：

$ ./bin/etcd
{"level":"info","ts":"2020-10-04T07:39:14.751+0800","caller":"etcdmain/etcd.go:69","msg":"Running: ","args":["./bin/etcd"]}
{"level":"info","ts":"2020-10-04T07:39:14.751+0800","caller":"etcdmain/etcd.go:94","msg":"failed to detect default host","error":"default host not supported on darwin_amd64"}
{"level":"warn","ts":"2020-10-04T07:39:14.751+0800","caller":"etcdmain/etcd.go:99","msg":"'data-dir' was empty; using default","data-dir":"default.etcd"}
{"level":"info","ts":"2020-10-04T07:39:14.751+0800","caller":"embed/etcd.go:113","msg":"configuring peer listeners","listen-peer-urls":["http://localhost:2380"]}
{"level":"info","ts":"2020-10-04T07:39:14.752+0800","caller":"embed/etcd.go:121","msg":"configuring client listeners","listen-client-urls":["http://localhost:2379"]}
{"level":"info","ts":"2020-10-04T07:39:14.753+0800","caller":"embed/etcd.go:266","msg":"starting an etcd server","etcd-version":"3.5.0-pre","git-sha":"ab4cc3cae","go-version":"go1.14.4","go-os":"darwin","go-arch":"amd64","max-cpu-set":4,"max-cpu-available":4,"member-initialized":false,"name":"default","data-dir":"default.etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"default.etcd/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":100000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://localhost:2380"],"listen-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"],"listen-client-urls":["http://localhost:2379"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"default=http://localhost:2380","initial-cluster-state":"new","initial-cluster-token":"etcd-cluster","quota-size-bytes":2147483648,"pre-vote":false,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","auto-compaction-mode":"periodic","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":""}
{"level":"info","ts":"2020-10-04T07:39:14.764+0800","caller":"etcdserver/backend.go:78","msg":"opened backend db","path":"default.etcd/member/snap/db","took":"9.908726ms"}
{"level":"info","ts":"2020-10-04T07:39:14.862+0800","caller":"etcdserver/raft.go:444","msg":"starting local member","local-member-id":"8e9e05c52164694d","cluster-id":"cdf818194e3a8c32"}
{"level":"info","ts":"2020-10-04T07:39:14.862+0800","caller":"raft/raft.go:1528","msg":"8e9e05c52164694d switched to configuration voters=()"}
{"level":"info","ts":"2020-10-04T07:39:14.862+0800","caller":"raft/raft.go:701","msg":"8e9e05c52164694d became follower at term 0"}
{"level":"info","ts":"2020-10-04T07:39:14.862+0800","caller":"raft/raft.go:383","msg":"newRaft 8e9e05c52164694d [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]"}
{"level":"info","ts":"2020-10-04T07:39:14.863+0800","caller":"raft/raft.go:701","msg":"8e9e05c52164694d became follower at term 1"}
{"level":"info","ts":"2020-10-04T07:39:14.863+0800","caller":"raft/raft.go:1528","msg":"8e9e05c52164694d switched to configuration voters=(10276657743932975437)"}
{"level":"warn","ts":"2020-10-04T07:39:14.888+0800","caller":"auth/store.go:1231","msg":"simple token is not cryptographically signed"}
{"level":"info","ts":"2020-10-04T07:39:14.912+0800","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
{"level":"info","ts":"2020-10-04T07:39:14.924+0800","caller":"etcdserver/server.go:752","msg":"starting etcd server","local-member-id":"8e9e05c52164694d","local-server-version":"3.5.0-pre","cluster-version":"to_be_decided"}
{"level":"info","ts":"2020-10-04T07:39:14.925+0800","caller":"etcdserver/server.go:640","msg":"started as single-node; fast-forwarding election ticks","local-member-id":"8e9e05c52164694d","forward-ticks":9,"forward-duration":"900ms","election-ticks":10,"election-timeout":"1s"}
{"level":"warn","ts":"2020-10-04T07:39:14.925+0800","caller":"etcdserver/metrics.go:212","msg":"failed to get file descriptor usage","error":"cannot get FDUsage on darwin"}
{"level":"info","ts":"2020-10-04T07:39:14.925+0800","caller":"raft/raft.go:1528","msg":"8e9e05c52164694d switched to configuration voters=(10276657743932975437)"}
{"level":"info","ts":"2020-10-04T07:39:14.925+0800","caller":"membership/cluster.go:385","msg":"added member","cluster-id":"cdf818194e3a8c32","local-member-id":"8e9e05c52164694d","added-peer-id":"8e9e05c52164694d","added-peer-peer-urls":["http://localhost:2380"]}
{"level":"info","ts":"2020-10-04T07:39:14.927+0800","caller":"embed/etcd.go:513","msg":"serving peer traffic","address":"127.0.0.1:2380"}
{"level":"info","ts":"2020-10-04T07:39:14.927+0800","caller":"embed/etcd.go:235","msg":"now serving peer/client/metrics","local-member-id":"8e9e05c52164694d","initial-advertise-peer-urls":["http://localhost:2380"],"listen-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"],"listen-client-urls":["http://localhost:2379"],"listen-metrics-urls":[]}
{"level":"info","ts":"2020-10-04T07:39:15.866+0800","caller":"raft/raft.go:788","msg":"8e9e05c52164694d is starting a new election at term 1"}
{"level":"info","ts":"2020-10-04T07:39:15.866+0800","caller":"raft/raft.go:714","msg":"8e9e05c52164694d became candidate at term 2"}
{"level":"info","ts":"2020-10-04T07:39:15.866+0800","caller":"raft/raft.go:848","msg":"8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 2"}
{"level":"info","ts":"2020-10-04T07:39:15.867+0800","caller":"raft/raft.go:766","msg":"8e9e05c52164694d became leader at term 2"}
{"level":"info","ts":"2020-10-04T07:39:15.867+0800","caller":"raft/node.go:327","msg":"raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 2"}
{"level":"info","ts":"2020-10-04T07:39:15.868+0800","caller":"etcdserver/server.go:2285","msg":"setting up initial cluster version","cluster-version":"3.5"}
{"level":"info","ts":"2020-10-04T07:39:15.876+0800","caller":"membership/cluster.go:523","msg":"set initial cluster version","cluster-id":"cdf818194e3a8c32","local-member-id":"8e9e05c52164694d","cluster-version":"3.5"}
{"level":"info","ts":"2020-10-04T07:39:15.876+0800","caller":"embed/serve.go:97","msg":"ready to serve client requests"}
{"level":"info","ts":"2020-10-04T07:39:15.876+0800","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
{"level":"info","ts":"2020-10-04T07:39:15.876+0800","caller":"etcdserver/server.go:2305","msg":"cluster version is updated","cluster-version":"3.5"}
{"level":"info","ts":"2020-10-04T07:39:15.876+0800","caller":"etcdserver/server.go:1863","msg":"published local member to cluster through raft","local-member-id":"8e9e05c52164694d","local-member-attributes":"{Name:default ClientURLs:[http://localhost:2379]}","request-path":"/0/members/8e9e05c52164694d/attributes","cluster-id":"cdf818194e3a8c32","publish-timeout":"7s"}
{"level":"info","ts":"2020-10-04T07:39:15.876+0800","caller":"etcdmain/main.go:47","msg":"notifying init daemon"}
{"level":"info","ts":"2020-10-04T07:39:15.876+0800","caller":"etcdmain/main.go:53","msg":"successfully notified init daemon"}
{"level":"info","ts":"2020-10-04T07:39:15.877+0800","caller":"embed/serve.go:139","msg":"serving client traffic insecurely; this is strongly discouraged!","address":"127.0.0.1:2379"}
  • name 表示節點名稱，預設為 default。
  • data-dir 表示 WAL 紀錄檔和 Snapshot 資料儲存目錄，預設為 ./default.etcd/ 目錄。
  • 使用 http://localhost:2380 和 etcd Cluster 中其他節點通訊。
  • 使用 http://localhost:2379 提供 HTTP API 服務，與使用者端通訊。
  • heartbeat 為 100ms，表示 Leader 多久傳送一次心跳到所有 Followers。
  • election-timeout 為 1s，該引數的作用是重新投票的超時時間，如果 Follow 在該時間間隔內沒有收到 Leader 發出的心跳包，就會觸發重新投票。
  • snapshot-count 為 100000，該引數的作用是指定有多少次事務被提交後觸發快照擷取動作並持久化到磁碟。
  • cluster-id 為 cdf818194e3a8c32。
  • raft.node 為 8e9e05c52164694d。
  • 啟動的時候，會執行 Raft，選舉出 Leader：elected leader 8e9e05c52164694d at term 2。

上述方法只是簡單的啟動了一個 etcd Server。當然，在生產環境中，通常使用 Systemd 來進行管理。

  • 建立相關目錄：
$ mkdir -p /var/lib/etcd/
$ mkdir -p /etc/etcd/config/
  • 設定 etcd 組態檔：
$ cat <<EOF | sudo tee /etc/etcd/config/etcd.conf
# 節點名稱
ETCD_NAME=$(hostname -s)
# 資料存放路徑
ETCD_DATA_DIR=/var/lib/etcd
EOF
  • 建立 systemd 組態檔：
$ cat <<EOF | sudo tee /etc/systemd/system/etcd.service

[Unit]
Description=Etcd Server
Documentation=https://github.com/coreos/etcd
After=network.target

[Service]
User=root
Type=notify
EnvironmentFile=-/opt/etcd/config/etcd.conf
ExecStart=~/workspace/etcd/bin
Restart=on-failure
RestartSec=10s
LimitNOFILE=40000

[Install]
WantedBy=multi-user.target
EOF
  • 啟動 etcd Server：
$ systemctl daemon-reload && systemctl enable etcd && systemctl start etcd

關鍵啟動選型清單

  • –name：指定 etcd Node 名稱，可以使用 hostname。
  • –data-dir：指定 etcd Server 持久化資料儲存目錄路徑。
  • –snapshot-count：指定有多少事務（transaction）被提交後，觸發擷取快照並持久化到磁碟。
  • –heartbeat-interval：指定 Leader 多久傳送一次心跳到 Followers。
  • –eletion-timeout：指定重新投票的超時時間，如果 Follow 在該時間間隔沒有收到 Leader 發出的心跳包，則會觸發重新投票。
  • –listen-peer-urls：指定和 Cluster 其他 Node 通訊的地址，比如：http://IP:2380，如果有多個，則使用逗號分隔。需要所有節點都能夠存取，所以不要使用 localhost。
  • –listen-client-urls：指定對外提供服務的地址，比如：http://IP:2379,http://127.0.0.1:2379。
  • –advertise-client-urls：對外通告的該節點的使用者端監聽地址，會告訴叢集中其他節點。
  • –initial-advertise-peer-urls：對外通告該節點的同伴（Peer）監聽地址，這個值會告訴叢集中其他節點。
  • –initial-cluster：指定叢集中所有節點的資訊，通常為 IP:Port 資訊，格式為：node1=http://ip1:2380,node2=http://ip2:2380,…。注意，這裡的 node1 就是 --name 指定的名字，ip1:2380 就是 --initial-advertise-peer-urls 指定的值。
  • –initial-cluster-state：新建叢集時，這個值為 new；假如已經存在了叢集，這個值為 existing。
  • –initial-cluster-token：建立叢集的 token，這個值每個叢集保持唯一。這樣的話，如果你要重新建立叢集，即使設定和之前一樣，也會再次生成新的叢集和節點 UUID；否則會導致多個叢集之間的衝突，造成未知的錯誤。

叢集部署

etcd Cluster 的部署，實際上就是多個主機上都部署 etcd Server，然後將它們加入到一個 Cluster 中。

在安裝和啟動 etcd 服務程序的時候，各個 Node 都需要知道 Cluster 中其他 Nodes 的資訊，一般是 IP:Port 資訊。根據使用者是否提前知曉（規劃）了每個 Node 的 IP 地址，有以下幾種不同的叢集部署方案：

  1. 靜態設定：在啟動 etcd Server 的時候，通過 --initial-cluster 引數設定好所有的節點資訊。
  2. 註冊到已有的 etcd Cluster：比如官方提供的 discovery.etcd.io。
  3. 使用 DNS 啟動。

靜態設定叢集

如果 etcd Cluster 中的成員是已知的，且具有固定的 IP 地址，就可以靜態的初始化一個叢集。

每個 Node 都使用如下環境變數：

ETCD_INITIAL_CLUSTER="radon=http://10.0.2.1:2380,neon=http://10.0.3.1:2380"
ETCD_INITIAL_CLUSTER_STATE=new

或者使用如下指令行引數來指定叢整合員：

--initial-cluster radon=http://10.0.2.1:2380,neon=http://10.0.3.1:2380
--initial-cluster-state new

初始化叢集：

etcd --name radon --initial-advertise-peer-urls http://10.0.2.1:2380
  --listen-peer-urls http://10.0.2.1:2380
  --listen-client-urls http://10.0.2.1:2379,http://127.0.0.1:2379
  --advertise-client-urls http://10.0.2.1:2380
  --initial-cluster-token etcd.gmem.cc
  --initial-cluster radon=http://10.0.2.1:2380,neon=http://10.0.3.1:2380
  --initial-cluster-state new

注：所有以 --initial-cluster* 開頭的選項，在第一次執行（Bootstrap）後都被忽略。

使用 TLS 加密，etcd 支援基於 TLS 加密的叢集內部、叢集外部（使用者端與叢集之間）的安全通訊，每個叢集節點都應該擁有被共用 CA 簽名的證書：

# 金鑰對、證書籤名請求
openssl genrsa -out radon.key 2048
export SAN_CFG=$(printf "\n[SAN]\nsubjectAltName=IP:127.0.0.1,IP:10.0.2.1,DNS:radon.gmem.cc")
openssl req -new -sha256 -key radon.key -out radon.csr \
    -subj "/C=CN/ST=BeiJing/O=Gmem Studio/CN=Server Radon" \
    -reqexts SAN -config <(cat /etc/ssl/openssl.cnf <(echo $SAN_CFG))

# 執行簽名
openssl x509 -req -sha256 -in radon.csr  -out radon.crt -CA ../ca.crt -CAkey ../ca.key -CAcreateserial -days 3650 \
     -extensions SAN -extfile <(echo "${SAN_CFG}")

初始化叢集命令需要修改為：

etcd --name radon --initial-advertise-peer-urls https://10.0.2.1:2380
  --listen-peer-urls https://10.0.2.1:2380
  --listen-client-urls https://10.0.2.1:2379,https://127.0.0.1:2379
  --advertise-client-urls https://10.0.2.1:2380
  --initial-cluster-token etcd.gmem.cc
  --initial-cluster radon=https://10.0.2.1:2380,neon=https://10.0.3.1:2380      # 指定叢整合員列表
  --initial-cluster-state new                                                                              # 初始化新叢集時使用  
  --initial-cluster-state existing                                                                        # 加入已有叢集時使用 

  # 使用者端 TLS 相關引數
  --client-cert-auth 
  --trusted-ca-file=/usr/share/ca-certificates/GmemCA.crt
  --cert-file=/opt/etcd/cert/radon.crt
  --key-file=/opt/etcd/cert/radon.key

  # 叢集內部 TLS 相關引數
  --peer-client-cert-auth
  --peer-trusted-ca-file=/usr/share/ca-certificates/GmemCA.crt
  --peer-cert-file=/opt/etcd/cert/radon.crt
  --peer-key-file=/opt/etcd/cert/radon.key