技術文章 » Kubernetes後臺資料庫etcd：安裝部署etcd叢集，資料備份與恢復

Kubernetes後臺資料庫etcd：安裝部署etcd叢集，資料備份與恢復

2022-09-15 18:00:18

目錄

一.系統環境

伺服器版本 docker軟體版本 CPU架構
CentOS Linux release 7.4.1708 (Core) Docker version 20.10.12 x86_64

二.前言

etcd 是兼顧一致性與高可用性的鍵值對資料庫，可以作為儲存 Kubernetes 所有叢集資料的後臺資料庫。保持 etcd 叢集的穩定對 Kubernetes 叢集的穩定性至關重要。

三.etcd資料庫

3.1 概述

etcd是使用Go語言開發的一個開源的、高可用的分散式key-value儲存系統，可以用於設定共用和服務的註冊和發現。類似專案有zookeeper和consul。
etcd具有以下特點：

  • 完全複製：叢集中的每個節點都可以使用完整的存檔
  • 高可用性：Etcd可用於避免硬體的單點故障或網路問題
  • 一致性：每次讀取都會返回跨多主機的最新寫入
  • 簡單：包括一個定義良好、面向使用者的API（gRPC）
  • 安全：實現了帶有可選的使用者端證書身份驗證的自動化TLS
  • 快速：每秒10000次寫入的基準速度
  • 可靠：使用Raft演演算法實現了強一致、高可用的服務儲存目錄

Kubernetes 叢集對etcd叢集有幾點要求：

  • 執行的 etcd 叢集個數成員為奇數。
  • etcd 是一個 leader-based 分散式系統。確保主節點定期向所有從節點傳送心跳，以保持叢集穩定。
  • 確保不發生資源不足：叢集的效能和穩定性對網路和磁碟 I/O 非常敏感。任何資源匱乏都會導致心跳超時， 從而導致叢集的不穩定。不穩定的情況表明沒有選出任何主節點。 在這種情況下，叢集不能對其當前狀態進行任何更改，這意味著不能排程新的 Pod。
  • 保持 etcd 叢集的穩定對 Kubernetes 叢集的穩定性至關重要。 因此，請在專用機器或隔離環境上執行 etcd 叢集， 以滿足所需資源需求。
  • 在生產中執行的 etcd 的最低推薦版本是 3.2.10+。

四.安裝部署etcd單節點

4.1 環境介紹

首先安裝etcd單節點，etcd架構：etcd1機器作為etcd的伺服器端，etcd2機器作為使用者端存取

伺服器 作業系統版本 CPU架構 程序 功能描述
etcd1/192.168.110.133 CentOS Linux release 7.4.1708 (Core) x86_64 etcd etcd伺服器端
etcd2/192.168.110.131 CentOS Linux release 7.4.1708 (Core) x86_64 etcd etcd使用者端

4.2 設定節點的基本環境

先設定節點的基本環境，所有節點都要同時設定，在此以etcd1作為範例

首先設定主機名

[root@localhost ~]# cat /etc/hostname
etcd1

設定IP地址（可選）

[root@localhost ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens32

[root@localhost ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens32
TYPE=Ethernet
BOOTPROTO=static
NAME=ens32
DEVICE=ens32
ONBOOT=yes
DNS1=114.114.114.114
IPADDR=192.168.110.133
NETMASK=255.255.255.0
GATEWAY=192.168.110.2
ZONE=trusted

#重啟網路卡
[root@localhost ~]# service network restart
Restarting network (via systemctl):                        [  確定  ]

[root@localhost ~]# systemctl restart NetworkManager

重啟機器之後看是否能存取網路

[root@etcd1 ~]# ping www.baidu.com
PING www.a.shifen.com (14.215.177.38) 56(84) bytes of data.
64 bytes from 14.215.177.38 (14.215.177.38): icmp_seq=1 ttl=128 time=31.1 ms
64 bytes from 14.215.177.38 (14.215.177.38): icmp_seq=2 ttl=128 time=30.5 ms
64 bytes from 14.215.177.38 (14.215.177.38): icmp_seq=3 ttl=128 time=31.9 ms
64 bytes from 14.215.177.38 (14.215.177.38): icmp_seq=4 ttl=128 time=30.6 ms
^C
--- www.a.shifen.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3007ms
rtt min/avg/max/mdev = 30.593/31.075/31.926/0.533 ms

設定IP和主機名對映

[root@etcd1 ~]# vim /etc/hosts

[root@etcd1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.110.133 etcd1
192.168.110.131 etcd2
192.168.110.132 etcd3

複製到另外兩個主機

[root@etcd1 ~]# scp /etc/hosts etcd3:/etc/hosts
root@etcd3's password: 
hosts                                                                                                                                                                          100%  224   281.3KB/s   00:00    

[root@etcd1 ~]# scp /etc/hosts etcd2:/etc/hosts
root@etcd2's password: 
hosts                                                                                                                                                                          100%  224   218.1KB/s   00:00

能相互ping通則正常

[root@etcd1 ~]# ping etcd1
PING etcd1 (192.168.110.133) 56(84) bytes of data.
64 bytes from etcd1 (192.168.110.133): icmp_seq=1 ttl=64 time=0.029 ms
64 bytes from etcd1 (192.168.110.133): icmp_seq=2 ttl=64 time=0.033 ms
64 bytes from etcd1 (192.168.110.133): icmp_seq=3 ttl=64 time=0.043 ms
^C
--- etcd1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 0.029/0.035/0.043/0.005 ms

[root@etcd1 ~]# ping etcd2
PING etcd2 (192.168.110.131) 56(84) bytes of data.
64 bytes from etcd2 (192.168.110.131): icmp_seq=1 ttl=64 time=1.61 ms
64 bytes from etcd2 (192.168.110.131): icmp_seq=2 ttl=64 time=1.92 ms
^C
--- etcd2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1010ms
rtt min/avg/max/mdev = 1.612/1.769/1.926/0.157 ms

[root@etcd1 ~]# ping etcd3
PING etcd3 (192.168.110.132) 56(84) bytes of data.
64 bytes from etcd3 (192.168.110.132): icmp_seq=1 ttl=64 time=0.484 ms
64 bytes from etcd3 (192.168.110.132): icmp_seq=2 ttl=64 time=2.65 ms
64 bytes from etcd3 (192.168.110.132): icmp_seq=3 ttl=64 time=2.65 ms
^C
--- etcd3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2021ms
rtt min/avg/max/mdev = 0.484/1.931/2.657/1.023 ms

關閉屏保（可選）

[root@etcd1 ~]# setterm -blank 0

設定yum源

[root@etcd1 ~]# rm -rf /etc/yum.repos.d/* ;wget ftp://ftp.rhce.cc/k8s/* -P /etc/yum.repos.d/

關閉selinux

[root@etcd1 ~]# cat /etc/selinux/config 

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three two values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted 

[root@etcd1 ~]# getenforce
Disabled

設定防火牆允許所有資料通過

[root@etcd1 ~]# firewall-cmd --set-default-zone=trusted
Warning: ZONE_ALREADY_SET: trusted
success

[root@etcd1 ~]# firewall-cmd --get-default-zone
trusted

4.3 安裝部署etcd單節點

etcd1機器安裝etcd

[root@etcd1 ~]# yum -y install etcd

etcd的組態檔為/etc/etcd/etcd.conf ,註釋如下：

[root@etcd1 ~]# vim /etc/etcd/etcd.conf 

#組態檔簡單註釋：etcd節點間通訊埠2380，使用者端存取etcd的埠為2379  
#資料目錄：ETCD_DATA_DIR="/var/lib/etcd/default.etcd
[root@etcd1 ~]# cat /etc/etcd/etcd.conf | egrep -v "^#|^$"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#ETCD_LISTEN_PEER_URLS="http://localhost:2380"
ETCD_LISTEN_CLIENT_URLS="http://localhost:2379"
#etcd名字
ETCD_NAME="default"

修改組態檔如下，新增etcd1的IP和埠

[root@etcd1 ~]# vim /etc/etcd/etcd.conf

[root@etcd1 ~]# cat /etc/etcd/etcd.conf
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="http://192.168.110.133:2380,http://localhost:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.110.133:2379,http://localhost:2379"
ETCD_NAME="default"

#[Clustering]
ETCD_ADVERTISE_CLIENT_URLS="http://localhost:2379"

啟動etcd

[root@etcd1 ~]# systemctl enable etcd --now
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.

[root@etcd1 ~]# systemctl status etcd 
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: active (running) since 一 2022-01-10 22:53:13 CST; 9s ago
 Main PID: 1148 (etcd)
   CGroup: /system.slice/etcd.service
           └─1148 /usr/bin/etcd --name=default --data-dir=/var/lib/etcd/default.etcd --listen-client-urls=http://192.168.110.133:2379,http://localhost:2379

1月 10 22:53:13 etcd1 etcd[1148]: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 2
1月 10 22:53:13 etcd1 etcd[1148]: setting up the initial cluster version to 3.3
1月 10 22:53:13 etcd1 etcd[1148]: published {Name:default ClientURLs:[http://localhost:2379]} to cluster cdf818194e3a8c32
1月 10 22:53:13 etcd1 etcd[1148]: set the initial cluster version to 3.3
1月 10 22:53:13 etcd1 etcd[1148]: enabled capabilities for version 3.3
1月 10 22:53:13 etcd1 etcd[1148]: ready to serve client requests
1月 10 22:53:13 etcd1 etcd[1148]: ready to serve client requests
1月 10 22:53:13 etcd1 etcd[1148]: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
1月 10 22:53:13 etcd1 etcd[1148]: serving insecure client requests on 192.168.110.133:2379, this is strongly discouraged!
1月 10 22:53:13 etcd1 systemd[1]: Started Etcd Server.

檢視etcd的成員，檢視有幾個節點

[root@etcd1 ~]# etcdctl member list
8e9e05c52164694d: name=default peerURLs=http://localhost:2380 clientURLs=http://localhost:2379 isLeader=true

檢視叢集健康狀態

[root@etcd1 ~]# etcdctl cluster-health
member 8e9e05c52164694d is healthy: got healthy result from http://localhost:2379
cluster is healthy

4.4 使用使用者端存取etcd服務

現在etcd2機器作為使用者端存取etcd1機器上的etcd服務

首先etcd2機器安裝etcd

[root@etcd2 ~]# yum -y install etcd

檢視etcdctl的幫助

[root@etcd2 ~]# etcdctl --help

連線192.168.110.133上的etcd服務，發現拒絕

[root@etcd2 ~]# etcdctl --endpoints http://192.168.110.133:2379 ls /
Error:  client: etcd cluster is unavailable or misconfigured; error #0: dial tcp [::1]:2379: connect: connection refused

error #0: dial tcp [::1]:2379: connect: connection refused

修改etcd1上的組態檔ETCD_ADVERTISE_CLIENT_URLS="http://192.168.110.133:2379,http://localhost:2379"，並重啟etcd服務

[root@etcd1 ~]# cat /etc/etcd/etcd.conf | egrep -v "^#|^$"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="http://192.168.110.133:2380,http://localhost:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.110.133:2379,http://localhost:2379"
ETCD_NAME="default"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.110.133:2379,http://localhost:2379"

#重啟
[root@etcd1 ~]# systemctl restart etcd

[root@etcd1 ~]# systemctl status etcd

此時使用者端連線192.168.110.133成功，ls /表示查詢根目錄下內容

[root@etcd2 ~]# etcdctl --endpoints http://192.168.110.133:2379 ls /

4.4.1 使用2版本API管理etcd

etcd寫入資料的時候有兩個版本：2版本和3版本，預設是2版本，Set environment variable ETCDCTL_API=3 to use v3 API or ETCDCTL_API=2 to use v2 API

現在還沒有資料

[root@etcd1 ~]# etcdctl ls /

etcd1建立資料

[root@etcd1 ~]# etcdctl mkdir /comeon

[root@etcd1 ~]# etcdctl ls /
/comeon

使用者端連線也可以看到資料

[root@etcd2 ~]# etcdctl --endpoints http://192.168.110.133:2379 ls /
/comeon

刪除資料

[root@etcd1 ~]# etcdctl rmdir /comeon

[root@etcd1 ~]# etcdctl ls /

4.4.2 使用3版本API管理etcd

現在使用3版本的API

[root@etcd1 ~]# export ETCDCTL_API=3

設定為3版本的API之後，help顯示的內容也會不同

[root@etcd1 ~]# etcdctl --help
NAME:
        etcdctl - A simple command line client for etcd3.

USAGE:
        etcdctl

VERSION:
        3.3.11

API VERSION:
        3.3


COMMANDS:
        get			Gets the key or a range of keys
        put			Puts the given key into the store
        del			Removes the specified key or range of keys [key, range_end)
        txn			Txn processes all the requests in one transaction
        compaction		Compacts the event history in etcd
        alarm disarm		Disarms all alarms
        alarm list		Lists all alarms
        defrag			Defragments the storage of the etcd members with given endpoints
        endpoint health		Checks the healthiness of endpoints specified in `--endpoints` flag
        endpoint status		Prints out the status of endpoints specified in `--endpoints` flag
        endpoint hashkv		Prints the KV history hash for each endpoint in --endpoints
        move-leader		Transfers leadership to another etcd cluster member.
        watch			Watches events stream on keys or prefixes
        version			Prints the version of etcdctl
        lease grant		Creates leases
        lease revoke		Revokes leases
        lease timetolive	Get lease information
        lease list		List all active leases
        lease keep-alive	Keeps leases alive (renew)
        member add		Adds a member into the cluster
        member remove		Removes a member from the cluster
        member update		Updates a member in the cluster
        member list		Lists all members in the cluster
        snapshot save		Stores an etcd node backend snapshot to a given file
        snapshot restore	Restores an etcd member snapshot to an etcd directory
        snapshot status		Gets backend snapshot status of a given file
        make-mirror		Makes a mirror at the destination etcd cluster
        migrate			Migrates keys in a v2 store to a mvcc store
        lock			Acquires a named lock
        elect			Observes and participates in leader election
        auth enable		Enables authentication
        auth disable		Disables authentication
        user add		Adds a new user
        user delete		Deletes a user
        user get		Gets detailed information of a user
        user list		Lists all users
        user passwd		Changes password of user
        user grant-role		Grants a role to a user
        user revoke-role	Revokes a role from a user
        role add		Adds a new role
        role delete		Deletes a role
        role get		Gets detailed information of a role
        role list		Lists all roles
        role grant-permission	Grants a key to a role
        role revoke-permission	Revokes a key from a role
        check perf		Check the performance of the etcd cluster
        help			Help about any command

......
  -w, --write-out="simple"			set the output format (fields, json, protobuf, simple, table)

寫資料

[root@etcd1 ~]# etcdctl put student1 99
OK

查資料

[root@etcd1 ~]# etcdctl get student1
student1
99

注意：2版本API和3版本API不可互用，從k8s1.5版本，etcd就開始使用3版本往etcd裡寫資料

[root@etcd2 ~]# export ETCDCTL_API=3

[root@etcd2 ~]# etcdctl --endpoints http://192.168.110.133:2379 get student1
student1
99

etcd單節點搭建完畢，接下來新增兩個節點變為etcd叢集。

五.安裝部署etcd叢集

5.1 環境介紹

etcd叢集架構：etcd1為leader，etcd2為follower，etcd3為follower

伺服器 作業系統版本 CPU架構 程序 功能描述
etcd1/192.168.110.133 CentOS Linux release 7.4.1708 (Core) x86_64 etcd leader
etcd2/192.168.110.131 CentOS Linux release 7.4.1708 (Core) x86_64 etcd follower
etcd3/192.168.110.132 CentOS Linux release 7.4.1708 (Core) x86_64 etcd follower

5.2 把etcd2機器加入叢集

首先還原環境變數

[root@etcd1 ~]# unset ETCDCTL_API

停止etcd1機器的etcd服務

[root@etcd1 ~]# systemctl stop etcd

[root@etcd1 ~]# systemctl status etcd
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since 二 2022-01-11 15:21:30 CST; 49s ago
  Process: 1582 ExecStart=/bin/bash -c GOMAXPROCS=$(nproc) /usr/bin/etcd --name="${ETCD_NAME}" --data-dir="${ETCD_DATA_DIR}" --listen-client-urls="${ETCD_LISTEN_CLIENT_URLS}" (code=killed, signal=TERM)
 Main PID: 1582 (code=killed, signal=TERM)

檢視etcd的資料目錄

[root@etcd1 ~]# cat /etc/etcd/etcd.conf | grep -i data
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"

刪除原始資料

[root@etcd1 ~]# rm -rf /var/lib/etcd/*

修改組態檔，把etcd2機器加入設定

[root@etcd1 ~]# vim /etc/etcd/etcd.conf

[root@etcd1 ~]# cat /etc/etcd/etcd.conf | egrep -v "^#|^$"
#設定資料目錄
ETCD_DATA_DIR="/var/lib/etcd/cluster.etcd"
ETCD_LISTEN_PEER_URLS="http://192.168.110.133:2380,http://localhost:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.110.133:2379,http://localhost:2379"
ETCD_NAME="etcd133"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.110.133:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.110.133:2379,http://localhost:2379"
#目前是兩個節點，所以這裡是兩個節點的etcd
ETCD_INITIAL_CLUSTER="etcd133=http://192.168.110.133:2380,etcd131=http://192.168.110.131:2380"
#叢集token
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
#新建立etcd叢集的時候ETCD_INITIAL_CLUSTER_STATE="new"，往已經存在的etcd叢集新增etcd節點時：ETCD_INITIAL_CLUSTER_STATE="existing"
ETCD_INITIAL_CLUSTER_STATE="new"

複製組態檔到etcd2

[root@etcd1 ~]# scp /etc/etcd/etcd.conf etcd2:/etc/etcd/etcd.conf
root@etcd2's password: 
etcd.conf                                                                                                                                                                       100% 1813     1.7MB/s   00:00

etcd2機器修改組態檔

[root@etcd2 ~]# unset ETCDCTL_API

[root@etcd2 ~]# vim /etc/etcd/etcd.conf 

[root@etcd2 ~]# cat /etc/etcd/etcd.conf | egrep -v "^#|^$"
ETCD_DATA_DIR="/var/lib/etcd/cluster.etcd"
ETCD_LISTEN_PEER_URLS="http://192.168.110.131:2380,http://localhost:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.110.131:2379,http://localhost:2379"
ETCD_NAME="etcd131"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.110.131:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.110.131:2379,http://localhost:2379"
ETCD_INITIAL_CLUSTER="etcd133=http://192.168.110.133:2380,etcd131=http://192.168.110.131:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

兩個節點都啟動etcd

[root@etcd1 ~]# systemctl start etcd 

[root@etcd1 ~]# systemctl status etcd

[root@etcd2 ~]# systemctl enable etcd --now
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.

[root@etcd2 ~]# systemctl status etcd

檢視etcd叢整合員，可以看到192.168.110.133節點是Leader

[root@etcd1 ~]# etcdctl member list
341a3c460c1c993a: name=etcd131 peerURLs=http://192.168.110.131:2380 clientURLs=http://192.168.110.131:2379,http://localhost:2379 isLeader=false
ab23bcc86cf3190b: name=etcd133 peerURLs=http://192.168.110.133:2380 clientURLs=http://192.168.110.133:2379,http://localhost:2379 isLeader=true

叢集健康狀態

[root@etcd1 ~]# etcdctl cluster-health
member 341a3c460c1c993a is healthy: got healthy result from http://192.168.110.131:2379
member ab23bcc86cf3190b is healthy: got healthy result from http://192.168.110.133:2379
cluster is healthy

現在兩個節點的etcd叢集搭建完畢，資料也同步了

[root@etcd1 ~]# etcdctl ls /

[root@etcd1 ~]# etcdctl mkdir /public

[root@etcd1 ~]# export ETCDCTL_API=3

[root@etcd1 ~]# etcdctl put student1 59
OK

[root@etcd2 ~]# etcdctl ls /
/public

[root@etcd2 ~]# export ETCDCTL_API=3

[root@etcd2 ~]# etcdctl get student1
student1
59

5.3 把etcd3機器加入叢集

現在新增一個節點etcd3到叢集

etcd3安裝etcd

[root@etcd3 ~]# yum -y install etcd

加入新節點的時候，使用API2版本

[root@etcd1 ~]# export ETCDCTL_API=2

執行新增節點命令，注意：ETCD_INITIAL_CLUSTER_STATE="existing"

[root@etcd1 ~]# etcdctl member add etcd132 http://192.168.110.132:2380
Added member named etcd132 with ID 7d816f4fa2bea295 to cluster

ETCD_NAME="etcd132"
ETCD_INITIAL_CLUSTER="etcd131=http://192.168.110.131:2380,etcd132=http://192.168.110.132:2380,etcd133=http://192.168.110.133:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

檢視叢整合員，發現192.168.110.132顯示不正常

[root@etcd1 ~]# etcdctl member list
341a3c460c1c993a: name=etcd131 peerURLs=http://192.168.110.131:2380 clientURLs=http://192.168.110.131:2379,http://localhost:2379 isLeader=false
7d816f4fa2bea295[unstarted]: peerURLs=http://192.168.110.132:2380
ab23bcc86cf3190b: name=etcd133 peerURLs=http://192.168.110.133:2380 clientURLs=http://192.168.110.133:2379,http://localhost:2379 isLeader=true

複製組態檔到etcd3

[root@etcd1 ~]# scp /etc/etcd/etcd.conf etcd3:/etc/etcd/etcd.conf
root@etcd3's password: 
etcd.conf                                                                                                                                                                       100% 1813     1.1MB/s   00:00

etcd3修改組態檔

 #注意：新增到一個已經存在的叢集，etcd133和etcd131組態檔不變，只修改etcd132組態檔
[root@etcd3 ~]# vim /etc/etcd/etcd.conf 

[root@etcd3 ~]# cat /etc/etcd/etcd.conf | egrep -v "^#|^$"
ETCD_DATA_DIR="/var/lib/etcd/cluster.etcd"
ETCD_LISTEN_PEER_URLS="http://192.168.110.132:2380,http://localhost:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.110.132:2379,http://localhost:2379"
ETCD_NAME="etcd132"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.110.132:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.110.132:2379,http://localhost:2379"
ETCD_INITIAL_CLUSTER="etcd133=http://192.168.110.133:2380,etcd131=http://192.168.110.131:2380,etcd132=http://192.168.110.132:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="existing"

etcd3啟動etcd

[root@etcd3 ~]# systemctl enable etcd --now
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.

檢視etcd叢整合員，etcd叢集有三個節點了

[root@etcd1 ~]# etcdctl member list
341a3c460c1c993a: name=etcd131 peerURLs=http://192.168.110.131:2380 clientURLs=http://192.168.110.131:2379,http://localhost:2379 isLeader=false
7d816f4fa2bea295: name=etcd132 peerURLs=http://192.168.110.132:2380 clientURLs=http://192.168.110.132:2379,http://localhost:2379 isLeader=false
ab23bcc86cf3190b: name=etcd133 peerURLs=http://192.168.110.133:2380 clientURLs=http://192.168.110.133:2379,http://localhost:2379 isLeader=true

自此etcd叢集搭建完畢，接下來進行etcd服務的管理。

六.etcd做快照備份資料

生產環境中，有些重要資料是要備份的，以免造成資料丟失。etcd對資料做快照可以進行資料備份。

設定API版本為3

[root@etcd1 ~]# export ETCDCTL_API=3

為了避免資料丟失，可以進行資料備份，即資料做快照，資料快照的help如下

[root@etcd1 ~]# etcdctl snap --help
NAME:
        snapshot - Manages etcd node snapshots

USAGE:
        etcdctl snapshot <subcommand>

API VERSION:
        3.3


COMMANDS:
        save	Stores an etcd node backend snapshot to a given file
        restore	Restores an etcd member snapshot to an etcd directory
        status	Gets backend snapshot status of a given file

GLOBAL OPTIONS:
      --cacert=""				verify certificates of TLS-enabled secure servers using this CA bundle
      --cert=""					identify secure client using this TLS certificate file
      --command-timeout=5s			timeout for short running command (excluding dial timeout)
      --debug[=false]				enable client-side debug logging
      --dial-timeout=2s				dial timeout for client connections
  -d, --discovery-srv=""			domain name to query for SRV records describing cluster endpoints
      --endpoints=[127.0.0.1:2379]		gRPC endpoints
      --hex[=false]				print byte strings as hex encoded strings
      --insecure-discovery[=true]		accept insecure SRV records describing cluster endpoints
      --insecure-skip-tls-verify[=false]	skip server certificate verification
      --insecure-transport[=true]		disable transport security for client connections
      --keepalive-time=2s			keepalive time for client connections
      --keepalive-timeout=6s			keepalive timeout for client connections
      --key=""					identify secure client using this TLS key file
      --user=""					username[:password] for authentication (prompt if password is not supplied)
  -w, --write-out="simple"			set the output format (fields, json, protobuf, simple, table)

進行快照

[root@etcd1 ~]# etcdctl snap save student.data
Snapshot saved at student.data

七.etcd恢復資料

刪除資料使用快照進行恢復

[root@etcd1 ~]# etcdctl del student1
1
[root@etcd1 ~]# etcdctl del student2
1

[root@etcd1 ~]# etcdctl get student1

[root@etcd1 ~]# etcdctl get student2

使用快照恢復資料的時候需要把快照複製到其他節點

[root@etcd1 ~]# scp student.data etcd2:~/
root@etcd2's password: 
student.data                                                                                                                                                                    100%   20KB  14.5MB/s   00:00    
[root@etcd1 ~]# scp student.data etcd3:~/
root@etcd3's password: 
student.data                                                                                                                                                                    100%   20KB   8.1MB/s   00:00

在所有節點停止etcd並清空資料

[root@etcd1 ~]# systemctl stop etcd
[root@etcd1 ~]# rm -rf /var/lib/etcd/*
[root@etcd1 ~]# chown etcd:etcd student.data 

[root@etcd2 ~]# systemctl stop etcd
[root@etcd2 ~]# rm -rf /var/lib/etcd/*
[root@etcd2 ~]# chown etcd:etcd student.data

[root@etcd3 ~]# systemctl stop etcd
[root@etcd3 ~]# rm -rf /var/lib/etcd/*
[root@etcd3 ~]# chown etcd:etcd student.data

在每個節點上使用快照恢復資料

[root@etcd1 ~]# etcdctl snapshot restore student.data --name etcd133 --initial-cluster etcd133=http://192.168.110.133:2380,etcd131=http://192.168.110.131:2380,etcd132=http://192.168.110.132:2380 --initial-advertise-peer-urls http://192.168.110.133:2380 --data-dir /var/lib/etcd/cluster.etcd
2022-01-11 16:44:08.375319 I | etcdserver/membership: added member 341a3c460c1c993a [http://192.168.110.131:2380] to cluster dd7594df5e81191b
2022-01-11 16:44:08.375393 I | etcdserver/membership: added member 4679fe0fcb37326d [http://192.168.110.132:2380] to cluster dd7594df5e81191b
2022-01-11 16:44:08.375404 I | etcdserver/membership: added member ab23bcc86cf3190b [http://192.168.110.133:2380] to cluster dd7594df5e81191b

[root@etcd2 ~]# etcdctl snapshot restore student.data --name etcd131 --initial-cluster etcd133=http://192.168.110.133:2380,etcd131=http://192.168.110.131:2380,etcd132=http://192.168.110.132:2380 --initial-advertise-peer-urls http://192.168.110.131:2380 --data-dir /var/lib/etcd/cluster.etcd
2022-01-11 16:45:18.378931 I | etcdserver/membership: added member 341a3c460c1c993a [http://192.168.110.131:2380] to cluster dd7594df5e81191b
2022-01-11 16:45:18.378991 I | etcdserver/membership: added member 4679fe0fcb37326d [http://192.168.110.132:2380] to cluster dd7594df5e81191b
2022-01-11 16:45:18.379000 I | etcdserver/membership: added member ab23bcc86cf3190b [http://192.168.110.133:2380] to cluster dd7594df5e81191b

[root@etcd3 ~]# etcdctl snapshot restore student.data --name etcd132 --initial-cluster etcd133=http://192.168.110.133:2380,etcd131=http://192.168.110.131:2380,etcd132=http://192.168.110.132:2380 --initial-advertise-peer-urls http://192.168.110.132:2380 --data-dir /var/lib/etcd/cluster.etcd
2022-01-11 16:46:26.826533 I | etcdserver/membership: added member 341a3c460c1c993a [http://192.168.110.131:2380] to cluster dd7594df5e81191b
2022-01-11 16:46:26.826584 I | etcdserver/membership: added member 4679fe0fcb37326d [http://192.168.110.132:2380] to cluster dd7594df5e81191b
2022-01-11 16:46:26.826595 I | etcdserver/membership: added member ab23bcc86cf3190b [http://192.168.110.133:2380] to cluster dd7594df5e81191b

修改所有節點資料目錄的屬主

[root@etcd1 ~]# chown -R etcd:etcd /var/lib/etcd/

[root@etcd2 ~]# chown -R etcd:etcd /var/lib/etcd/

[root@etcd3 ~]# chown -R etcd:etcd /var/lib/etcd/

啟動etcd

[root@etcd1 ~]# systemctl start etcd

[root@etcd2 ~]# systemctl start etcd

[root@etcd3 ~]# systemctl start etcd

可以發現資料已經恢復

[root@etcd1 ~]# etcdctl get student1
student1
59

[root@etcd3 ~]# etcdctl get student2
student2
62

八.Kubernetes(k8s)中以pod方式執行的etcd

etcd在Kubernetes叢集中可以以pod的方式執行，也可以以物理機部署的方式執行，本章講解以pod方式執行的etcd。

在此之前，需要有一套可以正常執行的Kubernetes叢集，關於Kubernetes(k8s)叢集的安裝部署，可以檢視部落格《Centos7 安裝部署Kubernetes(k8s)叢集》https://www.cnblogs.com/renshengdezheli/p/16686769.html

在k8s中etcd以pod的方式執行，那組態檔在哪裡，資料目錄在哪裡？

檢視etcd pod，k8s中etcd為etcd-k8scloude1

[root@k8scloude1 ~]# kubectl get pods
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-6b9fbfff44-4jzkj   1/1     Running   4          2d4h
calico-node-bdlgm                          1/1     Running   2          2d4h
calico-node-hx8bk                          1/1     Running   2          2d4h
calico-node-nsbfs                          1/1     Running   2          2d4h
coredns-545d6fc579-7wm95                   1/1     Running   2          2d4h
coredns-545d6fc579-87q8j                   1/1     Running   2          2d4h
etcd-k8scloude1                            1/1     Running   2          2d4h
kube-apiserver-k8scloude1                  1/1     Running   2          2d4h
kube-controller-manager-k8scloude1         1/1     Running   2          2d4h
kube-proxy-599xh                           1/1     Running   2          2d4h
kube-proxy-lpj8z                           1/1     Running   2          2d4h
kube-proxy-zxlk9                           1/1     Running   2          2d4h
kube-scheduler-k8scloude1                  1/1     Running   2          2d4h
metrics-server-bcfb98c76-k5dmj             1/1     Running   1          33h

在k8s中etcd以pod的形式執行， 此etcd的組態檔在哪？在/etc/kubernetes/manifests/etcd.yaml

[root@k8scloude1 ~]# ls /etc/kubernetes/manifests/etcd.yaml 
/etc/kubernetes/manifests/etcd.yaml

[root@k8scloude1 ~]# cat !$
cat /etc/kubernetes/manifests/etcd.yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/etcd.advertise-client-urls: https://192.168.110.130:2379
  creationTimestamp: null
  labels:
    component: etcd
    tier: control-plane
  name: etcd
  namespace: kube-system
spec:
  containers:
  - command:
    - etcd
    - --advertise-client-urls=https://192.168.110.130:2379
    - --cert-file=/etc/kubernetes/pki/etcd/server.crt
    - --client-cert-auth=true
    - --data-dir=/var/lib/etcd
    - --initial-advertise-peer-urls=https://192.168.110.130:2380
    - --initial-cluster=k8scloude1=https://192.168.110.130:2380
    - --key-file=/etc/kubernetes/pki/etcd/server.key
    - --listen-client-urls=https://127.0.0.1:2379,https://192.168.110.130:2379
    - --listen-metrics-urls=http://127.0.0.1:2381
    - --listen-peer-urls=https://192.168.110.130:2380
    - --name=k8scloude1
    - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
    - --peer-client-cert-auth=true
    - --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
    - --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
    - --snapshot-count=10000
    - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
    image: registry.aliyuncs.com/google_containers/etcd:3.4.13-0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 127.0.0.1
        path: /health
        port: 2381
        scheme: HTTP
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    name: etcd
    resources:
      requests:
        cpu: 100m
        ephemeral-storage: 100Mi
        memory: 100Mi
    startupProbe:
      failureThreshold: 24
      httpGet:
        host: 127.0.0.1
        path: /health
        port: 2381
        scheme: HTTP
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    volumeMounts:
    - mountPath: /var/lib/etcd
      name: etcd-data
    - mountPath: /etc/kubernetes/pki/etcd
      name: etcd-certs
  hostNetwork: true
  priorityClassName: system-node-critical
  volumes:
  - hostPath:
      path: /etc/kubernetes/pki/etcd
      type: DirectoryOrCreate
    name: etcd-certs
  - hostPath:
      path: /var/lib/etcd
      type: DirectoryOrCreate
    name: etcd-data
status: {}

可以發現掛載了資料卷，資料目錄在/var/lib/etcd/

[root@k8scloude1 ~]# ls /var/lib/etcd/
member

[root@k8scloude1 ~]# ls /var/lib/etcd/member/
snap  wal