k8s+crio+podman搭建叢集

前言

在傳統的k8s叢集中，我們都是使用docker engine做為底層的容器管理軟體的，而docker engine因為不是k8s親生的解決方案，所以實際使用中會有更多的分層。之前我們也講過，k8s為了呼叫docker engine，專門寫了一個dockershim做為CRI，而在1.20版本的時候，k8s就宣佈停止更新dockershim了，也就是說再往後的版本就不推薦使用k8s+dockershim+docker engine的方案了。

而k8s官方比較推薦的解決方案中，官方比較推薦的是cri-o或者containerd，前者是基於開放容器計劃(OCI)的實現，後者是基於docker的containerd，後脫離出來進行獨立開發的元件，現歸屬於CNCF組織。

CRI-O vs containerd vs docker daemon

這三者有啥區別呢？

首先，cri-o是cri的實現，可以直接呼叫底層的runc

其次，containerd是CRI-Containerd的實現，可以呼叫底層的runc

而docker則需要先呼叫dockershim，然後呼叫docker，再呼叫containerd，最後呼叫底層的runc

三者區別如圖：

k8s+crio+podman實現

podman安裝

可參考我前一篇文章

docker的平替--podman - eryoung2 - 部落格園

三臺機都需要安裝podman

k8s的安裝

可參考我另一篇文章

kubernetes 搭建叢集 - eryoung2 - 部落格園

三臺機都需要安裝kubelet/kubeadm/kubectl，並啟動kubelet

cri-o的安裝

Ubuntu(18.04)

1 準備

modprobe overlay  # 開啟overlay
modprobe br_netfilter  # 開啟netfilter
cat > /etc/sysctl.d/99-kubernetes-cri.conf <<EOF #核心處理
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
sysctl --system
swapoff -a #kube scheduler要求關閉swap

2 安裝CRI-O

# 指定版本
OS=xUbuntu_18.04
CRIO_VERSION=1.23
# 加源
echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /"|sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
echo "deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$CRIO_VERSION/$OS/ /"|sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$CRIO_VERSION.list
# 加key
curl -L https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$CRIO_VERSION/$OS/Release.key | sudo apt-key add -
curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | sudo apt-key add -
# 安裝
sudo apt update -y
sudo apt install cri-o cri-o-runc cri-tools -y

3 檢查CRI-O

root@home:~# apt show cri-o
Package: cri-o
Version: 1.23.3~0
Priority: optional
Section: devel
Maintainer: Peter Hunt <[email protected]>
Installed-Size: 98.3 MB
Depends: libgpgme11, libseccomp2, conmon, containers-common (>= 0.1.27) | golang-github-containers-common, tzdata
Suggests: cri-o-runc | runc (>= 1.0.0), containernetworking-plugins
Replaces: cri-o-1.19, cri-o-1.20, cri-o-1.21, cri-o-1.22
Homepage: https://github.com/cri-o/cri-o
Download-Size: 19.9 MB
APT-Manual-Installed: yes
APT-Sources: http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/1.23/xUbuntu_18.04  Packages
Description: OCI-based implementation of Kubernetes Container Runtime Interface.

N: Ignoring file 'ystemctlqq' in directory '/etc/apt/sources.list.d/' as it has no filename extension

systemctl enable crio.service
systemctl start crio.service

root@home:~# systemctl status crio
● crio.service - Container Runtime Interface for OCI (CRI-O)
   Loaded: loaded (/usr/lib/systemd/system/crio.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2022-07-13 01:26:03 CST; 16h ago
     Docs: https://github.com/cri-o/cri-o
 Main PID: 5338 (crio)
    Tasks: 15
   CGroup: /system.slice/crio.service
           └─5338 /usr/bin/crio

4 使用cri-o

檢視狀態

root@home:~# crictl info
{
  "status": {
    "conditions": [
      {
        "type": "RuntimeReady",
        "status": true,
        "reason": "",
        "message": ""
      },
      {
        "type": "NetworkReady",
        "status": true,
        "reason": "",
        "message": ""
      }
    ]
  }
}

檢視映象

root@home:~# crictl images
IMAGE                                TAG                 IMAGE ID            SIZE
docker.io/calico/cni                 v3.23.2             a87d3f6f1b8fd       263MB
docker.io/calico/node                v3.23.2             a3447b26d32c7       224MB
docker.io/library/nginx              latest              41b0e86104ba6       146MB
k8s.gcr.io/coredns/coredns           v1.8.6              a4ca41631cc7a       47MB
k8s.gcr.io/etcd                      3.5.3-0             aebe758cef4cd       301MB
k8s.gcr.io/kube-apiserver            v1.24.2             d3377ffb7177c       131MB
k8s.gcr.io/kube-controller-manager   v1.24.2             34cdf99b1bb3b       121MB
k8s.gcr.io/kube-proxy                v1.24.2             a634548d10b03       112MB
k8s.gcr.io/kube-scheduler            v1.24.2             5d725196c1f47       52.3MB
k8s.gcr.io/pause                     3.6                 6270bb605e12e       690kB
k8s.gcr.io/pause                     3.7                 221177c6082a8       718kB

Centos(7)

1 準備

VERSION=1.22
sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/CentOS_7/devel:kubic:libcontainers:stable.repo
sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:${VERSION}.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:${VERSION}/CentOS_7/devel:kubic:libcontainers:stable:cri-o:${VERSION}.repo

2 安裝

yum update -y
yum install cri-o cri-tools -y

3 檢視cri-o版本

[root@node1 systemd]# rpm -qi cri-o
Name        : cri-o
Epoch       : 0
Version     : 1.22.5
Release     : 2.2.el7
Architecture: x86_64
Install Date: 2022年07月13日 星期三 01時36分47秒
Group       : Unspecified
Size        : 236845729
License     : ASL 2.0
Signature   : RSA/SHA256, 2022年07月10日 星期日 12時53分28秒, Key ID 4d64390375060aa4
Source RPM  : cri-o-1.22.5-2.2.el7.src.rpm
Build Date  : 2022年07月10日 星期日 12時53分00秒
Build Host  : sheep87
Relocations : (not relocatable)
Vendor      : obs://build.opensuse.org/devel:kubic
URL         : https://github.com/cri-o/cri-o
Summary     : Open Container Initiative-based implementation of Kubernetes Container Runtime Interface
Description :
Open Container Initiative-based implementation of Kubernetes Container Runtime
Interface.

4 啟動cri-o

systemctl enable --now cri-o

5 檢視cri-o狀態

[root@node1 systemd]# systemctl status crio
● crio.service - Container Runtime Interface for OCI (CRI-O)
   Loaded: loaded (/usr/lib/systemd/system/crio.service; enabled; vendor preset: disabled)
   Active: active (running) since 三 2022-07-13 01:41:06 CST; 16h ago
     Docs: https://github.com/cri-o/cri-o
 Main PID: 24127 (crio)
    Tasks: 15
   Memory: 13.7M
   CGroup: /system.slice/crio.service
           └─24127 /usr/bin/crio

三臺機都安裝cri-o並啟動。

K8S啟動

在master上，跑下列命令：

kubeadm init --apiserver-advertise-address 192.168.1.150 --apiserver-bind-port 6443 --kubernetes-version 1.24.2 --pod-network-cidr 10.244.0.0/16

然後等5分鐘，就建立了一個k8s叢集的master node。

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.1.150:6443 --token gjxt6y.0wljlhfkjz90v12m --discovery-token-ca-cert-hash sha256:d69fc5929e442210c97ab85c05a8c2906f5819a74d5b0fa3481032d6a8f3fc07

在三臺機上跑這三條命令

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

否則會報錯。

The connection to the server localhost:8080 was refused - did you specify the right host or port?

檢查叢集

在以上所有內容都完成之後，我們可以看一下nodes或者pods

root@home:~# kubectl get nodes
NAME    STATUS   ROLES           AGE    VERSION
home    Ready    control-plane   179m   v1.24.2
node1   Ready    <none>          179m   v1.24.2
node2   Ready    <none>          179m   v1.24.2
root@home:~# kubectl get pods -A
NAMESPACE     NAME                           READY   STATUS    RESTARTS   AGE
kube-system   coredns-6d4b75cb6d-4wxjh       1/1     Running   0          179m
kube-system   coredns-6d4b75cb6d-7qxpv       1/1     Running   0          179m
kube-system   etcd-home                      1/1     Running   2          3h
kube-system   kube-apiserver-home            1/1     Running   2          3h
kube-system   kube-controller-manager-home   1/1     Running   2          3h
kube-system   kube-proxy-9w7mf               1/1     Running   0          179m
kube-system   kube-proxy-hpw6c               1/1     Running   0          179m
kube-system   kube-proxy-tbpr8               1/1     Running   0          179m
kube-system   kube-scheduler-home            1/1     Running   2          3h