雲原生之旅

2022-11-13 12:00:29

前言

上一篇文章 雲原生之旅 - 10)手把手教你安裝 Jenkins on Kubernetes 我們介紹了在 Kubernetes 上安裝 Jenkins,本文介紹下如何設定k8s pod作為Jenkins 構建job的 agent。

Jenkins master 和 agent 均以 pod 的形式執行在 Kubernetes 節點上。Master 執行在其中一個節點上,其設定資料 Jenkins home 使用儲存卷掛載,master pod重啟不會導致資料丟失。agent 執行在各個節點上,根據需求動態建立並自動釋放。這樣做的好處很多,比如高可用,高伸縮性,資源利用率高。

關鍵詞:Jenkins on Kubernetes 實踐,Jenkins 和 Kubernetes,在Kubernetes上安裝Jenkins,Jenkins 高可用安裝,Jenkins 動態伸縮構建, Kubernetes Pod as Jenkins build agent

 

準備

  1. 已搭建 Jenkins master on kubernetes 雲原生之旅 - 10)手把手教你安裝 Jenkins on Kubernetes
  2. 準備一個 Service Account,對目標 cluster 具有k8s admin許可權,以便部署。
  3. 防火牆已開通 Jenkins 出站到Docker hub,方便 push/pull image
  4. 防火牆已開通 Jenkins 到 目標 cluster,以便部署。

 

外掛安裝

  • Kubernetes Plugin
  • Google Kubernetes Engine Plugin (我的例子是部署到 GKE cluster)

 

Jenkins 設定

Manage Nodes and Clouds

1. Go to `Manage Jenkins` –> `Manage Nodes and Clouds`
2. Click `Configure Clouds`
3. Add a new Cloud select `Kubernetes`
4. Click `Kubernetes Cloud Detail
5. Enter `jenkins` namespace in `Kubernetes Namespace` field
6. Click `Test Connection` --> result show `Connected to Kubernetes v1.22.12-gke.2300`
7. Click `Save`
8. Enter `http://jenkins-service.jenkins.svc.cluster.local:8080` in `Jenkins URL` field
9. Enter `jenkins-agent:50000` in `Jenkins tunnel` field

10. Click `Add Pod Template` then `Pod Template Details`

11. Input `Name`=`jenkins-agent`, `Namespace`=`jenkins`, `Labels`=`kubeagent`
 

 

12. (Optional) 如果不新增 container template, the Jenkins Kubernetes plugin will use the default JNLP image from the Docker hub to spin up the agents.
如果你要覆蓋預設的jnlp image 可以 Click `Add Container` to add Container Template,
輸入 `Name`=`jnlp`, `Docker Image`=`your_registry/jenkins/inbound-agent:4.11-1-jdk11`

 

Ensure that you remove the sleep and 9999999 default argument from the container template.

 

Manage Credentials

  • Add `Usernames with password` for docker hub account/pwd,比如 wade_test_dockerhub
  • Add `Google Service Account from private key` 比如 gcp_sa_json_key

 Credentials 會在Jenkinsfile裡面用到。

### 本文首發於部落格園 https://www.cnblogs.com/wade-xu/p/16863955.html

 

Test a freestyle project

Go to Jenkins home –> New Item and create a freestyle project,命名為 quick-test
在 job description 部分, add the label `kubeagent` for `Restrict where this project can be run`.

 

這個label 和我們上面建立 pod template時用的label一致. 這樣的話 Jenkins就知道用哪個 pod template 作為 agent container.

 
隨便新增一個shell 作為build steps

 

 點Build Now

 檢視Console Output
Agent jenkins-agent-l7hw9 is provisioned from template jenkins-agent

......

Building remotely on jenkins-agent-l7hw9 (kubeagent) in workspace /home/jenkins/agent/workspace/quick-test
[quick-test] $ /bin/sh -xe /tmp/jenkins17573873264046707236.sh
+ echo test pipeline
test pipeline
Finished: SUCCESS

 ### 本文首發於部落格園 https://www.cnblogs.com/wade-xu/p/16863955.html

 

Jenkinsfile

CI

接著我們用 Jenkinsfile 寫一個 Declarative pipeline - build/push docker image 到docker hub
首先需要定義一個 pod.yaml 作為啟動 agent 的container
kind: Pod
spec:
  containers:  # list of containers that you want present for your build, you can define a default container in the Jenkinsfile
    - name: maven
      image: maven:3.5.4-jdk-8-slim
      command: ["tail", "-f", "/dev/null"]  # this or any command that is bascially a noop is required, this is so that you don't overwrite the entrypoint of the base container
      imagePullPolicy: Always # use cache or pull image for agent
      resources:  # request and limit the resources your build contaienr
        requests:
          memory: 4Gi
          cpu: 2
        limits:
          memory: 4Gi
          cpu: 2
      volumeMounts:
        - mountPath: /root/.m2 # maven .m2 cache directory
          name: maven-home
    - name: git
      image: bitnami/git:2.38.1
      imagePullPolicy: IfNotPresent
      command: ["tail", "-f", "/dev/null"]
      resources: # limit the resources your build contaienr
        limits:
          cpu: 100m
          memory: 256Mi
    - name: kubectl-kustomize
      image: line/kubectl-kustomize:1.25.3-4.5.7
      imagePullPolicy: IfNotPresent
      command: ["tail", "-f", "/dev/null"]
      resources: # limit the resources your build contaienr
        limits:
          cpu: 100m
          memory: 256Mi
    - name: docker
      image: docker:18.06.1
      command: ["tail", "-f", "/dev/null"]
      imagePullPolicy: Always
      volumeMounts:
        - name: docker
          mountPath: /var/run/docker.sock # We use the k8s host docker engine
  volumes:
    - name: docker
      hostPath:
        path: /var/run/docker.sock
    - name: maven-home
      persistentVolumeClaim:
        claimName: maven-repo-storage
build-pod.yaml

在Jenkinsfile裡面定義agent 使用這個yaml file

  agent {
    kubernetes {
      idleMinutes 3  // how long the pod will live after no jobs have run on it
      yamlFile './build-pod.yaml'  // path to the pod definition relative to the root of our project 
      defaultContainer 'docker'  // define a default container if more than a few stages use it, otherwise default to jnlp container
    }

下面步驟是 docker login/build/tag/push


  environment {
    DOCKER_HUB_REGISTRY='https://index.docker.io/v1/'
    DOCKER_HUB_CREDS = credentials('wade_test_dockerhub')
  }
stage('Build and Push Docker Image') {
      steps {
        script {
          dir(dir_path) {
            container('docker') {
                // docker login, Using single-quotes instead of double-quotes when referencing these sensitive environment variables prevents this type of leaking.
                sh 'echo $DOCKER_HUB_CREDS_PSW | docker login -u $DOCKER_HUB_CREDS_USR --password-stdin $DOCKER_HUB_REGISTRY'
                // build image with git tag
                sh """
                docker build -t $PROJECT_IMAGE_WITH_TAG .
                docker tag $PROJECT_IMAGE_WITH_TAG $DOCKER_HUB_CREDS_USR/$PROJECT_IMAGE_WITH_TAG
                """

                // push image_tag to docker hub
                sh """
                docker push $DOCKER_HUB_CREDS_USR/$PROJECT_IMAGE_WITH_TAG
                """
            }
          }
        }
      }
    }

我這裡沒有選擇用 docker.withRegistry

docker.withRegistry("$DOCKER_HUB_REGISTRY", "$DOCKER_HUB_CREDENTIAL") {}

因為會有不安全的log提示

WARNING! Using --password via the CLI is insecure. Use --password-stdin.

 

CI + Kustomize + CD

這個例子是上面的 CI 之後 加上 - 利用 Kustomize build K8S resource manifests 然後 CD 到一個 Cluster

Kustomize 可以參考 雲原生之旅 - 6)不能錯過的一款 Kubernetes 應用編排管理神器 Kustomize

    // assume your k8s manifests in another repo, mine is same repo, just in order to show git clone step
    stage('Checkout K8S manifests') {
      steps {
        script {
          dir(dir_path) {
            container('git') {
              if (! fileExists('learning_by_doing/README.md')) {
                sh """
                git clone https://github.com/wadexu007/learning_by_doing.git
                ls -lhrt
                """
              } else {
                  sh 'echo manifes repo already exist.'
              }
            }
          }
        }
      }
    }

    stage('Build manifests with Kustomize') {
      steps {
        script {
          dir(dir_path) {
            container('kubectl-kustomize') {
                sh """
                cd learning_by_doing/Kustomize/demo-manifests/services/demo-app/dev/
                kustomize edit set image $DOCKER_HUB_CREDS_USR/$PROJECT_IMAGE_WITH_TAG
                kustomize build > $WORKSPACE/$dir_path/deployment.yaml
                """
            }
          }
        }
      }
    }

    stage('Deploy to GKE test cluster') {
			environment{
				PROJECT_ID = 'xperiences-eng-cn-dev'
        CLUSTER_NAME = 'xpe-spark-test-gke'
        REGION = 'asia-east2'
        CREDENTIALS_ID = 'gcp_sa_json_key'
      }
      steps {
        script {
          dir(dir_path) {
            container('kubectl-kustomize') {
                sh """
                chown 1000:1000 deployment.yaml
                echo start to deploy to cluster $CLUSTER_NAME
                """
                step([
                  $class: 'KubernetesEngineBuilder',
                  projectId: env.PROJECT_ID,
                  clusterName: env.CLUSTER_NAME,
                  location: env.REGION,
                  manifestPattern: 'deployment.yaml',
                  credentialsId: env.CREDENTIALS_ID,
                  verifyDeployments: false])
                  // verifyDeployments does not work for non-default namespace
            }
          }
        }
      }
    }
View Code 

 

Pipeline: Input Step

這個例子是利用 Jenkins pipeline的 Input step 來做一個人工介入Approve的步驟。 然後再來一個多cluster 部署,選不同region 部署到不同的cluster的範例。
    stage('Wait for SRE Approval') {
      steps {
        timeout(time:72, unit:'HOURS') {
          input message: "Approved Prod deployment?", submitter: 'sre-team'
        }
      }
    }

    // deployment to multipe k8s clusters
    stage('Deploy to GKE Prod cluster') {
			environment{
				PROJECT_ID = 'sre-cn-dev'
        CREDENTIALS_ID = 'gcp_sa_json_key'
        CLUSTER_COMMON_NAME = 'demo-gke-prod'
      }
      steps {
        script {
          env.REGION = input message: 'Choose which region you want to deploy?',
                             parameters: [choice(name: 'Region',
                                                description: 'Select Region to Deloy',
                                                choices: ['europe-west1', 'us-central1'])
                                          ]
          dir(dir_path) {
            if ( env.REGION == "europe-west1" ) {
              def eu_cluster_name = env.CLUSTER_COMMON_NAME + "-eu"
              container('kubectl-kustomize') {
                  sh "echo deploy to cluster $eu_cluster_name in region: $REGION"
              }
            }
            if ( env.REGION == "us-central1" ) {
              def us_cluster_name = env.CLUSTER_COMMON_NAME + "-us"
              container('kubectl-kustomize') {
                  sh "echo deploy to cluster $us_cluster_name in region: $REGION"
              }
            }
          }
        }
      }
    }

所有例子均在我的 github repo

### 本文首發於部落格園 https://www.cnblogs.com/wade-xu/p/16863955.html

 

測試

現在你可以建立一個 Pipeline 或者 Multibranch Pipeline job 來測試。
Repository URL = `https://github.com/wadexu007/learning_by_doing`
Script Path, e.g. `Jenkins/k8s_pod_as_build_agent/demo-app-java/Jenkinsfile`
 
你會看到每啟動一個job 都會相應的產生一個pod 來作為Jenkins agent執行,結束後根據idleMinutes自動釋放。
 
 

總結

如果你已經成功建立並測試 CI/CD pipeline,可以繼續加強,比如加上 Post notifications
 

最佳實踐

  • 設定 resource requests and limits on each container in your Pod
  • 如果使用maven 構建 java專案,.m2 cache目錄需要 mount 出來,這樣加快後面的maven build速度。
  • 使用 Jenkins Shared Libraries 抽取Pipeline的共用程式碼
  • 在容器裡構建容器化應用(Run docker in docker) 我的例子是通過 mount docker.sock 利用k8s 主機 docker engine來實現的,這種方式需要 privileges mode 不安全,推薦使用Kaniko,下一篇文章會介紹。

 

感謝閱讀,如果您覺得本文的內容對您的學習有所幫助,您可以打賞和推薦,您的鼓勵是我創作的動力。