摘要

總算能讓devops執行以後，流水線卻卡在了deploy這一步。碰到了兩個比較大的問題，一個是無法使用k8sp自帶的kubeconfig認證去部署；一個是部署好了以後但是沒有辦法解析映象名。

版本資訊

k8s：v1.21.5
k8sp：v3.3.0

流水線概覽

Q問題描述

pipeline 在deploy 的階段總是報各種錯。

Q1.使用k8sp自帶kube認證產生報錯

      stage('deploy fail') {
        agent none
        steps {
          withCredentials([kubeconfigContent(credentialsId : 'kubeconfigger' ,variable : 'KUBECONFIGGER' ,)]) {
            kubernetesDeploy(enableConfigSubstitution: true, deleteResource: false, kubeconfigId: 'kubeconfigger', configs: 'hospital-manage/deploy/**')
          }

        }
      }

報錯內容如下：

Starting Kubernetes deployment
Loading configuration: /home/jenkins/agent/workspace/redp5lk5/rose/hospital-manage/deploy/deploy.yml
ERROR: ERROR: java.lang.RuntimeException: io.kubernetes.client.openapi.ApiException: 
hudson.remoting.ProxyException: java.lang.RuntimeException: io.kubernetes.client.openapi.ApiException: 
    at com.microsoft.jenkins.kubernetes.wrapper.ResourceManager.handleApiExceptionExceptNotFound(ResourceManager.java:180)
    at com.microsoft.jenkins.kubernetes.wrapper.V1ResourceManager$DeploymentUpdater.getCurrentResource(V1ResourceManager.java:213)
    at com.microsoft.jenkins.kubernetes.wrapper.V1ResourceManager$DeploymentUpdater.getCurrentResource(V1ResourceManager.java:201)
    at com.microsoft.jenkins.kubernetes.wrapper.ResourceManager$ResourceUpdater.createOrApply(ResourceManager.java:93)
    at com.microsoft.jenkins.kubernetes.wrapper.KubernetesClientWrapper.handleResource(KubernetesClientWrapper.java:289)
    at com.microsoft.jenkins.kubernetes.wrapper.KubernetesClientWrapper.apply(KubernetesClientWrapper.java:256)
    at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.doCall(DeploymentCommand.java:172)
    at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.call(DeploymentCommand.java:124)
    at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.call(DeploymentCommand.java:106)
    at hudson.remoting.UserRequest.perform(UserRequest.java:211)
    at hudson.remoting.UserRequest.perform(UserRequest.java:54)
    at hudson.remoting.Request$2.run(Request.java:376)
    at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
    at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:122)
    at java.base/java.lang.Thread.run(Unknown Source)
    Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 10.233.81.183/10.233.81.183:49480
        at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1797)
        at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
        at hudson.remoting.Channel.call(Channel.java:1001)
        at hudson.FilePath.act(FilePath.java:1256)
        at com.microsoft.jenkins.kubernetes.command.DeploymentCommand.execute(DeploymentCommand.java:68)
        at com.microsoft.jenkins.kubernetes.command.DeploymentCommand.execute(DeploymentCommand.java:45)
        at com.microsoft.jenkins.azurecommons.command.CommandService.runCommand(CommandService.java:88)
        at com.microsoft.jenkins.azurecommons.command.CommandService.execute(CommandService.java:96)
        at com.microsoft.jenkins.azurecommons.command.CommandService.executeCommands(CommandService.java:75)
        at com.microsoft.jenkins.azurecommons.command.BaseCommandContext.executeCommands(BaseCommandContext.java:77)
        at com.microsoft.jenkins.kubernetes.KubernetesDeploy.perform(KubernetesDeploy.java:42)
        at com.microsoft.jenkins.azurecommons.command.SimpleBuildStepExecution.run(SimpleBuildStepExecution.java:54)
        at com.microsoft.jenkins.azurecommons.command.SimpleBuildStepExecution.run(SimpleBuildStepExecution.java:35)
        at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: hudson.remoting.ProxyException: io.kubernetes.client.openapi.ApiException: 
    at io.kubernetes.client.openapi.ApiClient.handleResponse(ApiClient.java:979)
    at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:895)
    at io.kubernetes.client.openapi.apis.AppsV1Api.readNamespacedDeploymentWithHttpInfo(AppsV1Api.java:7299)
    at io.kubernetes.client.openapi.apis.AppsV1Api.readNamespacedDeployment(AppsV1Api.java:7275)
    at com.microsoft.jenkins.kubernetes.wrapper.V1ResourceManager$DeploymentUpdater.getCurrentResource(V1ResourceManager.java:210)
    ... 16 more
Api call failed with code 400, detailed message: {
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "the export parameter, deprecated since v1.14, is no longer supported",
  "reason": "BadRequest",
  "code": 400
}
Kubernetes deployment ended with HasError

從message來看，已經不支援這個什麼什麼引數了。從與小夥伴的溝通看，以前的版本是可以用的。但是新版本不支援了。

解決方案

採用以下寫法。
缺點：不支援圖形化流水線編輯，點選編輯該憑證會閃退。

法1

需要在環境變數處宣告 KUBECONFIG_CREDENTIAL_ID

    stage('deploy success') {
      agent none
      steps {
        container('maven') {
          withCredentials([kubeconfigFile(
                                           credentialsId: env.KUBECONFIG_CREDENTIAL_ID, variable: 'KUBECONFIG') ]) {
              sh 'envsubst < hospital-manage/deploy/deploy.yml | kubectl apply -f -'
            }

          }

        }
      }

法2

與法1沒什麼區別。主要區別就是shell中的命令。

    stage('deploy hospital-manage to dev') {
      agent none
      steps {
        container('maven') {
          withCredentials([kubeconfigFile(credentialsId: env.KUBECONFIG_CREDENTIAL_ID, variable: 'KUBECONFIG')]) {
            sh 'kubectl apply -f hospital-manage/deploy/**'
          }

        }

      }
    }

Q2：無法解析映象名

在deploy.yml檔案中，有關映象名的描述如下所示：

spec:
  containers:
    - image: $DOCKERHUB_NAMESPACE/server-gateway:SNAPSHOT-$BUILD_NUMBER

但是卻出現瞭如下結果

狀態資訊
初始化完成
狀態：True
容器組就緒
狀態：False
原因：ContainersNotReady
訊息：containers with unready status: [app]
所有容器就緒
狀態：False
原因：ContainersNotReady
訊息：containers with unready status: [app]
容器組排程完成
狀態：True
無法解析映象名稱

無法解析映象名，我點進去這個pod的yml以後，才發現，流水線沒能解析出這些變數。才導致了無法解析映象名。

解決方案

治標不治本方案

直接把映象寫死

belchance/ruoyi:hospital-manage_SNAPSHOT-10

替換環境變數方案

https://kubesphere.io/forum/d/5309-deploy-to-kubernets
來源於社群網友，方法就是使用 envsubst，把環境yml檔案裡的環境變數改了。
注意替換的字元要在JenkinsFile的環境變數區宣告，deploy.yml的位置要準確。

  agent none
  steps {
    container('nodejs') {
      sh 'envsubst \\'${REGISTRY},${ALIYUNHUB_NAMESPACE},${BUILD_NUMBER}\\' < deploy/deploy.yml > deploy/deploy2.yml'
      sh 'cat deploy/deploy2.yml'
    }
  }
}

envsubst用法介紹：

envsubst '$DOCKERHUB_NAME,$NUMBER' < deploy.yml 
envsubst '需要替換的環境變數' < target.file
command < file     將輸入重定向到 file。