Kubernetes Jenkins动态创建Slave

[TOC] # 0、前言 首先，我们考虑个问题，为何需要jenkins slave？其实在生产环境中，如果用单master，除非你单机器的配置特别高并且构建次数不多情况下，可以不考虑使用slave，但是，在构建次数上百次并且jenkins master运行在kubernetes环境中，借助kubernetes的灵活性，强烈推荐使用slave，master负责自动创建Slave Pod，然后将任务推送给Slave Pod，任务执行完毕后，Slave Pod会自动被回收/销毁。 # 1、Jenkins部署 jenkins交付进kubernetes 1.准备镜像文件 ```bash $ docker pull jenkins/jenkins:v2.204.1 $ docker tag 6097aa0af96e harbor.od.com/public/jenkins:v2.204.1 $ docker push harbor.od.com/public/jenkins:v2.204.1 ``` 2.资源配置文件 * rbac ```yaml $ vi https://img.qb5200.com/download-x/data/k8s-yaml/jenkins_slave/rbac.yaml apiVersion: v1 kind: ServiceAccount metadata: name: jenkins namespace: infra --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: jenkins rules: - apiGroups: ["extensions", "apps"] resources: ["deployments"] verbs: ["create", "delete", "get", "list", "watch", "patch", "update"] - apiGroups: [""] resources: ["services"] verbs: ["create", "delete", "get", "list", "watch", "patch", "update"] - apiGroups: [""] resources: ["pods"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/exec"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/log"] verbs: ["get","list","watch"] - apiGroups: [""] resources: ["secrets"] verbs: ["get"] --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: jenkins namespace: infra roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: jenkins subjects: - kind: ServiceAccount name: jenkins namespace: infra ``` > Jenkins会自动创建slave pod,所以需要给jenkins绑定权限 * deployment ```yaml $ vi https://img.qb5200.com/download-x/data/k8s-yaml/jenkins_slavehttps://img.qb5200.com/download-x/dp.yaml kind: Deployment apiVersion: extensions/v1beta1 metadata: name: jenkins namespace: infra labels: name: jenkins spec: replicas: 1 selector: matchLabels: name: jenkins template: metadata: labels: app: jenkins name: jenkins spec: serviceAccount: jenkins volumes: - name: data nfs: server: hdss7-200.host.com path: https://img.qb5200.com/download-x/data/nfs-volume/jenkins_home containers: - name: jenkins image: harbor.od.com/infra/jenkins:v2.204.1 imagePullPolicy: IfNotPresent ports: - containerPort: 8080 name: web protocol: TCP - containerPort: 50000 name: agent protocol: TCP env: - name: JAVA_OPTS value: "-Xms512G -Xmx512G -XX:PermSize=512m -XX:MaxPermSize=1024m -Duser.timezone=Asia/Shanghai" - name: TRY_UPGRADE_IF_NO_MARKER value: "true" volumeMounts: - name: data mountPath: /var/jenkins_home imagePullSecrets: - name: harbor securityContext: runAsUser: 0 strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 1 revisionHistoryLimit: 7 progressDeadlineSeconds: 600 ``` > 暴露web端口，还保留了agent端口，这个agent端口主要是用于 Jenkins 的 master 和 slave 之间通信使用的。 * service ```yaml $ vi https://img.qb5200.com/download-x/data/k8s-yaml/jenkins_slave/svc.yaml kind: Service apiVersion: v1 metadata: name: jenkins namespace: infra spec: ports: - name: web port: 80 targetPort: 8080 protocol: TCP - name: agent port: 50000 targetPort: 50000 protocol: TCP selector: app: jenkins ``` * ingress ```yaml $ vi https://img.qb5200.com/download-x/data/k8s-yaml/jenkins_slave/ingress.yaml kind: Ingress apiVersion: extensions/v1beta1 metadata: name: jenkins namespace: infra spec: rules: - host: jenkins.od.com http: paths: - path: / backend: serviceName: jenkins servicePort: 80 ``` 3.应用资源配置清单 ```bash $ kubectl apply -f http://k8s-yaml.od.com/jenkins_slave/rbac.yaml $ kubectl apply -f http://k8s-yaml.od.com/jenkins_slavehttps://img.qb5200.com/download-x/dp.yaml $ kubectl apply -f http://k8s-yaml.od.com/jenkins_slave/svc.yaml $ kubectl apply -f http://k8s-yaml.od.com/jenkins_slave/ingress.yaml ``` # 2、配置jenkins动态slave 初始化jenkins后，需要安装kubernetes 插件。 1.安装插件完成之后，点击 Manage Jenkins —> Configure System —> (拖到最下方)Add a new cloud —> 选择 Kubernetes，然后填写 Kubernetes 和 Jenkins 配置信息  填写kubernetes集群内部访问地址：https://kubernetes.default.svc.cluster.local，点击**Test Connection**，如果出现 Connection test successful 的提示信息证明 Jenkins 已经可以和 Kubernetes 系统正常通信了，然后下方的 Jenkins URL 地址：http://jenkins.infra.svc.cluster.local:8080 2.创建Pipeline动态构建测试 ```shell def label = "jenkins-slave-${UUID.randomUUID().toString()}" podTemplate(label: label, cloud: 'kubernetes') { node(label) { stage('Run shell') { sh 'sleep 10s' sh 'echo hello world.' } } } ``` 3.点击构建流水线 可以看到在jenkins的namespaces下自动创建了对应的agent pod 相当于就是一个jenkins 的node 当任务执行完成这个pod会自动退出这个pod默认会去pull一个`jenkins/jnlp-slave:x.xx-xx-alpine`的镜像 ```bash [root@hdss7-21 ~]# kubectl get pods -n infra NAME READY STATUS RESTARTS AGE jenkins-77b9c47874-qjgfd 1/1 Running 1 13h jenkins-slave-c07daa7b-31ef-41ea-825e-05c9c721edad-sb7h6-lpgwv 1/1 Running 0 18s ``` # 3、dubbo服务构建 我们在构建dubbo服务的时候，需要编译dubbo后制作镜像并推送到harbor，这时候就要用到maven和docker命令，所以需要自己去构建基础镜像，然后在pipeline里调用 > 注意：这里并不实现如何将dubbo服务交付到k8s，而是演示实现动态jenkins创建slave去构建dubbo服务 ## 3.1、制作dubbo镜像底包 底包需要有通用性，所有的dubbo微服务制作镜像时都会引用这个底包来制作，该底包主要实现一些功能，例如jmx监控（jmx_javaagent），和dubbo微服务的启动脚本，以及依赖的jdk环境 1.准备镜像（jdk环境） ```bash $ docker pull stanleyws/jre8:8u112 $ docker tag stanleyws/jre8:8u112 harbor.od.com/public/jre:8u112 $ docker push harbor.od.com/public/jre:8u112 ``` 2.自定义Dockerfile * https://img.qb5200.com/download-x/datahttps://img.qb5200.com/download-x/dockerfile/jre8/Dockerfile ```dockerfile FROM harbor.od.com/public/jre:8u112 RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\ echo 'Asia/Shanghai' >/etc/timezone ADD config.yml /opt/prom/config.yml ADD jmx_javaagent-0.3.1.jar /opt/prom/ WORKDIR /opt/project_dir ADD entrypoint.sh /entrypoint.sh CMD ["/entrypoint.sh"] ``` * config.yml（这是jmx_agent读取的配置文件） ```bash --- rules: - pattern: '.*' ``` * jmx_javaagent-0.3.1.jar ```bash # 采集jvm监控数据的jar包 $ wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.1/jmx_prometheus_javaagent-0.3.1.jar ``` * entrypoint.sh（不要忘了给该文件执行权限） ```bash #!/bin/sh M_OPTS="-Duser.timezone=Asia/Shanghai -javaagent:/opt/prom/jmx_javaagent-0.3.1.jar=$(hostname -i):${M_PORT:-"12346"}:/opt/prom/config.yml" C_OPTS=${C_OPTS} JAR_BALL=${JAR_BALL} exec java -jar ${M_OPTS} ${C_OPTS} ${JAR_BALL} ``` **按照上面的java启动选项解释** ```bash -agentpath:[=] // 加载java代理 -Duser.timezone: // 指定时区 C_OPTS=${C_OPTS} // 额外的启动参数，默认为空，可在k8s资源配置清单中添加额外参数 JAR_BALL=${JAR_BALL} // 启动的jar包名字，在k8s资源配置清单中指定 ``` 3.制作dubbo服务docker底包 ```bash $ ls -l total 372 -rw-r--r-- 1 root root 405 Jan 16 15:26 Dockerfile -rw-r--r-- 1 root root 41 Jan 16 15:28 config.yaml -rwxr-xr-x 1 root root 234 Jan 16 15:37 entrypoint.sh -rw-r--r-- 1 root root 367417 May 10 2018 jmx_prometheus_javaagent-0.3.1.jar $ docker build . -t harbor.od.com/base/jre8:8u112 $ docker push harbor.od.com/base/jre8:8u112 ``` ## 3.2、制作slave基础镜像 我们现在有个dubbo项目，需要用到maven构建项目，然后通过Docker打包成镜像并推送到Harbor，所以需要用到两个镜像，Maven以及Docker镜像。 ### 3.2.1、Maven镜像 该镜像主要用于构建java应用，这里就选择：maven:v3.3.9-jdk8 准备镜像文件（推送到本地仓库） ```bash $ docker pull maven:3.3.9-jdk-8-alpine $ docker tag dd9d4e1cd9db harbor.od.com/public/maven:v3.3.9-jdk8 $ docker push harbor.od.com/public/maven:v3.3.9-jdk8 ``` ### 3.2.2、Docker镜像 该镜像主要用于将dubbo项目打包成镜像并推送到harbor，但需要定制化一下镜像，需要将一台已经实现docker login 登录到harbor仓库所生成的配置文件，路径为：`/root/.docker/config.json`，与原始Docker镜像一起打包生成新的Docker镜像并推送到本地仓库。 1.准备镜像文件 ```bash $ docker pull docker:19.03 $ docker tag e036013d6d10 harbor.od.com/publichttps://img.qb5200.com/download-x/docker:v19.03 $ docker push harbor.od.com/publichttps://img.qb5200.com/download-x/docker:v19.03 ``` 2.Dockerfile * vim https://img.qb5200.com/download-x/datahttps://img.qb5200.com/download-x/dockerfilehttps://img.qb5200.com/download-x/docker/Dockerfile ```Dockerfile FROM harbor.od.com/publichttps://img.qb5200.com/download-x/docker:v19.03 USER root ADD config.json /root/.docker/config.json ``` 3.将`/root/.docker/config.json`文件拷贝到Dockerfile目录下 ```bash { "auths": { "harbor.od.com": { "auth": "YWRtaW46SGFyYm9yMTIzNDU=" } }, "HttpHeaders": { "User-Agent": "Docker-Client/19.03.6 (linux)" } ``` > 需要通过该文件才能访问到harbor仓库 4.制作并推送镜像（推送到本地仓库） ```bash $ docker build ./ -t harbor.od.com/publichttps://img.qb5200.com/download-x/docker:v19.03-1 $ docker push harbor.od.com/publichttps://img.qb5200.com/download-x/docker:v19.03-1 ``` ## 3.3、添加git key 我们这里使用到了Jenkins的Git插件来拉取代码，所以需要先创建一堆密钥，然后将公钥添加到git仓库，再将私钥添加到jenkins凭证，如下：  ## 3.4、创建dubbo流水线 1.添加参数化构建 * The String Parameter：app_name Describe：项目名称，例如dubbo-service * String Parameter：image_name Describe：docker镜像名称，格式：<仓库名>/<镜像名> 例如：apphttps://img.qb5200.com/download-x/dubbo-demo-service * String Parameter：git_repo Describe：项目所在的git中央仓库的地址：如https://gitee.com/jasonminghaohttps://img.qb5200.com/download-x/dubbo-demo-service.git * String Parameter：git_ver Describe：项目所在git中央仓库对应项目的分支或者版本号，例如master分支：*/master，commit ID：903b4e6 * String Parameter：image_ver Describe：镜像版本和git_ver一致即可，但是不要有任何的特殊符号 * String Parameter：add_tag Describe：docker镜像标签的一部分，日期时间戳，例如：20200121_1734 * String Parameter：mvn_dir Default：./ Describe：编译项目目录，默认为项目的根目录 * String Parameter：mvn_cmd Default：mvn clean package -Dmaven.test.skip=true Describe：执行mvn编译所用的命令 * Choice Parameter：base_image Choice Value： base/jre8:8u112 Describe：项目使用的docker底包镜像 2.pipeline ```bash podTemplate(containers: [ containerTemplate( name: 'maven', image: 'harbor.od.com/public/maven:v3.3.9-jdk8', ttyEnabled: true, command: 'cat'), containerTemplate( name: 'docker', ttyEnabled: true, image: 'harbor.od.com/publichttps://img.qb5200.com/download-x/docker:v19.03-1') ], volumes: [ nfsVolume(mountPath: '/root/.m2', readOnly: false, serverAddress: 'hdss7-200.host.com', serverPath: 'https://img.qb5200.com/download-x/data/nfs-volume/maven_repo/'), hostPathVolume(hostPath: '/runhttps://img.qb5200.com/download-x/docker.sock', mountPath: '/runhttps://img.qb5200.com/download-x/docker.sock') ]) { node(POD_LABEL) { stage('Get a Maven project') { checkout([ $class: 'GitSCM', branches: [[name: "${params.git_ver}"]], doGenerateSubmoduleConfigurations: false, extensions: [ [$class: 'AuthorInChangelog'], [$class: 'CloneOption',depth: 0,honorRefspec: true, noTags: true, reference: '', shallow: false] ], submoduleCfg: [], userRemoteConfigs: [[ credentialsId: 'git', url: "${params.git_repo}"] ]]) container('maven') { stage('Build a Maven project') { sh "ls -lha" sh "${params.mvn_cmd}" } } } stage('Docker build') { container('docker') { stage('create dir') { // /tmp目录创建一个临时用于构建镜像的工作目录，将jar包移动到该目录 sh "mkdir /tmp/${params.app_name}" sh "cd ${params.target_dir} && mkdir /tmp/${params.app_name}/project_dir && mv *.jar /tmp/${params.app_name}/project_dir" sh "ls -lha /tmp/${params.app_name}" } stage('docker build image') { // 动态生成Dockerfile，构建镜像并推送到harbor sh "cd /tmp/${params.app_name}/ && ls -lha" sh """ echo "FROM harbor.od.com/${params.base_image}" >/tmp/${params.app_name}/Dockerfile echo "ADD ./project_dir /opt/project_dir" >>/tmp/${params.app_name}/Dockerfile """ // writeFile 在当前pipeline脚本中不生效（抛弃使用） // writeFile file: "/tmp/${params.app_name}/Dockerfile", text: """FROM harbor.od.com/${params.base_image} ADD ./project_dir /opt/project_dir""" sh "ls -lha /tmp/${params.app_name}" sh "cat /tmp/${params.app_name}/Dockerfile" sh "cd /tmp/${params.app_name}/ && pwd && docker build ./ -t harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag} && docker push harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag} " } } } } } ``` >podTemplate只适用于pipeline ## 3.4、执行流水线构建 1.填写对应参数  2.查看`infra`名称空间下的Pod ```bash [root@hdss7-200 harbor]# kubectl get pods -n infra NAME READY STATUS RESTARTS AGE apollo-portal-57bc86966d-4tr6w 1/1 Running 8 37h dubbo-demo-slave-16-trktm-8x2d7-bw5dr 3/3 Running 0 53s dubbo-monitor-555c94f4b7-85plg 1/1 Running 32 7d14h jenkins-75fbb46546-f5ltc 1/1 Running 6 18h ``` > 可以看到jenkins slave会以在jenkins创建的项目名为命名来创建pod 3.构建结果