前言：

在上一篇文章中，我们已经在K8S集群部署了Jenkins、Harbor和EFK。作为本系列最后一篇文章，将通过实际案例串联所有的基础软件服务，基于K8S做DevOps。

整体的业务流程如下图所示：

一、一机多Jenkins Slave

由于业务需要，我们的自动化测试需要基于windows做web功能测试，每一个测试任务独占一个windows用户桌面，所以我们首先要给Jenkins配置几个Windows的Slave Node.在我之前的post《持续集成CI实施指南三–jenkins集成测试》中详细讲解了给Jenkins添加Node的方法步骤。本篇无需重复，但这里主要讲的是，如何在一台Windows服务器上搭建多个Jenkins Node，供多用户使用。

在目标机上建立多个用户，如下图所示：
用Administrator用户安装JDK
在Jenkins的节点管理建立三个Node，分别为WinTester01、WinTester02、WinTester03，配置如下
在目标机的Administrator，用IE打开Jenkins并进入节点管理，在WinTester01、WinTester02、WinTester03中分别点击“Launch”启动Slave
确认启动成功后，点击“File”下的“Install as service”
三个Slave都启动后，可以在服务管理器看到
除了Jenkins Slave1无需配置，Slave2和Slave3都需要右键进入属性，修改登录用户分别为JenkinsSlave2和JenkinsSlave3

通过上面的配置，可以在一台目标机部署三个用户对应三个Jenkins Slave以满足我们的业务需求。

二、二次开发Jenkins 钉钉通知插件

在整个DevOps的业务流程图上，我们想使用钉钉作为通知方式，相比邮件而言，实时性和扩展性都很高。在2018年4月，Jenkins的钉钉通知插件有两款，分别是Dingding JSON Pusher和Dingding notification plugin，前者长期未更新，已经不能使用，后者可以在非Pipeline模式下使用，对于Pipeline则有一些问题。虽然目前，Dingding notification plugin已经更新到1.9版本并支持了Pipeline，但在当时，我们不得不在1.4版本的基础上做二次开发。

整体开发经过参考《Jenkins项目实战之-钉钉提醒插件二次开发举例》，总体来说还是比较简单：

修改”src/main/java/com/ztbsuper/dingtalk/DingTalkNotifier.java”，钉钉的消息API类型有文本、link、markdown、card等，我们这里把通知接口改成文本类型

<span role="presentation"><span class="cm-keyword">public</span> <span class="cm-keyword">class</span> <span class="cm-def">DingTalkNotifier</span> <span class="cm-keyword">extends</span> <span class="cm-variable">Notifier</span> <span class="cm-keyword">implements</span> <span class="cm-variable">SimpleBuildStep</span> {</span>
<span role="presentation"></span>
<span role="presentation">    <span class="cm-keyword">private</span> <span class="cm-variable-3">String</span> <span class="cm-variable">accessToken</span>;</span>
<span role="presentation">    <span class="cm-keyword">private</span> <span class="cm-variable-3">String</span> <span class="cm-variable">message</span>;</span>
<span role="presentation">    <span class="cm-keyword">private</span> <span class="cm-variable-3">String</span> <span class="cm-variable">imageUrl</span>;</span>
<span role="presentation">    <span class="cm-keyword">private</span> <span class="cm-variable-3">String</span> <span class="cm-variable">messageUrl</span>;</span>
<span role="presentation"></span>
<span role="presentation">    <span class="cm-meta">@DataBoundConstructor</span></span>
<span role="presentation">    <span class="cm-keyword">public</span> <span class="cm-variable">DingTalkNotifier</span>(<span class="cm-variable-3">String</span> <span class="cm-variable">accessToken</span>, <span class="cm-variable-3">String</span> <span class="cm-variable">message</span>, <span class="cm-variable-3">String</span> <span class="cm-variable">imageUrl</span>, <span class="cm-variable-3">String</span> <span class="cm-variable">messageUrl</span>) {</span>
<span role="presentation">        <span class="cm-keyword">this</span>.<span class="cm-variable">accessToken</span> <span class="cm-operator">=</span> <span class="cm-variable">accessToken</span>; <span class="cm-comment">//钉钉的accesstoken</span></span>
<span role="presentation">        <span class="cm-keyword">this</span>.<span class="cm-variable">message</span> <span class="cm-operator">=</span> <span class="cm-variable">message</span>;    <span class="cm-comment">//消息主体</span></span>
<span role="presentation">        <span class="cm-keyword">this</span>.<span class="cm-variable">imageUrl</span> <span class="cm-operator">=</span> <span class="cm-variable">imageUrl</span>;  <span class="cm-comment">//缩略图</span></span>
<span role="presentation">        <span class="cm-keyword">this</span>.<span class="cm-variable">messageUrl</span> <span class="cm-operator">=</span> <span class="cm-variable">messageUrl</span>;  <span class="cm-comment">//消息的链接来源，一般是jenkins的build url</span></span>
<span role="presentation">    }</span>
<span role="presentation"></span>
<span role="presentation">    <span class="cm-keyword">public</span> <span class="cm-variable-3">String</span> <span class="cm-variable">getAccessToken</span>() {</span>
<span role="presentation">        <span class="cm-keyword">return</span> <span class="cm-variable">accessToken</span>;</span>
<span role="presentation">    }</span>
<span role="presentation">    <span class="cm-keyword">public</span> <span class="cm-variable-3">String</span> <span class="cm-variable">getMessage</span>() {</span>
<span role="presentation">        <span class="cm-keyword">return</span> <span class="cm-variable">message</span>;</span>
<span role="presentation">    }</span>
<span role="presentation">    <span class="cm-keyword">public</span> <span class="cm-variable-3">String</span> <span class="cm-variable">getImageUrl</span>() {</span>
<span role="presentation">        <span class="cm-keyword">return</span> <span class="cm-variable">imageUrl</span>;</span>
<span role="presentation">    }</span>
<span role="presentation">    <span class="cm-keyword">public</span> <span class="cm-variable-3">String</span> <span class="cm-variable">getMessageUrl</span>() {</span>
<span role="presentation">        <span class="cm-keyword">return</span> <span class="cm-variable">messageUrl</span>;</span>
<span role="presentation">    }</span>
<span role="presentation"></span>
<span role="presentation">    <span class="cm-meta">@Override</span></span>
<span role="presentation">    <span class="cm-keyword">public</span> <span class="cm-variable-3">void</span> <span class="cm-variable">perform</span>(<span class="cm-meta">@Nonnull</span> <span class="cm-variable">Run</span><span class="cm-operator"><?</span>, <span class="cm-operator">?></span> <span class="cm-variable">run</span>, <span class="cm-meta">@Nonnull</span> <span class="cm-variable">FilePath</span> <span class="cm-variable">filePath</span>, <span class="cm-meta">@Nonnull</span> <span class="cm-variable">Launcher</span> <span class="cm-variable">launcher</span>, <span class="cm-meta">@Nonnull</span> <span class="cm-variable">TaskListener</span> <span class="cm-variable">taskListener</span>) <span class="cm-keyword">throws</span> <span class="cm-variable">InterruptedException</span>, <span class="cm-variable">IOException</span> {</span>
<span role="presentation">        <span class="cm-variable-3">String</span> <span class="cm-variable">buildInfo</span> <span class="cm-operator">=</span> <span class="cm-variable">run</span>.<span class="cm-variable">getFullDisplayName</span>();</span>
<span role="presentation">        <span class="cm-keyword">if</span> (<span class="cm-operator">!</span><span class="cm-variable">StringUtils</span>.<span class="cm-variable">isBlank</span>(<span class="cm-variable">message</span>)) {</span>
<span role="presentation">            <span class="cm-variable">sendMessage</span>(<span class="cm-variable">LinkMessage</span>.<span class="cm-variable">builder</span>()</span>
<span role="presentation">                    .<span class="cm-variable">title</span>(<span class="cm-variable">buildInfo</span>)</span>
<span role="presentation">                    .<span class="cm-variable">picUrl</span>(<span class="cm-variable">imageUrl</span>)</span>
<span role="presentation">                    .<span class="cm-variable">text</span>(<span class="cm-variable">message</span>)</span>
<span role="presentation">                    .<span class="cm-variable">messageUrl</span>(<span class="cm-variable">messageUrl</span>)</span>
<span role="presentation">                    .<span class="cm-variable">build</span>());</span>
<span role="presentation">        }</span>
<span role="presentation">    }</span>
<span role="presentation"></span>
<span role="presentation">    <span class="cm-keyword">private</span> <span class="cm-variable-3">void</span> <span class="cm-variable">sendMessage</span>(<span class="cm-variable">DingMessage</span> <span class="cm-variable">message</span>) {</span>
<span role="presentation">        <span class="cm-variable">DingTalkClient</span> <span class="cm-variable">dingTalkClient</span> <span class="cm-operator">=</span> <span class="cm-variable">DingTalkClient</span>.<span class="cm-variable">getInstance</span>();</span>
<span role="presentation">        <span class="cm-keyword">try</span> {</span>
<span role="presentation">            <span class="cm-variable">dingTalkClient</span>.<span class="cm-variable">sendMessage</span>(<span class="cm-variable">accessToken</span>, <span class="cm-variable">message</span>);</span>
<span role="presentation">        } <span class="cm-keyword">catch</span> (<span class="cm-variable">IOException</span> <span class="cm-variable">e</span>) {</span>
<span role="presentation">            <span class="cm-variable">e</span>.<span class="cm-variable">printStackTrace</span>();</span>
<span role="presentation">        }</span>
<span role="presentation">    }</span>
<span role="presentation"></span>
<span role="presentation">    <span class="cm-meta">@Override</span></span>
<span role="presentation">    <span class="cm-keyword">public</span> <span class="cm-variable">BuildStepMonitor</span> <span class="cm-variable">getRequiredMonitorService</span>() {</span>
<span role="presentation">        <span class="cm-keyword">return</span> <span class="cm-variable">BuildStepMonitor</span>.<span class="cm-variable">NONE</span>;</span>
<span role="presentation">    }</span>
<span role="presentation"></span>
<span role="presentation">    <span class="cm-meta">@Symbol</span>(<span class="cm-string">"dingTalk"</span>)</span>
<span role="presentation">    <span class="cm-meta">@Extension</span></span>
<span role="presentation">    <span class="cm-keyword">public</span> <span class="cm-keyword">static</span> <span class="cm-keyword">final</span> <span class="cm-keyword">class</span> <span class="cm-def">DescriptorImpl</span> <span class="cm-keyword">extends</span> <span class="cm-variable">BuildStepDescriptor</span><span class="cm-operator"><</span><span class="cm-variable">Publisher</span><span class="cm-operator">></span> {</span>
<span role="presentation"></span>
<span role="presentation">        <span class="cm-meta">@Override</span></span>
<span role="presentation">        <span class="cm-keyword">public</span> <span class="cm-variable-3">boolean</span> <span class="cm-variable">isApplicable</span>(<span class="cm-variable">Class</span><span class="cm-operator"><?</span> <span class="cm-keyword">extends</span> <span class="cm-variable">AbstractProject</span><span class="cm-operator">></span> <span class="cm-variable">aClass</span>) {</span>
<span role="presentation">            <span class="cm-keyword">return</span> <span class="cm-atom">true</span>;</span>
<span role="presentation">        }</span>
<span role="presentation"></span>
<span role="presentation">        <span class="cm-meta">@Nonnull</span></span>
<span role="presentation">        <span class="cm-meta">@Override</span></span>
<span role="presentation">        <span class="cm-keyword">public</span> <span class="cm-variable-3">String</span> <span class="cm-variable">getDisplayName</span>() {</span>
<span role="presentation">            <span class="cm-keyword">return</span> <span class="cm-variable">Messages</span>.<span class="cm-variable">DingTalkNotifier_DescriptorImpl_DisplayName</span>();</span>
<span role="presentation">        }</span>
<span role="presentation">    }</span>
<span role="presentation">}</span>

www#gaodaima.com来源gao@daima#com搞(%代@#码@网搞代码

用maven打包
maven需要安装java环境，为了方便，我直接run一个maven的docker image，编译完成后把hpi文件send出来
在jenkins的插件管理页面上传hpi文件
在钉钉群中开启自定义机器人
找到accesstoken

在jenkins pipeline中可以使用以下命令发送信息到钉钉群

<span role="presentation">dingTalk accessToken:"2fccafaexxxx",message:"信息",imageUrl:"图片地址",messageUrl:"消息链接"</span>

三、 DevOps解决方案

针对每一个软件项目增加部署目录，目录结构如下：

_deploy
- master
  - deployment.yaml
  - Dockerfile
  - other files
- test
  - deployment.yaml
  - Dockerfile
  - other files

master和test文件夹用于区分测试环境与生产环境的部署配置

Dockerfile和other files用于生成应用或服务的镜像

如前端vue和nodejs项目的Dockerfile：

<span role="presentation"><span class="cm-comment"># 前端项目运行环境的Image，从Harbor获取</span></span>
<span role="presentation"><span class="cm-variable-2">FROM</span> xxx/xxx/frontend:1.0.0 </span>
<span role="presentation"><span class="cm-variable-2">RUN</span> mkdir -p /workspace/build && mkdir -p /workspace/run</span>
<span role="presentation"><span class="cm-variable-2">COPY</span> . /workspace/build</span>
<span role="presentation"><span class="cm-comment"># 编译，生成执行文件，并删除源文件</span></span>
<span role="presentation"><span class="cm-variable-2">RUN</span> cd /workspace/build/frontend && </span>
<span role="presentation">    cnpm install && </span>
<span role="presentation">    npm run test && </span>
<span role="presentation">    cp -r /workspace/build/app/* /workspace/run && </span>
<span role="presentation">    rm -rf /workspace/build && </span>
<span role="presentation">    cd /workspace/run && </span>
<span role="presentation">    cnpm install </span>
<span role="presentation"><span class="cm-comment"># 运行项目，用npm run test或run prod区分测试和生产环境</span></span>
<span role="presentation"><span class="cm-variable-2">CMD</span> cd /workspace/run && npm run test</span>

又如dotnet core项目的Dockerfile：

<span role="presentation"><span class="cm-comment"># dotnet项目编译环境的Image，从Harbor获取</span></span>
<span role="presentation"><span class="cm-variable-2">FROM</span> xxx/xxx/aspnetcore-build:2 AS builder</span>
<span role="presentation"><span class="cm-variable-2">WORKDIR</span> /app</span>
<span role="presentation"><span class="cm-variable-2">COPY</span> . .</span>
<span role="presentation"><span class="cm-comment"># 编译</span></span>
<span role="presentation"><span class="cm-variable-2">RUN</span> cd /app/xxx</span>
<span role="presentation"><span class="cm-variable-2">RUN</span> pwd && ls -al && dotnet restore</span>
<span role="presentation"><span class="cm-variable-2">RUN</span> dotnet publish -c Release -o publish</span>
<span role="presentation"></span>
<span role="presentation"><span class="cm-comment"># dotnet项目运行环境的Image，从Harbor获取</span></span>
<span role="presentation"><span class="cm-variable-2">FROM</span> xxx/xxx/aspnetcore:2</span>
<span role="presentation"><span class="cm-variable-2">WORKDIR</span> /publish</span>
<span role="presentation"><span class="cm-variable-2">COPY</span> --from=builder /app/xxx/publish .</span>
<span role="presentation"><span class="cm-comment"># 重命名配置文件，中缀test、prod用于区分测试环境和生产环境</span></span>
<span role="presentation"><span class="cm-variable-2">RUN</span> mv appsettings.test.json appsettings.json</span>
<span role="presentation"><span class="cm-comment"># 运行</span></span>
<span role="presentation"><span class="cm-variable-2">ENTRYPOINT</span> ["dotnet", "xxx.dll"]</span>

deployent.yaml用于执行应用或服务在k8s上的部署

由于deployment有很多配置项可以抽离成公共配置，所以deployment的配置有很多占位变量，占位变量用两个#中间加变量名表示，如下所示：

<span role="presentation"><span class="cm-atom">apiVersion</span><span class="cm-meta">: </span>v1  </span>
<span role="presentation"><span class="cm-atom">kind</span><span class="cm-meta">: </span>Namespace  </span>
<span role="presentation"><span class="cm-atom">metadata</span><span class="cm-meta">:  </span></span>
<span role="presentation"><span class="cm-atom">    name</span><span class="cm-meta">: </span><span class="cm-comment">#namespace#  </span></span>
<span role="presentation"><span class="cm-atom">    labels</span><span class="cm-meta">:  </span></span>
<span role="presentation"><span class="cm-atom">      name</span><span class="cm-meta">: </span><span class="cm-comment">#namespace#  </span></span>
<span role="presentation"><span class="cm-def">---</span></span>
<span role="presentation"><span class="cm-atom">apiVersion</span><span class="cm-meta">: </span>v1</span>
<span role="presentation"><span class="cm-atom">data</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-atom">  .dockerconfigjson</span><span class="cm-meta">: </span>xxxxxxxxxxxxxxxxxxxxxx</span>
<span role="presentation"><span class="cm-atom">kind</span><span class="cm-meta">: </span>Secret</span>
<span role="presentation"><span class="cm-atom">metadata</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-atom">  name</span><span class="cm-meta">: </span>regcred</span>
<span role="presentation"><span class="cm-atom">  namespace</span><span class="cm-meta">: </span><span class="cm-comment">#namespace#  </span></span>
<span role="presentation"><span class="cm-atom">type</span><span class="cm-meta">: </span>kubernetes.io/dockerconfigjson</span>
<span role="presentation"><span class="cm-def">---</span></span>
<span role="presentation"><span class="cm-atom">apiVersion</span><span class="cm-meta">: </span>extensions/v1beta1</span>
<span role="presentation"><span class="cm-atom">kind</span><span class="cm-meta">: </span>Deployment</span>
<span role="presentation"><span class="cm-atom">metadata</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-atom">  name</span><span class="cm-meta">: </span><span class="cm-comment">#app#-deploy</span></span>
<span role="presentation"><span class="cm-atom">  namespace</span><span class="cm-meta">: </span><span class="cm-comment">#namespace# </span></span>
<span role="presentation"><span class="cm-atom">  labels</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-atom">    app</span><span class="cm-meta">: </span><span class="cm-comment">#app#-deploy</span></span>
<span role="presentation"><span class="cm-atom">spec</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-atom">  replicas</span><span class="cm-meta">: </span><span class="cm-comment">#replicas#</span></span>
<span role="presentation"><span class="cm-atom">  strategy</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-atom">    type</span><span class="cm-meta">: </span>Recreate</span>
<span role="presentation"><span class="cm-atom">  template</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-atom">    metadata</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-atom">      labels</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-atom">        app</span><span class="cm-meta">: </span><span class="cm-comment">#app#</span></span>
<span role="presentation"><span class="cm-atom">    spec</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-atom">      containers</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-meta">      - </span><span class="cm-atom">image</span><span class="cm-meta">:  </span><span class="cm-comment">#image#  </span></span>
<span role="presentation"><span class="cm-atom">        name</span><span class="cm-meta">: </span><span class="cm-comment">#app#</span></span>
<span role="presentation"><span class="cm-atom">        ports</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-meta">        - </span><span class="cm-atom">containerPort</span><span class="cm-meta">: </span><span class="cm-comment">#port#</span></span>
<span role="presentation"><span class="cm-atom">          name</span><span class="cm-meta">: </span><span class="cm-comment">#app#</span></span>
<span role="presentation"><span class="cm-atom">        securityContext</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-atom">          privileged</span><span class="cm-meta">: </span><span class="cm-comment">#privileged#</span></span>
<span role="presentation"><span class="cm-atom">        volumeMounts</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-meta">        - </span><span class="cm-atom">name</span><span class="cm-meta">: </span>log-volume</span>
<span role="presentation"><span class="cm-atom">          mountPath</span><span class="cm-meta">: </span><span class="cm-comment">#log#</span></span>
<span role="presentation"><span class="cm-meta">      - </span><span class="cm-atom">image</span><span class="cm-meta">:  </span><span class="cm-comment">#filebeatImage#  </span></span>
<span role="presentation"><span class="cm-atom">        name</span><span class="cm-meta">: </span>filebeat</span>
<span role="presentation"><span class="cm-atom">        args</span><span class="cm-meta">: [</span></span>
<span role="presentation">          <span class="cm-string">"-c"</span><span class="cm-meta">,</span> <span class="cm-string">"/etc/filebeat.yml"</span></span>
<span role="presentation">        <span class="cm-meta">]</span></span>
<span role="presentation"><span class="cm-atom">        securityContext</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-atom">          runAsUser</span><span class="cm-meta">: </span><span class="cm-number">0</span></span>
<span role="presentation"><span class="cm-atom">        volumeMounts</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-meta">        - </span><span class="cm-atom">name</span><span class="cm-meta">: </span>config</span>
<span role="presentation"><span class="cm-atom">          mountPath</span><span class="cm-meta">: </span>/etc/filebeat.yml</span>
<span role="presentation"><span class="cm-atom">          readOnly</span><span class="cm-meta">: </span><span class="cm-keyword">true</span></span>
<span role="presentation"><span class="cm-atom">          subPath</span><span class="cm-meta">: </span>filebeat.yml</span>
<span role="presentation"><span class="cm-meta">        - </span><span class="cm-atom">name</span><span class="cm-meta">: </span>log-volume</span>
<span role="presentation"><span class="cm-atom">          mountPath</span><span class="cm-meta">: </span>/var/log/container/</span>
<span role="presentation"><span class="cm-atom">      volumes</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-meta">      - </span><span class="cm-atom">name</span><span class="cm-meta">: </span>config</span>
<span role="presentation"><span class="cm-atom">        configMap</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-atom">          defaultMode</span><span class="cm-meta">: </span><span class="cm-number">0600</span></span>
<span role="presentation"><span class="cm-atom">          name</span><span class="cm-meta">: </span>filebeat-config</span>
<span role="presentation"><span class="cm-meta">      - </span><span class="cm-atom">name</span><span class="cm-meta">: </span>log-volume</span>
<span role="presentation"><span class="cm-atom">        emptyDir</span><span class="cm-meta">: {}</span></span>
<span role="presentation"><span class="cm-atom">      imagePullSecrets</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-meta">      - </span><span class="cm-atom">name</span><span class="cm-meta">: </span>regcred</span>
<span role="presentation"><span class="cm-def">---</span></span>
<span role="presentation"><span class="cm-atom">apiVersion</span><span class="cm-meta">: </span>v1</span>
<span role="presentation"><span class="cm-atom">kind</span><span class="cm-meta">: </span>ConfigMap</span>
<span role="presentation"><span class="cm-atom">metadata</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-atom">  name</span><span class="cm-meta">: </span>filebeat-config</span>
<span role="presentation"><span class="cm-atom">  namespace</span><span class="cm-meta">: </span><span class="cm-comment">#namespace#</span></span>
<span role="presentation"><span class="cm-atom">  labels</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-atom">    app</span><span class="cm-meta">: </span>filebeat</span>
<span role="presentation"><span class="cm-atom">data</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-atom">  filebeat.yml</span><span class="cm-meta">: |</span>-</span>
<span role="presentation"><span class="cm-atom">    filebeat.inputs</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-meta">    - </span><span class="cm-atom">type</span><span class="cm-meta">: </span>log</span>
<span role="presentation"><span class="cm-atom">      enabled</span><span class="cm-meta">: </span><span class="cm-keyword">true</span></span>
<span role="presentation"><span class="cm-atom">      paths</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-meta">        - </span>/var/log/container/*.log</span>
<span role="presentation"><span class="cm-atom">    output.elasticsearch</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-atom">      hosts</span><span class="cm-meta">: [</span><span class="cm-string">"#es#"</span><span class="cm-meta">]</span></span>
<span role="presentation"><span class="cm-atom">    tags</span><span class="cm-meta">: [</span><span class="cm-string">"#namespace#-#app#"</span><span class="cm-meta">]</span></span>
<span role="presentation"><span class="cm-def">---</span></span>
<span role="presentation"><span class="cm-atom">apiVersion</span><span class="cm-meta">: </span>v1</span>
<span role="presentation"><span class="cm-atom">kind</span><span class="cm-meta">: </span>Service</span>
<span role="presentation"><span class="cm-atom">metadata</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-atom">  name</span><span class="cm-meta">: </span><span class="cm-comment">#app#-service</span></span>
<span role="presentation"><span class="cm-atom">  namespace</span><span class="cm-meta">: </span><span class="cm-comment">#namespace# </span></span>
<span role="presentation"><span class="cm-atom">  labels</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-atom">    app</span><span class="cm-meta">: </span><span class="cm-comment">#app#-service</span></span>
<span role="presentation"><span class="cm-atom">spec</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-atom">  ports</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-meta">    - </span><span class="cm-atom">port</span><span class="cm-meta">: </span><span class="cm-number">80</span></span>
<span role="presentation"><span class="cm-atom">      targetPort</span><span class="cm-meta">: </span><span class="cm-comment">#port#    </span></span>
<span role="presentation"><span class="cm-atom">  selector</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-atom">    app</span><span class="cm-meta">: </span><span class="cm-comment">#app#</span></span>
<span role="presentation"><span class="cm-def">---</span></span>
<span role="presentation"><span class="cm-atom">apiVersion</span><span class="cm-meta">: </span>extensions/v1beta1</span>
<span role="presentation"><span class="cm-atom">kind</span><span class="cm-meta">: </span>Ingress</span>
<span role="presentation"><span class="cm-atom">metadata</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-atom">  name</span><span class="cm-meta">: </span><span class="cm-comment">#app#-ingress</span></span>
<span role="presentation"><span class="cm-atom">  namespace</span><span class="cm-meta">: </span><span class="cm-comment">#namespace#  </span></span>
<span role="presentation"><span class="cm-atom">  annotations</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-atom">      nginx.ingress.kubernetes.io/proxy-body-size</span><span class="cm-meta">: </span><span class="cm-string">"0"</span></span>
<span role="presentation"><span class="cm-atom">      nginx.ingress.kubernetes.io/rewrite-target</span><span class="cm-meta">: </span>/</span>
<span role="presentation"><span class="cm-atom">spec</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-atom">  rules</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-meta">  - </span><span class="cm-atom">host</span><span class="cm-meta">: </span><span class="cm-comment">#host#  </span></span>
<span role="presentation"><span class="cm-atom">    http</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-atom">      paths</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-meta">      - </span><span class="cm-atom">path</span><span class="cm-meta">: </span><span class="cm-comment">#urlPath#</span></span>
<span role="presentation"><span class="cm-atom">        backend</span><span class="cm-meta">:</span></span>
<span role="presentation"><span class="cm-atom">          serviceName</span><span class="cm-meta">: </span><span class="cm-comment">#app#-service</span></span>
<span role="presentation"><span class="cm-atom">          servicePort</span><span class="cm-meta">: </span><span class="cm-number">80</span></span>

其中几个关键变量的解释如下：

dockerconfigjson：因为所有的镜像需要从Harbor获取，而Harbor的镜像如果设置为私有权限，就需要提供身份验证，这里的dockerconfigjson就是Harbor的身份信息。生成dockerconfigjson的方法如下：
- 进入K8S任何一个节点，删除” ~/.docker/config.json ” 文件
- 使用命令” docker login harbor地址”登录harbor
- 通过命令” cat ~/.docker/config.json “可以看到harbor的身份验证信息
- 使用命令” cat /root/.docker/config.json | base64 -w 0 “对信息编码，将生成后的编码填写到deployment.yaml的dockerconfigjson节点即可
namespace：同一个项目的不同k8s组件应置于同一个namespace，所以namespace可统一配置，在我们的项目实践中，生产环境的namespace为” 项目名 “，测试环境的namespace为” 项目名-test “
app：应用或服务名称
image：应用或服务的镜像地址
replicas：副本数量
port：应用或服务的Pod开放端口
log：应用或服务的日志路径，在本系列的第二篇文章中，提到我们的日志方案是给每个应用或服务配一个filebeat，放在同一Pod中，这里只需告知应用或服务的日志的绝对路径，filebeat就能将日志传递到ES中，日志的tag命名方式为” namespace-app”
host：在本系列的第一篇文章中，讲了使用nginx ingress做服务暴露与负载。这里的host就是给nginx ingress设置的域名，端口默认都是80，如果需要https，则在外层使用阿里云SLB转发
urlPath：很多情况下，如微服务，需要通过相同的域名，不同的一级目录将请求分发到不同的后台，在nginx中，就是location的配置与反向代理，比如host的配置是确定了域名aaa.bbb.com，而urlPath的配置是确定aaa.bbb.com/user/getuser将会被转发到用户服务podIP:podPort/getuser中

以上所有的占位变量都是在Pipeline Script中赋值，关于Jenkins Pipeline的相关内容介绍这里不再多讲，还是去看官方文档靠谱。我们这里将k8s的部署文件deployment.yaml与Jenkinsfile结合，即可做到一个deployment.yaml能适配所有项目，一个Pipeline Script模板能适配所有项目，针对不同的项目，只需在Pipeline Script中给占位变量赋值，大大降低了配置复杂度。下面是一个项目的Jenkins配置示例：

对于一个项目，我们只需配置Trigger和Pipeline，上图“Do not allow concurrent builds ”也是通过Pipeline的配置生成的。Pipeline Script示例如下：

<span role="presentation"><span class="cm-variable">pipeline</span> {</span>
<span role="presentation">    <span class="cm-comment">// 指定项目在label为jnlp-agent的节点上构建，也就是Jenkins Slave in Pod</span></span>
<span role="presentation">    <span class="cm-variable">agent</span> { <span class="cm-variable">label</span> <span class="cm-string">"jnlp-agent"</span> } </span>
<span role="presentation">    <span class="cm-comment">// 对应Do not allow concurrent builds </span></span>
<span role="presentation">    <span class="cm-variable">options</span> {</span>
<span role="presentation">        <span class="cm-variable">disableConcurrentBuilds</span>()</span>
<span role="presentation">    }</span>
<span role="presentation">    <span class="cm-variable">environment</span> { </span>
<span role="presentation">        <span class="cm-comment">// ------ 以下内容，每个项目可能均有不同，按需修改 ------ </span></span>
<span role="presentation">        <span class="cm-comment">//author：用于钉钉通知</span></span>
<span role="presentation">        <span class="cm-variable">author</span><span class="cm-operator">=</span><span class="cm-string">"张三"</span></span>
<span role="presentation">        <span class="cm-comment">// branch: 分支，一般是test、 master，对应git从哪个分支拉取代码，也对应究竟执行_deploy文件夹下的test配置还是master配置</span></span>
<span role="presentation">        <span class="cm-variable">branch</span> <span class="cm-operator">=</span> <span class="cm-string">"test"</span></span>
<span role="presentation">        <span class="cm-comment">// namespace: myproject-test, myproject，命名空间一般是项目名称，测试环境加test</span></span>
<span role="presentation">        <span class="cm-variable">namespace</span> <span class="cm-operator">=</span> <span class="cm-string">"myproject-test"</span></span>
<span role="presentation">        <span class="cm-comment">// hostname：对应deployment中的host</span></span>
<span role="presentation">        <span class="cm-variable">host</span> <span class="cm-operator">=</span> <span class="cm-string">"test.aaa.bbb.com"</span></span>
<span role="presentation">        <span class="cm-comment">// appname：对应deployment中的app</span></span>
<span role="presentation">        <span class="cm-variable">app</span> <span class="cm-operator">=</span> <span class="cm-string">"myserver"</span></span>
<span role="presentation">        <span class="cm-comment">// port：对应deployment中的port</span></span>
<span role="presentation">        <span class="cm-variable">port</span><span class="cm-operator">=</span> <span class="cm-string">"80"</span></span>
<span role="presentation">        <span class="cm-comment">// replicas：对应deployment中的replicas</span></span>
<span role="presentation">        <span class="cm-variable">replicas</span> <span class="cm-operator">=</span> <span class="cm-number">2</span></span>
<span role="presentation">        <span class="cm-comment">//git repo path：git的地址</span></span>
<span role="presentation">        <span class="cm-variable">git</span><span class="cm-operator">=</span><span class="cm-string">"[email protected]/xxx.git"</span></span>
<span role="presentation">         <span class="cm-comment">//log：对应deployment中的log</span></span>
<span role="presentation">        <span class="cm-variable">log</span><span class="cm-operator">=</span><span class="cm-string">"/publish/logs/"</span></span>
<span role="presentation">        <span class="cm-comment">// ------ 以下内容，一般所有的项目都一样，不经常修改 ------</span></span>
<span role="presentation">        <span class="cm-comment">// harbor inner address</span></span>
<span role="presentation">        <span class="cm-variable">repoHost</span> <span class="cm-operator">=</span> <span class="cm-string">"192.168.0.1:23280"</span></span>
<span role="presentation">        <span class="cm-comment">// harbor的账号密码信息，在jenkins中配置用户名/密码形式的认证信息，命名成harbor即可</span></span>
<span role="presentation">        <span class="cm-variable">harborCreds</span> <span class="cm-operator">=</span> <span class="cm-variable">credentials</span>(<span class="cm-string">"harbor"</span>)</span>
<span role="presentation">        <span class="cm-comment">// filebeat的镜像地址</span></span>
<span role="presentation">        <span class="cm-variable">filebeatImage</span><span class="cm-operator">=</span><span class="cm-string">"${repoHost}/common/filebeat:6.3.1"</span></span>
<span role="presentation">        <span class="cm-comment">// es的内网访问地址</span></span>
<span role="presentation">        <span class="cm-variable">es</span><span class="cm-operator">=</span><span class="cm-string">"elasticsearch-logging.kube-system:9200"</span></span>
<span role="presentation">    }</span>
<span role="presentation">    <span class="cm-comment">// ------ 以下内容无需修改 ------</span></span>
<span role="presentation">    <span class="cm-variable">stages</span> {</span>
<span role="presentation">         <span class="cm-comment">// 开始构建前清空工作目录</span></span>
<span role="presentation">         <span class="cm-variable">stage</span> (<span class="cm-string">"CleanWS"</span>){ </span>
<span role="presentation">            <span class="cm-variable">steps</span> {</span>
<span role="presentation">                <span class="cm-variable">script</span> {</span>
<span role="presentation">                    <span class="cm-keyword">try</span>{</span>
<span role="presentation">                       <span class="cm-variable">deleteDir</span>()</span>
<span role="presentation">                    }<span class="cm-keyword">catch</span>(<span class="cm-variable">err</span>){</span>
<span role="presentation">                        <span class="cm-variable">echo</span> <span class="cm-string">"${err}"</span></span>
<span role="presentation">                        <span class="cm-variable">sh</span> <span class="cm-string">"exit 1"</span></span>
<span role="presentation">                    }</span>
<span role="presentation">                }  </span>
<span role="presentation">            }  </span>
<span role="presentation">        }</span>
<span role="presentation">        <span class="cm-comment">// 拉取</span></span>
<span role="presentation">        <span class="cm-variable">stage</span> (<span class="cm-string">"CheckOut"</span>){ </span>
<span role="presentation">            <span class="cm-variable">steps</span> {</span>
<span role="presentation">                <span class="cm-variable">script</span> {</span>
<span role="presentation">                    <span class="cm-keyword">try</span>{</span>
<span role="presentation">                      <span class="cm-variable">checkout</span>([<span class="cm-variable">$class</span>: <span class="cm-string">"GitSCM"</span>, <span class="cm-variable">branches</span>: [[<span class="cm-variable">name</span>: <span class="cm-string">"*/${branch}"</span>]], <span class="cm-variable">doGenerateSubmoduleConfigurations</span>: <span class="cm-atom">false</span>, <span class="cm-variable">extensions</span>: [], <span class="cm-variable">submoduleCfg</span>: [], <span class="cm-variable">userRemoteConfigs</span>: [[<span class="cm-variable">credentialsId</span>: <span class="cm-string">"gitlab"</span>, <span class="cm-variable">url</span>: <span class="cm-string">"${git}"</span>]]])</span>
<span role="presentation">                    }<span class="cm-keyword">catch</span>(<span class="cm-variable">err</span>){</span>
<span role="presentation">                        <span class="cm-variable">echo</span> <span class="cm-string">"${err}"</span></span>
<span role="presentation">                        <span class="cm-variable">sh</span> <span class="cm-string">"exit 1"</span></span>
<span role="presentation">                    }</span>
<span role="presentation">                }  </span>
<span role="presentation">            }  </span>
<span role="presentation">        }</span>
<span role="presentation">       <span class="cm-comment">// 构建</span></span>
<span role="presentation">        <span class="cm-variable">stage</span> (<span class="cm-string">"Build"</span>){ </span>
<span role="presentation">            <span class="cm-variable">steps</span> {</span>
<span role="presentation">                <span class="cm-variable">script</span> {</span>
<span role="presentation">                    <span class="cm-keyword">try</span>{</span>
<span role="presentation">                        <span class="cm-comment">// 登录 harbor </span></span>
<span role="presentation">                        <span class="cm-variable">sh</span> <span class="cm-string">"docker login -u ${harborCreds_USR} -p ${harborCreds_PSW} ${repoHost}"</span></span>
<span role="presentation">                        <span class="cm-variable">sh</span> <span class="cm-string">"date +%Y%m%d%H%m%S > timestamp"</span></span>
<span role="presentation">                        <span class="cm-comment">// 镜像tag用时间戳代表</span></span>
<span role="presentation">                        <span class="cm-variable">tag</span> <span class="cm-operator">=</span> <span class="cm-variable">readFile</span>(<span class="cm-string">"timestamp"</span>).<span class="cm-variable">replace</span>(<span class="cm-string">"
"</span>, <span class="cm-string">""</span>).<span class="cm-variable">replace</span>(<span class="cm-string">""</span>, <span class="cm-string">""</span>)</span>
<span role="presentation">                        <span class="cm-variable">repoPath</span> <span class="cm-operator">=</span> <span class="cm-string">"${repoHost}/${namespace}/${app}:${tag}"</span></span>
<span role="presentation">                        <span class="cm-comment">// 根据分支，进入_deploy下对应的不同文件夹，通过dockerfile打包镜像</span></span>
<span role="presentation">                        <span class="cm-variable">sh</span> <span class="cm-string">"cp _deploy/${branch}/* ./"</span></span>
<span role="presentation">                        <span class="cm-variable">sh</span> <span class="cm-string">"docker login -u ${harborCreds_USR} -p ${harborCreds_PSW} ${repoHost}"</span></span>
<span role="presentation">                        <span class="cm-variable">sh</span> <span class="cm-string">"docker build -t ${repoPath}  ."</span></span>
<span role="presentation">                    }<span class="cm-keyword">catch</span>(<span class="cm-variable">err</span>){</span>
<span role="presentation">                        <span class="cm-variable">echo</span> <span class="cm-string">"${err}"</span></span>
<span role="presentation">                        <span class="cm-variable">sh</span> <span class="cm-string">"exit 1"</span></span>
<span role="presentation">                    }</span>
<span role="presentation">                }  </span>
<span role="presentation">            }  </span>
<span role="presentation">        }</span>
<span role="presentation">        <span class="cm-comment">// 镜像推送到harbor</span></span>
<span role="presentation">        <span class="cm-variable">stage</span> (<span class="cm-string">"Push"</span>){</span>
<span role="presentation">            <span class="cm-variable">steps</span> {</span>
<span role="presentation">                <span class="cm-variable">script</span> {</span>
<span role="presentation">                    <span class="cm-keyword">try</span>{</span>
<span role="presentation">                        <span class="cm-variable">sh</span> <span class="cm-string">"docker push ${repoPath}"</span></span>
<span role="presentation">                    }<span class="cm-keyword">catch</span>(<span class="cm-variable">err</span>){</span>
<span role="presentation">                        <span class="cm-variable">echo</span> <span class="cm-string">"${err}"</span></span>
<span role="presentation">                        <span class="cm-variable">sh</span> <span class="cm-string">"exit 1"</span></span>
<span role="presentation">                    }</span>
<span role="presentation">                }   </span>
<span role="presentation">            }</span>
<span role="presentation">        }</span>
<span role="presentation">        <span class="cm-comment">// 使用pipeline script中复制的变量替换deployment.yaml中的占位变量，执行deployment.yaml进行部署</span></span>
<span role="presentation">        <span class="cm-variable">stage</span> (<span class="cm-string">"Deploy"</span>){</span>
<span role="presentation">            <span class="cm-variable">steps</span> {</span>
<span role="presentation">                <span class="cm-variable">script</span> {</span>
<span role="presentation">                    <span class="cm-keyword">try</span>{</span>
<span role="presentation">                        <span class="cm-variable">sh</span> <span class="cm-string">"sed -i "s|#namespace#|${namespace}|g" deployment.yaml"</span></span>
<span role="presentation">                        <span class="cm-variable">sh</span> <span class="cm-string">"sed -i "s|#app#|${app}|g" deployment.yaml"</span></span>
<span role="presentation">                        <span class="cm-variable">sh</span> <span class="cm-string">"sed -i "s|#image#|${repoPath}|g" deployment.yaml"</span></span>
<span role="presentation">                        <span class="cm-variable">sh</span> <span class="cm-string">"sed -i "s|#port#|${port}|g" deployment.yaml"</span></span>
<span role="presentation">                        <span class="cm-variable">sh</span> <span class="cm-string">"sed -i "s|#host#|${host}|g" deployment.yaml"</span></span>
<span role="presentation">                        <span class="cm-variable">sh</span> <span class="cm-string">"sed -i "s|#replicas#|${replicas}|g" deployment.yaml"</span></span>
<span role="presentation">                        <span class="cm-variable">sh</span> <span class="cm-string">"sed -i "s|#log#|${log}|g" deployment.yaml"</span></span>
<span role="presentation">                        <span class="cm-variable">sh</span> <span class="cm-string">"sed -i "s|#filebeatImage#|${filebeatImage}|g" deployment.yaml"</span></span>
<span role="presentation">                        <span class="cm-variable">sh</span> <span class="cm-string">"sed -i "s|#es#|${es}|g" deployment.yaml"</span></span>
<span role="presentation">                        <span class="cm-variable">sh</span> <span class="cm-string">"sed -i "s|#redisImage#|${redisImage}|g" deployment.yaml"</span></span>
<span role="presentation">                        <span class="cm-variable">sh</span> <span class="cm-string">"cat deployment.yaml"</span></span>
<span class="cm-tab-wrap-hack" role="presentation">                        <span class="cm-variable">sh</span> <span class="cm-string">"kubectl apply -f deployment.yaml"</span>   </span>
<span role="presentation">                    }<span class="cm-keyword">catch</span>(<span class="cm-variable">err</span>){</span>
<span role="presentation">                        <span class="cm-variable">echo</span> <span class="cm-string">"${err}"</span></span>
<span role="presentation">                        <span class="cm-variable">sh</span> <span class="cm-string">"exit 1"</span></span>
<span role="presentation">                    }</span>
<span role="presentation">                }</span>
<span role="presentation">            }</span>
<span role="presentation">        }</span>
<span role="presentation">    }</span>
<span role="presentation">    <span class="cm-variable">post</span> {</span>
<span role="presentation">        <span class="cm-comment">// 使用钉钉插件进行通知</span></span>
<span role="presentation">        <span class="cm-variable">always</span> {</span>
<span role="presentation">            <span class="cm-variable">script</span> {   </span>
<span role="presentation">                <span class="cm-variable">def</span> <span class="cm-variable">msg</span> <span class="cm-operator">=</span> <span class="cm-string">"【${author}】你把服务器搞挂了，老詹喊你回家改BUG！"</span></span>
<span role="presentation">                <span class="cm-variable">def</span> <span class="cm-variable">imageUrl</span> <span class="cm-operator">=</span> <span class="cm-string">"https://www.iconsdb.com/icons/preview/red/x-mark-3-xxl-2.png"</span></span>
<span role="presentation">                <span class="cm-keyword">if</span> (<span class="cm-variable">currentBuild</span>.<span class="cm-variable">currentResult</span><span class="cm-operator">==</span><span class="cm-string">"SUCCESS"</span>){</span>
<span role="presentation">                    <span class="cm-variable">imageUrl</span><span class="cm-operator">=</span> <span class="cm-string">"http://icons.iconarchive.com/icons/paomedia/small-n-flat/1024/sign-check-icon-2.png"</span></span>
<span role="presentation">                    <span class="cm-variable">msg</span> <span class="cm-operator">=</span><span class="cm-string">"【${author}】发布成功，干得不错！"</span></span>
<span role="presentation">                }</span>
<span role="presentation">                <span class="cm-variable">dingTalk</span> <span class="cm-variable">accessToken</span>:<span class="cm-string">"xxxx"</span>,<span class="cm-variable">message</span>:<span class="cm-string">"${msg}"</span>,<span class="cm-variable">imageUrl</span>:<span class="cm-string">"${imageUrl}"</span>,<span class="cm-variable">messageUrl</span>:<span class="cm-string">"${BUILD_URL}"</span></span>
<span role="presentation">            }</span>
<span role="presentation">        }</span>
<span role="presentation">    }</span>
<span role="presentation">}</span>

发布完成后，可以参考《持续集成CI实施指南三–jenkins集成测试》，做持续测试，测试结果也可通过钉钉通知。最后我们利用自建的运维平台，监控阿里云ECS状态、K8S各组件状态、监控ES中的日志并做异常抓取和报警。形成一整套DevOps模式。

综上，对于每个项目，我们只需维护Dockerfile，并在Jenkins创建持续集成项目时，填写项目所需的参数变量。进阶情况下，也可定制性的修改deployment文件与pipeline script，满足不同的业务需要。至此，完结，撒花！

来源：http://wurang.net/alicloud_kubernetes_03/

搞代码网（gaodaima.com）提供的所有资源部分来自互联网，如果有侵犯您的版权或其他权益，请说明详细缘由并提供版权或权益证明然后发送到邮箱[email protected]‍，我们会在看到邮件的第一时间内为您处理，或直接联系QQ：872152909。本网站采用BY-NC-SA协议进行授权
转载请注明原文链接：阿里云Kubernetes实战3–DevOps

一、一机多Jenkins Slave

二、 二次开发Jenkins 钉钉通知插件

三、 DevOps解决方案

Hi，您需要填写昵称和邮箱！

二、二次开发Jenkins 钉钉通知插件