• 欢迎访问搞代码网站，推荐使用最新版火狐浏览器和Chrome浏览器访问本网站！
  • 如果您觉得本站非常有看点，那么赶紧使用Ctrl+D 收藏搞代码吧
> k8s > Kubernetes安装之证书验证

Kubernetes安装之证书验证

k8s 搞java代码 3年前 (2022-05-13) 19次浏览 已收录 0个评论

前言

昨晚（Apr 9,2017）金山软件的opsnull发布了一个开源项目和我一步步部署kubernetes集群，下文是结合我之前部署kubernetes的过程打造的kubernetes环境和opsnull的文章创建 kubernetes 各组件 TLS 加密通信的证书和秘钥的实践。之前安装过程中一直使用的是非加密方式，一直到后来使用Fluentd和ElasticSearch收集Kubernetes集群日志时发现有权限验证问题，所以为了深入研究kubernentes。

Kubernentes中的身份验证

kubernetes 系统的各组件需要使用 TLS 证书对通信进行加密，本文档使用 CloudFlare 的 PKI 工具集 cfssl 来生成 Certificate Authority (CA) 和其它证书；

生成的 CA 证书和秘钥文件如下：

  • ca-key.pem
  • ca.pem
  • kubernetes-key.pem
  • kubernetes.pem
  • kube-proxy.pem
  • kube-proxy-key.pem
  • admin.pem
  • admin-key.pem

使用证书的组件如下：

  • etcd：使用 ca.pem、kubernetes-key.pem、kubernetes.pem；
  • kube-apiserver：使用 ca.pem、kubernetes-key.pem、kubernetes.pem；
  • kubelet：使用 ca.pem；
  • kube-proxy：使用 ca.pem、kube-proxy-key.pem、kube-proxy.pem；
  • kubectl：使用 ca.pem、admin-key.pem、admin.pem；

kube-controller、kube-scheduler 当前需要和 kube-apiserver 部署在同一台机器上且使用非安全端口通信，故不需要证书。

安装 CFSSL

方式一：直接使用二进制源码包安装

<code class="language-bash hljs ">$ wget https://pkg.cfssl.org/R1.<span class="hljs-number">2</span>/cfssl_linux-amd64
$ chmod +x cfssl_linux-amd64
$ <span class="hljs-built_in">sudo</span> mv cfssl_linux-amd64 /root/local/bin/cfssl

$ wget https://pkg.cfssl.org/R1.<span class="hljs-number">2</span>/cfssljson_linux-amd64
$ chmod +x cfssljson_linux-amd64
$ <span class="hljs-built_in">sudo</span> mv cfssljson_linux-amd64 /root/local/bin/cfssljson

$ wget https://pkg.cfssl.org/R1.<span class="hljs-number">2</span>/cfssl-certinfo_linux-amd64
$ chmod +x cfssl-certinfo_linux-amd64
$ <span class="hljs-built_in">sudo</span> mv cfssl-certinfo_linux-amd64 /root/local/bin/cfssl-certinfo

$ <span class="hljs-keyword">export</span> PATH=/root/local/bin:<span class="hljs-variable">$PATH</span>
</code>

www#gaodaima.com来源gaodai#ma#com搞*!代#%^码网搞代码

方式二：使用go命令安装

我们的系统中安装了Go1.7.5，使用以下命令安装更快捷：

<code class=" hljs perl"><span class="hljs-variable">$go</span> get -u github.com/cloudflare/cfssl/cmd/...
<span class="hljs-variable">$echo</span> <span class="hljs-variable">$GOPATH</span>
/usr/<span class="hljs-keyword">local</span>
<span class="hljs-variable">$ls</span> /usr/<span class="hljs-keyword">local</span>/bin/cfssl*
cfssl cfssl-bundle cfssl-certinfo cfssljson cfssl-newkey cfssl-scan
</code>

在$GOPATH/bin目录下得到以cfssl开头的几个命令。

创建 CA (Certificate Authority)

创建 CA 配置文件

<code class="language-bash hljs ">$ mkdir /root/ssl
$ <span class="hljs-built_in">cd</span> /root/ssl
$ cfssl print-defaults config > config.json
$ cfssl print-defaults csr > csr.json
$ cat ca-config.json
{
  <span class="hljs-string">"signing"</span>: {
    <span class="hljs-string">"default"</span>: {
      <span class="hljs-string">"expiry"</span>: <span class="hljs-string">"8760h"</span>
    },
    <span class="hljs-string">"profiles"</span>: {
      <span class="hljs-string">"kubernetes"</span>: {
        <span class="hljs-string">"usages"</span>: [
            <span class="hljs-string">"signing"</span>,
            <span class="hljs-string">"key encipherment"</span>,
            <span class="hljs-string">"server auth"</span>,
            <span class="hljs-string">"client auth"</span>
        ],
        <span class="hljs-string">"expiry"</span>: <span class="hljs-string">"8760h"</span>
      }
    }
  }
}
</code>

字段说明

  • ca-config.json：可以定义多个 profiles，分别指定不同的过期时间、使用场景等参数；后续在签名证书时使用某个 profile；
  • signing：表示该证书可用于签名其它证书；生成的 ca.pem 证书中 CA=TRUE；
  • server auth：表示client可以用该 CA 对server提供的证书进行验证；
  • client auth：表示server可以用该CA对client提供的证书进行验证；

创建 CA 证书签名请求

<code class="language-bash hljs ">$ cat ca-csr.json
{
  <span class="hljs-string">"CN"</span>: <span class="hljs-string">"kubernetes"</span>,
  <span class="hljs-string">"key"</span>: {
    <span class="hljs-string">"algo"</span>: <span class="hljs-string">"rsa"</span>,
    <span class="hljs-string">"size"</span>: <span class="hljs-number">2048</span>
  },
  <span class="hljs-string">"names"</span>: [
    {
      <span class="hljs-string">"C"</span>: <span class="hljs-string">"CN"</span>,
      <span class="hljs-string">"ST"</span>: <span class="hljs-string">"BeiJing"</span>,
      <span class="hljs-string">"L"</span>: <span class="hljs-string">"BeiJing"</span>,
      <span class="hljs-string">"O"</span>: <span class="hljs-string">"k8s"</span>,
      <span class="hljs-string">"OU"</span>: <span class="hljs-string">"System"</span>
    }
  ]
}
</code>
  • “CN”：Common Name，kube-apiserver 从证书中提取该字段作为请求的用户名 (User Name)；浏览器使用该字段验证网站是否合法；
  • “O”：Organization，kube-apiserver 从证书中提取该字段作为请求用户所属的组 (Group)；

生成 CA 证书和私钥

<code class="language-bash hljs ">$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca
$ ls ca*
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem
</code>

创建 Kubernetes 证书

创建 kubernetes 证书签名请求

<code class="language-bash hljs ">$ cat kubernetes-csr.json
{
    <span class="hljs-string">"CN"</span>: <span class="hljs-string">"kubernetes"</span>,
    <span class="hljs-string">"hosts"</span>: [
      <span class="hljs-string">"127.0.0.1"</span>,
      <span class="hljs-string">"172.20.0.112"</span>,
      <span class="hljs-string">"172.20.0.113"</span>,
      <span class="hljs-string">"172.20.0.114"</span>,
      <span class="hljs-string">"172.20.0.115"</span>,
      <span class="hljs-string">"10.254.0.1"</span>,
      <span class="hljs-string">"kubernetes"</span>,
      <span class="hljs-string">"kubernetes.default"</span>,
      <span class="hljs-string">"kubernetes.default.svc"</span>,
      <span class="hljs-string">"kubernetes.default.svc.cluster"</span>,
      <span class="hljs-string">"kubernetes.default.svc.cluster.local"</span>
    ],
    <span class="hljs-string">"key"</span>: {
        <span class="hljs-string">"algo"</span>: <span class="hljs-string">"rsa"</span>,
        <span class="hljs-string">"size"</span>: <span class="hljs-number">2048</span>
    },
    <span class="hljs-string">"names"</span>: [
        {
            <span class="hljs-string">"C"</span>: <span class="hljs-string">"CN"</span>,
            <span class="hljs-string">"ST"</span>: <span class="hljs-string">"BeiJing"</span>,
            <span class="hljs-string">"L"</span>: <span class="hljs-string">"BeiJing"</span>,
            <span class="hljs-string">"O"</span>: <span class="hljs-string">"k8s"</span>,
            <span class="hljs-string">"OU"</span>: <span class="hljs-string">"System"</span>
        }
    ]
}
</code>
  • 如果 hosts 字段不为空则需要指定授权使用该证书的 IP 或域名列表，由于该证书后续被 etcd 集群和 kubernetes master 集群使用，所以上面分别指定了 etcd 集群、kubernetes master 集群的主机 IP 和 kubernetes 服务的服务 IP（一般是 kue-apiserver 指定的 service-cluster-ip-range 网段的第一个IP，如 10.254.0.1。

生成 kubernetes 证书和私钥

<code class="language-bash hljs ">$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
$ ls kuberntes*
kubernetes.csr  kubernetes-csr.json  kubernetes-key.pem  kubernetes.pem
</code>

或者直接在命令行上指定相关参数：

<code class="language-bash hljs ">$ <span class="hljs-built_in">echo</span> <span class="hljs-string">"{"CN":"kubernetes","hosts":[""],"key":{"algo":"rsa","size":2048}}"</span> | cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes -hostname=<span class="hljs-string">"127.0.0.1,10.64.3.7,10.254.0.1,kubernetes,kubernetes.default"</span> - | cfssljson -bare kubernetes
</code>

创建 Admin 证书

创建 admin 证书签名请求

<code class="language-bash hljs ">$ cat admin-csr.json
{
  <span class="hljs-string">"CN"</span>: <span class="hljs-string">"admin"</span>,
  <span class="hljs-string">"hosts"</span>: [],
  <span class="hljs-string">"key"</span>: {
    <span class="hljs-string">"algo"</span>: <span class="hljs-string">"rsa"</span>,
    <span class="hljs-string">"size"</span>: <span class="hljs-number">2048</span>
  },
  <span class="hljs-string">"names"</span>: [
    {
      <span class="hljs-string">"C"</span>: <span class="hljs-string">"CN"</span>,
      <span class="hljs-string">"ST"</span>: <span class="hljs-string">"BeiJing"</span>,
      <span class="hljs-string">"L"</span>: <span class="hljs-string">"BeiJing"</span>,
      <span class="hljs-string">"O"</span>: <span class="hljs-string">"system:masters"</span>,
      <span class="hljs-string">"OU"</span>: <span class="hljs-string">"System"</span>
    }
  ]
}
</code>
  • 后续 kube-apiserver 使用 RBAC 对客户端(如 kubelet、kube-proxy、Pod)请求进行授权；
  • kube-apiserver 预定义了一些 RBAC 使用的 RoleBindings，如 cluster-admin 将 Groupsystem:masters 与 Role cluster-admin 绑定，该 Role 授予了调用kube-apiserver 的所有 API的权限；
  • OU 指定该证书的 Group 为 system:masters，kubelet 使用该证书访问 kube-apiserver时 ，由于证书被 CA 签名，所以认证通过，同时由于证书用户组为经过预授权的system:masters，所以被授予访问所有 API 的权限；

生成 admin 证书和私钥

<code class="language-bash hljs ">$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
$ ls admin*
admin.csr  admin-csr.json  admin-key.pem  admin.pem
</code>

创建 Kube-Proxy 证书

创建 kube-proxy 证书签名请求

<code class="language-bash hljs ">$ cat kube-proxy-csr.json
{
  <span class="hljs-string">"CN"</span>: <span class="hljs-string">"system:kube-proxy"</span>,
  <span class="hljs-string">"hosts"</span>: [],
  <span class="hljs-string">"key"</span>: {
    <span class="hljs-string">"algo"</span>: <span class="hljs-string">"rsa"</span>,
    <span class="hljs-string">"size"</span>: <span class="hljs-number">2048</span>
  },
  <span class="hljs-string">"names"</span>: [
    {
      <span class="hljs-string">"C"</span>: <span class="hljs-string">"CN"</span>,
      <span class="hljs-string">"ST"</span>: <span class="hljs-string">"BeiJing"</span>,
      <span class="hljs-string">"L"</span>: <span class="hljs-string">"BeiJing"</span>,
      <span class="hljs-string">"O"</span>: <span class="hljs-string">"k8s"</span>,
      <span class="hljs-string">"OU"</span>: <span class="hljs-string">"System"</span>
    }
  ]
}
</code>
  • CN 指定该证书的 User 为 system:kube-proxy；
  • kube-apiserver 预定义的 RoleBinding cluster-admin 将User system:kube-proxy 与 Rolesystem:node-proxier 绑定，该 Role 授予了调用 kube-apiserver Proxy 相关 API 的权限；

生成 kube-proxy 客户端证书和私钥

<code class="language-bash hljs ">$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes  kube-proxy-csr.json | cfssljson -bare kube-proxy
$ ls kube-proxy*
kube-proxy.csr  kube-proxy-csr.json  kube-proxy-key.pem  kube-proxy.pem
</code>

校验证书

以 kubernetes 证书为例

使用 Opsnssl 命令

<code class="language-bash hljs ">$ openssl x509  -noout -text -in  kubernetes.pem
...
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=CN, ST=BeiJing, L=BeiJing, O=k8s, OU=System, CN=Kubernetes
        Validity
            Not Before: Apr  <span class="hljs-number">5</span> <span class="hljs-number">05</span>:<span class="hljs-number">36</span>:<span class="hljs-number">00</span> <span class="hljs-number">2017</span> GMT
            Not After : Apr  <span class="hljs-number">5</span> <span class="hljs-number">05</span>:<span class="hljs-number">36</span>:<span class="hljs-number">00</span> <span class="hljs-number">2018</span> GMT
        Subject: C=CN, ST=BeiJing, L=BeiJing, O=k8s, OU=System, CN=kubernetes
...
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                DD:<span class="hljs-number">52</span>:<span class="hljs-number">04</span>:<span class="hljs-number">43</span>:<span class="hljs-number">10</span>:<span class="hljs-number">13</span>:A9:<span class="hljs-number">29</span>:<span class="hljs-number">24</span>:<span class="hljs-number">17</span>:<span class="hljs-number">3</span>A:<span class="hljs-number">0</span>E:D7:<span class="hljs-number">14</span>:DB:<span class="hljs-number">36</span>:F8:<span class="hljs-number">6</span>C:E0:E0
            X509v3 Authority Key Identifier:
                keyid:<span class="hljs-number">44</span>:<span class="hljs-number">04</span>:<span class="hljs-number">3</span>B:<span class="hljs-number">60</span>:BD:<span class="hljs-number">69</span>:<span class="hljs-number">78</span>:<span class="hljs-number">14</span>:<span class="hljs-number">68</span>:AF:A0:<span class="hljs-number">41</span>:<span class="hljs-number">13</span>:F6:<span class="hljs-number">17</span>:<span class="hljs-number">07</span>:<span class="hljs-number">13</span>:<span class="hljs-number">63</span>:<span class="hljs-number">58</span>:CD

            X509v3 Subject Alternative Name:
                DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster, DNS:kubernetes.default.svc.cluster.local, IP Address:<span class="hljs-number">127.0</span>.<span class="hljs-number">0.1</span>, IP Address:<span class="hljs-number">172.20</span>.<span class="hljs-number">0.112</span>, IP Address:<span class="hljs-number">172.20</span>.<span class="hljs-number">0.113</span>, IP Address:<span class="hljs-number">172.20</span>.<span class="hljs-number">0.114</span>, IP Address:<span class="hljs-number">172.20</span>.<span class="hljs-number">0.115</span>, IP Address:<span class="hljs-number">10.254</span>.<span class="hljs-number">0.1</span>
...
</code>
  • 确认 Issuer 字段的内容和 ca-csr.json 一致；
  • 确认 Subject 字段的内容和 kubernetes-csr.json 一致；
  • 确认 X509v3 Subject Alternative Name 字段的内容和 kubernetes-csr.json 一致；
  • 确认 X509v3 Key Usage、Extended Key Usage 字段的内容和 ca-config.json 中 kubernetesprofile 一致；

使用 Cfssl-Certinfo 命令

<code class="language-bash hljs ">$ cfssl-certinfo -cert kubernetes.pem
...
{
  <span class="hljs-string">"subject"</span>: {
    <span class="hljs-string">"common_name"</span>: <span class="hljs-string">"kubernetes"</span>,
    <span class="hljs-string">"country"</span>: <span class="hljs-string">"CN"</span>,
    <span class="hljs-string">"organization"</span>: <span class="hljs-string">"k8s"</span>,
    <span class="hljs-string">"organizational_unit"</span>: <span class="hljs-string">"System"</span>,
    <span class="hljs-string">"locality"</span>: <span class="hljs-string">"BeiJing"</span>,
    <span class="hljs-string">"province"</span>: <span class="hljs-string">"BeiJing"</span>,
    <span class="hljs-string">"names"</span>: [
      <span class="hljs-string">"CN"</span>,
      <span class="hljs-string">"BeiJing"</span>,
      <span class="hljs-string">"BeiJing"</span>,
      <span class="hljs-string">"k8s"</span>,
      <span class="hljs-string">"System"</span>,
      <span class="hljs-string">"kubernetes"</span>
    ]
  },
  <span class="hljs-string">"issuer"</span>: {
    <span class="hljs-string">"common_name"</span>: <span class="hljs-string">"Kubernetes"</span>,
    <span class="hljs-string">"country"</span>: <span class="hljs-string">"CN"</span>,
    <span class="hljs-string">"organization"</span>: <span class="hljs-string">"k8s"</span>,
    <span class="hljs-string">"organizational_unit"</span>: <span class="hljs-string">"System"</span>,
    <span class="hljs-string">"locality"</span>: <span class="hljs-string">"BeiJing"</span>,
    <span class="hljs-string">"province"</span>: <span class="hljs-string">"BeiJing"</span>,
    <span class="hljs-string">"names"</span>: [
      <span class="hljs-string">"CN"</span>,
      <span class="hljs-string">"BeiJing"</span>,
      <span class="hljs-string">"BeiJing"</span>,
      <span class="hljs-string">"k8s"</span>,
      <span class="hljs-string">"System"</span>,
      <span class="hljs-string">"Kubernetes"</span>
    ]
  },
  <span class="hljs-string">"serial_number"</span>: <span class="hljs-string">"174360492872423263473151971632292895707129022309"</span>,
  <span class="hljs-string">"sans"</span>: [
    <span class="hljs-string">"kubernetes"</span>,
    <span class="hljs-string">"kubernetes.default"</span>,
    <span class="hljs-string">"kubernetes.default.svc"</span>,
    <span class="hljs-string">"kubernetes.default.svc.cluster"</span>,
    <span class="hljs-string">"kubernetes.default.svc.cluster.local"</span>,
    <span class="hljs-string">"127.0.0.1"</span>,
    <span class="hljs-string">"10.64.3.7"</span>,
    <span class="hljs-string">"10.254.0.1"</span>
  ],
  <span class="hljs-string">"not_before"</span>: <span class="hljs-string">"2017-04-05T05:36:00Z"</span>,
  <span class="hljs-string">"not_after"</span>: <span class="hljs-string">"2018-04-05T05:36:00Z"</span>,
  <span class="hljs-string">"sigalg"</span>: <span class="hljs-string">"SHA256WithRSA"</span>,
...
</code>

分发证书

将生成的证书和秘钥文件（后缀名为.pem）拷贝到所有机器的 /etc/kubernetes/ssl 目录下备用；

<code class="language-bash hljs ">$ <span class="hljs-built_in">sudo</span> mkdir -p /etc/kubernetes/ssl
$ <span class="hljs-built_in">sudo</span> cp *.pem /etc/kubernetes/ssl
</code>

参考

  • Generate self-signed certificates
  • Setting up a Certificate Authority and Creating TLS Certificates
  • Client Certificates V/s Server Certificates
  • 数字证书及 CA 的扫盲介绍

搞代码网（gaodaima.com）提供的所有资源部分来自互联网，如果有侵犯您的版权或其他权益，请说明详细缘由并提供版权或权益证明然后发送到邮箱[email protected]‍，我们会在看到邮件的第一时间内为您处理，或直接联系QQ：872152909。本网站采用BY-NC-SA协议进行授权
转载请注明原文链接：Kubernetes安装之证书验证

喜欢 (0)赏
[搞代码]
分享 (0)
发表我的评论
取消评论

表情 贴图 加粗 删除线 居中 斜体 签到

Hi，您需要填写昵称和邮箱！

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址