kubernetes 准入策略 —— Kyverno

2020-08-23 tech kubernetes 25 mins 1 图 8855 字

官方github地址：https://github.com/nirmata/kyverno

做过运维的同学应该有过这种体会，假如我们维护了某些组件，这些组件所在的机器如果任由各个团队自己管理，一旦出问题定位很麻烦，没有人知道机器曾经发生了什么事。这种场景同样适用于kubernetes集群运维。没有规矩不成方圆，而我们又没办法要求所有人的用户都能按照标准写yaml，要统一化集群环境：

要么我们将yaml收藏，只提供UI界面
要么我们在用户提交yaml前进行标准化验证，不符合标准的调整为标准的或拒绝运行

第一种办法未免过于粗暴，还是使用方法2。kubernetes 提供了 Admission Controller 对资源进行审查，我们可以基于 Admission Controller 进行开发完成我们的逻辑。

有需求就有市场！ Kyverno 就是这样一个工具：

验证资源：对资源定义进行检查，不符合条件的资源拒绝创建，从而保证集群资源的合规性。
修改资源：在资源定义中进行注入，强制资源部分行为的一致性。
生成资源：在资源创建时，同时创建相关的资源。

除了 Kyverno，cncf 也官宣了 Open Policy Agent (OPA) 用作 Kubernetes 准入控制器，个人感觉目前 OPA 过于灵活，并且目前还不支持generate和mutate，等待社区慢慢完善了。目前先使用 kyverno 解决手头的麻烦。

一、安装

kubectl create -f https://raw.githubusercontent.com/nirmata/kyverno/master/definitions/release/install.yaml

这是我当时的拉到的安装文件，做个备份https://cdn.kelu.org/blog/2020/08/kyverno.yaml

默认会安装在 kyverno 这个namespace下。

1598255025850

可以看到 kyverno 生成了很多crd和权限相关的资源。安装完成后，可以在 configmap 中修改不需要处理的资源类型，格式为：[<Kind>,<Namespace>,<Name>]。

每个 kyverno policy 包含一个或者多个规则，每个规则都有一个match，exclude可选，以及一个mutate，validate或generate。

二、用法

常用命令：

经常要看事件来确定规则是否生效：

kubectl get events -A -w

查看已经配置的规则：

kubectl get ClusterPolicy
kubectl get cpol

快速生成pod：

kubectl run busybox --image=busybox  --restart=Never -- sleep 1000000

快速生成pod的yaml：

kubectl run busybox --image=busybox  --restart=Never --dry-run -o yaml  -- sleep 1000000 > pod.yaml

快速生成deployment：

kubectl create deployment nginx --image=nginx

hostpath:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: test
  name: test-hp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      volumes:
        - name: test-volume
          hostPath:
            path: /app/kelu/test-volume  
            type: ""
      containers:
      - image: nginx
        name: test-hp
        volumeMounts:
        - mountPath: /test-pd
          name: test-volume

三、实践

相关实践在官方GitHub上有详细解说，这里我记录其中的一部分，并做了一小部分调整。

1. 默认添加 namespace 的 quotas/limitrange

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-ns-quota
spec:
  rules:
  - name: generate-resourcequota
    match:
      resources:
        kinds:
        - Namespace
    generate:
      kind: ResourceQuota
      name: default-resourcequota
      namespace: ""
      data:
        spec:
          hard:
            requests.cpu: '1'
            requests.memory: '1Gi'
            limits.cpu: '1'
            limits.memory: '1Gi'
            persistentvolumeclaims: "20"
            requests.storage: 100Gi
            pods: "100"
            services: "100"
  - name: generate-limitrange
    match:
      resources:
        kinds:
        - Namespace
    generate:
      kind: LimitRange
      name: default-limitrange
      namespace: ""
      data:
        spec:
          limits:
          - default:
              cpu: 200m
              memory: 200Mi
            defaultRequest:
              cpu: 200m
              memory: 200Mi
            type: Container

2. pod必须包含 rquest 和 limit 才允许运行

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-pod-requests-limits
spec:
  validationFailureAction: enforce
  rules:
  - name: validate-resources
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "CPU and memory resource requests and limits are required"
      pattern:
        spec:
          containers:
          - resources:
              requests:
                memory: "?*"
                cpu: "?*"
              limits:
                memory: "?*"

3. 禁止绑定 docker sock

需要绑定的容器，得在kyverno生成的init-config里添加。

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-docker-sock-mount
spec:
  validationFailureAction: enforce
  rules:
  - name: validate-docker-sock-mount
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "Use of the Docker Unix socket is not allowed"
      pattern:
        spec:
          =(volumes):
            - =(hostPath):
                path: "!/var/run/docker.sock"

4. 禁止特权容器

因为默认不对 kube-system 进行应用，所以可以放心对所有容器禁止特权。

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-privileged
spec:
  validationFailureAction: enforce
  rules:
  - name: validate-privileged
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "Privileged mode is not allowed. Set privileged to false"
      pattern:
        spec:
          containers:
          - =(securityContext):
              =(privileged): false
  - name: validate-allowPrivilegeEscalation
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "Privileged mode is not allowed. Set allowPrivilegeEscalation to false"
      pattern:
        spec:
          containers:
          - securityContext:
              allowPrivilegeEscalation: false

5. 禁止 hostpid 和hostipc

避免Pod容器对主机进程空间具有可见性

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-host-pid-ipc
spec:
  validationFailureAction: enforce
  rules:
  - name: validate-hostPID-hostIPC
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "Use of host PID and IPC namespaces is not allowed"
      pattern:
        spec:
          =(hostPID): "false"
          =(hostIPC): "false"

6. 禁止绑定 hostpath

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: 
  name: disallow-bind-mounts
spec:
  validationFailureAction: enforce
  rules: 
  - name: validate-hostPath
    match: 
      resources: 
        kinds: 
        - Pod
    validate: 
      message: "Host path volumes are not allowed"
      pattern: 
        spec: 
          =(volumes): 
          - X(hostPath): null

7. 默认增加默认磁盘限制

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: set-ephemeral-storage
spec:
  rules:
    - name: set-ephemeral-storage
      match:
        resources:
          kinds:
            - Pod
      mutate:
        overlay:
          spec:
            containers:
              - (name): "*"
                resources:
                  limits:
                    +(ephemeral-storage): "40Gi"
                  requests:
                    +(ephemeral-storage): "100Mi"

8. 必须包含nodeselect或affinity

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-no-nodeselector-and-affinity
spec:
  validationFailureAction: enforce
  rules:
    - name: disallow-no-nodeselector-and-affinity
      match:
        resources:
          kinds:
            - Deployment
            - StatefuleSet
            - DaemonSet
          name: "*"
      validate:
        message: "no nodeselector and affinity"
        anyPattern:
          - spec:
              template:
                spec:
                  affinity: "*"
          - spec:
              template:
                spec:
                  nodeSelector: "*"

测试yaml：

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: test
  name: test-hp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
      - image: nginx
        name: test-hp
      nodeSelector:
        kubernetes.io/hostname: kind-control-plane

9. 只允许使用某镜像仓库的镜像

apiVersion : kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-external-image
spec:
  validationFailureAction: enforce
  rules:
  - name: require-image-tag
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "Only internal images are allowed"  
      pattern:
        spec:
          containers:
          - image: "image.kelu.org/*"

10. 不允许使用latest镜像

apiVersion : kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-latest-tag
spec:
  validationFailureAction: enforce
  rules:
  - name: require-image-tag
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "An image tag is required"  
      pattern:
        spec:
          containers:
          - image: "*:*"
  - name: validate-image-tag
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "Using a mutable image tag e.g. 'latest' is not allowed"
      pattern:
        spec:
          containers:
          - image: "!*:latest"

11. 禁止使用特定的nodeport端口

apiVersion : kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-nodeport-29000
spec:
  validationFailureAction: enforce
  rules:
  - name: disallow-nodeport-29000
    match:
      resources:
        kinds:
        - Service
    validate:
      message: "disallow-nodeport-29000"  
      pattern:
        spec:
          type: NodePort
          ports:
           - nodePort: "!29000"
  - name: disallow-lb-29000
    match:
      resources:
        kinds:
        - Service
    validate:
      message: "disallow-LoadBalancer-29000"  
      pattern:
        spec:
          type: LoadBalancer
          ports:
           - nodePort: "!29000"

12. 禁止配置external-ip

apiVersion : kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-external-ip
spec:
  validationFailureAction: enforce
  rules:
  - name: disallow-external-ip
    match:
      resources:
        kinds:
        - Service
    validate:
      message: "externalIPs are not allowed."  
      pattern:
        spec:
          X(externalIPs): "null"

参考资料

Sample Policies