官方github地址：https://github.com/nirmata/kyverno
做过运维的同学应该有过这种体会，假如我们维护了某些组件，这些组件所在的机器如果任由各个团队自己管理，一旦出问题定位很麻烦，没有人知道机器曾经发生了什么事。 这种场景同样适用于kubernetes集群运维。没有规矩不成方圆，而我们又没办法要求所有人的用户都能按照标准写yaml，要统一化集群环境：
- 要么我们将yaml收藏，只提供UI界面
- 要么我们在用户提交yaml前进行标准化验证，不符合标准的调整为标准的或拒绝运行
第一种办法未免过于粗暴，还是使用方法2。kubernetes 提供了 Admission Controller 对资源进行审查，我们可以基于 Admission Controller 进行开发完成我们的逻辑。
有需求就有市场！ Kyverno 就是这样一个工具：
- 验证资源：对资源定义进行检查，不符合条件的资源拒绝创建，从而保证集群资源的合规性。
- 修改资源：在资源定义中进行注入，强制资源部分行为的一致性。
- 生成资源：在资源创建时，同时创建相关的资源。
除了 Kyverno，cncf 也官宣了 Open Policy Agent (OPA) 用作 Kubernetes 准入控制器，个人感觉目前 OPA 过于灵活，并且目前还不支持generate和mutate，等待社区慢慢完善了。目前先使用 kyverno 解决手头的麻烦。
一、安装
kubectl create -f https://raw.githubusercontent.com/nirmata/kyverno/master/definitions/release/install.yaml
这是我当时的拉到的安装文件，做个备份https://cdn.kelu.org/blog/2020/08/kyverno.yaml
默认会安装在 kyverno 这个namespace下。
可以看到 kyverno 生成了很多crd和权限相关的资源。安装完成后，可以在 configmap 中修改不需要处理的资源类型，格式为：[<Kind>,<Namespace>,<Name>]。
每个 kyverno policy 包含一个或者多个规则，每个规则都有一个match，exclude可选，以及一个mutate，validate或generate。
二、用法
常用命令：
经常要看事件来确定规则是否生效：
查看已经配置的规则：
kubectl get ClusterPolicy
kubectl get cpol
快速生成pod：
kubectl run busybox --image=busybox --restart=Never -- sleep 1000000
快速生成pod的yaml：
kubectl run busybox --image=busybox --restart=Never --dry-run -o yaml -- sleep 1000000 > pod.yaml
快速生成deployment：
kubectl create deployment nginx --image=nginx
hostpath:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: test
name: test-hp
spec:
replicas: 1
selector:
matchLabels:
app: test
template:
metadata:
labels:
app: test
spec:
volumes:
- name: test-volume
hostPath:
path: /app/kelu/test-volume
type: ""
containers:
- image: nginx
name: test-hp
volumeMounts:
- mountPath: /test-pd
name: test-volume
三、实践
相关实践在官方GitHub上有详细解说，这里我记录其中的一部分，并做了一小部分调整。
1. 默认添加 namespace 的 quotas/limitrange
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: add-ns-quota
spec:
rules:
- name: generate-resourcequota
match:
resources:
kinds:
- Namespace
generate:
kind: ResourceQuota
name: default-resourcequota
namespace: ""
data:
spec:
hard:
requests.cpu: '1'
requests.memory: '1Gi'
limits.cpu: '1'
limits.memory: '1Gi'
persistentvolumeclaims: "20"
requests.storage: 100Gi
pods: "100"
services: "100"
- name: generate-limitrange
match:
resources:
kinds:
- Namespace
generate:
kind: LimitRange
name: default-limitrange
namespace: ""
data:
spec:
limits:
- default:
cpu: 200m
memory: 200Mi
defaultRequest:
cpu: 200m
memory: 200Mi
type: Container
2. pod必须包含 rquest 和 limit 才允许运行
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-pod-requests-limits
spec:
validationFailureAction: enforce
rules:
- name: validate-resources
match:
resources:
kinds:
- Pod
validate:
message: "CPU and memory resource requests and limits are required"
pattern:
spec:
containers:
- resources:
requests:
memory: "?*"
cpu: "?*"
limits:
memory: "?*"
3. 禁止绑定 docker sock
需要绑定的容器，得在kyverno生成的init-config里添加。
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: disallow-docker-sock-mount
spec:
validationFailureAction: enforce
rules:
- name: validate-docker-sock-mount
match:
resources:
kinds:
- Pod
validate:
message: "Use of the Docker Unix socket is not allowed"
pattern:
spec:
=(volumes):
- =(hostPath):
path: "!/var/run/docker.sock"
4. 禁止特权容器
因为默认不对 kube-system 进行应用，所以可以放心对所有容器禁止特权。
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: disallow-privileged
spec:
validationFailureAction: enforce
rules:
- name: validate-privileged
match:
resources:
kinds:
- Pod
validate:
message: "Privileged mode is not allowed. Set privileged to false"
pattern:
spec:
containers:
- =(securityContext):
=(privileged): false
- name: validate-allowPrivilegeEscalation
match:
resources:
kinds:
- Pod
validate:
message: "Privileged mode is not allowed. Set allowPrivilegeEscalation to false"
pattern:
spec:
containers:
- securityContext:
allowPrivilegeEscalation: false
5. 禁止 hostpid 和hostipc
避免Pod容器对主机进程空间具有可见性
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: disallow-host-pid-ipc
spec:
validationFailureAction: enforce
rules:
- name: validate-hostPID-hostIPC
match:
resources:
kinds:
- Pod
validate:
message: "Use of host PID and IPC namespaces is not allowed"
pattern:
spec:
=(hostPID): "false"
=(hostIPC): "false"
6. 禁止绑定 hostpath
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: disallow-bind-mounts
spec:
validationFailureAction: enforce
rules:
- name: validate-hostPath
match:
resources:
kinds:
- Pod
validate:
message: "Host path volumes are not allowed"
pattern:
spec:
=(volumes):
- X(hostPath): null
7. 默认增加 默认磁盘限制
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: set-ephemeral-storage
spec:
rules:
- name: set-ephemeral-storage
match:
resources:
kinds:
- Pod
mutate:
overlay:
spec:
containers:
- (name): "*"
resources:
limits:
+(ephemeral-storage): "40Gi"
requests:
+(ephemeral-storage): "100Mi"
8. 必须包含nodeselect或affinity
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: disallow-no-nodeselector-and-affinity
spec:
validationFailureAction: enforce
rules:
- name: disallow-no-nodeselector-and-affinity
match:
resources:
kinds:
- Deployment
- StatefuleSet
- DaemonSet
name: "*"
validate:
message: "no nodeselector and affinity"
anyPattern:
- spec:
template:
spec:
affinity: "*"
- spec:
template:
spec:
nodeSelector: "*"
测试yaml：
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: test
name: test-hp
spec:
replicas: 1
selector:
matchLabels:
app: test
template:
metadata:
labels:
app: test
spec:
containers:
- image: nginx
name: test-hp
nodeSelector:
kubernetes.io/hostname: kind-control-plane
9. 只允许使用某镜像仓库的镜像
apiVersion : kyverno.io/v1
kind: ClusterPolicy
metadata:
name: disallow-external-image
spec:
validationFailureAction: enforce
rules:
- name: require-image-tag
match:
resources:
kinds:
- Pod
validate:
message: "Only internal images are allowed"
pattern:
spec:
containers:
- image: "image.kelu.org/*"
10. 不允许使用latest镜像
apiVersion : kyverno.io/v1
kind: ClusterPolicy
metadata:
name: disallow-latest-tag
spec:
validationFailureAction: enforce
rules:
- name: require-image-tag
match:
resources:
kinds:
- Pod
validate:
message: "An image tag is required"
pattern:
spec:
containers:
- image: "*:*"
- name: validate-image-tag
match:
resources:
kinds:
- Pod
validate:
message: "Using a mutable image tag e.g. 'latest' is not allowed"
pattern:
spec:
containers:
- image: "!*:latest"
11. 禁止使用特定的nodeport端口
apiVersion : kyverno.io/v1
kind: ClusterPolicy
metadata:
name: disallow-nodeport-29000
spec:
validationFailureAction: enforce
rules:
- name: disallow-nodeport-29000
match:
resources:
kinds:
- Service
validate:
message: "disallow-nodeport-29000"
pattern:
spec:
type: NodePort
ports:
- nodePort: "!29000"
- name: disallow-lb-29000
match:
resources:
kinds:
- Service
validate:
message: "disallow-LoadBalancer-29000"
pattern:
spec:
type: LoadBalancer
ports:
- nodePort: "!29000"
12. 禁止配置external-ip
apiVersion : kyverno.io/v1
kind: ClusterPolicy
metadata:
name: disallow-external-ip
spec:
validationFailureAction: enforce
rules:
- name: disallow-external-ip
match:
resources:
kinds:
- Service
validate:
message: "externalIPs are not allowed."
pattern:
spec:
X(externalIPs): "null"
参考资料
