准入控制器

发表于 | 分类于 | 阅读次数

背景

在请求经过身份验证和授权之后，对象持久化之前，对Kubernetes API 服务器的请求进行拦截。

准入控制器可能执行validating或者mutating或两者兼而有之。Mutating 控制器可以修改他们的处理的资源对象，Validating 控制器不会，如果任何一个阶段中的任何控制器拒绝了请求，则会立即拒绝整个请求，并将错误返回给最终的用户。

1
2
3
4
5
# 打开准入控制器
kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger ...
# 关闭准入控制器
kube-apiserver --disable-admission-plugins=PodNodeSelector,AlwaysDeny ...

在 apiserver 内部，有两个特殊的 controllers：MutatingAdmissionWebhook 和 ValidatingAdmissionWebhook，通过它们提供的协议，用户能够将自定义 webhook 集成到 admission controller 控制流中。

顾名思义，mutating admission webhook 可以拦截并修改请求资源，validating admission webhook 只能拦截并校验请求资源，但不能修改它们。分成两类的一个好处是，后者可以被 apiserver 并发执行，只要任一失败，即可快速结束请求。

k8s api request lifecycle

向集群发出POST请求，会序列化成AdmissionReview

1
2
3
4
5
6
7
8
9
10
11
12
{
  "apiVersion": "admission.k8s.io/v1",
  "kind": "AdmissionReview",
  ...
  "request": {
    # Random uid uniquely identifying this admission call
    "uid": "705ab4f5-6393-11e8-b7cc-42010a800002",
    # object is the new object being admitted.
    "object": {"apiVersion":"v1","kind":"Pod", ...},
    ...
  }
}

修改，校验 request.object 中的反序列后内容（需要我们提供https自定义服务完成），最后将结果返回给 apiserver。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# 成功
{
  "apiVersion": "admission.k8s.io/v1",
  "kind": "AdmissionReview",
  "response": {
    "uid": "<value from request.uid>",
    "allowed": true,
  }
}



# 失败
{
  "apiVersion": "admission.k8s.io/v1",
  "kind": "AdmissionReview",
  "response": {
    "uid": "<value from request.uid>", # 唯一确定请求
    "allowed": false, # 表示通过或者不通过
    "status": {   # 提供错误提示
      "code": 402,
      "status": "Failure",
      "message": "#ctr is using env vars",
      "reason": "#ctr is using env vars"
    }
  }
}

开启校验 for Mac

Docker for mac 修改kube-apiserver参数（How to enable admission-pulgins in kubernetes of docker-desktop）

1
2
3
4
5
6
7
8
9
10
11
12
13
# 确保admissionregistration.k8s.io/v1API 已启

kubectl api-versions |grep admission

# 只需从您的 Mac 终端运行它，它就会将您放入一个对 Docker VM 具有完全权限的容器中
docker run -it --rm --privileged --pid=host justincormack/nsenter1

# 修改
/etc/kubernetes/manifests/kube-apiserver.yaml

- --enable-admission-plugins=NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook

# 重启docker-desktop

证书准备

既然是提供https服务，自然需要提供证书，Kubernetes 证书有效期为 1 年，复杂的生产环境可以考虑certmanager，它有自动更新、自动注入等一系列生命周期管理功能。

mac上面调试访问本机服务使用host.docker.internal域名

k8s Certificate Signing Requests

证书签名请求

  • 使用openssl，通过配置，生产key和csr文件。
  • 使用 Kubernetes CertificateSigningRequest 和 kubectl approve 签名服务证书
  • 将服务私钥和证书，存储到 Kubernetes Secret 中
  • 如果采用集群外部署，注意在 csr.conf 中指定好域名或 IP 地址
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
#!/bin/bash

set -e

usage() {
    cat <<EOF
Generate certificate suitable for use with an webhook service.

This script uses k8s' CertificateSigningRequest API to a generate a
certificate signed by k8s CA suitable for use with webhook
services. This requires permissions to create and approve CSR. See
https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster for
detailed explantion and additional instructions.

The server key/cert k8s CA cert are stored in a k8s secret.

usage: ${0} [OPTIONS]

The following flags are required.

       --service          Service name of webhook.
       --namespace        Namespace where webhook service and secret reside.
       --secret           Secret name for CA certificate and server certificate/key pair.
EOF
    exit 1
}

while [[ $# -gt 0 ]]; do
    case ${1} in
        --service)
            service="$2"
            shift
            ;;
        --secret)
            secret="$2"
            shift
            ;;
        --namespace)
            namespace="$2"
            shift
            ;;
        *)
            usage
            ;;
    esac
    shift
done

[ -z ${service} ] && echo "ERROR: --service flag is required" && exit 1
[ -z ${secret} ] && echo "ERROR: --secret flag is required" && exit 1
[ -z ${namespace} ] && namespace=default

if [ ! -x "$(command -v openssl)" ]; then
    echo "openssl not found"
    exit 1
fi

csrName=${service}.${namespace}
tmpdir=$(mktemp -d)
echo "creating certs in tmpdir ${tmpdir} "

cat <<EOF >> ${tmpdir}/csr.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = ${service}
DNS.2 = ${service}.${namespace}
DNS.3 = ${service}.${namespace}.svc
IP.1 = 192.168.1.10  # change it to your IP address
EOF

openssl genrsa -out ${tmpdir}/server-key.pem 2048
openssl req -new -key ${tmpdir}/server-key.pem -subj "/CN=${service}.${namespace}.svc" -out ${tmpdir}/server.csr -config ${tmpdir}/csr.conf

# clean-up any previously created CSR for our service. Ignore errors if not present.
kubectl delete csr ${csrName} 2>/dev/null || true

# create  server cert/key CSR and  send to k8s API
cat <<EOF | kubectl create -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: ${csrName}
spec:
  groups:
  - system:authenticated
  request: $(cat ${tmpdir}/server.csr | base64 | tr -d '\n')
  usages:
  - digital signature
  - key encipherment
  - server auth
EOF

# verify CSR has been created
while true; do
    kubectl get csr ${csrName}
    if [ "$?" -eq 0 ]; then
        break
    fi
done

# approve and fetch the signed certificate
kubectl certificate approve ${csrName}
# verify certificate has been signed
for x in $(seq 10); do
    serverCert=$(kubectl get csr ${csrName} -o jsonpath='{.status.certificate}')
    if [[ ${serverCert} != '' ]]; then
        break
    fi
    sleep 1
done
if [[ ${serverCert} == '' ]]; then
    echo "ERROR: After approving csr ${csrName}, the signed certificate did not appear on the resource. Giving up after 10 attempts." >&2
    exit 1
fi
echo ${serverCert} | openssl base64 -d -A -out ${tmpdir}/server-cert.pem


# create the secret with CA cert and server cert/key
kubectl create secret tls ${secret} \
        --key="${tmpdir}/server-key.pem" \
        --cert="${tmpdir}/server-cert.pem" \
        --dry-run -o yaml |
    kubectl -n ${namespace} apply -f -

CertManager

cert-manager-自签名

  • 安装cert-manager(kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.3/cert-manager.yaml)或者helm chart

  • apply Issuer CR 和Certificate CR，引用 Issuer 签发证书

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: customwebhook-selfsigned-issuer
  namespace: default
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: customwebhook-tls-secret
spec:
  duration: 8760h
  renewBefore: 8000h
  commonName: customwebhook.default
  isCA: false
  privateKey:
    algorithm: RSA
    encoding: PKCS1
    size: 2048
  usages:
    - digital signature
    - key encipherment
    - server auth
  dnsNames:
    - customwebhook    
    - customwebhook.default
    - customwebhook.default.svc  # 给集群内调用
    - host.docker.internal  # 给集群外面用 方便调试
#  ipAddresses:
#    - 192.168.1.10 # change it to your IP addresses
  issuerRef:
    kind: Issuer
    name: customwebhook-selfsigned-issuer
  secretName: customwebhook-tls-secret  # 生产的secretName

  • 查看secretName

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    # 查看
kubectl describe secret customwebhook-tls-secret
  
  
Name:         customwebhook-tls-secret
Namespace:    default
Labels:       <none>
Annotations:  cert-manager.io/alt-names: ....
  
Type:  kubernetes.io/tls
  
Data
====
ca.crt:   1237 bytes
tls.crt:  1237 bytes
tls.key:  1675 bytes
  
# 获取方便调试
kubectl get secrets/customwebhook-tls-secret -o jsonpath="{.data['tls\.key']}" | base64 -d > tls.key
  
kubectl get secrets/customwebhook-tls-secret -o jsonpath="{.data['tls\.crt']}" | base64 -d > tls.crt

向 apiserver 注册 admission webhook

获取caBundle , cert-manager 自动注入 不需要caBundle

1
2
3
# 获取集群CA 

kubectl get configmap -n kube-system extension-apiserver-authentication -o=jsonpath='{.data.client-ca-file}' | base64 | tr -d '\n'

官方文档 解析入参出参 WebhookConfiguration 配置，按需要配置ValidatingWebhookConfiguration，MutatingWebhookConfiguration，其中一个或全部

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: customwebhook-outercluster
  annotations:
    # cert-manager 自动注入 不需要caBundle
    cert-manager.io/inject-ca-from: default/customwebhook-tls-secret
webhooks:
  - admissionReviewVersions:
      - v1
    clientConfig:
			caBundle: "<Kubernetes CA> or <cert-manager CA>"
      url: 'https://host.docker.internal:8000/validate' # 集群外部署，使用此方式时，注释 service ref
      service:                                  #---------------------#             
        name: customwebhook                     #---------------------#             
        namespace: default                      #       集群内部署      #            
        port: 443                               # 使用此方式时，注释 url #            
        path: /validate                         #---------------------#            

    failurePolicy: Fail
    matchPolicy: Exact
    name: customwebhook.valid.outercluster
    rules:
      - apiGroups:
          - ""
        apiVersions:
          - v1
        operations:
          - CREATE
          - UPDATE
        resources:
          - pods
        scope: '*'
    objectSelector:
      matchExpressions:
        - key: app  # 排除自己，拦截所有
          operator: NotIn
          values:
            - customwebhook
    sideEffects: None
    timeoutSeconds: 30
---
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: customwebhook-outercluster
  annotations:
    cert-manager.io/inject-ca-from: default/customwebhook-tls-secret
webhooks:
  - admissionReviewVersions: # admissionReviewVersions 请求的版本
      - v1
    clientConfig: # 客户端配置
      #      caBundle: # ca证书
      #      service: # 调用服务相关配置，这里是一个k8s的service，访问地址是<name>.<namespace>.svc:<port>/<path>
      #        name: mutating-test
      #        namespace: testing-tools
      #        path: /mutation-deployment
      #        port: 8000
      url: 'https://host.docker.internal:8000/mutate'
    failurePolicy: Fail # 调用失败策略，Ignore为忽略错误, failed表示admission会处理错误
    matchPolicy: Exact
    name: customwebhook.mutate.outercluster
    namespaceSelector: {} # 命名空间过滤条件
    objectSelector: # 对象过滤条件
      matchExpressions:
        - key: app
          operator: NotIn
          values:
            - customwebhook
    # reinvocationPolicy表示再调度策略，因为webhook本身没有顺序性，因此每个修改后可能又被其他webhook修改，所以提供
    # 一个策略表示是否需要被多次调用，Never 表示只会调度一次，IfNeeded 表示资源被修改后会再调度这个webhook
    reinvocationPolicy: Never
    rules: # 规则
      - apiGroups:
          - ""
        apiVersions:
          - v1
        operations:
          - CREATE
          - UPDATE
        resources:
          - pods
        scope: '*' # 匹配范围，"*" 匹配所有资源，但不包括子资源，"*/*" 匹配所有资源，包括子资源
    sideEffects: None # 这个表示webhook是否存在副作用，主要针对 dryRun 的请求
    timeoutSeconds: 30

代码参考

官方代码很好了解一些入参出参的写法，而sig做了相应的封装。

main.go

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
package main

import (
	"context"
	"crypto/tls"
	"encoding/json"
	"fmt"
	"io"
	"net/http"
	"os"
	"os/signal"
	"path/filepath"
	"syscall"

	admissionv1 "k8s.io/api/admission/v1"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/runtime/serializer"
	"k8s.io/klog/v2"
	"sigs.k8s.io/controller-runtime/pkg/webhook/admission"
)

const (
	tlsKeyName  = "tls.key"
	tlsCertName = "tls.crt"
)

func waitExit() {
	exitChan := make(chan os.Signal, 1)
	signal.Notify(exitChan, syscall.SIGINT, syscall.SIGTERM, syscall.SIGHUP)

	<-exitChan
	klog.Infof("Got OS shutdown signal, shutting down webhook server gracefully...")
}

var (
	runtimeScheme = runtime.NewScheme()
	codecs        = serializer.NewCodecFactory(runtimeScheme)
	decoder, _    = admission.NewDecoder(runtimeScheme)
	deserializer  = codecs.UniversalDeserializer()
)

// 统一处理入参出参，类似装饰器后面可以搞
func serve(w http.ResponseWriter, r *http.Request) {
	var body []byte
	if r.Body != nil {
		if data, err := io.ReadAll(r.Body); err == nil {
			body = data
		}
	}
	if len(body) == 0 {
		klog.Error("empty body")
		http.Error(w, "empty body", http.StatusBadRequest)
		return
	}
	// verify the content type is accurate
	contentType := r.Header.Get("Content-Type")
	if contentType != "application/json" {
		klog.Errorf("contentType=%s, expect application/json", contentType)
		return
	}
	var response admission.Response
	ar := admissionv1.AdmissionReview{}
	if _, _, err := deserializer.Decode(body, nil, &ar); err != nil {
		msg := fmt.Sprintf("Request could not be decoded: %v", err)
		klog.Error(msg)
		response = admission.Errored(http.StatusInternalServerError, err)

	} else {
		if r.URL.Path == "/mutate" {
			response = mutate(ar.Request)
		} else if r.URL.Path == "/validate" {
			response = validate(ar.Request)
		}
	}

	if err := response.Complete(admission.Request{AdmissionRequest: *ar.Request}); err != nil {
		klog.Errorf("unable to get response: %v", err)
		http.Error(w, fmt.Sprintf("could not get response: %v", err), http.StatusInternalServerError)
	}

	responseAdmissionReview := admissionv1.AdmissionReview{
		TypeMeta: ar.TypeMeta,
		Response: &response.AdmissionResponse,
	}
	resp, err := json.Marshal(responseAdmissionReview)
	if err != nil {
		klog.Errorf("Can't encode response: %v", err)
		http.Error(w, fmt.Sprintf("could not encode response: %v", err), http.StatusInternalServerError)
	}
	w.Header().Set("Content-Type", "application/json")

	if _, err := w.Write(resp); err != nil {
		klog.Errorf("Can't write response: %v", err)
		http.Error(w, fmt.Sprintf("could not write response: %v", err), http.StatusInternalServerError)
	}

}

func main() {
	port := ":8000"
	if certDir := os.Getenv("CERT_DIR"); certDir != "" {
		certFile := filepath.Join(certDir, tlsCertName)
		keyFile := filepath.Join(certDir, tlsKeyName)
		pair, err := tls.LoadX509KeyPair(certFile, keyFile)
		if err != nil {
			klog.Errorf("Failed to load key pair in $s : %v", certDir, err)
		}

		mux := http.NewServeMux()
		mux.HandleFunc("/validate", serve)
		mux.HandleFunc("/mutate", serve)

		server := &http.Server{
			Addr:      port,
			TLSConfig: &tls.Config{Certificates: []tls.Certificate{pair}},
			Handler:   mux,
		}

		klog.Infof("Server started Listen to %s", port)
		go func() {
			if err := server.ListenAndServeTLS("", ""); err != nil {
				klog.Errorf("Failed to listen and serve webhook server: %v", err)
			}
		}()

		waitExit()
		server.Shutdown(context.Background())

	}

}

webhook.go 核心方法

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
package main

import (
	"encoding/json"
	"fmt"
	"net/http"

	admissionv1 "k8s.io/api/admission/v1"
	corev1 "k8s.io/api/core/v1"
	"k8s.io/klog/v2"
	"sigs.k8s.io/controller-runtime/pkg/webhook/admission"
)

// 修改pod label
func mutate(req *admissionv1.AdmissionRequest) admission.Response {
	klog.Infof("Call MutatingWebhookConfiguration")
	pod := corev1.Pod{}
	// Get pod object from request
	if err := json.Unmarshal(req.Object.Raw, &pod); err != nil {
		return admission.Errored(http.StatusForbidden, err)
	}
	klog.Infof("mutate pod %s", pod.Name)
	if pod.Labels == nil {
		pod.Labels = map[string]string{}
	}
	pod.Labels["app"] = pod.Name

	newObj, err := json.Marshal(pod)
	if err != nil {
		return admission.Errored(http.StatusInternalServerError, err)
	}

	return admission.PatchResponseFromRaw(req.Object.Raw, newObj)
}

// 检查pod image 名字
func validate(req *admissionv1.AdmissionRequest) admission.Response {
	klog.Infof("Call ValidatingWebhookConfiguration")
	pod := corev1.Pod{}

	// Get pod object from request
	if err := json.Unmarshal(req.Object.Raw, &pod); err != nil {
		return admission.Errored(http.StatusForbidden, err)
	}
	klog.Infof("validating pod %s", pod.Name)

	for _, ctr := range pod.Spec.Containers {
		if ctr.Image != "ebpf-test" {
			return admission.Denied(fmt.Sprintf("%s image name not good", ctr.Name))
		}
	}
	return admission.Allowed("")
}

参考