自定义资源类型（CRD）
Kubernetes 系统的扩展和增强既包括扩展 API Server 所支持的资源类型及相关声明式功能的实现，以及消除集群的单点以实现集群的高可用等，也包括如何将系统增强为一个完整意义上的 PaaS 平台，并以 DevOps 文化为驱动改善工作流程等。
Kubernetes API 默认提供的众多的功能性资源类型可用于容器编排以解决多数场景中的编排需求。但是，有些场景也需要借助于额外的或更高级别的编排抽象，例如引入集群外部的一些服务并以资源对象的形式进行管理，或者把 Kubernetes 的多个标准资源对象合并为一个单一的更高级别的资源抽象等等。而这类的 API 扩展抽象通常也应该兼容 Kubernetes 系统的基本特性，如支持 kubectl 管理工具、CRUD 及 watch 机制、标签、etcd 存储、认证、授权、RBAC 及审计，等等，从而使得用户可将精力集中于构建业务逻辑本身。
目前，扩展 Kubernetes API 的常用方式有三种：
- 使用 CRD（CustomResourceDefinitions）自定义资源类型。
- 开发自定义的 API Server 并聚合至主 API Server。
- 定制扩展 Kubernetes 源码。
其中，CRD 最易用但限制较多，自定义 API Server 更富有弹性但代码工作量偏大，而仅在必须添加新的核心类型才能确保专用的 Kubernetes 集群功能正常时才应该定制系统源码。
CRD 并非设计用来取代 Kubernetes 的原生资源类型，而是用于补充一种简单易用的更为灵活和更高级别的自定义 API 资源的方式。虽然目前在功能上仍存在不少的局限，但对于大多数的需求场景来说，CRD 的表现已经足够好，因此在满足需求的前提下是首选的 API 资源类型扩展方案。
一、创建 CRD 对象
CRD 自 Kubernetes 1.7 版开始引入，并自 1.8 版起完全取代其前身 TPR(ThirdPartyResources)，其设计目标是通过 CRD 可以向 API 中增加新资源类型，无须修改 Kubernetes 源代码就能扩展它支持使用 API 资源类型。该功能大大提高了 Kubernetes 的可扩展能力。
CRD 本身也是一种资源类型，是一种无须编码就可以扩展原生 Kubernetes API 接口的形式，隶属于集群级别。实例化出特定的对象之后，它会在 API 上注册生成 GVR 类型 URL 端点，并能够作为一种资源类型被使用并实例化相应的对象。自定义资源类型之前，选定其使用的 API 群租名称、版本及新建的资源类型名称，根据这些信息即可创建自定义资源类型，并创建自定义类型的资源对象。
CRD 高级特性：
-
- 支持 subresource，比如 status 或者 scale。
- finalizer，在删除自定义资源的同时去做垃圾回收。
- controller，通过 client-go 中的 informer，实现更强大的功能。
- crd 的 validation 可对资源中的字段进行验证，设置默认值。
CustomResourceDefinition 字段定义如下：
[root@k8s-master01-test-2-26 ~]# kubectl explain customresourcedefinition
KIND: CustomResourceDefinition
VERSION: apiextensions.k8s.io/v1
DESCRIPTION:
CustomResourceDefinition represents a resource that should be exposed on
the API server. Its name MUST be in the format <.spec.name>.<.spec.group>.
FIELDS:
apiVersion <string>
APIVersion defines the versioned schema of this representation of an
object. Servers should convert recognized schemas to the latest internal
value, and may reject unrecognized values. More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
kind <string>
Kind is a string value representing the REST resource this object
represents. Servers may infer this from the endpoint the client submits
requests to. Cannot be updated. In CamelCase. More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata <Object>
Standard object's metadata More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
spec <Object> -required-
spec describes how the user wants the resources to appear
status <Object>
status indicates the actual state of the CustomResourceDefinition
[root@k8s-master01-test-2-26 ~]#
CustomResourceDefinition.spec 字段定义如下：
[root@k8s-master01-test-2-26 ~]# kubectl explain customresourcedefinition.spec
KIND: CustomResourceDefinition
VERSION: apiextensions.k8s.io/v1
RESOURCE: spec <Object>
DESCRIPTION:
spec describes how the user wants the resources to appear
CustomResourceDefinitionSpec describes how a user wants their resource to
appear
FIELDS:
conversion <Object>
conversion defines conversion settings for the CRD.
group <string> -required-
group is the API group of the defined custom resource. The custom resources
are served under `/apis/<group>/...`. Must match the name of the
CustomResourceDefinition (in the form `<names.plural>.<group>`).
names <Object> -required-
names specify the resource and kind names for the custom resource.
preserveUnknownFields <boolean>
preserveUnknownFields indicates that object fields which are not specified
in the OpenAPI schema should be preserved when persisting to storage.
apiVersion, kind, metadata and known fields inside metadata are always
preserved. This field is deprecated in favor of setting
`x-preserve-unknown-fields` to true in
`spec.versions[*].schema.openAPIV3Schema`. See
https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#pruning-versus-preserving-unknown-fields
for details.
scope <string> -required-
scope indicates whether the defined custom resource is cluster- or
namespace-scoped. Allowed values are `Cluster` and `Namespaced`.
versions <[]Object> -required-
versions is the list of all API versions of the defined custom resource.
Version names are used to compute the order in which served versions are
listed in API discovery. If the version string is "kube-like", it will sort
above non "kube-like" version strings, which are ordered lexicographically.
"Kube-like" versions start with a "v", then are followed by a number (the
major version), then optionally the string "alpha" or "beta" and another
number (the minor version). These are sorted first by GA > beta > alpha
(where GA is a version with no suffix such as beta or alpha), and then by
comparing major version, then minor version. An example sorted list of
versions: v10, v2, v1, v11beta2, v10beta3, v3beta1, v12alpha1, v11alpha2,
foo1, foo10.
[root@k8s-master01-test-2-26 ~]#
customresourcedefinition.spce 常用字段定义如下：
- conversion <Object>: 定义 CRD 的不同版本之间的格式转换方式。有两个子选项: strategy 和 webhook。
- group <string> -required-: 定义了自定义资源所属的 API 组。自定义资源在`/apis/<group>/…`下提供。必须与 CustomResourceDefinition（格式为 `<names.plural>.<group>`）相同。
- names <Object> -required-: 指定自定义资源的资源和种类名称。
- preserveUnknownFields <boolean>: 表示未指定的对象字段。
- scope <string> -required-: 指示定义的自定义资源是 cluster or Namespace-scoped。允许的值为 'Cluster' 和 'Namespaced'。
- versions <[]Object> -required-: 定义自定义资源的所有API版本的列表。
先看看系统中有哪些 CRD 资源，然后我们再自己创建一个 CRD 资源:
[root@k8s-master01-test-2-26 ~]# kubectl get CustomResourceDefinition
NAME CREATED AT
accessors.storage.kubesphere.io 2022-07-01T02:20:20Z
alertmanagerconfigs.monitoring.coreos.com 2022-07-01T02:23:04Z
alertmanagers.monitoring.coreos.com 2022-07-01T02:23:08Z
applications.app.k8s.io 2022-07-01T02:19:04Z
applications.argoproj.io 2022-07-01T02:22:10Z
applications.gitops.kubesphere.io 2022-07-01T02:22:38Z
applicationsets.argoproj.io 2022-07-01T02:22:10Z
appprojects.argoproj.io 2022-07-01T02:22:11Z
argocdextensions.argoproj.io 2022-07-01T02:22:11Z
authorizationpolicies.security.istio.io 2022-07-01T02:21:41Z
bgpconfigurations.crd.projectcalico.org 2022-07-01T02:12:34Z
bgppeers.crd.projectcalico.org 2022-07-01T02:12:34Z
blockaffinities.crd.projectcalico.org 2022-07-01T02:12:35Z
clusterconfigurations.installer.kubesphere.io 2022-07-01T02:12:51Z
clusterdashboards.monitoring.kubesphere.io 2022-07-01T02:28:13Z
clusterinformations.crd.projectcalico.org 2022-07-01T02:12:35Z
clusters.cluster.kubesphere.io 2022-07-01T02:19:19Z
clustertemplates.devops.kubesphere.io 2022-07-01T02:22:36Z
configs.notification.kubesphere.io 2022-07-01T02:27:24Z
dashboards.monitoring.kubesphere.io 2022-07-01T02:28:13Z
destinationrules.networking.istio.io 2022-07-01T02:21:42Z
devopsprojects.devops.kubesphere.io 2022-07-01T02:22:37Z
envoyfilters.networking.istio.io 2022-07-01T02:21:42Z
exporters.events.kubesphere.io 2022-07-01T02:21:33Z
federatedrolebindings.iam.kubesphere.io 2022-07-01T02:19:28Z
federatedroles.iam.kubesphere.io 2022-07-01T02:19:31Z
federatedusers.iam.kubesphere.io 2022-07-01T02:19:33Z
felixconfigurations.crd.projectcalico.org 2022-07-01T02:12:35Z
filters.logging.kubesphere.io 2022-07-01T02:17:40Z
fluentbitconfigs.logging.kubesphere.io 2022-07-01T02:17:41Z
fluentbits.logging.kubesphere.io 2022-07-01T02:17:41Z
gateways.gateway.kubesphere.io 2022-07-01T02:19:23Z
gateways.networking.istio.io 2022-07-01T02:21:42Z
gitrepositories.devops.kubesphere.io 2022-07-01T02:22:37Z
globalnetworkpolicies.crd.projectcalico.org 2022-07-01T02:12:35Z
globalnetworksets.crd.projectcalico.org 2022-07-01T02:12:35Z
globalrolebindings.iam.kubesphere.io 2022-07-01T02:19:35Z
globalroles.iam.kubesphere.io 2022-07-01T02:19:38Z
groupbindings.iam.kubesphere.io 2022-07-01T02:19:41Z
groups.iam.kubesphere.io 2022-07-01T02:19:44Z
helmapplications.application.kubesphere.io 2022-07-01T02:19:07Z
helmapplicationversions.application.kubesphere.io 2022-07-01T02:19:09Z
helmcategories.application.kubesphere.io 2022-07-01T02:19:12Z
helmreleases.application.kubesphere.io 2022-07-01T02:19:14Z
helmrepos.application.kubesphere.io 2022-07-01T02:19:17Z
hostendpoints.crd.projectcalico.org 2022-07-01T02:12:35Z
inputs.logging.kubesphere.io 2022-07-01T02:17:41Z
ipamblocks.crd.projectcalico.org 2022-07-01T02:12:35Z
ipamblocks.network.kubesphere.io 2022-07-01T02:20:00Z
ipamconfigs.crd.projectcalico.org 2022-07-01T02:12:35Z
ipamhandles.crd.projectcalico.org 2022-07-01T02:12:36Z
ipamhandles.network.kubesphere.io 2022-07-01T02:20:03Z
ippools.crd.projectcalico.org 2022-07-01T02:12:36Z
ippools.network.kubesphere.io 2022-07-01T02:20:06Z
istiooperators.install.istio.io 2022-07-01T02:21:42Z
jaegers.jaegertracing.io 2022-07-01T02:23:06Z
kialis.kiali.io 2022-07-01T02:23:39Z
kubecontrollersconfigurations.crd.projectcalico.org 2022-07-01T02:12:36Z
loginrecords.iam.kubesphere.io 2022-07-01T02:19:46Z
namespacenetworkpolicies.network.kubesphere.io 2022-07-01T02:20:09Z
networkpolicies.crd.projectcalico.org 2022-07-01T02:12:36Z
networksets.crd.projectcalico.org 2022-07-01T02:12:36Z
nginxes.gateway.kubesphere.io 2022-07-01T02:19:25Z
notificationmanagers.notification.kubesphere.io 2022-07-01T02:27:24Z
outputs.logging.kubesphere.io 2022-07-01T02:17:41Z
parsers.logging.kubesphere.io 2022-07-01T02:17:41Z
peerauthentications.security.istio.io 2022-07-01T02:21:42Z
pipelineruns.devops.kubesphere.io 2022-07-01T02:22:37Z
pipelines.devops.kubesphere.io 2022-07-01T02:22:37Z
podmonitors.monitoring.coreos.com 2022-07-01T02:23:12Z
probes.monitoring.coreos.com 2022-07-01T02:23:27Z
prometheuses.monitoring.coreos.com 2022-07-01T02:24:56Z
prometheusrules.monitoring.coreos.com 2022-07-01T02:24:11Z
receivers.notification.kubesphere.io 2022-07-01T02:27:25Z
requestauthentications.security.istio.io 2022-07-01T02:21:42Z
resourcequotas.quota.kubesphere.io 2022-07-01T02:20:12Z
rolebases.iam.kubesphere.io 2022-07-01T02:19:49Z
rulers.events.kubesphere.io 2022-07-01T02:21:33Z
rules.auditing.kubesphere.io 2022-07-01T02:21:23Z
rules.events.kubesphere.io 2022-07-01T02:21:33Z
s2ibinaries.devops.kubesphere.io 2022-07-01T02:22:39Z
s2ibuilders.devops.kubesphere.io 2022-07-01T02:22:39Z
s2ibuildertemplates.devops.kubesphere.io 2022-07-01T02:22:39Z
s2iruns.devops.kubesphere.io 2022-07-01T02:22:40Z
serviceentries.networking.istio.io 2022-07-01T02:21:43Z
servicemonitors.monitoring.coreos.com 2022-07-01T02:24:17Z
servicepolicies.servicemesh.kubesphere.io 2022-07-01T02:20:14Z
sidecars.networking.istio.io 2022-07-01T02:21:43Z
strategies.servicemesh.kubesphere.io 2022-07-01T02:20:18Z
telemetries.telemetry.istio.io 2022-07-01T02:21:43Z
templates.devops.kubesphere.io 2022-07-01T02:22:37Z
thanosrulers.monitoring.coreos.com 2022-07-01T02:24:26Z
users.iam.kubesphere.io 2022-07-01T02:19:52Z
virtualservices.networking.istio.io 2022-07-01T02:21:43Z
volumesnapshotclasses.snapshot.storage.k8s.io 2022-07-01T02:15:14Z
volumesnapshotcontents.snapshot.storage.k8s.io 2022-07-01T02:15:14Z
volumesnapshots.snapshot.storage.k8s.io 2022-07-01T02:15:14Z
webhooks.auditing.kubesphere.io 2022-07-01T02:21:23Z
workloadentries.networking.istio.io 2022-07-01T02:21:43Z
workloadgroups.networking.istio.io 2022-07-01T02:21:43Z
workspacerolebindings.iam.kubesphere.io 2022-07-01T02:19:55Z
workspaceroles.iam.kubesphere.io 2022-07-01T02:19:58Z
workspaces.tenant.kubesphere.io 2022-07-01T02:20:23Z
workspacetemplates.tenant.kubesphere.io 2022-07-01T02:20:25Z
[root@k8s-master01-test-2-26 ~]#
下面的配置清单中定义了一个名为 user.auth.zuoyang.tech 的 CRD 资源对象，它的群组名称为为 auth.zuoyang.tech，仅支持一个版本级别 v1alpha1，复数形式为 usters，隶属于名称空间级别，因此，它的对象在 API 上的 URL 路径前缀为 /apis/auth.zuoyang.tech/v1alpha1/namespaces/NS_NAME/users/:
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: users.auth.zuoyang.tech
spec:
group: auth.zuoyang.tech
names:
kind: User
plural: users
singular: user
shortNames:
- u
scope: Namespaced
versions:
- served: true
storage: true
name: v1alpha1
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
userID:
type: integer
minimum: 1
maximum: 65535
groups :
type: array
items:
type: string
email:
type: string
password:
type: string
format: password
required: ["userID","groups"]
配置清单中，spec 嵌套使用的字段都能够见名知意，这里需要特别说明的是，metadata.name 字段的值必须等同于 spec 字段中的 "<name.plural>.<group>" 合并起来的形式。
清单中的 CRD 资源创建于 Kubernetes 集群中之后，继而创建一个新的 CustomResourceDefinition 类型的对象，例如，使用下面的命令列出集群上的 CRD 对象时，命令结果将显示出如下资源名称及创建时间状态信息：
[root@k8s-master01-test-2-26 ~]# kubectl get crd |grep zuoyang
users.auth.zuoyang.tech 2022-07-07T07:48:05Z
[root@k8s-master01-test-2-26 ~]#
现在，在 API 群组 auth.zuoyang.tech/v1beta1 中，users 已经是一个名称空间级别的可用资源类型，用不可按需创建出任意数量的 users 类型的对象，下面就是一个资源清单示例，它定义了一个名为 admin 的 users 对象：
apiVersion: auth.zuoyang.tech/v1alpha1
kind: User
metadata:
name: admin
namespace: default
spec:
userID: 1
email: 1635230007@qq.com
groups:
- superusers
- adminstrators
password: zuoyang.tech
[root@k8s-master01-test-2-26 ~]# kubectl apply -f user-cr-demo.yaml
user.auth.zuoyang.tech/admin created
"kubectl get users" 可以查看自定义资源的状态信息：
[root@k8s-master01-test-2-26 ~]# kubectl get users -n default
NAME AGE
admin 2m39s
[root@k8s-master01-test-2-26 ~]# kubectl describe users
Name: admin
Namespace: default
Labels: <none>
Annotations: <none>
API Version: auth.zuoyang.tech/v1alpha1
Kind: User
Metadata:
Creation Timestamp: 2022-07-07T07:59:20Z
Generation: 1
Managed Fields:
API Version: auth.zuoyang.tech/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:email:
f:groups:
f:password:
f:userID:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2022-07-07T07:59:20Z
Resource Version: 390747
UID: e18c3482-cf85-4898-9ad7-340d47d87572
Spec:
Email: 1635230007@qq.com
Groups:
superusers
adminstrators
Password: zuoyang.tech
User ID: 1
Events: <none>
[root@k8s-master01-test-2-26 ~]#
根据 API 对象 GVR 格式的 URL 规范，users 资源的对象 admin 的引用路径为 /apis/auth.zuoyang.tech/v1alpha1/namespaces/default/users/admin，这一点已通过 descreibe 命令予以证实。而要删除自定义的 users 对象 admin，只要使用通用格式的 kubeclt 命令即可，如 "kubectl delete users admin"，或者使用陈述式对象配置命令 “kubectl delete -f <filename>”。
[root@k8s-master01-test-2-26 ~]# kubectl delete users admin
user.auth.zuoyang.tech "admin" deleted
[root@k8s-master01-test-2-26 ~]# kubectl get users -n default
No resources found in default namespace.
[root@k8s-master01-test-2-26 ~]#
二、定义资源格式验证
除了对操作请求进行身份验证和授权检查之外，对象配置的变动在存储 etcd 之前还需要经由准入控制器的核验，尤其是验证型（validation）控制器会检查传入的对象格式是否符合有效格式，包括是否设定了不符合定义的数据类型值，以及是否违反了字段的限制规则等。
Kubernetes 自 1.9 版本起支持为 CRD 定义验证（validation 字段）机制用以定义 CRD 可用的有效字段等，这在将设计的自定义资源公开时非常重要。自定义资源可使用 OpenAPI 模式声明验证规则，该模式是 JSON 模式的子集。需要注意的是，OpenAPI 架构并不能支持 JSON 架构的所有功能，而 CRD 验证也不能支持 OpenAPI 架构的所有功能，不过对大多数情况来说，它足够使用了。
下面是一个配置清单（就是上面那个yaml）。它分别定义了 userID、groups、email 和 password 字段的数据类型，并指定了 userID 字段的取值范围，以及 password 字段的数据格式，而且还通过 required 指定 userID 和 groups 是必选字段：
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: users.auth.zuoyang.tech
spec:
group: auth.zuoyang.tech
names:
kind: User
plural: users
singular: user
shortNames:
- u
scope: Namespaced
versions:
- served: true
storage: true
name: v1alpha1
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
userID:
type: integer
minimum: 1
maximum: 65535
groups :
type: array
items:
type: string
email:
type: string
password:
type: string
format: password
required: ["userID","groups"]
限制了每个字段的数据类型之后，验证机制会检查创建或修改操作中提供的每个字段值等是否违反了格式要求，任何的核验失败都会导致写入操作被拒绝。
自 Kubernetes 1.11 版本开始，kubeclt 即使用服器侧对象信息打印机制，这意味着将由 API Server 决定 kubectl get 命令结果会显示哪些字段。在 CRD 资源中，可在 spec.additionalPrinterColumns 中嵌套定义在对象的详细信息中要打印的字段列表。
下面的配置清单片段定义了要为 users 类型的对象显示相关的四个字段，将其合并至前面定义的 CRD 对象 users 配置清单的 spec 字段中，重新应用到集群中并确保其正常生效：
[root@k8s-master01-test-2-26 ~]# kubectl explain customresourcedefinition.spec.versions.additionalPrinterColumns
KIND: CustomResourceDefinition
VERSION: apiextensions.k8s.io/v1
RESOURCE: additionalPrinterColumns <[]Object>
DESCRIPTION:
additionalPrinterColumns specifies additional columns returned in Table
output. See
https://kubernetes.io/docs/reference/using-api/api-concepts/#receiving-resources-as-tables
for details. If no columns are specified, a single column displaying the
age of the custom resource is used.
CustomResourceColumnDefinition specifies a column for server side printing.
FIELDS:
description <string>
description is a human readable description of this column.
format <string>
format is an optional OpenAPI type definition for this column. The 'name'
format is applied to the primary identifier column to assist in clients
identifying column is the resource name. See
https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md#data-types
for details.
jsonPath <string> -required-
jsonPath is a simple JSON path (i.e. with array notation) which is
evaluated against each custom resource to produce the value for this
column.
name <string> -required-
name is a human readable name for the column.
priority <integer>
priority is an integer defining the relative importance of this column
compared to others. Lower numbers are considered higher priority. Columns
that may be omitted in limited space scenarios should be given a priority
greater than 0.
type <string> -required-
type is an OpenAPI type definition for this column. See
https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md#data-types
for details.
[root@k8s-master01-test-2-26 ~]#
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: users.auth.zuoyang.tech
spec:
group: auth.zuoyang.tech
names:
kind: User
plural: users
singular: user
shortNames:
- u
scope: Namespaced
versions:
- served: true
storage: true
name: v1alpha1
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
userID:
type: integer
minimum: 1
maximum: 65535
groups :
type: array
items:
type: string
email:
type: string
password:
type: string
format: password
required: ["userID","groups"]
additionalPrinterColumns:
- name: userID
type: integer
description: The user ID.
jsonPath: .spec.userID
- name: groups
type: string
description: The goups of the user.
jsonPath: .spec.goups
- name: email
type: string
description: The email of the user.
jsonPath: .spec.email
- name: password
type: string
description: The password of the user account.
jsonPath: .spec.password
[root@k8s-master01-test-2-26 ~]# kubectl get crd users.auth.zuoyang.tech
NAME CREATED AT
users.auth.zuoyang.tech 2022-07-07T08:54:41Z
[root@k8s-master01-test-2-26 ~]# kubectl describe crd users.auth.zuoyang.tech
Name: users.auth.zuoyang.tech
Namespace:
Labels: <none>
Annotations: <none>
API Version: apiextensions.k8s.io/v1
Kind: CustomResourceDefinition
Metadata:
Creation Timestamp: 2022-07-07T08:54:41Z
Generation: 1
Managed Fields:
API Version: apiextensions.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
f:acceptedNames:
f:kind:
f:listKind:
f:plural:
f:shortNames:
f:singular:
f:conditions:
k:{"type":"Established"}:
.:
f:lastTransitionTime:
f:message:
f:reason:
f:status:
f:type:
k:{"type":"NamesAccepted"}:
.:
f:lastTransitionTime:
f:message:
f:reason:
f:status:
f:type:
Manager: kube-apiserver
Operation: Update
Subresource: status
Time: 2022-07-07T08:54:41Z
API Version: apiextensions.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
f:conversion:
.:
f:strategy:
f:group:
f:names:
f:kind:
f:listKind:
f:plural:
f:shortNames:
f:singular:
f:scope:
f:versions:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2022-07-07T08:54:41Z
Resource Version: 405679
UID: c98f0ba2-7fb4-4e0f-85f4-a55173b5aa64
Spec:
Conversion:
Strategy: None
Group: auth.zuoyang.tech
Names:
Kind: User
List Kind: UserList
Plural: users
Short Names:
u
Singular: user
Scope: Namespaced
Versions:
Additional Printer Columns:
Description: The user ID.
Json Path: .spec.userID
Name: userID
Type: integer
Description: The goups of the user.
Json Path: .spec.goups
Name: groups
Type: string
Description: The email of the user.
Json Path: .spec.email
Name: email
Type: string
Description: The password of the user account.
Json Path: .spec.password
Name: password
Type: string
Name: v1alpha1
Schema:
openAPIV3Schema:
Properties:
Spec:
Properties:
Email:
Type: string
Groups:
Items:
Type: string
Type: array
Password:
Format: password
Type: string
User ID:
Maximum: 65535
Minimum: 1
Type: integer
Required:
userID
groups
Type: object
Type: object
Served: true
Storage: true
Status:
Accepted Names:
Kind: User
List Kind: UserList
Plural: users
Short Names:
u
Singular: user
Conditions:
Last Transition Time: 2022-07-07T08:54:41Z
Message: no conflicts found
Reason: NoConflicts
Status: True
Type: NamesAccepted
Last Transition Time: 2022-07-07T08:54:41Z
Message: the initial names have been accepted
Reason: InitialNamesAccepted
Status: True
Type: Established
Stored Versions:
v1alpha1
Events: <none>
[root@k8s-master01-test-2-26 ~]#
三、子资源
前面自定义资源对象 admin 在其详细状态信息输出中没有类似核心资源的 status 字段，该字段是一种用于保存对象当前状态的子资源。
在 Kubernetes 系统的声明式 API 中，status 字段至关重要，它由 Kubernetes 系统自行维护，相关的控制器循环中持续与 API Server 进行通信，并负责确保 status 字段中的状态匹配 spec 字段中定义的期望状态。
在 CRD 中为自定义资源启用 status 字段的方式非常简单，只需要为其定义 spec.subresources.status 字段即可，其内嵌的字段等由系统自行维护，用户无须提供任何额外的配置。它的使用格式如下：
crd.spec.versions.subresources 字段定义如下：
[root@k8s-master01-test-2-26 ~]# kubectl explain crd.spec.versions.subresources
KIND: CustomResourceDefinition
VERSION: apiextensions.k8s.io/v1
RESOURCE: subresources <Object>
DESCRIPTION:
subresources specify what subresources this version of the defined custom
resource have.
CustomResourceSubresources defines the status and scale subresources for
CustomResources.
FIELDS:
scale <Object>
scale indicates the custom resource should serve a `/scale` subresource
that returns an `autoscaling/v1` Scale object.
status <map[string]>
status indicates the custom resource should serve a `/status` subresource.
When enabled: 1. requests to the custom resource primary endpoint ignore
changes to the `status` stanza of the object. 2. requests to the custom
resource `/status` subresource ignore changes to anything other than the
`status` stanza of the object.
[root@k8s-master01-test-2-26 ~]#
crd.spec.versions.subresources.scale 字段定义如下：
[root@k8s-master01-test-2-26 ~]# kubectl explain crd.spec.versions.subresources.scale
KIND: CustomResourceDefinition
VERSION: apiextensions.k8s.io/v1
RESOURCE: scale <Object>
DESCRIPTION:
scale indicates the custom resource should serve a `/scale` subresource
that returns an `autoscaling/v1` Scale object.
CustomResourceSubresourceScale defines how to serve the scale subresource
for CustomResources.
FIELDS:
labelSelectorPath <string>
labelSelectorPath defines the JSON path inside of a custom resource that
corresponds to Scale `status.selector`. Only JSON paths without the array
notation are allowed. Must be a JSON Path under `.status` or `.spec`. Must
be set to work with HorizontalPodAutoscaler. The field pointed by this JSON
path must be a string field (not a complex selector struct) which contains
a serialized label selector in string form. More info:
https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions#scale-subresource
If there is no value under the given path in the custom resource, the
`status.selector` value in the `/scale` subresource will default to the
empty string.
specReplicasPath <string> -required-
specReplicasPath defines the JSON path inside of a custom resource that
corresponds to Scale `spec.replicas`. Only JSON paths without the array
notation are allowed. Must be a JSON Path under `.spec`. If there is no
value under the given path in the custom resource, the `/scale` subresource
will return an error on GET.
statusReplicasPath <string> -required-
statusReplicasPath defines the JSON path inside of a custom resource that
corresponds to Scale `status.replicas`. Only JSON paths without the array
notation are allowed. Must be a JSON Path under `.status`. If there is no
value under the given path in the custom resource, the `status.replicas`
value in the `/scale` subresource will default to 0.
[root@k8s-master01-test-2-26 ~]#
crd.spec.versions.subresources.status字段定义如下：
KIND: CustomResourceDefinition
VERSION: apiextensions.k8s.io/v1
DESCRIPTION:
status indicates the custom resource should serve a `/status` subresource.
When enabled: 1. requests to the custom resource primary endpoint ignore
changes to the `status` stanza of the object. 2. requests to the custom
resource `/status` subresource ignore changes to anything other than the
`status` stanza of the object.
CustomResourceSubresourceStatus defines how to serve the status subresource
for CustomResources. Status is represented by the `.status` JSON path
inside of a CustomResource. When set, * exposes a /status subresource for
the custom resource * PUT requests to the /status subresource take a custom
resource object, and ignore changes to anything except the status stanza *
PUT/POST/PATCH requests to the custom resource ignore changes to the status
stanza
CustomResourceSubresourceStatus defines how to serve the status subresource
for CustomResources. Status is represented by the `.status` JSON path
inside of a CustomResource. When set, * exposes a /status subresource for
the custom resource * PUT requests to the /status subresource take a custom
resource object, and ignore changes to anything except the status stanza *
PUT/POST/PATCH requests to the custom resource ignore changes to the status
stanza
[root@k8s-master01-test-2-26 ~]#
将前面配置清单中 CRD 对象 users 的配置中，再改造下，完成活动对象的修改即可在其实例化出的对象上通过 /status 获取状态信息，例如，对象 admin 的状态引用路径为 /apis/auth.zuoyang.tech/v1alpha1/namespaces/default/users/admin/status。在没有相应资源控制器的情形下，活动对象状态的非计划内变动既不会实时反映到 status 中，也不会问 spec 定义的期望状态转换。
事实上，如果有相应的资源控制器维护自定义的资源类型时，还可以配置自定义资源对象时使用 scale 子资源和 status 子资源协同实现类似 Deployment 或 StatefulSet 等控制器一样对象规模的伸缩功能。例子如下：
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: users.auth.zuoyang.tech
spec:
group: auth.zuoyang.tech
names:
kind: User
plural: users
singular: user
shortNames:
- u
scope: Namespaced
versions:
- served: true
storage: true
name: v1alpha1
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
userID:
type: integer
minimum: 1
maximum: 65535
groups :
type: array
items:
type: string
email:
type: string
password:
type: string
format: password
required: ["userID","groups"]
additionalPrinterColumns:
- name: userID
type: integer
description: The user ID.
jsonPath: .spec.userID
- name: groups
type: string
description: The goups of the user.
jsonPath: .spec.goups
- name: email
type: string
description: The email of the user.
jsonPath: .spec.email
- name: password
type: string
description: The password of the user account.
jsonPath: .spec.password
subresources:
status: {}
scale:
labelSelectorPath: .status.labelSelector
specReplicasPath: .spec.replicas
statusReplicasPath: .status.replicas
crd.spec.versions.subresources.scale 的各内嵌字段介绍如下：
- specReplicasPath <string> -required-: 引用定义在 spec 中用于表示期望的副本数量的字段 jsonPath 格式，如 .spec.relicas。
- statusReplicasPath <string> -required-： 引用保存在 status 中用于表示对象的当前副本数量的字段，jsonPath 格式，如 .status.replicas。
- labelSelectorPath <string>: 可选字段，但若要与 HPA 结合使用则是必选字段，用于引用 status 中用于表示使用的标签选择器字段，jsonPath 格式，如 .status.labelSelector。
为某 CRD 资源定义 scale 子资源之后，即可实例化出支持规模伸缩的自定义资源对象。但必须存在一个相应的资源控制器来持续维护相应的自定义资源对象，以确保其当前状态匹配期望的状态。满足条件后，类似于 Deployment 资源对象等，使用 "kubectl scale" 命令就能完成对自定义资源对象规模的手动伸缩。
四、使用资源类别
类别（categories）[英[ˈkætɪgəriz]美[ˈkætəˌgɔriz]] 是 Kubernetes 1.10 版本引入的一种分组组织自定义资源的方法，定义 CRD 对象时为其指定一个或多个类别，可以通过 kubectl get <category-name> 命令列出该类别中的所有自定义资源对象，all 就是一个常用的内建资源类别。
例如，为前面定义的 CRD 对象 users 的 spec.names 字段中额外内嵌如下配置，便能使得 users 资源类型下的所有对象都隶属于 all 类别：
[root@k8s-master01-test-2-26 ~]# kubectl explain crd.spec.names
KIND: CustomResourceDefinition
VERSION: apiextensions.k8s.io/v1
RESOURCE: names <Object>
DESCRIPTION:
names specify the resource and kind names for the custom resource.
CustomResourceDefinitionNames indicates the names to serve this
CustomResourceDefinition
FIELDS:
categories <[]string>
categories is a list of grouped resources this custom resource belongs to
(e.g. 'all'). This is published in API discovery documents, and used by
clients to support invocations like `kubectl get all`.
kind <string> -required-
kind is the serialized kind of the resource. It is normally CamelCase and
singular. Custom resource instances will use this value as the `kind`
attribute in API calls.
listKind <string>
listKind is the serialized kind of the list for this resource. Defaults to
"`kind`List".
plural <string> -required-
plural is the plural name of the resource to serve. The custom resources
are served under `/apis/<group>/<version>/.../<plural>`. Must match the
name of the CustomResourceDefinition (in the form
`<names.plural>.<group>`). Must be all lowercase.
shortNames <[]string>
shortNames are short names for the resource, exposed in API discovery
documents, and used by clients to support invocations like `kubectl get
<shortname>`. It must be all lowercase.
singular <string>
singular is the singular name of the resource. It must be all lowercase.
Defaults to lowercased `kind`.
[root@k8s-master01-test-2-26 ~]#
explain crd.spec.names.categories 内嵌字段定义如下：
[root@k8s-master01-test-2-26 ~]# kubectl explain crd.spec.names.categories
KIND: CustomResourceDefinition
VERSION: apiextensions.k8s.io/v1
FIELD: categories <[]string>
DESCRIPTION:
categories is a list of grouped resources this custom resource belongs to
(e.g. 'all'). This is published in API discovery documents, and used by
clients to support invocations like `kubectl get all`.
[root@k8s-master01-test-2-26 ~]#
五、多版本支持
Kubernetes 原生资源的一个引人瞩目的特性是它们能够在 API 版本之间自动和透明地迁移。资源的使用者可以使用混合 API 版本，并且都能够获得他们所期望的资源版本。而 Kubernetes 1.11 版本开始，CRD 支持多个版本，但它们之间的转换必须手动完成。
在 CRD 上使用多版本机制时，将 spec.version 字段替换为 spec.versions 字段，并将支持的各版本以对象列表的形式给出定义即可。不过，多版本并存时，仅其中一个版本且必须有一个版本应标记为存储（customresourcedefinition.spec.versions.storage 字段）版本。
下面的配置清单片断中定义了两个 API 版本，其中仅 v1alpha1 标记为了 storage:
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: users.auth.zuoyang.tech
spec:
group: auth.zuoyang.tech
names:
kind: User
plural: users
singular: user
shortNames:
- u
scope: Namespaced
versions:
- served: true
storage: true
name: v1alpha1
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
userID:
type: integer
minimum: 1
maximum: 65535
groups :
type: array
items:
type: string
email:
type: string
password:
type: string
format: password
required: ["userID","groups"]
- served: true
storage: false
name: v1alpha2
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
userID:
type: integer
minimum: 1
maximum: 65535
groups :
type: array
items:
type: string
email:
type: string
password:
type: string
format: password
required: ["userID","groups"]
additionalPrinterColumns:
- name: userID
type: integer
description: The user ID.
jsonPath: .spec.userID
- name: groups
type: string
description: The goups of the user.
jsonPath: .spec.goups
- name: email
type: string
description: The email of the user.
jsonPath: .spec.email
- name: password
type: string
description: The password of the user account.
jsonPath: .spec.password
subresources:
status: {}
scale:
labelSelectorPath: .status.labelSelector
specReplicasPath: .spec.replicas
statusReplicasPath: .status.replicas
六、自定义控制器基础
仅借助于 CRD 完成资源自定义本身并不能为用户带来太多的价值，它只是资源类型的定义，只是提供了 JSON 格式的数据范式及存储相关数据的能力，至于如何执行数据相关的业务逻辑，时刻确保将 status 中的状态移向 spec 中的状态则是由封装于控制器中的代码来负责实现的。
相应的，为自定义资源类型提供业务逻辑代码的控制器需要由用户自行开发，并运行于 API Server 的客户端程序（通常是托管运行于 kubernetes 系统之上，类似于 Ingress 控制器），这就是所谓的自定义控制器（Custom Controller）。
换句话讲，某特定的 CRD 资源的相关对象发生变动时，如何确保它的当前状态不断地接近期望的状态并非 API Server 的功能，而是相关的专用控制器组件。
事实上，自定义控制器不仅仅能够用于管理 CRD 资源，用户也完全可以仅针对系统内建的核心类型对象开啊更高级的控制器，这些对象类型包括 Service、Deployment、ConfigMap。不过，对于每个 CRD 的自定义类别来说，至少应该存在一个相关的自定义控制器。
简单的来说，控制器负责持续监视资源变动，根据资源的 spec 及 status 中的信息执行某些操作逻辑，并将执行结果更新至资源的 status 中。自定义控制器同样担负着类似的职责，只不过一个特定的控制器通常仅负责一部分特定的资源类型，并执行专有管理逻辑。
Kubernetes 系统内建了许多控制器，如 NodeController、ServiceController 等，它们打包在一起并作为一个统一守护进程 kube-controller-manager。这些控制器基本上都遵循同一钟模式以完成资源管理操作，该模式通常可描述为三种特性：
-
- 声明式 API：用户自定义期望的状态而非要执行的特定操作，如使用 kubectl apply 命令将请求发送至 API Server 存储于 etcd 中，并由 API Server 做出处理响应。
- 异步：客户端的请求于 API Server 存储完成之后即返回成功信息，而无须等待控制器执行的和解循环结束。
- 水平处理（level-based）：同一对象有多次变动事件待处理时，控制器仅需要处理其最新一次的变动。这种根据最新观察结果而非历史变化来和解 status 和 spec 的机制即为水平处理机制。
定义控制器实现自定义资源管理行为的最佳方法同样是遵循此类控制器模式进行程序开发，目前，大部分开发接口都是基于 Golang 语言来实现的，相关代码位于客户端库的 client-go/go/tools/cache 和 client-go/util/workqueue 目录中。
简单的来说，控制器包含两个重要组件，Informer/SharedInformer 和 Workqueue，前者负责监视资源对象当前状态的更改，并将事件发送至后者，而后由处理函数进行处理，如下图：
Informer 主要由 Listwatcher、ResourceEvenetHandler 和 ResyncPeriod 三类函数组件进行构造。
-
- Listwather: 是应用于特定名称空间中特定资源对象的列表函数（listFunc）和监视函数（watchFunc），并结合字段选择器为控制器精确限定关注的资源对象。
- ResourceEvenetHandler: 负责处理资源对象状态改变产生的相关通知，其分别使用 AddFunc、UpdateFunc、DeleteFunc 三个函数完成对象的创建、更新和删除操作。
- ResyncPeriod 定义控制器遍历缓存中的所有项目并触发运行 UpdateFunc 的频率，即和解循环的执行频度。