• 【原创】Kuberneters-HelmV3.3.1入门介绍及实践


    一、为什么需要Helm?

    Kubernetes目前已成为容器编排的事实标准,随着传统架构向微服务容器化架构的转变,从一个巨大的单体的应用切分为多个微服务,每个微服务可独立部署和扩展,实现了敏捷开发和快速部署,但是由于从大一个应用变成了多个微服务,导致服务数大幅增加,对于Kubernetes来说,针对每个服务需要部署如deployment、statufulset、service、pod 等资源文件,而对于一个复杂的应用来说,可能会有很多类似上面的资源文件,其中还有更新或回滚的需求,若要执行,需要修改和维护相关的大量资源文件,这时候,如何针对每个服务涉及到资源集合到一个整体来实现部署、升级和回滚是亟待需要解决的难题,Helm就是在这个背景下诞生的。

    二、Helm的简介和使用场景

    那么Helm是如何解决上述难题的呢?它其实是将Kubernetes资源(如deployment、statufulset、service、pod) 打包到一个chart中,而chart被打包并推送保存到chart仓库(repo),然后通过chart仓库用来存储和分享chart包。Helm可直接拉取并安装chart包,生成helm维度整体的应用,其应用会包含如deployment、statufulset、service、pod的资源,并支持以一个整体Helm应用的维度来进行版本控制、打包、发布、删除、更新等操作。针对用户来说主要适用如下场景:

    1、将常用搭配、标准化的多个资源文件放在同一个chart包中,方便统一的版本控制、安装、部署、升级、回滚和删除

    2、可以大大简化了使用Kubernetes部署的难度,降低了用户使用门槛,只需配置参数即可一键部署

    三、Helm实践

    本次演示Helm的版本号如下:

    [root@k8s-master my-second-helm]# helm version
    version.BuildInfo{Version:"v3.3.1", GitCommit:"249e5215cde0c3fa72e27eb7a30e8d55c9696144", GitTreeState:"clean", GoVersion:"go1.14.7"}

    1、安装

    1)制作chart包的形式

    Helm 应用的安装既可以从远程的repo进行拉取chart包进行安装,repo中提供了很多官方的chart包,若无个性化的需求,可以类似公开镜像一样进行拉取并安装,若需要单独调整,可以在已有的chart包进行更新再次打包,然后拉取更新后的chart包进行安装,接下来先来实践下后者,我们先新建一个helm的目录,然后在此目录下执行如下命令:

    [root@k8s-master helm]# helm create my-second-helm 
    Creating my-second-helm
    [root@k8s-master helm]# ls
    my-first-helm  my-second-helm
    [root@k8s-master helm]# cd my-second-helm/
    [root@k8s-master my-second-helm]# ls
    charts  Chart.yaml  templates  values.yaml
    [root@k8s-master helm]# tree my-second-helm
    my-second-helm
    ├── charts
    ├── Chart.yaml
    ├── templates
    │   ├── deployment.yaml
    │   ├── _helpers.tpl
    │   ├── hpa.yaml
    │   ├── ingress.yaml
    │   ├── NOTES.txt
    │   ├── serviceaccount.yaml
    │   ├── service.yaml
    │   └── tests
    │       └── test-connection.yaml
    └── values.yaml
    
    3 directories, 10 files

    我们在前面的操作中只是指定了这个文件夹的名字my-second-helm,但是在这个文件下默认创建了多个目录和文件,接下里我们一起看下默认的文件中都包含哪些内容,各自充当什么作用,先一起看下Chart.yaml这个文件

    [root@k8s-master my-second-helm]# cat Chart.yaml 
    apiVersion: v2
    name: my-second-helm
    description: A Helm chart for Kubernetes
    
    # A chart can be either an 'application' or a 'library' chart.
    #
    # Application charts are a collection of templates that can be packaged into versioned archives
    # to be deployed.
    #
    # Library charts provide useful utilities or functions for the chart developer. They're included as
    # a dependency of application charts to inject those utilities and functions into the rendering
    # pipeline. Library charts do not define any templates and therefore cannot be deployed.
    type: application
    
    # This is the chart version. This version number should be incremented each time you make changes
    # to the chart and its templates, including the app version.
    # Versions are expected to follow Semantic Versioning (https://semver.org/)
    version: 0.1.0
    
    # This is the version number of the application being deployed. This version number should be
    # incremented each time you make changes to the application. Versions are not expected to
    # follow Semantic Versioning. They should reflect the version the application is using.
    appVersion: 1.16.0

    可以看到Chart.yaml这个文件主要是用来对chart进行的描述,声明了当前 Chart 的名称、版本等基本信息,如果我们自己制作的chart有相关的描述都可以写入到文件,这些信息会在该 Chart 被放入仓库后,供用户浏览检索,方便用户了解这个chart。

    templates这个目录下,我们看到了大家比较熟悉的资源,我们看下deployment.yaml的构成

    [root@k8s-master templates]# cat deployment.yaml 
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: {{ include "my-second-helm.fullname" . }}
      labels:
        {{- include "my-second-helm.labels" . | nindent 4 }}
    spec:
    {{- if not .Values.autoscaling.enabled }}
      replicas: {{ .Values.replicaCount }}
    {{- end }}
      selector:
        matchLabels:
          {{- include "my-second-helm.selectorLabels" . | nindent 6 }}
      template:
        metadata:
        {{- with .Values.podAnnotations }}
          annotations:
            {{- toYaml . | nindent 8 }}
        {{- end }}
          labels:
            {{- include "my-second-helm.selectorLabels" . | nindent 8 }}
        spec:
          {{- with .Values.imagePullSecrets }}
          imagePullSecrets:
            {{- toYaml . | nindent 8 }}
          {{- end }}
          serviceAccountName: {{ include "my-second-helm.serviceAccountName" . }}
          securityContext:
            {{- toYaml .Values.podSecurityContext | nindent 8 }}
          containers:
            - name: {{ .Chart.Name }}
              securityContext:
                {{- toYaml .Values.securityContext | nindent 12 }}
              image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
              imagePullPolicy: {{ .Values.image.pullPolicy }}
              ports:
                - name: http
                  containerPort: 80
                  protocol: TCP
              livenessProbe:
                httpGet:
                  path: /
                  port: http
              readinessProbe:
                httpGet:
                  path: /
                  port: http
              resources:
                {{- toYaml .Values.resources | nindent 12 }}
          {{- with .Values.nodeSelector }}
          nodeSelector:
            {{- toYaml . | nindent 8 }}
          {{- end }}
          {{- with .Values.affinity }}
          affinity:
            {{- toYaml . | nindent 8 }}
          {{- end }}
          {{- with .Values.tolerations }}
          tolerations:
            {{- toYaml . | nindent 8 }}
          {{- end }}

    细心的读者可能发现了,这个kind为deployment的资源与我们平常自己写的有些出入,主要是出现了很多类似{}的内容,如:replicas: {{ .Values.replicaCount }},平常我们是用replicas: 数字的形式,而这里已经换成了变量,而整个文件出现了很多类似的变量,想必读者也猜到了原来Helm是通过设置变量来统一管理很多需要输入的值,若有多个文件引用同一个变量,就再也不用同时修改多个文件了,细心的读者可能会发现,即便对K8s不太了解的用户,也可以仅设置变量的值也可以轻松部署应用,岂不可以大大降低部署的难度,确实如此,这就是helm的一个主要的特点,让部署变的简单,特别是常用的标准的配置我们可将其制作成chart一键部署,让我们再回到replicas: {{ .Values.replicaCount }}这个变量上,既然存在变量,那变量的设值在哪里呢? 我们可以看到变量的组成中存在Values,而和我们templates同级别的目录选还存在一个很重要的文件values.yaml,我们看下这个文件里面的内容:

    [root@k8s-master my-second-helm]# cat values.yaml 
    # Default values for my-second-helm.
    # This is a YAML-formatted file.
    # Declare variables to be passed into your templates.
    
    replicaCount: 1
    
    image:
      repository: nginx
      pullPolicy: IfNotPresent
      # Overrides the image tag whose default is the chart appVersion.
      tag: ""
    
    imagePullSecrets: []
    nameOverride: ""
    fullnameOverride: ""
    
    serviceAccount:
      # Specifies whether a service account should be created
      create: true
      # Annotations to add to the service account
      annotations: {}
      # The name of the service account to use.
      # If not set and create is true, a name is generated using the fullname template
      name: ""
    
    podAnnotations: {}
    
    podSecurityContext: {}
      # fsGroup: 2000
    
    securityContext: {}
      # capabilities:
      #   drop:
      #   - ALL
      # readOnlyRootFilesystem: true
      # runAsNonRoot: true
      # runAsUser: 1000
    
    service:
      type: ClusterIP
      port: 80
    
    ingress:
      enabled: false
      annotations: {}
        # kubernetes.io/ingress.class: nginx
        # kubernetes.io/tls-acme: "true"
      hosts:
        - host: chart-example.local
          paths: []
      tls: []
      #  - secretName: chart-example-tls
      #    hosts:
      #      - chart-example.local
    
    resources: {}
      # We usually recommend not to specify default resources and to leave this as a conscious
      # choice for the user. This also increases chances charts run on environments with little
      # resources, such as Minikube. If you do want to specify resources, uncomment the following
      # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
      # limits:
      #   cpu: 100m
      #   memory: 128Mi
      # requests:
      #   cpu: 100m
      #   memory: 128Mi
    
    autoscaling:
      enabled: false
      minReplicas: 1
      maxReplicas: 100
      targetCPUUtilizationPercentage: 80
      # targetMemoryUtilizationPercentage: 80
    
    nodeSelector: {}
    
    tolerations: []
    
    affinity: {}

    可以看到这个文件包含的是整个chart中各个资源下变量的值,如replicaCount: 1,前面replicas: {{ .Values.replicaCount }},因此上面的deployment的replicas为1,其他变量的值类似,其中{}表示未指定,用户可根据需要进行自定义。

    接下来我们通过vim 修改下value.yaml文件中replicaCount: 2内容然后进行安装部署,但是安装部署之前需要先打包,我们在上面的操作都是针对目录和文件的,helm只能针对包进行安装,因此我们需先打包,打包之前,最好先验证下我们之前在各个文件输入的值是否合法,可以通过操作进行校验

    [root@k8s-master my-second-helm]# helm lint --strict my-second-helm
    ==> Linting my-second-helm
    Error unable to check Chart.yaml file in chart: stat my-second-helm/Chart.yaml: no such file or directory
    
    Error: 1 chart(s) linted, 1 chart(s) failed
    [root@k8s-master my-second-helm]# cd ..
    [root@k8s-master helm]# ls
    my-first-helm  my-second-helm
    [root@k8s-master helm]# helm lint --strict my-second-helm
    ==> Linting my-second-helm
    [INFO] Chart.yaml: icon is recommended
    
    1 chart(s) linted, 0 chart(s) failed

    可以看到在进行校验的时候,需要回到chart的根目录,0 chart(s) failed显示这个chart的之前输入的value值应无问题,接下来可以执行打包操作了,如下所示,红色字体就是我们打包后的结果

    [root@k8s-master helm]# helm package my-second-helm
    Successfully packaged chart and saved it to: /home/James/zhanglei/helm/my-second-helm-0.1.0.tgz
    [root@k8s-master helm]# ls
    my-first-helm  my-second-helm  my-second-helm-0.1.0.tgz

    这个就是我们所说的chart包,接下来终于可以执行安装的操作了

    [root@k8s-master helm]# helm install my-second-helm-test my-second-helm-0.1.0.tgz
    NAME: my-second-helm-test 
    LAST DEPLOYED: Sat Sep 12 21:48:18 2020
    NAMESPACE: default
    STATUS: deployed                # 状态
    REVISION: 1                      
    NOTES:
    1. Get the application URL by running these commands:
      export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=my-second-helm,app.kubernetes.io/instance=my-second-helm-test" -o jsonpath="{.items[0].metadata.name}")
      echo "Visit http://127.0.0.1:8080 to use your application"
      kubectl --namespace default port-forward $POD_NAME 8080:80
    [root@k8s-master helm]# helm list
    NAME                   NAMESPACE    REVISION    UPDATED                                    STATUS    CHART                   APP VERSION
    my-second-helm-test    default      1           2020-09-12 21:48:18.480694046 +0800 CST    deployed  my-second-helm-0.1.0    1.16.0 

    可以看到在install后面指定的就是部署名称,在helm list中可追溯到时通过哪个版本的chart包进行的安装,可以看到状态为deployed,我们之前设定了deployment的副本实例数为2,我们一起来验证下:

    [root@k8s-master helm]# kubectl get pod -o wide |grep my-second-helm
    my-second-helm-test-566f5d8757-x27zf 1/1 Running 0 5m25s 10.122.235.244 k8s-master <none> <none>
    my-second-helm-test-566f5d8757-zdm47 1/1 Running 0 5m25s 10.122.235.247 k8s-master <none> <none>

    可以看到2个Pod已经running了,我们测试这个nginx如下所示为正常了,也就是说我们通过helm install chart包的方式成功安装了NGINX的应用

    [root@k8s-master helm]# curl 10.122.235.244:80
    <!DOCTYPE html>
    <html>
    <head>
    <title>Welcome to nginx!</title>
    <style>
        body {
             35em;
            margin: 0 auto;
            font-family: Tahoma, Verdana, Arial, sans-serif;
        }
    </style>
    </head>
    <body>
    <h1>Welcome to nginx!</h1>
    <p>If you see this page, the nginx web server is successfully installed and
    working. Further configuration is required.</p>
    
    <p>For online documentation and support please refer to
    <a href="http://nginx.org/">nginx.org</a>.<br/>
    Commercial support is available at
    <a href="http://nginx.com/">nginx.com</a>.</p>
    
    <p><em>Thank you for using nginx.</em></p>
    </body>

    至此,我们演示了通过在本地目录里面新建了一个默认的chart目录,然后对value中的值进行修改,最后的再次打包,然后安装的全过程,有读者不免发问了,这个过程太麻烦了,我只需要类型像公有镜像,并不想制作chart,是否可以远程直接拉取chart包安装?答案是:可以!接下来我将介绍第二种创建helm 应用的方式。

    2)远程拉取chart包

    但是在拉取公有的chart包之前,需要先配置好chart repo,如下所示是我已经配置好的chart repo,里面提供常用的chart包

    [root@k8s-master my-first-helm]# helm repo list
    NAME         URL                                                 
    stable       http://mirror.azure.cn/kubernetes/charts            
    incubator    http://mirror.azure.cn/kubernetes/charts-incubator  
    svc-cat      http://mirror.azure.cn/kubernetes/svc-catalog-charts

    我们在正式拉取chart包之前,很多时候想在对应的仓库中先查找下有想要的chart包,可以通过如下操作进行查看下,比如我想查看下有无nginx的chart

    [root@k8s-master my-first-helm]# helm search repo nginx
    NAME                           CHART VERSION    APP VERSION    DESCRIPTION                                       
    stable/nginx-ingress           1.41.3           v0.34.1        DEPRECATED! An nginx Ingress controller that us...
    stable/nginx-ldapauth-proxy    0.1.4            1.13.5         nginx proxy with ldapauth                         
    stable/nginx-lego              0.3.1                           Chart for nginx-ingress-controller and kube-lego  
    stable/gcloud-endpoints        0.1.2            1              DEPRECATED Develop, deploy, protect and monitor...

    可以看到,通过search命令,可查询到仓库中已有的NGINX的chart包,其中stable是repo的类型,CHART VERSION:chart包的版本,APP VERSION:应用版本,DESCRIPTION:针对此chart包的描述,看到这里是不是有印象,这正式是我们在上文提到Chart.yaml文件里面的信息,接下来我们尝试直接从远程仓库进行拉取并完成安装的操作

    [root@k8s-master helm]# helm install nginx-test stable/nginx-ingress  
    WARNING: This chart is deprecated
    NAME: nginx-test
    LAST DEPLOYED: Sat Sep 12 23:47:59 2020
    NAMESPACE: default
    STATUS: deployed
    REVISION: 1
    TEST SUITE: None
    NOTES:
    *******************************************************************************************************
    * DEPRECATED, please use https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx *
    *******************************************************************************************************
    
    
    The nginx-ingress controller has been installed.
    It may take a few minutes for the LoadBalancer IP to be available.
    You can watch the status by running 'kubectl --namespace default get services -o wide -w nginx-test-nginx-ingress-controller'
    
    An example Ingress that makes use of the controller:
    
      apiVersion: extensions/v1beta1
      kind: Ingress
      metadata:
        annotations:
          kubernetes.io/ingress.class: nginx
        name: example
        namespace: foo
      spec:
        rules:
          - host: www.example.com
            http:
              paths:
                - backend:
                    serviceName: exampleService
                    servicePort: 80
                  path: /
        # This section is only required if TLS is to be enabled for the Ingress
        tls:
            - hosts:
                - www.example.com
              secretName: example-tls
    
    If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:
    
      apiVersion: v1
      kind: Secret
      metadata:
        name: example-tls
        namespace: foo
      data:
        tls.crt: <base64 encoded cert>
        tls.key: <base64 encoded key>
      type: kubernetes.io/tls
    [root@k8s-master helm]# helm list
    NAME                   NAMESPACE    REVISION    UPDATED                                    STATUS    CHART                   APP VERSION
    my-second-helm-test    default      1           2020-09-12 21:48:18.480694046 +0800 CST    deployed  my-second-helm-0.1.0    1.16.0     
    nginx-test             default      1           2020-09-12 23:47:59.084999904 +0800 CST    deployed  nginx-ingress-1.41.3    v0.34.1    

    可以看到helm应用已经部署成功其应用名称为nginx-test。

    2、升级

    1)通过修改文件内容升级

    可以针对chart目录的文件内容更新后再进行升级,如vim values.yaml文件,我们将之前replicaCount的值设置为4

    [root@k8s-master my-second-helm]# vim values.yaml 
    
    # Default values for my-second-helm.
    # This is a YAML-formatted file.
    # Declare variables to be passed into your templates.
    
    replicaCount: 4
    [root@k8s-master my-second-helm]# helm upgrade -f values.yaml my-second-helm-test ./
    Release "my-second-helm-test" has been upgraded. Happy Helming!   # 显示升级成功
    NAME: my-second-helm-test
    LAST DEPLOYED: Sun Sep 13 10:41:27 2020
    NAMESPACE: default
    STATUS: deployed
    REVISION: 5                # release的版本号为5
    NOTES:
    1. Get the application URL by running these commands:
      export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=my-second-helm,app.kubernetes.io/instance=my-second-helm-test" -o jsonpath="{.items[0].metadata.name}")
      echo "Visit http://127.0.0.1:8080 to use your application"
      kubectl --namespace default port-forward $POD_NAME 8080:80
    [root@k8s-master my-second-helm]# kubectl get pod |grep my-second
    my-second-helm-test-566f5d8757-8v8wc        1/1     Running   0          6m32s
    my-second-helm-test-566f5d8757-kfh5s        1/1     Running   0          11h
    my-second-helm-test-566f5d8757-s4286        1/1     Running   0          86s
    my-second-helm-test-566f5d8757-t2hhx        1/1     Running   0          11h

    如上看到4个pod实例已经正常Running了,升级成功。

    2)通过--set参数升级

    其格式为:helm upgrade -f 指定目录/指定文件 --set 指定文件参数=value  部署名称 chart目录

    [root@k8s-master helm]# helm upgrade -f my-second-helm/values.yaml --set replicaCount=3 my-second-helm-test ./my-second-helm
    Release "my-second-helm-test" has been upgraded. Happy Helming!
    NAME: my-second-helm-test
    LAST DEPLOYED: Sun Sep 13 11:08:30 2020
    NAMESPACE: default
    STATUS: deployed
    REVISION: 6              #  版本6
    NOTES:
    1. Get the application URL by running these commands:
      export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=my-second-helm,app.kubernetes.io/instance=my-second-helm-test" -o jsonpath="{.items[0].metadata.name}")
      echo "Visit http://127.0.0.1:8080 to use your application"
      kubectl --namespace default port-forward $POD_NAME 8080:80
    [root@k8s-master helm]# kubectl get pod | grep my-se
    my-second-helm-test-566f5d8757-8v8wc        1/1     Running   0          32m
    my-second-helm-test-566f5d8757-kfh5s        1/1     Running   0          12h
    my-second-helm-test-566f5d8757-t2hhx        1/1     Running   0          12h

    在上个版本5中我们在values.yaml修改replicaCount的值为4,版本6中通过--set的方式我们修改了replicaCount的值为3,这个是否会同步到values.yaml中呢?

    [root@k8s-master my-second-helm]# cat values.yaml 
    # Default values for my-second-helm.
    # This is a YAML-formatted file.
    # Declare variables to be passed into your templates.
    
    replicaCount: 4

    通过查看,replicaCount的值任然是版本5的值,并未发生改变,也就是说通过--set的值并不会影响到chart目录下源文件的值,另外从这种方式的升级来看,我们得知helm应用的升级是针对已经部署后的应用进行升级,并非针对chart版本的版本进行升级,这些地方都需要注意下!

    3、回滚

    升级操作成功后,每升级一次版本都会加1,在上面的例子里版本号已经6了,若我们想回到版本5该如何进行操作呢?我们先看下整个release的部署历史:

    [root@k8s-master helm]# helm history my-second-helm-test
    REVISION    UPDATED                     STATUS        CHART                   APP VERSION    DESCRIPTION     
    1           Sat Sep 12 21:48:18 2020    superseded    my-second-helm-0.1.0    1.16.0         Install complete
    2           Sun Sep 13 00:07:17 2020    superseded    my-second-helm-0.1.0    1.16.0         Upgrade complete
    3           Sun Sep 13 10:30:36 2020    superseded    my-second-helm-0.1.0    1.16.0         Upgrade complete
    4           Sun Sep 13 10:36:21 2020    superseded    my-second-helm-0.1.0    1.16.0         Upgrade complete
    5           Sun Sep 13 10:41:27 2020    superseded    my-second-helm-0.1.0    1.16.0         Upgrade complete
    6           Sun Sep 13 11:08:30 2020    deployed      my-second-helm-0.1.0    1.16.0         Upgrade complete

    可以看到针对同一个release来说,最终生效的只有一个版本,其状态为:deployed,其他版本为superseded的状态,接下来我们将版本6回滚到版本5

    [root@k8s-master my-second-helm]# helm rollback my-second-helm-test 5
    Rollback was a success! Happy Helming!
    [root@k8s-master my-second-helm]# kubectl get pod |grep my-second-test
    [root@k8s-master my-second-helm]# kubectl get pod |grep my-second
    my-second-helm-test-566f5d8757-8v8wc        1/1     Running   0          51m
    my-second-helm-test-566f5d8757-kfh5s        1/1     Running   0          12h
    my-second-helm-test-566f5d8757-njqbn        1/1     Running   0          61s
    my-second-helm-test-566f5d8757-t2hhx        1/1     Running   0          12h

    版本5:4个Pod,版本6:3个Pod,可以看到命令执行完成之后立马就回滚且生效了,而回滚的操作命令也很简单,其格式为:helm rollback 部署名称 版本号

     4、删除

    若想删除已经部署成功的helm应用该如何操作呢?helm应用的删除其chart下的资源如Pod是否会保留?我们带着这些问题实践下:

    [root@k8s-master my-second-helm]# helm delete my-second-helm-test
    release "my-second-helm-test" uninstalled
    [root@k8s-master my-second-helm]# kubectl get all |grep my-second-helm
    [root@k8s-master my-second-helm]# 

    删除的格式为:helm delete 部署名称, 经验证,一旦删除helm的应用,其关联生成的所有资源会被全部删除掉,也就是说,helm提供给用户是一个集合了所需资源的整体应用,其新建和删除也都是以整体应用为维度。

    四、总结

    本文介绍了什么是Helm,在哪些场景下适用Helm,然后介绍了Helm应用的安装、部署、升级、回滚和删除等操作,Helm应用是一个资源的集合,以整体为用户提供服务,通过这个服务,可以大大简化用户部署的难度,另外还针对应用的版本进行了历史管理,升级和回滚的操作减少了用户自己重新部署和维护版本的成本,在产品设计中,我们可将整个部署helm的流程体现在UI化达到进一步降低部署应用的门槛,校验chart》安装chart-》部署-》升级-》回滚-》删除,设计的重点保证这个业务主流程信息可以清晰的传递给用户,针对重点的操作,需要在页面中重点凸显出来,且要方便用户实际进行操作。

     作者简介:云计算容器DockerK8sServerless方向产品经理,学点技术,为更好地设计产品。

                                         

  • 相关阅读:
    DbgPrint格式 输出
    string 类常用函数[转]
    pragma warning[转]
    连接符
    ubuntu ftp server
    关于dex

    Topology中各函数调用顺序
    C# 错误捕捉
    操作word,Excel,PPT
  • 原文地址:https://www.cnblogs.com/gdut1425/p/13657848.html
Copyright © 2020-2023  润新知