• 010.Kubernetes二进制master节点部署kube-apiserver


    一 获取Kubernetes二进制文件

    1.1 master节点服务

    kubernetes master 节点运行如下组件:

    • kube-apiserver
    • kube-scheduler
    • kube-controller-manager
    • kube-nginx

    kube-apiserver、kube-scheduler 和 kube-controller-manager 均以多实例模式运行:

    kube-scheduler 和 kube-controller-manager 会自动选举产生一个 leader 实例,其它实例处于阻塞模式,当 leader 挂了后,重新选举产生新的 leader,从而保证服务可用性。

    kube-apiserver 是无状态的,需要通过 kube-nginx 进行代理访问,从而保证服务可用性。

    1.2 安装Kubernetes

      1 [root@master01 ~]# cd /opt/k8s/work
      2 [root@master01 work]# wget https://storage.googleapis.com/kubernetes-release/release/v1.18.3/kubernetes-server-linux-amd64.tar.gz
      3 [root@master01 work]# tar -xzvf kubernetes-server-linux-amd64.tar.gz
      4 [root@master01 work]# cd kubernetes
      5 [root@master01 kubernetes]# tar -xzvf kubernetes-src.tar.gz

    提示:本步骤操作仅需要在master01节点操作。

    1.3 分发Kubernetes

      1 [root@master01 ~]# cd /opt/k8s/work
      2 [root@master01 work]# source /root/environment.sh
      3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
      4   do
      5     echo ">>> ${master_ip}"
      6     scp -rp kubernetes/server/bin/{apiextensions-apiserver,kube-apiserver,kube-controller-manager,kube-proxy,kube-scheduler,kubeadm,kubectl,kubelet,mounter} root@${master_ip}:/opt/k8s/bin/
      7     ssh root@${master_ip} "chmod +x /opt/k8s/bin/*"
      8   done

    提示:本步骤操作仅需要在master01节点操作。

    二 创建kube-apiserver证书

    2.1 高可用apiserver介绍

    本实验部署一个三实例 kube-apiserver 集群的步骤,它们通过 kube-nginx 进行代理访问,从而保证服务可用性。

    2.2 创建Kubernetes证书和私钥

      1 [root@master01 ~]# cd /opt/k8s/work
      2 [root@master01 work]# source /root/environment.sh
      3 [root@master01 work]# cat > kubernetes-csr.json <<EOF
      4 {
      5   "CN": "kubernetes",
      6   "hosts": [
      7     "127.0.0.1",
      8     "172.24.8.71",
      9     "172.24.8.72",
     10     "172.24.8.73",
     11     "172.24.8.100",
     12     "${CLUSTER_KUBERNETES_SVC_IP}",
     13     "kubernetes",
     14     "kubernetes.default",
     15     "kubernetes.default.svc",
     16     "kubernetes.default.svc.cluster",
     17     "kubernetes.default.svc.cluster.local."
     18   ],
     19   "key": {
     20     "algo": "rsa",
     21     "size": 2048
     22   },
     23   "names": [
     24     {
     25       "C": "CN",
     26       "ST": "Shanghai",
     27       "L": "Shanghai",
     28       "O": "k8s",
     29       "OU": "System"
     30     }
     31   ]
     32 }
     33 EOF		#创建Kubernetes的CA证书请求文件

    提示:本步骤操作仅需要在master01节点操作。

    解释:

    hosts 字段指定授权使用该证书的 IP 和域名列表,这里列出了 master 节点 IP、kubernetes 服务的 IP 和域名;

    kubernetes 服务 IP 是 apiserver 自动创建的,一般是 --service-cluster-ip-range 参数指定的网段的第一个IP,后续可以通过下面命令获取:

    # kubectl get svc kubernetes

      1 [root@master01 ~]# cd /opt/k8s/work
      2 [root@master01 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem 
      3 -ca-key=/opt/k8s/work/ca-key.pem -config=/opt/k8s/work/ca-config.json 
      4 -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes	#生成密钥和证书

    提示:本步骤操作仅需要在master01节点操作。

    2.3 分发证书和私钥

      1 [root@master01 ~]# cd /opt/k8s/work
      2 [root@master01 work]# source /root/environment.sh
      3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
      4   do
      5     echo ">>> ${master_ip}"
      6     ssh root@${master_ip} "mkdir -p /etc/kubernetes/cert"
      7     scp kubernetes*.pem root@${master_ip}:/etc/kubernetes/cert/
      8   done

    提示:本步骤操作仅需要在master01节点操作。

    三 配置kube-apiserver审计

    3.1 创建加密配置文件

      1 [root@master01 ~]# cd /opt/k8s/work
      2 [root@master01 work]# source /root/environment.sh
      3 [root@master01 work]# cat > encryption-config.yaml <<EOF
      4 kind: EncryptionConfig
      5 apiVersion: v1
      6 resources:
      7   - resources:
      8       - secrets
      9     providers:
     10       - aescbc:
     11           keys:
     12             - name: key1
     13               secret: ${ENCRYPTION_KEY}
     14       - identity: {}
     15 EOF

    提示:本步骤操作仅需要在master01节点操作。

    3.2 分发加密配置文件

      1 [root@master01 ~]# cd /opt/k8s/work
      2 [root@master01 work]# source /root/environment.sh
      3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
      4   do
      5     echo ">>> ${master_ip}"
      6     scp encryption-config.yaml root@${master_ip}:/etc/kubernetes/
      7   done

    提示:本步骤操作仅需要在master01节点操作。

    3.3 创建审计策略文件

      1 [root@master01 ~]# cd /opt/k8s/work
      2 [root@master01 work]# source /root/environment.sh
      3 [root@master01 work]# cat > audit-policy.yaml <<EOF
      4 apiVersion: audit.k8s.io/v1beta1
      5 kind: Policy
      6 rules:
      7   # The following requests were manually identified as high-volume and low-risk, so drop them.
      8   - level: None
      9     resources:
     10       - group: ""
     11         resources:
     12           - endpoints
     13           - services
     14           - services/status
     15     users:
     16       - 'system:kube-proxy'
     17     verbs:
     18       - watch
     19 
     20   - level: None
     21     resources:
     22       - group: ""
     23         resources:
     24           - nodes
     25           - nodes/status
     26     userGroups:
     27       - 'system:nodes'
     28     verbs:
     29       - get
     30 
     31   - level: None
     32     namespaces:
     33       - kube-system
     34     resources:
     35       - group: ""
     36         resources:
     37           - endpoints
     38     users:
     39       - 'system:kube-controller-manager'
     40       - 'system:kube-scheduler'
     41       - 'system:serviceaccount:kube-system:endpoint-controller'
     42     verbs:
     43       - get
     44       - update
     45 
     46   - level: None
     47     resources:
     48       - group: ""
     49         resources:
     50           - namespaces
     51           - namespaces/status
     52           - namespaces/finalize
     53     users:
     54       - 'system:apiserver'
     55     verbs:
     56       - get
     57 
     58   # Don't log HPA fetching metrics.
     59   - level: None
     60     resources:
     61       - group: metrics.k8s.io
     62     users:
     63       - 'system:kube-controller-manager'
     64     verbs:
     65       - get
     66       - list
     67 
     68   # Don't log these read-only URLs.
     69   - level: None
     70     nonResourceURLs:
     71       - '/healthz*'
     72       - /version
     73       - '/swagger*'
     74 
     75   # Don't log events requests.
     76   - level: None
     77     resources:
     78       - group: ""
     79         resources:
     80           - events
     81 
     82   # node and pod status calls from nodes are high-volume and can be large, don't log responses for expected updates from nodes
     83   - level: Request
     84     omitStages:
     85       - RequestReceived
     86     resources:
     87       - group: ""
     88         resources:
     89           - nodes/status
     90           - pods/status
     91     users:
     92       - kubelet
     93       - 'system:node-problem-detector'
     94       - 'system:serviceaccount:kube-system:node-problem-detector'
     95     verbs:
     96       - update
     97       - patch
     98 
     99   - level: Request
    100     omitStages:
    101       - RequestReceived
    102     resources:
    103       - group: ""
    104         resources:
    105           - nodes/status
    106           - pods/status
    107     userGroups:
    108       - 'system:nodes'
    109     verbs:
    110       - update
    111       - patch
    112 
    113   # deletecollection calls can be large, don't log responses for expected namespace deletions
    114   - level: Request
    115     omitStages:
    116       - RequestReceived
    117     users:
    118       - 'system:serviceaccount:kube-system:namespace-controller'
    119     verbs:
    120       - deletecollection
    121 
    122   # Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,
    123   # so only log at the Metadata level.
    124   - level: Metadata
    125     omitStages:
    126       - RequestReceived
    127     resources:
    128       - group: ""
    129         resources:
    130           - secrets
    131           - configmaps
    132       - group: authentication.k8s.io
    133         resources:
    134           - tokenreviews
    135   # Get repsonses can be large; skip them.
    136   - level: Request
    137     omitStages:
    138       - RequestReceived
    139     resources:
    140       - group: ""
    141       - group: admissionregistration.k8s.io
    142       - group: apiextensions.k8s.io
    143       - group: apiregistration.k8s.io
    144       - group: apps
    145       - group: authentication.k8s.io
    146       - group: authorization.k8s.io
    147       - group: autoscaling
    148       - group: batch
    149       - group: certificates.k8s.io
    150       - group: extensions
    151       - group: metrics.k8s.io
    152       - group: networking.k8s.io
    153       - group: policy
    154       - group: rbac.authorization.k8s.io
    155       - group: scheduling.k8s.io
    156       - group: settings.k8s.io
    157       - group: storage.k8s.io
    158     verbs:
    159       - get
    160       - list
    161       - watch
    162 
    163   # Default level for known APIs
    164   - level: RequestResponse
    165     omitStages:
    166       - RequestReceived
    167     resources:
    168       - group: ""
    169       - group: admissionregistration.k8s.io
    170       - group: apiextensions.k8s.io
    171       - group: apiregistration.k8s.io
    172       - group: apps
    173       - group: authentication.k8s.io
    174       - group: authorization.k8s.io
    175       - group: autoscaling
    176       - group: batch
    177       - group: certificates.k8s.io
    178       - group: extensions
    179       - group: metrics.k8s.io
    180       - group: networking.k8s.io
    181       - group: policy
    182       - group: rbac.authorization.k8s.io
    183       - group: scheduling.k8s.io
    184       - group: settings.k8s.io
    185       - group: storage.k8s.io
    186 
    187   # Default level for all other requests.
    188   - level: Metadata
    189     omitStages:
    190       - RequestReceived
    191 EOF

    提示:本步骤操作仅需要在master01节点操作。

    3.4 分发策略文件

      1 [root@master01 ~]# cd /opt/k8s/work
      2 [root@master01 work]# source /root/environment.sh
      3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
      4   do
      5     echo ">>> ${master_ip}"
      6     scp audit-policy.yaml root@${master_ip}:/etc/kubernetes/audit-policy.yaml
      7   done

    提示:有关审计知识参考:https://www.cnblogs.com/liabio/p/11696081.html。

    提示:本步骤操作仅需要在master01节点操作。

    四 配置metrics-server

    4.1 创建访问 metrics-server的证书和密钥

      1 [root@master01 ~]# cd /opt/k8s/work
      2 [root@master01 work]# cat > proxy-client-csr.json <<EOF
      3 {
      4   "CN": "system:metrics-server",
      5   "hosts": [],
      6   "key": {
      7     "algo": "rsa",
      8     "size": 2048
      9   },
     10   "names": [
     11     {
     12       "C": "CN",
     13       "ST": "Shanghai",
     14       "L": "Shanghai",
     15       "O": "k8s",
     16       "OU": "System"
     17     }
     18   ]
     19 }
     20 EOF		#创建metrics-server的CA证书请求文件

    提示:本步骤操作仅需要在master01节点操作。

    解释:

    CN 名称需要位于 kube-apiserver 的 --requestheader-allowed-names 参数中,否则后续访问 metrics 时会提示权限不足。

      1 [root@master01 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem 
      2 -ca-key=/opt/k8s/work/ca-key.pem -config=/opt/k8s/work/ca-config.json 
      3 -profile=kubernetes proxy-client-csr.json | cfssljson -bare proxy-client #生成密钥和证书

    提示:本步骤操作仅需要在master01节点操作。

    4.2 分发证书和私钥

      1 [root@master01 ~]# cd /opt/k8s/work
      2 [root@master01 work]# source /root/environment.sh
      3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
      4   do
      5     echo ">>> ${master_ip}"
      6     scp proxy-client*.pem root@${master_ip}:/etc/kubernetes/cert/
      7   done

    提示:本步骤操作仅需要在master01节点操作。

    五 配置kube-apiserver

    5.1 创建kube-apiserver的systemd

      1 [root@master01 ~]# cd /opt/k8s/work
      2 [root@master01 work]# source /root/environment.sh
      3 [root@master01 work]# cat > kube-apiserver.service.template <<EOF
      4 [Unit]
      5 Description=Kubernetes API Server
      6 Documentation=https://github.com/GoogleCloudPlatform/kubernetes
      7 After=network.target
      8 
      9 [Service]
     10 WorkingDirectory=${K8S_DIR}/kube-apiserver
     11 ExecStart=/opt/k8s/bin/kube-apiserver \
     12   --insecure-port=0 \
     13   --secure-port=6443 \
     14   --bind-address=##MASTER_IP## \
     15   --advertise-address=##MASTER_IP## \
     16   --default-not-ready-toleration-seconds=360 \
     17   --default-unreachable-toleration-seconds=360 \
     18   --feature-gates=DynamicAuditing=true \
     19   --max-mutating-requests-inflight=2000 \
     20   --max-requests-inflight=4000 \
     21   --default-watch-cache-size=200 \
     22   --delete-collection-workers=2 \
     23   --encryption-provider-config=/etc/kubernetes/encryption-config.yaml \
     24   --etcd-cafile=/etc/kubernetes/cert/ca.pem \
     25   --etcd-certfile=/etc/kubernetes/cert/kubernetes.pem \
     26   --etcd-keyfile=/etc/kubernetes/cert/kubernetes-key.pem \
     27   --etcd-servers=${ETCD_ENDPOINTS} \
     28   --tls-cert-file=/etc/kubernetes/cert/kubernetes.pem \
     29   --tls-private-key-file=/etc/kubernetes/cert/kubernetes-key.pem \
     30   --audit-dynamic-configuration \
     31   --audit-log-maxage=30 \
     32   --audit-log-maxbackup=3 \
     33   --audit-log-maxsize=100 \
     34   --audit-log-truncate-enabled=true \
     35   --audit-log-path=${K8S_DIR}/kube-apiserver/audit.log \
     36   --audit-policy-file=/etc/kubernetes/audit-policy.yaml \
     37   --profiling \
     38   --anonymous-auth=false \
     39   --client-ca-file=/etc/kubernetes/cert/ca.pem \
     40   --enable-bootstrap-token-auth=true \
     41   --requestheader-allowed-names="system:metrics-server" \
     42   --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \
     43   --requestheader-extra-headers-prefix=X-Remote-Extra- \
     44   --requestheader-group-headers=X-Remote-Group \
     45   --requestheader-username-headers=X-Remote-User \
     46   --service-account-key-file=/etc/kubernetes/cert/ca.pem \
     47   --authorization-mode=Node,RBAC \
     48   --runtime-config=api/all=true \
     49   --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction \
     50   --allow-privileged=true \
     51   --apiserver-count=3 \
     52   --event-ttl=168h \
     53   --kubelet-certificate-authority=/etc/kubernetes/cert/ca.pem \
     54   --kubelet-client-certificate=/etc/kubernetes/cert/kubernetes.pem \
     55   --kubelet-client-key=/etc/kubernetes/cert/kubernetes-key.pem \
     56   --kubelet-https=true \
     57   --kubelet-timeout=10s \
     58   --proxy-client-cert-file=/etc/kubernetes/cert/proxy-client.pem \
     59   --proxy-client-key-file=/etc/kubernetes/cert/proxy-client-key.pem \
     60   --service-cluster-ip-range=${SERVICE_CIDR} \
     61   --service-node-port-range=${NODE_PORT_RANGE} \
     62   --logtostderr=true \
     63   --v=2
     64 Restart=on-failure
     65 RestartSec=10
     66 Type=notify
     67 LimitNOFILE=65536
     68 
     69 [Install]
     70 WantedBy=multi-user.target
     71 EOF

    提示:本步骤操作仅需要在master01节点操作。

    使用 --audit-policy-file 标志将包含策略的文件传递给 kube-apiserver。如果不设置该标志,则不记录事件。

    解释:

    • --advertise-address:apiserver 对外通告的 IP(kubernetes 服务后端节点 IP);
    • --default-*-toleration-seconds:设置节点异常相关的阈值;
    • --max-*-requests-inflight:请求相关的最大阈值;
    • --etcd-*:访问 etcd 的证书和 etcd 服务器地址;
    • --experimental-encryption-provider-config:指定用于加密 etcd 中 secret 的配置;
    • --bind-address: https 监听的 IP,不能为 127.0.0.1,否则外界不能访问它的安全端口 6443;
    • --secret-port:https 监听端口;
    • --insecure-port=0:关闭监听 http 非安全端口(8080);
    • --tls-*-file:指定 apiserver 使用的证书、私钥和 CA 文件;
    • --audit-*:配置审计策略和审计日志文件相关的参数;
    • --client-ca-file:验证 client (kue-controller-manager、kube-scheduler、kubelet、kube-proxy 等)请求所带的证书;
    • --enable-bootstrap-token-auth:启用 kubelet bootstrap 的 token 认证;
    • --requestheader-*:kube-apiserver 的 aggregator layer 相关的配置参数,proxy-client & HPA 需要使用;
    • --requestheader-client-ca-file:用于签名 --proxy-client-cert-file 和 --proxy-client-key-file 指定的证书;在启用了 metric aggregator 时使用;
    • --requestheader-allowed-names:不能为空,值为逗号分割的 --proxy-client-cert-file 证书的 CN 名称,这里设置为 "aggregator";
    • --service-account-key-file:签名 ServiceAccount Token 的公钥文件,kube-controller-manager 的 --service-account-private-key-file 指定私钥文件,两者配对使用;
    • --runtime-config=api/all=true: 启用所有版本的 APIs,如 autoscaling/v2alpha1;
    • --authorization-mode=Node,RBAC、--anonymous-auth=false: 开启 Node 和 RBAC 授权模式,拒绝未授权的请求;
    • --enable-admission-plugins:启用一些默认关闭的 plugins;
    • --allow-privileged:运行执行 privileged 权限的容器;
    • --apiserver-count=3:指定 apiserver 实例的数量;
    • --event-ttl:指定 events 的保存时间;
    • --kubelet-*:如果指定,则使用 https 访问 kubelet APIs;需要为证书对应的用户(上面 kubernetes*.pem 证书的用户为 kubernetes) 用户定义 RBAC 规则,否则访问 kubelet API 时提示未授权;
    • --proxy-client-*:apiserver 访问 metrics-server 使用的证书;
    • --service-cluster-ip-range: 指定 Service Cluster IP 地址段;
    • --service-node-port-range: 指定 NodePort 的端口范围。

    提示:如果 kube-apiserver 机器没有运行 kube-proxy,则还需要添加 --enable-aggregator-routing=true 参数。

    注意:requestheader-client-ca-file 指定的 CA 证书,必须具有 client auth and server auth;

    如果 --requestheader-allowed-names 为空,或者 --proxy-client-cert-file 证书的 CN 名称不在 allowed-names 中,则后续查看 node 或 pods 的 metrics 失败,提示:

      1 [root@master01 ~]# kubectl top nodes
      2 Error from server (Forbidden): nodes.metrics.k8s.io is forbidden: User "aggregator" cannot list resource "nodes" in API group "metrics.k8s.io" at the cluster scope

    5.2 分发systemd

      1 [root@master01 ~]# cd /opt/k8s/work
      2 [root@master01 work]# source /root/environment.sh
      3 [root@master01 work]# for (( i=0; i < 3; i++ ))
      4   do
      5     sed -e "s/##MASTER_NAME##/${MASTER_NAMES[i]}/" -e "s/##MASTER_IP##/${MASTER_IPS[i]}/" kube-apiserver.service.template > kube-apiserver-${MASTER_IPS[i]}.service
      6   done
      7 [root@master01 work]# ls kube-apiserver*.service			#替换相应的IP
      8 
      9 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
     10   do
     11     echo ">>> ${master_ip}"
     12     scp kube-apiserver-${master_ip}.service root@${master_ip}:/etc/systemd/system/kube-apiserver.service
     13   done						#分发systemd

    提示:本步骤操作仅需要在master01节点操作。

    六 启动并验证

    6.1 启动kube-apiserver服务

      1 [root@master01 ~]# cd /opt/k8s/work
      2 [root@master01 work]# source /root/environment.sh
      3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
      4   do
      5     echo ">>> ${master_ip}"
      6     ssh root@${master_ip} "mkdir -p ${K8S_DIR}/kube-apiserver"
      7     ssh root@${master_ip} "systemctl daemon-reload && systemctl enable kube-apiserver && systemctl restart kube-apiserver"
      8   done

    提示:本步骤操作仅需要在master01节点操作。

    6.2 检查kube-apiserver服务

      1 [root@master01 ~]# cd /opt/k8s/work
      2 [root@master01 work]# source /root/environment.sh
      3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
      4   do
      5     echo ">>> ${master_ip}"
      6     ssh root@${master_ip} "systemctl status kube-apiserver |grep 'Active:'"
      7   done

    提示:本步骤操作仅需要在master01节点操作。

    clipboard

    6.3 查看kube-apiserver写入etcd的数据

      1 [root@master01 ~]# cd /opt/k8s/work
      2 [root@master01 work]# source /root/environment.sh
      3 [root@master01 work]# ETCDCTL_API=3 etcdctl 
      4     --endpoints=${ETCD_ENDPOINTS} 
      5     --cacert=/opt/k8s/work/ca.pem 
      6     --cert=/opt/k8s/work/etcd.pem 
      7     --key=/opt/k8s/work/etcd-key.pem 
      8     get /registry/ --prefix --keys-only

    提示:本步骤操作仅需要在master01节点操作。

    6.4 检查集群信息

      1 [root@master01 work]# kubectl cluster-info
      2 [root@master01 work]# kubectl get all --all-namespaces
      3 [root@master01 work]# kubectl get componentstatuses
      4 [root@master01 work]# netstat -lnpt|grep 6443		#检查 kube-apiserver 监听的端口

    clipboard

    提示:

    如果执行 kubectl 命令式时输出如下错误信息,则说明使用的 ~/.kube/config 文件不对,检查《009.Kubernetes二进制master节点部署kubectl》是否正确完成,然后再执行该命令:

    The connection to the server localhost:8080 was refused - did you specify the right host or port?

    执行 kubectl get componentstatuses 命令时,apiserver 默认向 127.0.0.1 发送请求。目前controller-manager和scheduler还未部署,因此显示Unhealthy;

    6443: 接收 https 请求的安全端口,对所有请求做认证和授权;

    16443:Nginx反向代理监听端口;

    由于关闭了非安全端口,故没有监听 8080。

    6.5 授权

    授予 kube-apiserver 访问 kubelet API 的权限。

    在执行 kubectl exec、run、logs 等命令时,apiserver 会将请求转发到 kubelet 的 https 端口。本实验定义 RBAC 规则,授权 apiserver 使用的证书(kubernetes.pem)用户名(CN:kuberntes)访问 kubelet API 的权限:

      1 [root@master01 ~]# kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes

    提示:本步骤操作仅需要在master01节点操作。

  • 相关阅读:
    phpstorm 2017 关掉变量提示 parameter name hints,改变打开方式
    RMAN-06496: must use the TO clause when the database is mounted or open
    FAL[client]: Failed to request gap sequence GAP
    搭建rac对单实例的MAA
    Linux下安装Oracle 10g(redhat 4)
    ora-01031:insufficient privileges
    ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
    checking for oracle home incompatibilities failed
    用root帐号切换其他帐号提示 su: warning: cannot change directory to /home/oracle: Permission denied
    an error occured during the file system check
  • 原文地址:https://www.cnblogs.com/itzgr/p/11873920.html
Copyright © 2020-2023  润新知