简介：
Velero 是一个云原生的灾难恢复和迁移工具，采用 Go 语言编写，可以安全的备份、恢复和迁移Kubernetes集群资源和持久卷。velero是备份整个 k8s 集群所有的数据，胜过 etcd 备份。
Velero目前包含以下特性：
支持Kubernetes集群数据备份和恢复
支持复制当前Kubernetes集群的资源到其它Kubernetes集群
支持复制生产环境到开发以及测试环境
Velero组件一共分两部分，分别是服务端和客户端。
服务端：运行在Kubernetes集群中
客户端：运行在本地的velero命令行工具，需要在机器上已配置好kubectl及集群kubeconfig
velero使用场景
灾备场景：提供备份恢复k8s集群的能力
迁移场景：提供拷贝集群资源到其他集群的能力（复制同步开发，测试，生产环境的集群配置，简化环境配置）
velero与etcd备份区别
直接备份 Etcd 是将集群的全部资源备份起来，而 Velero 可以对 Kubernetes 集群内对象级别进行备份。
除了对 Kubernetes 集群进行整体备份外，Velero 还可以通过对 Type、Namespace、Label
等对象进行分类备份或者恢复。
github地址：https://github.com/vmware-tanzu/velero
Velero 架构图
Velero 工作原理
1、本地 Velero 客户端发送备份指令。
2、Kubernetes 集群内就会创建一个 Backup 对象。
3、BackupController 监测 Backup 对象并开始备份过程。
4、BackupController 会向 API Server 查询相关数据。
5、BackupController 将查询到的数据备份到远端的对象存储。
支持备份存储
1、AWS S3 以及兼容 S3 的存储，例如：Minio
2、Azure BloB 存储
3、Google Cloud 存储
4、Aliyun OSS 存储( https://github.com/AliyunContainerService/velero-plugin)
保障数据一致性
对象存储的数据是唯一的数据源，也就是说 Kubernetes 集群内的控制器会检查远程的 OSS 存储，发现有备份就会在集群内创建相关 CRD 。如果发现远端存储没有当前集群内的 CRD 所关联的存储数据，那么就会删除当前集群内的 CRD。
部署 Velero 客户端
1、下载tar包并解压
https://github.com/vmware-tanzu/velero
2、加入环境变量并赋予权限
[root@master2 ~]# cp velero-v1.4.2-linux-amd64/velero /usr/local/bin/
[root@master2 ~]# chmod +x /usr/local/bin/velero
3、查看 velero velero CLI 版本
[root@master2 ~]# velero
Velero is a tool for managing disaster recovery, specifically for Kubernetes
cluster resources. It provides a simple, configurable, and operationally robust
way to back up your application state and associated data.
If you're familiar with kubectl, Velero supports a similar model, allowing you to
execute commands such as 'velero get backup' and 'velero create schedule'. The same
operations can also be performed as 'velero backup get' and 'velero schedule create'.
Usage:
velero [command]
Available Commands:
backup Work with backups
backup-location Work with backup storage locations
bug Report a Velero bug
client Velero client related commands
completion Output shell completion code for the specified shell (bash or zsh)
create Create velero resources
delete Delete velero resources
describe Describe velero resources
get Get velero resources
help Help about any command
install Install Velero
plugin Work with plugins
restic Work with restic
restore Work with restores
schedule Work with schedules
snapshot-location Work with snapshot locations
version Print the velero version and associated image
Flags:
--add_dir_header If true, adds the file directory to the header
--alsologtostderr log to standard error as well as files
--features stringArray Comma-separated list of features to enable for this Velero process. Combines with values from $HOME/.config/velero/config.json if present
-h, --help help for velero
--kubeconfig string Path to the kubeconfig file to use to talk to the Kubernetes apiserver. If unset, try the environment variable KUBECONFIG, as well as in-cluster configuration
--kubecontext string The context to use to talk to the Kubernetes apiserver. If unset defaults to whatever your current-context is (kubectl config current-context)
--log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)
--log_dir string If non-empty, write log files in this directory
--log_file string If non-empty, use this log file
--log_file_max_size uint Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
--logtostderr log to standard error instead of files (default true)
-n, --namespace string The namespace in which Velero should operate (default "velero")
--skip_headers If true, avoid header prefixes in the log messages
--skip_log_headers If true, avoid headers when opening log files
--stderrthreshold severity logs at or above this threshold go to stderr (default 2)
-v, --v Level number for the log level verbosity
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
Use "velero [command] --help" for more information about a command.
部署 Velero 服务端
1、查看 00-minio-deployment.yaml 文件中的 MINIO_ACCESS_KEY 和 MINIO_SECRET_KEY 值
cat velero-v1.4.2-linux-amd64/examples/minio/00-minio-deployment.yaml
# Copyright 2017 the Velero contributors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
---
apiVersion: v1
kind: Namespace
metadata:
name: velero
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: velero
name: minio
labels:
component: minio
spec:
strategy:
type: Recreate
selector:
matchLabels:
component: minio
template:
metadata:
labels:
component: minio
spec:
volumes:
- name: storage
emptyDir: {}
- name: config
emptyDir: {}
containers:
- name: minio
image: minio/minio:latest
imagePullPolicy: IfNotPresent
args:
- server
- /storage
- --config-dir=/config
env:
- name: MINIO_ACCESS_KEY # 要查找的值
value: "minio"
- name: MINIO_SECRET_KEY # 要查找的值
value: "minio123"
ports:
- containerPort: 9000
volumeMounts:
- name: storage
mountPath: "/storage"
- name: config
mountPath: "/config"
---
apiVersion: v1
kind: Service
metadata:
namespace: velero
name: minio
labels:
component: minio
spec:
# ClusterIP is recommended for production environments.
# Change to NodePort if needed per documentation,
# but only if you run Minio in a test/trial environment, for example with Minikube.
type: NodePort
ports:
- port: 9000
targetPort: 9000
protocol: TCP
selector:
component: minio
---
apiVersion: batch/v1
kind: Job
metadata:
namespace: velero
name: minio-setup
labels:
component: minio
spec:
template:
metadata:
name: minio-setup
spec:
restartPolicy: OnFailure
volumes:
- name: config
emptyDir: {}
containers:
- name: mc
image: minio/mc:latest
imagePullPolicy: IfNotPresent
command:
- /bin/sh
- -c
- "mc --config-dir=/config config host add velero http://minio:9000 minio minio123 && mc --config-dir=/config mb -p velero/velero"
volumeMounts:
- name: config
mountPath: "/config"
2、克隆 velero-plugin 插件项目
git clone https://github.com/AliyunContainerService/velero-plugin
3、进入克隆的插件项目把两个值添加到 credentials-velero 文件中
[root@master1 ~]# cat /root/velero-plugin/install/credentials-velero
ALIBABA_CLOUD_ACCESS_KEY_ID= minio
ALIBABA_CLOUD_ACCESS_KEY_SECRET= minio123
4、更改 00-minio-deployment.yaml 文件中的 type 类型为 NodePort
[root@master1 ~]# cat /root/velero-v1.4.2-linux-amd64/examples/minio/00-minio-deployment.yaml
# Copyright 2017 the Velero contributors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
---
apiVersion: v1
kind: Namespace
metadata:
name: velero
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: velero
name: minio
labels:
component: minio
spec:
strategy:
type: Recreate
selector:
matchLabels:
component: minio
template:
metadata:
labels:
component: minio
spec:
volumes:
- name: storage
emptyDir: {}
- name: config
emptyDir: {}
containers:
- name: minio
image: minio/minio:latest
imagePullPolicy: IfNotPresent
args:
- server
- /storage
- --config-dir=/config
env:
- name: MINIO_ACCESS_KEY
value: "minio"
- name: MINIO_SECRET_KEY
value: "minio123"
ports:
- containerPort: 9000
volumeMounts:
- name: storage
mountPath: "/storage"
- name: config
mountPath: "/config"
---
apiVersion: v1
kind: Service
metadata:
namespace: velero
name: minio
labels:
component: minio
spec:
# ClusterIP is recommended for production environments.
# Change to NodePort if needed per documentation,
# but only if you run Minio in a test/trial environment, for example with Minikube.
type: NodePort # 已经改过了
ports:
- port: 9000
targetPort: 9000
protocol: TCP
selector:
component: minio
---
apiVersion: batch/v1
kind: Job
metadata:
namespace: velero
name: minio-setup
labels:
component: minio
spec:
template:
metadata:
name: minio-setup
spec:
restartPolicy: OnFailure
volumes:
- name: config
emptyDir: {}
containers:
- name: mc
image: minio/mc:latest
imagePullPolicy: IfNotPresent
command:
- /bin/sh
- -c
- "mc --config-dir=/config config host add velero http://minio:9000 minio minio123 && mc --config-dir=/config mb -p velero/velero"
volumeMounts:
- name: config
mountPath: "/config"
5、部署文件
kubectl apply -f /root/velero-v1.4.2-linux-amd64/examples/minio/00-minio-deployment.yaml
velero install
--provider aws
--plugins velero/velero-plugin-for-aws:v1.2.0
--namespace velero
--bucket velerobak
--default-volumes-to-restic
--use-restic
--secret-file ./credentials-velero
--use-volume-snapshots=false
--backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://{minio_service_ip}:暴露端口号
6、验证结果
提示这个代表部署成功
[root@master1 ~]# kubectl get pods -n velero
NAME READY STATUS RESTARTS AGE
minio-7b4ff54f67-k77kx 1/1 Running 0 3h58m
minio-setup-nwwr7 0/1 Completed 2 3h58m
restic-m6g9s 1/1 Running 0 3h46m
velero-8dc7498d9-kgrgx 1/1 Running 0 3h46m
7、Velero 在 Kubernetes 集群中创建的 CRD
[root@master1 ~]# kubectl -n velero get crds -l component=velero
NAME CREATED AT
backups.velero.io 2021-10-04T11:44:26Z
backupstoragelocations.velero.io 2021-10-04T11:44:26Z
deletebackuprequests.velero.io 2021-10-04T11:44:26Z
downloadrequests.velero.io 2021-10-04T11:44:26Z
podvolumebackups.velero.io 2021-10-04T11:44:26Z
podvolumerestores.velero.io 2021-10-04T11:44:26Z
resticrepositories.velero.io 2021-10-04T11:44:26Z
restores.velero.io 2021-10-04T11:44:26Z
schedules.velero.io 2021-10-04T11:44:26Z
serverstatusrequests.velero.io 2021-10-04T11:44:26Z
volumesnapshotlocations.velero.io 2021-10-04T11:44:26Z
8、查看 velero 命令及版本
[root@master1 ~]# velero create backup NAME [flags]
Error: accepts 1 arg(s), received 2
Usage:
velero create backup NAME [flags]
Examples:
# create a backup containing all resources
velero backup create backup1
# create a backup including only the nginx namespace
velero backup create nginx-backup --include-namespaces nginx
# create a backup excluding the velero and default namespaces
velero backup create backup2 --exclude-namespaces velero,default
# view the YAML for a backup that doesn't snapshot volumes, without sending it to the server
velero backup create backup3 --snapshot-volumes=false -o yaml
# wait for a backup to complete before returning from the command
velero backup create backup4 --wait
Flags:
--exclude-namespaces stringArray namespaces to exclude from the backup
--exclude-resources stringArray resources to exclude from the backup, formatted as resource.group, such as storageclasses.storage.k8s.io
--from-schedule string create a backup from the template of an existing schedule. Cannot be used with any other filters.
-h, --help help for backup
--include-cluster-resources optionalBool[=true] include cluster-scoped resources in the backup
--include-namespaces stringArray namespaces to include in the backup (use '*' for all namespaces) (default *)
--include-resources stringArray resources to include in the backup, formatted as resource.group, such as storageclasses.storage.k8s.io (use '*' for all resources)
--label-columns stringArray a comma-separated list of labels to be displayed as columns
--labels mapStringString labels to apply to the backup
-o, --output string Output display format. For create commands, display the object but do not send it to the server. Valid formats are 'table', 'json', and 'yaml'. 'table' is not valid for the install command.
-l, --selector labelSelector only back up resources matching this label selector (default <none>)
--show-labels show labels in the last column
--snapshot-volumes optionalBool[=true] take snapshots of PersistentVolumes as part of the backup
--storage-location string location in which to store the backup
--ttl duration how long before the backup can be garbage collected (default 720h0m0s)
--volume-snapshot-locations strings list of locations (at most one per provider) where volume snapshots should be stored
-w, --wait wait for the operation to complete
Global Flags:
--add_dir_header If true, adds the file directory to the header
--alsologtostderr log to standard error as well as files
--features stringArray Comma-separated list of features to enable for this Velero process. Combines with values from $HOME/.config/velero/config.json if present
--kubeconfig string Path to the kubeconfig file to use to talk to the Kubernetes apiserver. If unset, try the environment variable KUBECONFIG, as well as in-cluster configuration
--kubecontext string The context to use to talk to the Kubernetes apiserver. If unset defaults to whatever your current-context is (kubectl config current-context)
--log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)
--log_dir string If non-empty, write log files in this directory
--log_file string If non-empty, use this log file
--log_file_max_size uint Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
--logtostderr log to standard error instead of files (default true)
-n, --namespace string The namespace in which Velero should operate (default "velero")
--skip_headers If true, avoid header prefixes in the log messages
--skip_log_headers If true, avoid headers when opening log files
--stderrthreshold severity logs at or above this threshold go to stderr (default 2)
-v, --v Level number for the log level verbosity
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
An error occurred: accepts 1 arg(s), received 2
[root@master1 ~]# velero version
Client:
Version: v1.4.2
Git commit: 56a08a4d695d893f0863f697c2f926e27d70c0c5
Server:
Version: v1.4.2