【案例】 实例已删除 如何恢复

 

问题：主要是 分布式的物理备份 ， 我又不能手动来恢复。

思路： 1.要先创建一个实例 ，然后将备份移动到该实例的路径下，然后进行的恢复 

1.在赤兔上创建新的实例
2.在hdfs 上根据实例信息 创建相关目录 group_1610847854_45 ，set_1610848126_1   如果有多个分片 也都要创建
hadoop fs -mkdir -p /tdsqlbackup/tdsqlzk/group_1610847854_45/autocoldbackup/sets/set_1610848126_1/binlog
hadoop fs -mkdir -p /tdsqlbackup/tdsqlzk/group_1610847854_45/autocoldbackup/sets/set_1610848126_1/errlog
hadoop fs -mkdir -p /tdsqlbackup/tdsqlzk/group_1610847854_45/autocoldbackup/sets/set_1610848126_1/slowlog
hadoop fs -mkdir -p /tdsqlbackup/tdsqlzk/group_1610847854_45/autocoldbackup/sets/set_1610848126_1/xtrabackup

3.将备份文件，和日志文件拷贝到相应目录 （注意 cp 的时候需要把分片数加上）
cp 物理文件 和 binlog 
hadoop fs -cp -d  /tdsqlbackup/tdsqlzk/group_1610846781_37/autocoldbackup/sets/set_1610847062_1/xtrabackup/xtrabackup+1610847791+20210117+094311+10.85.10.53+4002+171279522+20210117+094325++xbstream.lz4  /tdsqlbackup/tdsqlzk/group_1610847854_45/autocoldbackup/sets/set_1610848126_1/xtrabackup/xtrabackup+1610847791+20210117+094311+10.85.10.53+4002+171279522+20210117+094325+1+xbstream.lz4
hadoop fs -cp -d  /tdsqlbackup/tdsqlzk/group_1610846781_37/autocoldbackup/sets/set_1610847062_1/xtrabackup/xtrabackup+1610847791+20210117+094311+10.85.10.53+4002+171279522+20210117+094325++xbstream.lz4.ok  /tdsqlbackup/tdsqlzk/group_1610847854_45/autocoldbackup/sets/set_1610848126_1/xtrabackup/xtrabackup+1610847791+20210117+094311+10.85.10.53+4002+171279522+20210117+094325+1+xbstream.lz4.ok

hadoop fs -cp -d  /tdsqlbackup/tdsqlzk/group_1610846781_37/autocoldbackup/sets/set_1610847062_1/binlog/binlog+1610847031+20210117+093031+10.85.10.53+4002+171279522+binlog.000001.lz4  /tdsqlbackup/tdsqlzk/group_1610847854_45/autocoldbackup/sets/set_1610848126_1/binlog/binlog+1610847031+20210117+093031+10.85.10.53+4002+171279522+binlog.000001.lz4

4.在赤兔上进行恢复 ，注意回档时间如果选择当前的话， 要有相应的binlog 
  测试好像至少需要在一个binlog的时间点后在， 不然会报    can not get route  in the retreat time
    我的猜想：1.这个时间是必须要一个binlog  
             2.这个时间点需要新创建的实例后面， 而恢复的时候直接恢复到没有binlog 为止 (测试这种可能性较大)
               新实例创建结束时间： 在 日志管理 -> 控制台操作日志  的   创建分布式实例任务流程 => 任务开始时间  +  执行时间  = 实例创建结束的时间
  
  
#原理
1.在还原的时候，点击下一步时，需要去flush log 日志，所需需要一个可用的实例。
2.在恢复的时候是去扫描是否有可用的备份文件，跟文件名无关， 不会去匹配文件名
  所以只需要把 备份文件 cp 到 目录就可以了。
  
3.在恢复的时候 指定时间点 没有恢复成后，  有待继续 测试
  使用当前的时间点（默认的当前的）， 恢复成功

4.没有binlog 也是恢复不成功的。  
   恢复：明确要去找binlog 
    ERROR 57093,backupfilemgn.cpp:228:pickValidBinlog,tid:0x7f6f16d5e880,serverid is not legal serverid = 171279522
    ERROR 57093,binlogmgn.cpp:67:getAvailableBinlog,tid:0x7f6f16d5e880,unable to get the available binlog
    ERROR 57093,recoverset.cpp:161:recover,tid:0x7f6f16d5e880,Binlog Recover Failed:Binlog Error:Check the binlog failure on HDFS.

#注意有时候 赤兔上反回  
   提示:	【-2007】the zookeeper operation timeout
   可以等待，后台执行完成在看。  因为任务已经下出下去了。

#总结：
1.恢复的时候选 使用当前的时间点（默认的当前的）
2.恢复的时候至少需要一个binlog 文件。
  没有binlog ,备份片的数据也会还原，只是状态不可用， 可以尝试手动调整。
  

测试1:  回当时间必须设置到, 新实例创建后的时间点

#1. 查看新实例创建完成时间：

#2.回当时间设置在 52分：回档成功

#问题 ：提示找到分版，但是不可用  ，   可能是指定的时间太靠前了。  
#       在次测试，就用当前时间恢复成轼， 不手动指定时间。
[2021-01-17 11:36:25 190425] ERROR 81370,xtrabackupmgn.cpp:128:getAvailableImage,tid:0x7fe7bbf4c880,There are 1 of backup files , but no one available.
[2021-01-17 11:36:25 190564] ERROR 81370,recoverset.cpp:153:recover,tid:0x7fe7bbf4c880,Xtrabackup Recover Failed:Xtrabackup Error:Unable to get available image.

测试2:  只有备份文件，没有binlog 

1.删掉binlog 
[tdsql@tdsql1 ~]$ hadoop fs -rm /tdsqlbackup/tdsqlzk/group_1610847854_45/autocoldbackup/sets/set_1610848126_1/binlog/*                       # */
Deleted /tdsqlbackup/tdsqlzk/group_1610847854_45/autocoldbackup/sets/set_1610848126_1/binlog/binlog+1610847031+20210117+093031+10.85.10.53+4002+171279522+binlog.000001.lz4

2. 在次，不指定时间 进行恢复 ，也恢复失败
 [2021-01-17 12:54:55 457702] ERROR 57093,backupfilemgn.cpp:228:pickValidBinlog,tid:0x7f6f16d5e880,serverid is not legal serverid = 171279522
[2021-01-17 12:54:55 457818] ERROR 57093,binlogmgn.cpp:67:getAvailableBinlog,tid:0x7f6f16d5e880,unable to get the available binlog
[2021-01-17 12:54:55 457863] ERROR 57093,recoverset.cpp:161:recover,tid:0x7f6f16d5e880,Binlog Recover Failed:Binlog Error:Check the binlog failure on HDFS.

3.但实际上，备份片的数据已经还源，只是状态不可用。

1. set 状态处理回档中

2.实例状态处理不可用

在zk中找到相应的 group 和 set 
#1.更改set 状态 ，将回档中 101   调到正常 0
get /tdsqlzk/group_1610858871_151/sets/set@set_1610859155_1/setrun@set_1610859155_1
[zk: localhost:2181(CONNECTED) 8] get /tdsqlzk/group_1610858871_151/sets/set@set_1610859155_1/setrun@set_1610859155_1
{"cgroup_cpu":"","degrade_flag":0,"history_ids":[{"id":"set_1610859155_1","timestamp":1610859155,"type":0}],"kpstatus":101,"master":{"alive":"0","city":"default","election":true,"hb_err":"0","idc":"IDC_CQ_YB_9527_01","idc_weight":"100","losthbtime":"0","name":"10.85.10.51_4013","sqlasyn":"1","weight":"1","zone":"default"},"password":"gYNPpki%7Q5WDD3WhL","proxy":[{"name":"10.85.10.51_15002"},{"name":"10.85.10.53_15002"},{"name":"10.85.10.52_15002"}],"read_only":"0","resource_info":{"cpu":100,"data_disk":16000,"log_disk":4000,"mem":1500},"set":"set_1610859155_1","slave":[{"alive":"0","city":"default","election":true,"hb_err":"0","idc":"IDC_CQ_YB_9527_02","idc_weight":"100","losthbtime":"0","name":"10.85.10.52_4013","sqlasyn":"1","weight":"1","zone":"default"},{"alive":"0","city":"default","election":true,"hb_err":"0","idc":"IDC_CQ_YB_9527_03","idc_weight":"100","losthbtime":"0","name":"10.85.10.53_4013","sqlasyn":"1","weight":"1","zone":"default"}],"specid":32768,"status":0,"uniqueid":"unique_1326296854_1610859155","user":"tdsqlsys_normal"}

手动设置 SET 值   将kpstatus 改为 0 

[zk: localhost:2181(CONNECTED) 8] set /tdsqlzk/group_1610858871_151/sets/set@set_1610859155_1/setrun@set_1610859155_1 {"cgroup_cpu":"","degrade_flag":0,"history_ids":[{"id":"set_1610859155_1","timestamp":1610859155,"type":0}],"kpstatus":101,"master":{"alive":"0","city":"default","election":true,"hb_err":"0","idc":"IDC_CQ_YB_9527_01","idc_weight":"100","losthbtime":"0","name":"10.85.10.51_4013","sqlasyn":"1","weight":"1","zone":"default"},"password":"gYNPpki%7Q5WDD3WhL","proxy":[{"name":"10.85.10.51_15002"},{"name":"10.85.10.53_15002"},{"name":"10.85.10.52_15002"}],"read_only":"0","resource_info":{"cpu":100,"data_disk":16000,"log_disk":4000,"mem":1500},"set":"set_1610859155_1","slave":[{"alive":"0","city":"default","election":true,"hb_err":"0","idc":"IDC_CQ_YB_9527_02","idc_weight":"100","losthbtime":"0","name":"10.85.10.52_4013","sqlasyn":"1","weight":"1","zone":"default"},{"alive":"0","city":"default","election":true,"hb_err":"0","idc":"IDC_CQ_YB_9527_03","idc_weight":"100","losthbtime":"0","name":"10.85.10.53_4013","sqlasyn":"1","weight":"1","zone":"default"}],"specid":32768,"status":0,"uniqueid":"unique_1326296854_1610859155","user":"tdsqlsys_normal"}

2.将实例中的不可用调到 正常0 
get /tdsqlzk/group_1610858871_151/routes/status@group
{"kpstatus":"30000","status":"1"}

set /tdsqlzk/group_1610858871_151/routes/status@group {"status":"0"}

3.测试登录  ，我这里是新创建的账号   可以登录 ，如果可以用的话， 此时建议将数据做一个逻辑备份。

[root@tdsql1 ~]# mysql -utest -ptest -h10.85.10.51 -P15002
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 4561
Server version: 5.7.17-11-V2.0R540D002-20191226-1152-log Source distribution
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [(none)]> 

来自为知笔记(Wiz)