br 备份数据失败

提问参考模版:

  • nebula 版本:3.4.1
  • 部署方式:分布式
  • 安装方式: RPM
  • 是否上生产环境:Y
  • 问题的具体描述
  • 相关的 meta / storage / graph info 日志信息(尽量使用文本形式方便检索)

采用社区版的 br 和 agent 做数据备份,目前 agent 已经启动,并且注册在服务上。

但是执行本地备份的时候,报错,具体日志如下。

[work@bdbl-inf-bce436-768ec.bdbl.baidu.com nebula-all]$ ./br backup full --meta "10.122.107.172:8559" --storage "local:///home/work/bakcup"
{"level":"info","meta address":"10.122.107.172:8559","msg":"Try to connect meta service.","time":"2023-03-20T11:44:31.096Z"}
{"level":"info","meta address":"10.122.107.172:8559","msg":"Connect meta server successfully.","time":"2023-03-20T11:44:31.096Z"}
{"level":"info","meta address":"10.161.85.24:8559","msg":"Try to connect meta service.","time":"2023-03-20T11:44:31.096Z"}
{"level":"info","meta address":"10.161.85.24:8559","msg":"Connect meta server successfully.","time":"2023-03-20T11:44:31.097Z"}
{"level":"info","meta address":"10.122.108.171:8559","msg":"Try to connect meta service.","time":"2023-03-20T11:44:31.097Z"}
{"level":"info","meta address":"10.122.108.171:8559","msg":"Connect meta server successfully.","time":"2023-03-20T11:44:31.100Z"}
{"level":"info","meta address":"10.161.85.24:8559","msg":"Try to connect meta service.","time":"2023-03-20T11:44:31.100Z"}
{"level":"info","meta address":"10.161.85.24:8559","msg":"Connect meta server successfully.","time":"2023-03-20T11:44:31.100Z"}
{"level":"info","meta address":"10.122.107.172:8559","msg":"Try to connect meta service.","time":"2023-03-20T11:44:31.101Z"}
{"level":"info","meta address":"10.122.107.172:8559","msg":"Connect meta server successfully.","time":"2023-03-20T11:44:31.101Z"}
{"host info":"map[10.122.107.172:10.122.107.172:8779[STORAGE]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.122.107.172:8346[AGENT]: (data: , root: ) | 10.122.107.172:8669[GRAPH]: (data: , root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.122.107.172:8559[META]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) 10.122.108.171:10.122.108.171:8779[STORAGE]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.122.108.171:8346[AGENT]: (data: , root: ) | 10.122.108.171:8669[GRAPH]: (data: , root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.122.108.171:8559[META]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) 10.161.85.24:10.161.85.24:8779[STORAGE]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.161.85.24:8346[AGENT]: (data: , root: ) | 10.161.85.24:8669[GRAPH]: (data: , root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.161.85.24:8559[META]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64)]","level":"info","msg":"Get cluster topology from the nebula.","time":"2023-03-20T11:44:31.101Z"}
Start to backup cluster...
{"backup info":"map[all spaces:true backup name:BACKUP_2023_03_20_11_44_31 backups:demo_shareholding: space-id 4, hosts: 3;test: space-id 74, hosts: 3;test_space: space-id 188, hosts: 3;demo_basketballplayer: space-id 1, hosts: 3 created time:2023-03-20 11:47:37 +0800 CST full backup:true meta files:/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula/0/checkpoints/BACKUP_2023_03_20_11_44_31/__disk_parts__.sst,/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula/0/checkpoints/BACKUP_2023_03_20_11_44_31/__local_id__.sst,/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula/0/checkpoints/BACKUP_2023_03_20_11_44_31/__stats__.sst,/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula/0/checkpoints/BACKUP_2023_03_20_11_44_31/__indexes__.sst,/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula/0/checkpoints/BACKUP_2023_03_20_11_44_31/__edges__.sst,/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula/0/checkpoints/BACKUP_2023_03_20_11_44_31/__tags__.sst,/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula/0/checkpoints/BACKUP_2023_03_20_11_44_31/__parts__.sst,/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula/0/checkpoints/BACKUP_2023_03_20_11_44_31/__spaces__.sst,/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula/0/checkpoints/BACKUP_2023_03_20_11_44_31/__sessions__.sst,/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula/0/checkpoints/BACKUP_2023_03_20_11_44_31/__machines__.sst,/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula/0/checkpoints/BACKUP_2023_03_20_11_44_31/__zones__.sst,/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula/0/checkpoints/BACKUP_2023_03_20_11_44_31/__users__.sst,/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula/0/checkpoints/BACKUP_2023_03_20_11_44_31/__last_update_time__.sst,/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula/0/checkpoints/BACKUP_2023_03_20_11_44_31/__id__.sst,/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula/0/checkpoints/BACKUP_2023_03_20_11_44_31/__index__.sst]","level":"info","msg":"Create backup in nebula machine's local.","name":"BACKUP_2023_03_20_11_44_31","time":"2023-03-20T11:44:31.171Z"}
Backup failed, will try to clean the remaining garbage...
{"level":"info","meta address":"10.122.107.172:8559","msg":"Try to connect meta service.","time":"2023-03-20T11:44:31.172Z"}
{"level":"info","meta address":"10.122.107.172:8559","msg":"Connect meta server successfully.","time":"2023-03-20T11:44:31.172Z"}
{"level":"info","meta address":"10.161.85.24:8559","msg":"Try to connect meta service.","time":"2023-03-20T11:44:31.172Z"}
{"level":"info","meta address":"10.161.85.24:8559","msg":"Connect meta server successfully.","time":"2023-03-20T11:44:31.172Z"}
{"level":"info","meta address":"10.122.108.171:8559","msg":"Try to connect meta service.","time":"2023-03-20T11:44:31.173Z"}
{"level":"info","meta address":"10.122.108.171:8559","msg":"Connect meta server successfully.","time":"2023-03-20T11:44:31.173Z"}
{"level":"info","meta address":"10.161.85.24:8559","msg":"Try to connect meta service.","time":"2023-03-20T11:44:31.173Z"}
{"level":"info","meta address":"10.161.85.24:8559","msg":"Connect meta server successfully.","time":"2023-03-20T11:44:31.174Z"}
{"level":"info","meta address":"10.122.107.172:8559","msg":"Try to connect meta service.","time":"2023-03-20T11:44:31.175Z"}
{"level":"info","meta address":"10.122.107.172:8559","msg":"Connect meta server successfully.","time":"2023-03-20T11:44:31.175Z"}
{"host info":"map[10.122.107.172:10.122.107.172:8779[STORAGE]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.122.107.172:8346[AGENT]: (data: , root: ) | 10.122.107.172:8669[GRAPH]: (data: , root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.122.107.172:8559[META]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) 10.122.108.171:10.122.108.171:8779[STORAGE]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.122.108.171:8346[AGENT]: (data: , root: ) | 10.122.108.171:8669[GRAPH]: (data: , root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.122.108.171:8559[META]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) 10.161.85.24:10.161.85.24:8779[STORAGE]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.161.85.24:8346[AGENT]: (data: , root: ) | 10.161.85.24:8669[GRAPH]: (data: , root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.161.85.24:8559[META]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64)]","level":"info","msg":"Get cluster topology from the nebula.","time":"2023-03-20T11:44:31.175Z"}
{"backup name":"BACKUP_2023_03_20_11_44_31","level":"info","msg":"Start to cleanup data in nebula cluster.","time":"2023-03-20T11:44:31.175Z"}
{"backup name":"BACKUP_2023_03_20_11_44_31","level":"info","msg":"Start cleanup data in external storage.","time":"2023-03-20T11:44:31.206Z"}
Error: cleanup BACKUP_2023_03_20_11_44_31 failed when backup failed: clean external storage data failed: remove local:///home/work/bakcup/BACKUP_2023_03_20_11_44_31 in external storage failed: local:///home/work/bakcup/BACKUP_2023_03_20_11_44_31 not found
Usage:
  br backup full [flags]

Flags:
  -h, --help   help for full

Global Flags:
      --debug                  Output log in debug level or not
      --log string             Specify br detail log path (default "br.log")
      --meta string            Specify meta server
      --s3.access_key string   S3 Option: set access key id
      --s3.endpoint string     S3 Option: set the S3 endpoint URL, please specify the http or https scheme explicitly
      --s3.region string       S3 Option: set region or location to upload or download backup
      --s3.secret_key string   S3 Option: set secret key for access id
      --spaces stringArray     (EXPERIMENTAL)space names.
                                   By this option, user can specify which spaces to backup. Now this feature is still experimental.
                                   If not specified, will backup all spaces.
                                   
      --storage string         backup target url, format: <SCHEME>://<PATH>.
                                   <SCHEME>: a string indicating which backend type. optional: local, s3.
                                   now only s3-compatible is supported.
                                   example:
                                   for local - "local:///the/local/path/to/backup"
                                   for s3  - "s3://example/url/to/the/backup"

{"level":"fatal","msg":"cleanup BACKUP_2023_03_20_11_44_31 failed when backup failed: clean external storage data failed: remove local:///home/work/bakcup/BACKUP_2023_03_20_11_44_31 in external storage failed: local:///home/work/bakcup/BACKUP_2023_03_20_11_44_31 not found","time":"2023-03-20T11:44:31.207Z"}

辛苦相关同学看一下

你好,如果使用的是本地备份,需要所有服务所在节点,以及使用br工具的节点,都存在–storage 指定的目录(/home/work/bakcup),可以检查下是否已经创建

感谢提示,仔细检查了一下,发现是路径写错了 backup 写成了 ba kc up

数据备份成功后,恢复数据。遇到机器无法结束进程

备份的数据

恢复命令

./br restore full --meta "10.122.107.172:8559" --storage "local:///home/work/backup" --name BACKUP_2023_03_22_10_52_09

相关日志

{"level":"info","meta address":"10.122.107.172:8559","msg":"Try to connect meta service.","time":"2023-03-23T16:33:02.311Z"}
{"level":"info","meta address":"10.122.107.172:8559","msg":"Connect meta server successfully.","time":"2023-03-23T16:33:02.313Z"}
{"level":"info","meta address":"10.122.108.171:8559","msg":"Try to connect meta service.","time":"2023-03-23T16:33:02.313Z"}
{"level":"info","meta address":"10.122.108.171:8559","msg":"Connect meta server successfully.","time":"2023-03-23T16:33:02.314Z"}
{"level":"info","meta address":"10.122.108.171:8559","msg":"Try to connect meta service.","time":"2023-03-23T16:33:02.314Z"}
{"level":"info","meta address":"10.122.108.171:8559","msg":"Connect meta server successfully.","time":"2023-03-23T16:33:02.317Z"}
{"level":"info","meta address":"10.161.85.24:8559","msg":"Try to connect meta service.","time":"2023-03-23T16:33:02.317Z"}
{"level":"info","meta address":"10.161.85.24:8559","msg":"Connect meta server successfully.","time":"2023-03-23T16:33:02.318Z"}
{"level":"info","meta address":"10.122.107.172:8559","msg":"Try to connect meta service.","time":"2023-03-23T16:33:02.319Z"}
{"level":"info","meta address":"10.122.107.172:8559","msg":"Connect meta server successfully.","time":"2023-03-23T16:33:02.320Z"}
{"host info":"map[10.122.107.172:10.122.107.172:8779[STORAGE]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.122.107.172:8346[AGENT]: (data: , root: ) | 10.122.107.172:8669[GRAPH]: (data: , root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.122.107.172:8559[META]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) 10.122.108.171:10.122.108.171:8779[STORAGE]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.122.108.171:8346[AGENT]: (data: , root: ) | 10.122.108.171:8669[GRAPH]: (data: , root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.122.108.171:8559[META]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) 10.161.85.24:10.161.85.24:8779[STORAGE]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.161.85.24:8346[AGENT]: (data: , root: ) | 10.161.85.24:8669[GRAPH]: (data: , root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.161.85.24:8559[META]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64)]","level":"info","msg":"Get cluster topology from the nebula.","time":"2023-03-23T16:33:02.320Z"}
{"backup":"BACKUP_2023_03_22_10_52_09","level":"info","msg":"Check backup dir successfully.","time":"2023-03-23T16:33:02.320Z","uri":"local:///home/work/backup/BACKUP_2023_03_22_10_52_09"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.108.171","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-23T16:33:02.321Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.108.171","level":"info","msg":"Stop services.","role":"GRAPH","time":"2023-03-23T16:33:02.348Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.108.171","level":"info","msg":"Stop services.","role":"META","time":"2023-03-23T16:33:02.373Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.161.85.24","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-23T16:33:02.398Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.161.85.24","level":"info","msg":"Stop services.","role":"GRAPH","time":"2023-03-23T16:33:02.427Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.161.85.24","level":"info","msg":"Stop services.","role":"META","time":"2023-03-23T16:33:02.456Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.107.172","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-23T16:33:02.485Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.107.172","level":"info","msg":"Stop services.","role":"GRAPH","time":"2023-03-23T16:33:02.514Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.107.172","level":"info","msg":"Stop services.","role":"META","time":"2023-03-23T16:33:02.540Z"}
{"backup":"BACKUP_2023_03_22_10_52_09","level":"info","msg":"Stop cluster successfully.","time":"2023-03-23T16:33:02.567Z"}
{"addr":"10.122.108.171:8779","backup path":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula_old_1679560382","level":"info","msg":"Backup origin storage data path successfully.","origin path":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula","time":"2023-03-23T16:33:02.567Z"}
{"addr":"10.161.85.24:8779","backup path":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula_old_1679560382","level":"info","msg":"Backup origin storage data path successfully.","origin path":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula","time":"2023-03-23T16:33:02.568Z"}
{"addr":"10.122.107.172:8779","backup path":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula_old_1679560382","level":"info","msg":"Backup origin storage data path successfully.","origin path":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula","time":"2023-03-23T16:33:02.568Z"}
{"addr":"10.122.108.171:8559","backup path":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula_old_1679560382","level":"info","msg":"Backup origin meta data path successfully.","origin path":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula","time":"2023-03-23T16:33:02.568Z"}
{"addr":"10.161.85.24:8559","backup path":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula_old_1679560382","level":"info","msg":"Backup origin meta data path successfully.","origin path":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula","time":"2023-03-23T16:33:02.569Z"}
{"addr":"10.122.107.172:8559","backup path":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula_old_1679560382","level":"info","msg":"Backup origin meta data path successfully.","origin path":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula","time":"2023-03-23T16:33:02.569Z"}
{"backup":"BACKUP_2023_03_22_10_52_09","level":"info","msg":"Backup origin cluster data successfully.","time":"2023-03-23T16:33:02.569Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.107.172","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-23T16:33:02.876Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.107.172","level":"info","msg":"Stop services.","role":"GRAPH","time":"2023-03-23T16:33:02.899Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.107.172","level":"info","msg":"Stop services.","role":"META","time":"2023-03-23T16:33:02.922Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.108.171","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-23T16:33:02.949Z"}
{"error":"stop services in host 10.122.108.171 failed: agent, stop service failed: rpc error: code = Unknown desc = exit status 1","level":"info","msg":"Stop all services failed, try times=1.","time":"2023-03-23T16:33:02.973Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.108.171","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-23T16:33:03.973Z"}
{"error":"stop services in host 10.122.108.171 failed: agent, stop service failed: rpc error: code = Unknown desc = exit status 1","level":"info","msg":"Stop all services failed, try times=2.","time":"2023-03-23T16:33:03.997Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.108.171","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-23T16:33:05.998Z"}
{"error":"stop services in host 10.122.108.171 failed: agent, stop service failed: rpc error: code = Unknown desc = exit status 1","level":"info","msg":"Stop all services failed, try times=3.","time":"2023-03-23T16:33:06.023Z"}
Fix failed when restore failed stop services in host 10.122.108.171 failed: agent, stop service failed: rpc error: code = Unknown desc = exit status 1
Error: download meta data to cluster failed: download meta files from local:///home/work/backup/BACKUP_2023_03_22_10_52_09/meta to /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta failed: rpc error: code = Unknown desc = source external uri: local:///home/work/backup/BACKUP_2023_03_22_10_52_09/meta does not exist
Usage:
  br restore full [flags]

Flags:
  -h, --help   help for full

Global Flags:
      --concurrency int        Max concurrency for download data (default 5)
      --debug                  Output log in debug level or not
      --log string             Specify br detail log path (default "br.log")
      --meta string            Specify meta server
      --name string            Specify backup name
      --s3.access_key string   S3 Option: set access key id
      --s3.endpoint string     S3 Option: set the S3 endpoint URL, please specify the http or https scheme explicitly
      --s3.region string       S3 Option: set region or location to upload or download backup
      --s3.secret_key string   S3 Option: set secret key for access id
      --storage string         backup target url, format: <SCHEME>://<PATH>.
                                   <SCHEME>: a string indicating which backend type. optional: local, s3.
                                   now only s3-compatible is supported.
                                   example:
                                   for local - "local:///the/local/path/to/backup"
                                   for s3  - "s3://example/url/to/the/backup"

{"level":"fatal","msg":"download meta data to cluster failed: download meta files from local:///home/work/backup/BACKUP_2023_03_22_10_52_09/meta to /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta failed: rpc error: code = Unknown desc = source external uri: local:///home/work/backup/BACKUP_2023_03_22_10_52_09/meta does not exist","time":"2023-03-23T16:33:09.025Z"}

相关日志显示未 10.122.108.171 这台机器无法结束进程。

实际看机器情况

image

这台机器上的进程已经结束了

你好,看错误日志是说meta目录没找到,请问你的nebula集群有多个metad服务吗。使用BR备份数据到本地,默认只备份 leader metad 的数据,因此当有多个 metad 节点时,需要手动将备份后的 leader metad 的目录拷贝到其他 follower meatd 节点的对应目录。

@kqzh 大佬
集群目前有三个 metad,

目前将lead的meta已经备份到各个节点的备份数据下
leader

follower


{"level":"info","meta address":"10.122.107.172:8559","msg":"Try to connect meta service.","time":"2023-03-23T17:18:59.451Z"}
{"level":"info","meta address":"10.122.107.172:8559","msg":"Connect meta server successfully.","time":"2023-03-23T17:18:59.455Z"}
{"level":"info","meta address":"10.122.108.171:8559","msg":"Try to connect meta service.","time":"2023-03-23T17:18:59.456Z"}
{"level":"info","meta address":"10.122.108.171:8559","msg":"Connect meta server successfully.","time":"2023-03-23T17:18:59.457Z"}
{"level":"info","meta address":"10.122.108.171:8559","msg":"Try to connect meta service.","time":"2023-03-23T17:18:59.457Z"}
{"level":"info","meta address":"10.122.108.171:8559","msg":"Connect meta server successfully.","time":"2023-03-23T17:18:59.462Z"}
{"level":"info","meta address":"10.161.85.24:8559","msg":"Try to connect meta service.","time":"2023-03-23T17:18:59.463Z"}
{"level":"info","meta address":"10.161.85.24:8559","msg":"Connect meta server successfully.","time":"2023-03-23T17:18:59.464Z"}
{"level":"info","meta address":"10.122.107.172:8559","msg":"Try to connect meta service.","time":"2023-03-23T17:18:59.464Z"}
{"level":"info","meta address":"10.122.107.172:8559","msg":"Connect meta server successfully.","time":"2023-03-23T17:18:59.465Z"}
{"host info":"map[10.122.107.172:10.122.107.172:8779[STORAGE]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.122.107.172:8346[AGENT]: (data: , root: ) | 10.122.107.172:8669[GRAPH]: (data: , root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.122.107.172:8559[META]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) 10.122.108.171:10.122.108.171:8779[STORAGE]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.122.108.171:8346[AGENT]: (data: , root: ) | 10.122.108.171:8669[GRAPH]: (data: , root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.122.108.171:8559[META]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) 10.161.85.24:10.161.85.24:8779[STORAGE]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.161.85.24:8346[AGENT]: (data: , root: ) | 10.161.85.24:8669[GRAPH]: (data: , root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.161.85.24:8559[META]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64)]","level":"info","msg":"Get cluster topology from the nebula.","time":"2023-03-23T17:18:59.465Z"}
{"backup":"BACKUP_2023_03_22_10_52_09","level":"info","msg":"Check backup dir successfully.","time":"2023-03-23T17:18:59.465Z","uri":"local:///home/work/backup/BACKUP_2023_03_22_10_52_09"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.108.171","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-23T17:18:59.467Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.108.171","level":"info","msg":"Stop services.","role":"GRAPH","time":"2023-03-23T17:18:59.498Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.108.171","level":"info","msg":"Stop services.","role":"META","time":"2023-03-23T17:18:59.524Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.161.85.24","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-23T17:18:59.549Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.161.85.24","level":"info","msg":"Stop services.","role":"GRAPH","time":"2023-03-23T17:18:59.579Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.161.85.24","level":"info","msg":"Stop services.","role":"META","time":"2023-03-23T17:18:59.608Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.107.172","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-23T17:18:59.636Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.107.172","level":"info","msg":"Stop services.","role":"GRAPH","time":"2023-03-23T17:18:59.666Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.107.172","level":"info","msg":"Stop services.","role":"META","time":"2023-03-23T17:18:59.696Z"}
{"backup":"BACKUP_2023_03_22_10_52_09","level":"info","msg":"Stop cluster successfully.","time":"2023-03-23T17:18:59.724Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.161.85.24","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-23T17:19:00.036Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.161.85.24","level":"info","msg":"Stop services.","role":"GRAPH","time":"2023-03-23T17:19:00.063Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.161.85.24","level":"info","msg":"Stop services.","role":"META","time":"2023-03-23T17:19:00.091Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.107.172","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-23T17:19:00.117Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.107.172","level":"info","msg":"Stop services.","role":"GRAPH","time":"2023-03-23T17:19:00.142Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.107.172","level":"info","msg":"Stop services.","role":"META","time":"2023-03-23T17:19:00.167Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.108.171","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-23T17:19:00.192Z"}
{"error":"stop services in host 10.122.108.171 failed: agent, stop service failed: rpc error: code = Unknown desc = exit status 1","level":"info","msg":"Stop all services failed, try times=1.","time":"2023-03-23T17:19:00.216Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.161.85.24","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-23T17:19:01.217Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.161.85.24","level":"info","msg":"Stop services.","role":"GRAPH","time":"2023-03-23T17:19:01.247Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.161.85.24","level":"info","msg":"Stop services.","role":"META","time":"2023-03-23T17:19:01.274Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.107.172","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-23T17:19:01.301Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.107.172","level":"info","msg":"Stop services.","role":"GRAPH","time":"2023-03-23T17:19:01.329Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.107.172","level":"info","msg":"Stop services.","role":"META","time":"2023-03-23T17:19:01.355Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.108.171","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-23T17:19:01.380Z"}
{"error":"stop services in host 10.122.108.171 failed: agent, stop service failed: rpc error: code = Unknown desc = exit status 1","level":"info","msg":"Stop all services failed, try times=2.","time":"2023-03-23T17:19:01.406Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.161.85.24","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-23T17:19:03.407Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.161.85.24","level":"info","msg":"Stop services.","role":"GRAPH","time":"2023-03-23T17:19:03.436Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.161.85.24","level":"info","msg":"Stop services.","role":"META","time":"2023-03-23T17:19:03.463Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.107.172","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-23T17:19:03.491Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.107.172","level":"info","msg":"Stop services.","role":"GRAPH","time":"2023-03-23T17:19:03.519Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.107.172","level":"info","msg":"Stop services.","role":"META","time":"2023-03-23T17:19:03.545Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.108.171","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-23T17:19:03.569Z"}
{"error":"stop services in host 10.122.108.171 failed: agent, stop service failed: rpc error: code = Unknown desc = exit status 1","level":"info","msg":"Stop all services failed, try times=3.","time":"2023-03-23T17:19:03.594Z"}
Fix failed when restore failed stop services in host 10.122.108.171 failed: agent, stop service failed: rpc error: code = Unknown desc = exit status 1
Error: backup origin data path failed: move dir from /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula to /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula_old_1679563139 failed: agent, move dir failed: rpc error: code = Unknown desc = /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula does not exist
Usage:
  br restore full [flags]

Flags:
  -h, --help   help for full

Global Flags:
      --concurrency int        Max concurrency for download data (default 5)
      --debug                  Output log in debug level or not
      --log string             Specify br detail log path (default "br.log")
      --meta string            Specify meta server
      --name string            Specify backup name
      --s3.access_key string   S3 Option: set access key id
      --s3.endpoint string     S3 Option: set the S3 endpoint URL, please specify the http or https scheme explicitly
      --s3.region string       S3 Option: set region or location to upload or download backup
      --s3.secret_key string   S3 Option: set secret key for access id
      --storage string         backup target url, format: <SCHEME>://<PATH>.
                                   <SCHEME>: a string indicating which backend type. optional: local, s3.
                                   now only s3-compatible is supported.
                                   example:
                                   for local - "local:///the/local/path/to/backup"
                                   for s3  - "s3://example/url/to/the/backup"

{"level":"fatal","msg":"backup origin data path failed: move dir from /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula to /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula_old_1679563139 failed: agent, move dir failed: rpc error: code = Unknown desc = /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula does not exist","time":"2023-03-23T17:19:06.596Z"}

了解,需要排查下服务停止不掉的原因,检查下agent权限是否够,正常情况下是能停掉的。如果想看具体错误,可以编译下br最新master的代码GitHub - vesoft-inc/nebula-br: Backup and restore utility for Nebula Graph ,加了一个优化可以把错误原因提前打出来,3.3.0里似乎被吞了

@kqzh
目前编译了 master 上的代码,但是看报错好像没有什么区别

{"level":"info","meta address":"10.122.107.172:8559","msg":"Try to connect meta service.","time":"2023-03-23T19:19:03.069Z"}
{"level":"info","meta address":"10.122.107.172:8559","msg":"Connect meta server successfully.","time":"2023-03-23T19:19:03.070Z"}
{"level":"info","meta address":"10.122.108.171:8559","msg":"Try to connect meta service.","time":"2023-03-23T19:19:03.070Z"}
{"level":"info","meta address":"10.122.108.171:8559","msg":"Connect meta server successfully.","time":"2023-03-23T19:19:03.071Z"}
{"level":"info","meta address":"10.122.107.172:8559","msg":"Try to connect meta service.","time":"2023-03-23T19:19:03.071Z"}
{"level":"info","meta address":"10.122.107.172:8559","msg":"Connect meta server successfully.","time":"2023-03-23T19:19:03.073Z"}
{"level":"info","meta address":"10.122.108.171:8559","msg":"Try to connect meta service.","time":"2023-03-23T19:19:03.073Z"}
{"level":"info","meta address":"10.122.108.171:8559","msg":"Connect meta server successfully.","time":"2023-03-23T19:19:03.074Z"}
{"level":"info","meta address":"10.161.85.24:8559","msg":"Try to connect meta service.","time":"2023-03-23T19:19:03.075Z"}
{"level":"info","meta address":"10.161.85.24:8559","msg":"Connect meta server successfully.","time":"2023-03-23T19:19:03.076Z"}
{"host info":"map[10.122.107.172:10.122.107.172:8779[STORAGE]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.122.107.172:8346[AGENT]: (data: , root: ) | 10.122.107.172:8669[GRAPH]: (data: , root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.122.107.172:8559[META]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) 10.122.108.171:10.122.108.171:8779[STORAGE]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.122.108.171:8346[AGENT]: (data: , root: ) | 10.122.108.171:8669[GRAPH]: (data: , root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.122.108.171:8559[META]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) 10.161.85.24:10.161.85.24:8779[STORAGE]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.161.85.24:8346[AGENT]: (data: , root: ) | 10.161.85.24:8669[GRAPH]: (data: , root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.161.85.24:8559[META]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64)]","level":"info","msg":"Get cluster topology from the nebula.","time":"2023-03-23T19:19:03.076Z"}
{"backup":"BACKUP_2023_03_22_10_52_09","level":"info","msg":"Check backup dir successfully.","time":"2023-03-23T19:19:03.076Z","uri":"local:///home/work/backup/BACKUP_2023_03_22_10_52_09"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.107.172","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-23T19:19:03.077Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.107.172","level":"info","msg":"Stop services.","role":"GRAPH","time":"2023-03-23T19:19:03.106Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.107.172","level":"info","msg":"Stop services.","role":"META","time":"2023-03-23T19:19:03.133Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.108.171","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-23T19:19:03.161Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.108.171","level":"info","msg":"Stop services.","role":"GRAPH","time":"2023-03-23T19:19:03.188Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.108.171","level":"info","msg":"Stop services.","role":"META","time":"2023-03-23T19:19:03.214Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.161.85.24","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-23T19:19:03.239Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.161.85.24","level":"info","msg":"Stop services.","role":"GRAPH","time":"2023-03-23T19:19:03.268Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.161.85.24","level":"info","msg":"Stop services.","role":"META","time":"2023-03-23T19:19:03.296Z"}
{"backup":"BACKUP_2023_03_22_10_52_09","level":"info","msg":"Stop cluster successfully.","time":"2023-03-23T19:19:03.324Z"}
{"addr":"10.122.108.171:8779","backup path":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula_old_1679570343","level":"info","msg":"Backup origin storage data path successfully.","origin path":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula","time":"2023-03-23T19:19:03.325Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.161.85.24","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-23T19:19:03.636Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.161.85.24","level":"info","msg":"Stop services.","role":"GRAPH","time":"2023-03-23T19:19:03.663Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.161.85.24","level":"info","msg":"Stop services.","role":"META","time":"2023-03-23T19:19:03.689Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.107.172","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-23T19:19:03.716Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.107.172","level":"info","msg":"Stop services.","role":"GRAPH","time":"2023-03-23T19:19:03.741Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.107.172","level":"info","msg":"Stop services.","role":"META","time":"2023-03-23T19:19:03.766Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.108.171","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-23T19:19:03.790Z"}
{"error":"stop services in host 10.122.108.171 failed: agent, stop service failed: rpc error: code = Unknown desc = exit status 1","level":"info","msg":"Stop all services failed, try times=1.","time":"2023-03-23T19:19:03.815Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.108.171","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-23T19:19:04.816Z"}
{"error":"stop services in host 10.122.108.171 failed: agent, stop service failed: rpc error: code = Unknown desc = exit status 1","level":"info","msg":"Stop all services failed, try times=2.","time":"2023-03-23T19:19:04.840Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.108.171","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-23T19:19:06.841Z"}
{"error":"stop services in host 10.122.108.171 failed: agent, stop service failed: rpc error: code = Unknown desc = exit status 1","level":"info","msg":"Stop all services failed, try times=3.","time":"2023-03-23T19:19:06.866Z"}
Fix failed when restore failed stop services in host 10.122.108.171 failed: agent, stop service failed: rpc error: code = Unknown desc = exit status 1
Error: backup origin data path failed: move dir from /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula to /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula_old_1679570343 failed: agent, move dir failed: rpc error: code = Unknown desc = /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula does not exist
Usage:
  br restore full [flags]

Flags:
  -h, --help   help for full

Global Flags:
      --concurrency int        Max concurrency for download data (default 5)
      --debug                  Output log in debug level or not
      --log string             Specify br detail log path (default "br.log")
      --meta string            Specify meta server
      --name string            Specify backup name
      --s3.access_key string   S3 Option: set access key id
      --s3.endpoint string     S3 Option: set the S3 endpoint URL, please specify the http or https scheme explicitly
      --s3.region string       S3 Option: set region or location to upload or download backup
      --s3.secret_key string   S3 Option: set secret key for access id
      --storage string         backup target url, format: <SCHEME>://<PATH>.
                                   <SCHEME>: a string indicating which backend type. optional: local, s3.
                                   now only s3-compatible is supported.
                                   example:
                                   for local - "local:///the/local/path/to/backup"
                                   for s3  - "s3://example/url/to/the/backup"

{"level":"fatal","msg":"backup origin data path failed: move dir from /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula to /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula_old_1679570343 failed: agent, move dir failed: rpc error: code = Unknown desc = /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula does not exist","time":"2023-03-23T19:19:09.868Z"}

你好,不好意思,是我记错了,我之前只加了备份的,恢复那里没有加错误提前打印。nebula-br/restore.go at master · vesoft-inc/nebula-br · GitHub
可以在这个位置加一行代码打印err日志,编译完再试一下,应该会有恢复失败的错误原因,也欢迎给我们贡献pr

添加了相关日志

{"level":"info","meta address":"10.122.107.172:8559","msg":"Try to connect meta service.","time":"2023-03-24T11:00:57.898Z"}
{"level":"info","meta address":"10.122.107.172:8559","msg":"Connect meta server successfully.","time":"2023-03-24T11:00:57.899Z"}
{"level":"info","meta address":"10.122.108.171:8559","msg":"Try to connect meta service.","time":"2023-03-24T11:00:57.900Z"}
{"level":"info","meta address":"10.122.108.171:8559","msg":"Connect meta server successfully.","time":"2023-03-24T11:00:57.902Z"}
{"level":"info","meta address":"10.161.85.24:8559","msg":"Try to connect meta service.","time":"2023-03-24T11:00:57.903Z"}
{"level":"info","meta address":"10.161.85.24:8559","msg":"Connect meta server successfully.","time":"2023-03-24T11:00:57.904Z"}
{"level":"info","meta address":"10.122.107.172:8559","msg":"Try to connect meta service.","time":"2023-03-24T11:00:57.904Z"}
{"level":"info","meta address":"10.122.107.172:8559","msg":"Connect meta server successfully.","time":"2023-03-24T11:00:57.904Z"}
{"host info":"map[10.122.107.172:10.122.107.172:8779[STORAGE]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.122.107.172:8346[AGENT]: (data: , root: ) | 10.122.107.172:8669[GRAPH]: (data: , root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.122.107.172:8559[META]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) 10.122.108.171:10.122.108.171:8779[STORAGE]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.122.108.171:8346[AGENT]: (data: , root: ) | 10.122.108.171:8669[GRAPH]: (data: , root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.122.108.171:8559[META]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) 10.161.85.24:10.161.85.24:8779[STORAGE]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.161.85.24:8346[AGENT]: (data: , root: ) | 10.161.85.24:8669[GRAPH]: (data: , root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64) | 10.161.85.24:8559[META]: (data: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta, root: /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64)]","level":"info","msg":"Get cluster topology from the nebula.","time":"2023-03-24T11:00:57.905Z"}
{"backup":"BACKUP_2023_03_22_10_52_09","level":"info","msg":"Check backup dir successfully.","time":"2023-03-24T11:00:57.905Z","uri":"local:///home/work/backup/BACKUP_2023_03_22_10_52_09"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.108.171","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-24T11:00:57.906Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.108.171","level":"info","msg":"Stop services.","role":"GRAPH","time":"2023-03-24T11:00:57.931Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.108.171","level":"info","msg":"Stop services.","role":"META","time":"2023-03-24T11:00:57.955Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.161.85.24","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-24T11:00:57.980Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.161.85.24","level":"info","msg":"Stop services.","role":"GRAPH","time":"2023-03-24T11:00:58.011Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.161.85.24","level":"info","msg":"Stop services.","role":"META","time":"2023-03-24T11:00:58.042Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.107.172","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-24T11:00:58.074Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.107.172","level":"info","msg":"Stop services.","role":"GRAPH","time":"2023-03-24T11:00:58.111Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.107.172","level":"info","msg":"Stop services.","role":"META","time":"2023-03-24T11:00:58.138Z"}
{"backup":"BACKUP_2023_03_22_10_52_09","level":"info","msg":"Stop cluster successfully.","time":"2023-03-24T11:00:58.167Z"}
{"addr":"10.122.108.171:8779","backup path":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula_old_1679626858","level":"info","msg":"Backup origin storage data path successfully.","origin path":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula","time":"2023-03-24T11:00:58.167Z"}
{"addr":"10.161.85.24:8779","backup path":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula_old_1679626858","level":"info","msg":"Backup origin storage data path successfully.","origin path":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula","time":"2023-03-24T11:00:58.168Z"}
{"addr":"10.122.107.172:8779","backup path":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula_old_1679626858","level":"info","msg":"Backup origin storage data path successfully.","origin path":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula","time":"2023-03-24T11:00:58.168Z"}
{"addr":"10.122.108.171:8559","backup path":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula_old_1679626858","level":"info","msg":"Backup origin meta data path successfully.","origin path":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula","time":"2023-03-24T11:00:58.169Z"}
{"addr":"10.161.85.24:8559","backup path":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula_old_1679626858","level":"info","msg":"Backup origin meta data path successfully.","origin path":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula","time":"2023-03-24T11:00:58.169Z"}
{"addr":"10.122.107.172:8559","backup path":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula_old_1679626858","level":"info","msg":"Backup origin meta data path successfully.","origin path":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula","time":"2023-03-24T11:00:58.169Z"}
{"backup":"BACKUP_2023_03_22_10_52_09","level":"info","msg":"Backup origin cluster data successfully.","time":"2023-03-24T11:00:58.169Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.108.171","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-24T11:00:58.495Z"}
{"error":"stop services in host 10.122.108.171 failed: agent, stop service failed: rpc error: code = Unknown desc = exit status 1","level":"info","msg":"Stop all services failed, try times=1.","time":"2023-03-24T11:00:58.521Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.107.172","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-24T11:00:59.522Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.107.172","level":"info","msg":"Stop services.","role":"GRAPH","time":"2023-03-24T11:00:59.547Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.107.172","level":"info","msg":"Stop services.","role":"META","time":"2023-03-24T11:00:59.572Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.108.171","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-24T11:00:59.598Z"}
{"error":"stop services in host 10.122.108.171 failed: agent, stop service failed: rpc error: code = Unknown desc = exit status 1","level":"info","msg":"Stop all services failed, try times=2.","time":"2023-03-24T11:00:59.624Z"}
{"dir":"/home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64","host":"10.122.108.171","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-03-24T11:01:01.625Z"}
{"error":"stop services in host 10.122.108.171 failed: agent, stop service failed: rpc error: code = Unknown desc = exit status 1","level":"info","msg":"Stop all services failed, try times=3.","time":"2023-03-24T11:01:01.650Z"}
Fix failed when restore failed stop services in host 10.122.108.171 failed: agent, stop service failed: rpc error: code = Unknown desc = exit status 1
Error: download meta data to cluster failed: download meta files from local:///home/work/backup/BACKUP_2023_03_22_10_52_09/meta to /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta failed: rpc error: code = Unknown desc = source external uri: local:///home/work/backup/BACKUP_2023_03_22_10_52_09/meta does not exist
Usage:
  br restore full [flags]

Flags:
  -h, --help   help for full

Global Flags:
      --concurrency int        Max concurrency for download data (default 5)
      --debug                  Output log in debug level or not
      --log string             Specify br detail log path (default "br.log")
      --meta string            Specify meta server
      --name string            Specify backup name
      --s3.access_key string   S3 Option: set access key id
      --s3.endpoint string     S3 Option: set the S3 endpoint URL, please specify the http or https scheme explicitly
      --s3.region string       S3 Option: set region or location to upload or download backup
      --s3.secret_key string   S3 Option: set secret key for access id
      --storage string         backup target url, format: <SCHEME>://<PATH>.
                                   <SCHEME>: a string indicating which backend type. optional: local, s3.
                                   now only s3-compatible is supported.
                                   example:
                                   for local - "local:///the/local/path/to/backup"
                                   for s3  - "s3://example/url/to/the/backup"

{"level":"fatal","msg":"download meta data to cluster failed: download meta files from local:///home/work/backup/BACKUP_2023_03_22_10_52_09/meta to /home/work/nebula-all/nebula-graph-3.4.1.el7.x86_64/data/meta failed: rpc error: code = Unknown desc = source external uri: local:///home/work/backup/BACKUP_2023_03_22_10_52_09/meta does not exist","time":"2023-03-24T11:01:04.652Z"}

日志末尾提示了meta 数据不对

重新检查了一下备份数据,应该将 meta 文件同步到其他节点下,而不是 BACKUP_xxxx.meta 文件同步到其他节点下。

重新同步后,备份成功。

@kqzh 感谢大佬的帮助

2 个赞

此话题已在最后回复的 7 天后被自动关闭。不再允许新回复。