br 数据恢复失败

问答 问题
1

nebula 版本：3.4.1
部署方式: 分布式
安装方式：RPM
是否上生产环境 是
硬件信息
磁盘（ 推荐使用 SSD）
CPU、内存信息 32c, 128GB

在使用备份工具备份成功之后,想用备份数据恢复图数据库中的数据,此时遇到了问题:
恢复命令:
./br restore full --meta “xxxxx:9559” --storage “local:///data/datum/nebula_data_backup” --name BACKUP_2023_06_27_11_31_51

运行信息:
{“level”:“info”,“msg”:“Download storage data to cluster successfully.”,“time”:“2023-06-27T14:07:30.142Z”}
{“level”:“info”,“msg”:“All services are OK.”,“time”:“2023-06-27T14:07:30.409Z”}
Error: start meta service failed: start meta service 10.130.115.135:9559 by agent failed: agent, start service failed: rpc error: code = Unknown desc = exit status 1

去对应的节点下,找到对应的meta日志信息,如下:
E20230627 14:07:27.982591 8796 FileUtils.cpp:377] Failed to read the directory “/data/src/nebula-3.4.1/data/meta/nebula” (2): No such file or directory

但是在对应的目录下,能找到相应的文件:
drwxr-xr-x 3 root root 15 2023-06-27 14:07:27.981882217 +0800 nebula

所以有点看不懂了,请求热心大佬帮助!!!

2

您好，这个错误和meta日志里的error没关系，看起来是因为agent启动meta服务失败了，可以手动在nebula-3.4.1/scripts目录，执行下nebula.service start metad看看，meta是否能正常启动

3

大佬,是可以正常启动的

4

看起来没什么问题，请问重复执行restore，还是会报一样的错吗

5

是的,都是这个错误

6

请问agent和nebula，都是使用root启动的吗，正常流程这样跑是没问题的，也可能是因为agent权限问题

7

都是用root启动的

8

请问您用的br和agent版本是多少，我去复现下看看

1 个赞
9

BR工具

wget https://github.com/vesoft-inc/nebula-br/releases/download/v3.3.0/br-3.3.0-linux-amd64

Agent工具

wget https://github.com/vesoft-inc/nebula-agent/releases/download/v3.4.0/agent-3.4.0-linux-amd64

10

谢谢大佬了,另外我这里br工具部署在第四个节点上,第一到第三个节点部署nebula集群,然后为了备份,使用的是nfs,将第四个节点的一块盘挂到了nebula集群的各个节点上

1 个赞
11

大佬, 你能复现问题吗?

12

您好，不好意思之前忘了回复您，我在本地没有复现出来，感觉可能是环境上的一些问题

13

好的,谢谢大佬, 那我再试试看

1 个赞
14

大佬, 我又重试了一下, 还是有问题, 另外让我感到奇怪的是日志信息中前面还是现实meta已经启动成功了,但是后面却是现实启动失败, 这是怎么回事啊

image
image1142×179 41.8 KB

15 
storage "local:///data/cloudera/parcels/nebula/nebula-graph-3.4.1/backup/" --name BACKUP_2023_07_28_14_30_05
{"level":"info","meta address":"10.129.88.74:9559","msg":"Try to connect meta service.","time":"2023-07-28T16:39:11.703Z"}
{"level":"info","meta address":"10.129.88.74:9559","msg":"Connect meta server successfully.","time":"2023-07-28T16:39:11.704Z"}
{"level":"info","meta address":"10.129.88.82:9559","msg":"Try to connect meta service.","time":"2023-07-28T16:39:11.704Z"}
{"level":"info","meta address":"10.129.88.82:9559","msg":"Connect meta server successfully.","time":"2023-07-28T16:39:11.727Z"}
{"level":"info","meta address":"10.129.88.82:9559","msg":"Try to connect meta service.","time":"2023-07-28T16:39:11.728Z"}
{"level":"info","meta address":"10.129.88.82:9559","msg":"Connect meta server successfully.","time":"2023-07-28T16:39:11.728Z"}
{"level":"info","meta address":"10.129.88.25:9559","msg":"Try to connect meta service.","time":"2023-07-28T16:39:11.729Z"}
{"level":"info","meta address":"10.129.88.25:9559","msg":"Connect meta server successfully.","time":"2023-07-28T16:39:11.731Z"}
{"level":"info","meta address":"10.129.88.74:9559","msg":"Try to connect meta service.","time":"2023-07-28T16:39:11.731Z"}
{"level":"info","meta address":"10.129.88.74:9559","msg":"Connect meta server successfully.","time":"2023-07-28T16:39:11.732Z"}
{"host info":"map[10.129.88.25:10.129.88.25:9779[STORAGE]: (data: /data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/storage, root: /data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64) | 10.129.88.25:8888[AGENT]: (data: , root: ) | 10.129.88.25:9669[GRAPH]: (data: , root: /data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64) | 10.129.88.25:9559[META]: (data: /data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/meta, root: /data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64) 10.129.88.74:10.129.88.74:9779[STORAGE]: (data: /data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/storage, root: /data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64) | 10.129.88.74:8888[AGENT]: (data: , root: ) | 10.129.88.74:9669[GRAPH]: (data: , root: /data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64) | 10.129.88.74:9559[META]: (data: /data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/meta, root: /data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64) 10.129.88.82:10.129.88.82:9779[STORAGE]: (data: /data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/storage, root: /data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64) | 10.129.88.82:8888[AGENT]: (data: , root: ) | 10.129.88.82:9669[GRAPH]: (data: , root: /data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64) | 10.129.88.82:9559[META]: (data: /data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/meta, root: /data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64)]","level":"info","msg":"Get cluster topology from the nebula.","time":"2023-07-28T16:39:11.732Z"}
{"backup":"BACKUP_2023_07_28_14_30_05","level":"info","msg":"Check backup dir successfully.","time":"2023-07-28T16:39:11.732Z","uri":"local:///data/cloudera/parcels/nebula/nebula-graph-3.4.1/backup/BACKUP_2023_07_28_14_30_05"}
{"dir":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64","host":"10.129.88.74","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-07-28T16:39:11.738Z"}
{"dir":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64","host":"10.129.88.74","level":"info","msg":"Stop services.","role":"GRAPH","time":"2023-07-28T16:39:11.758Z"}
{"dir":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64","host":"10.129.88.74","level":"info","msg":"Stop services.","role":"META","time":"2023-07-28T16:39:11.778Z"}
{"dir":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64","host":"10.129.88.82","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-07-28T16:39:11.802Z"}
{"dir":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64","host":"10.129.88.82","level":"info","msg":"Stop services.","role":"GRAPH","time":"2023-07-28T16:39:11.822Z"}
{"dir":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64","host":"10.129.88.82","level":"info","msg":"Stop services.","role":"META","time":"2023-07-28T16:39:11.841Z"}
{"dir":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64","host":"10.129.88.25","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-07-28T16:39:11.867Z"}
{"dir":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64","host":"10.129.88.25","level":"info","msg":"Stop services.","role":"GRAPH","time":"2023-07-28T16:39:11.900Z"}
{"dir":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64","host":"10.129.88.25","level":"info","msg":"Stop services.","role":"META","time":"2023-07-28T16:39:11.935Z"}
{"backup":"BACKUP_2023_07_28_14_30_05","level":"info","msg":"Stop cluster successfully.","time":"2023-07-28T16:39:11.975Z"}
{"addr":"10.129.88.82:9779","backup path":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula_old_1690533551","level":"info","msg":"Backup origin storage data path successfully.","origin path":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula","time":"2023-07-28T16:39:11.976Z"}
{"addr":"10.129.88.25:9779","backup path":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula_old_1690533551","level":"info","msg":"Backup origin storage data path successfully.","origin path":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula","time":"2023-07-28T16:39:11.976Z"}
{"addr":"10.129.88.74:9779","backup path":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula_old_1690533551","level":"info","msg":"Backup origin storage data path successfully.","origin path":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula","time":"2023-07-28T16:39:11.977Z"}
{"addr":"10.129.88.82:9559","backup path":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula_old_1690533551","level":"info","msg":"Backup origin meta data path successfully.","origin path":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula","time":"2023-07-28T16:39:11.977Z"}
{"addr":"10.129.88.25:9559","backup path":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula_old_1690533551","level":"info","msg":"Backup origin meta data path successfully.","origin path":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula","time":"2023-07-28T16:39:11.978Z"}
{"addr":"10.129.88.74:9559","backup path":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula_old_1690533551","level":"info","msg":"Backup origin meta data path successfully.","origin path":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula","time":"2023-07-28T16:39:11.978Z"}
{"backup":"BACKUP_2023_07_28_14_30_05","level":"info","msg":"Backup origin cluster data successfully.","time":"2023-07-28T16:39:11.978Z"}
{"level":"info","msg":"Download meta data to cluster successfully.","time":"2023-07-28T16:39:12.138Z"}
{"addr":"10.129.88.25:9779","external":"local:///data/cloudera/parcels/nebula/nebula-graph-3.4.1/backup/BACKUP_2023_07_28_14_30_05/data/10.129.88.25:9779/data0/","level":"info","local":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula","msg":"Download storage data successfully.","time":"2023-07-28T16:39:12.555Z"}
{"addr":"10.129.88.74:9779","external":"local:///data/cloudera/parcels/nebula/nebula-graph-3.4.1/backup/BACKUP_2023_07_28_14_30_05/data/10.129.88.74:9779/data0/","level":"info","local":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula","msg":"Download storage data successfully.","time":"2023-07-28T16:39:12.935Z"}
{"addr":"10.129.88.82:9779","external":"local:///data/cloudera/parcels/nebula/nebula-graph-3.4.1/backup/BACKUP_2023_07_28_14_30_05/data/10.129.88.82:9779/data0/","level":"info","local":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula","msg":"Download storage data successfully.","time":"2023-07-28T16:39:13.314Z"}
{"level":"info","msg":"Download storage data to cluster successfully.","time":"2023-07-28T16:39:13.314Z"}
{"addr":"10.129.88.82:9559","level":"info","msg":"Start meta service successfully.","time":"2023-07-28T16:39:13.337Z"}
{"dir":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64","host":"10.129.88.82","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-07-28T16:39:13.685Z"}
{"dir":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64","host":"10.129.88.82","level":"info","msg":"Stop services.","role":"GRAPH","time":"2023-07-28T16:39:13.702Z"}
{"dir":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64","host":"10.129.88.82","level":"info","msg":"Stop services.","role":"META","time":"2023-07-28T16:39:13.718Z"}
{"dir":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64","host":"10.129.88.25","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-07-28T16:39:13.733Z"}
{"dir":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64","host":"10.129.88.25","level":"info","msg":"Stop services.","role":"GRAPH","time":"2023-07-28T16:39:13.765Z"}
{"dir":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64","host":"10.129.88.25","level":"info","msg":"Stop services.","role":"META","time":"2023-07-28T16:39:13.796Z"}
{"dir":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64","host":"10.129.88.74","level":"info","msg":"Stop services.","role":"STORAGE","time":"2023-07-28T16:39:13.827Z"}
{"dir":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64","host":"10.129.88.74","level":"info","msg":"Stop services.","role":"GRAPH","time":"2023-07-28T16:39:13.848Z"}
{"dir":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64","host":"10.129.88.74","level":"info","msg":"Stop services.","role":"META","time":"2023-07-28T16:39:13.866Z"}
{"backup path":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula_old_1690533551","level":"info","msg":"Moveback origin STORAGE data path successfully","name":"STORAGE[10.129.88.82:9779]","origin path":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula","time":"2023-07-28T16:39:13.900Z"}
{"backup path":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula_old_1690533551","level":"info","msg":"Moveback origin STORAGE data path successfully","name":"STORAGE[10.129.88.25:9779]","origin path":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula","time":"2023-07-28T16:39:13.926Z"}
{"backup path":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula_old_1690533551","level":"info","msg":"Moveback origin STORAGE data path successfully","name":"STORAGE[10.129.88.74:9779]","origin path":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/storage/nebula","time":"2023-07-28T16:39:13.945Z"}
{"backup path":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula_old_1690533551","level":"info","msg":"Moveback origin META data path successfully","name":"META[10.129.88.74:9559]","origin path":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula","time":"2023-07-28T16:39:13.945Z"}
{"backup path":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula_old_1690533551","level":"info","msg":"Moveback origin META data path successfully","name":"META[10.129.88.82:9559]","origin path":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula","time":"2023-07-28T16:39:13.947Z"}
{"backup path":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula_old_1690533551","level":"info","msg":"Moveback origin META data path successfully","name":"META[10.129.88.25:9559]","origin path":"/data/cloudera/parcels/nebula/nebula-graph-3.4.1/nebula-graph-3.4.1.el7.x86_64/data/meta/nebula","time":"2023-07-28T16:39:13.949Z"}
{"addr":"10.129.88.82:9779","level":"info","msg":"Start STORAGE[10.129.88.82:9779] by agent successfully.","time":"2023-07-28T16:39:14.277Z"}
{"addr":"10.129.88.82:9669","level":"info","msg":"Start GRAPH[10.129.88.82:9669] by agent successfully.","time":"2023-07-28T16:39:14.304Z"}
{"addr":"10.129.88.82:9559","level":"info","msg":"Start META[10.129.88.82:9559] by agent successfully.","time":"2023-07-28T16:39:14.330Z"}
{"addr":"10.129.88.25:9559","level":"info","msg":"Start META[10.129.88.25:9559] by agent successfully.","time":"2023-07-28T16:39:14.373Z"}
{"addr":"10.129.88.74:9559","level":"info","msg":"Start META[10.129.88.74:9559] by agent successfully.","time":"2023-07-28T16:39:14.397Z"}
Error: start meta service failed: start meta service 10.129.88.25:9559 by agent failed: agent, start service failed: rpc error: code = Unknown desc = exit status 1
Usage:
  br restore full [flags]

Flags:
  -h, --help   help for full

Global Flags:
      --concurrency int        Max concurrency for download data (default 5)
      --debug                  Output log in debug level or not
      --log string             Specify br detail log path (default "br.log")
      --meta string            Specify meta server
      --name string            Specify backup name
      --s3.access_key string   S3 Option: set access key id
      --s3.endpoint string     S3 Option: set the S3 endpoint URL, please specify the http or https scheme explicitly
      --s3.region string       S3 Option: set region or location to upload or download backup
      --s3.secret_key string   S3 Option: set secret key for access id
      --storage string         backup target url, format: <SCHEME>://<PATH>.
                                   <SCHEME>: a string indicating which backend type. optional: local, s3.
                                   now only s3-compatible is supported.
                                   example:
                                   for local - "local:///the/local/path/to/backup"
                                   for s3  - "s3://example/url/to/the/backup"

{"level":"fatal","msg":"start meta service failed: start meta service 10.129.88.25:9559 by agent failed: agent, start service failed: rpc error: code = Unknown desc = exit status 1","time":"2023-07-28T16:39:14.397Z"}
16

您好，这是因为在正常恢复流程中，启动失败，触发了回滚机制，会重新启动服务，才会有这些日志

17

我猜失败原因可能是因为meta还没停成功，就立马启动，产生了报错，我在最新的代码里加了一个等待时间来避免这个问题，可以用最新的master分支试试

1 个赞
18

此话题已在最后回复的 30 天后被自动关闭。不再允许新回复。