数据迁移失败

问答 问题
,
1

提问参考模版：

  • nebula 版本：2.6.2
  • 部署方式：分布式
  • 安装方式：RPM
  • 是否上生产环境：N
  • 硬件信息
    • 磁盘（ 推荐使用 SSD）机械硬盘
    • CPU、内存信息
  • 问题的具体描述：需求：把老服务器（2.6.2版本，单节点）的数据迁移到新集群（2.6.2，三台）
    根据查看社区讨论理解的方案：
    1新集群先配置好一台
    2修改了meta\storage\graph配置文件的ip为该机器内网ip并启动所有服务，
    3复制老服务器的data文件夹到该机器
    4停掉服务，修改meta\storage\graph配置文件的ip为集群的ip，建立好集群后balance data
    问题：完成了前三步，服务启动正常，show spaces正常，数据也能查询，但是show hosts，显示不出来spaces
  • 相关的 meta / storage / graph info 日志信息（尽量使用文本形式方便检索）
meta日志
E0713 12:07:42.559993 207849 HBProcessor.cpp:41] Reject wrong cluster host "172.16.0.164":9779!
E0713 12:07:52.560696 207849 HBProcessor.cpp:41] Reject wrong cluster host "172.16.0.164":9779!
E0713 12:08:02.561421 207849 HBProcessor.cpp:41] Reject wrong cluster host "172.16.0.164":9779!
E0713 12:08:12.562183 207849 HBProcessor.cpp:41] Reject wrong cluster host "172.16.0.164":9779!
E0713 12:08:22.562942 207849 HBProcessor.cpp:41] Reject wrong cluster host "172.16.0.164":9779!
E0713 12:08:32.563702 207849 HBProcessor.cpp:41] Reject wrong cluster host "172.16.0.164":9779!
E0713 12:08:42.564458 207849 HBProcessor.cpp:41] Reject wrong cluster host "172.16.0.164":9779!
E0713 12:08:52.565181 207849 HBProcessor.cpp:41] Reject wrong cluster host "172.16.0.164":9779!
E0713 12:09:02.565897 207849 HBProcessor.cpp:41] Reject wrong cluster host "172.16.0.164":9779!
E0713 12:09:12.566578 207849 HBProcessor.cpp:41] Reject wrong cluster host "172.16.0.164":9779!
E0713 12:09:22.567338 207849 HBProcessor.cpp:41] Reject wrong cluster host "172.16.0.164":9779!
E0713 12:09:32.568101 207849 HBProcessor.cpp:41] Reject wrong cluster host "172.16.0.164":9779!
E0713 12:09:42.568899 207849 HBProcessor.cpp:41] Reject wrong cluster host "172.16.0.164":9779!
E0713 12:09:52.569716 207849 HBProcessor.cpp:41] Reject wrong cluster host "172.16.0.164":9779!
E0713 12:10:02.570554 207849 HBProcessor.cpp:41] Reject wrong cluster host "172.16.0.164":9779!
E0713 12:10:12.571342 207849 HBProcessor.cpp:41] Reject wrong cluster host "172.16.0.164":9779!
E0713 12:10:12.799590 207836 JobDescription.cpp:179] Loading Job Description FailedE_KEY_NOT_FOUND
E0713 12:10:12.799628 207836 JobManager.cpp:90] [JobManager] load an invalid job from queue 0
E0713 12:10:12.816622 207849 HBProcessor.cpp:41] Reject wrong cluster host "172.16.0.164":9779!

0713 12:10:54.399240 208434 Balancer.cpp:296] Check Replica failed: Not enough alive host hold the part 1 Part: 1
E0713 12:10:54.399436 208434 Balancer.cpp:225] Generate tasks on space 1792 failed
E0713 12:10:54.399446 208434 Balancer.cpp:41] Create balance plan failed
E0713 12:10:54.399456 208434 BalanceProcessor.cpp:117] Balance Failed: E_NO_VALID_HOST
E0713 15:45:56.785190 208434 Balancer.cpp:950] No active hosts
E0713 15:45:56.785214 208434 Balancer.cpp:862] Building leader balance plan failed Space: 1792
E0713 15:45:56.785319 208434 Balancer.cpp:950] No active hosts
E0713 15:45:56.785351 208434 Balancer.cpp:862] Building leader balance plan failed Space: 1
E0713 15:45:56.785452 208434 Balancer.cpp:950] No active hosts
E0713 15:45:56.785463 208434 Balancer.cpp:862] Building leader balance plan failed Space: 3629
E0713 15:45:56.785492 208434 Balancer.cpp:950] No active hosts
E0713 15:45:56.785501 208434 Balancer.cpp:862] Building leader balance plan failed Space: 114
E0713 15:45:56.785530 208434 Balancer.cpp:950] No active hosts
E0713 15:45:56.785539 208434 Balancer.cpp:862] Building leader balance plan failed Space: 136
E0713 15:45:56.785571 208434 Balancer.cpp:950] No active hosts
E0713 15:45:56.785580 208434 Balancer.cpp:862] Building leader balance plan failed Space: 1229
E0713 15:45:56.785620 208434 Balancer.cpp:950] No active hosts
E0713 15:45:56.785629 208434 Balancer.cpp:862] Building leader balance plan failed Space: 242

storage日志：
E0713 12:09:22.567543 207784 MetaClient.cpp:70] Heartbeat failed, status:Wrong cluster!
E0713 12:09:32.568275 207784 MetaClient.cpp:70] Heartbeat failed, status:Wrong cluster!
E0713 12:09:42.569108 207784 MetaClient.cpp:70] Heartbeat failed, status:Wrong cluster!
E0713 12:09:52.569914 207784 MetaClient.cpp:70] Heartbeat failed, status:Wrong cluster!
E0713 12:10:02.570753 207784 MetaClient.cpp:70] Heartbeat failed, status:Wrong cluster!
E0713 12:10:12.571542 207784 MetaClient.cpp:70] Heartbeat failed, status:Wrong cluster!
E0713 12:10:12.816752 207784 MetaClient.cpp:70] Heartbeat failed, status:Wrong cluster!
E0713 12:10:12.816771 207784 MetaClient.cpp:112] Connect to the MetaServer Failed
E0713 12:10:12.816780 207784 StorageServer.cpp:163] waitForMetadReady error!
E0713 12:10:12.816788 207784 StorageDaemon.cpp:161] Storage server start failed

Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
W0713 12:10:23.027097 208366 FileBasedClusterIdMan.cpp:43] Open file failed, error No such file or directory
graph日志：
I0713 12:02:19.666379 207764 GraphDaemon.cpp:124] Starting Graph HTTP Service
I0713 12:02:19.668459 207766 WebService.cpp:124] Web service started on HTTP[19669], HTTP2[19670]
I0713 12:02:19.668509 207764 GraphDaemon.cpp:138] Number of networking IO threads: 8
I0713 12:02:19.668524 207764 GraphDaemon.cpp:147] Number of worker threads: 8
I0713 12:02:19.668696 207764 MetaClient.cpp:57] Create meta client to "172.16.0.164":9559
I0713 12:02:22.716820 207764 MetaClient.cpp:3000] Load leader ok
I0713 12:02:22.716910 207764 MetaClient.cpp:117] Register time task for heartbeat!
I0713 12:02:22.726025 207764 GraphDaemon.cpp:180] Starting nebula-graphd on 172.16.0.164:9669
I0713 12:02:48.301097 207880 GraphService.cpp:63] Authenticating user root from [::ffff:223.104.106.61]:4982
E0713 12:05:49.778800 207882 QueryInstance.cpp:108] SyntaxError: syntax error near `host'
E0713 12:05:51.838559 207885 QueryInstance.cpp:108] SyntaxError: syntax error near `hosts'
E0713 12:06:17.873505 207881 QueryInstance.cpp:108] SyntaxError: syntax error near `hosts'
E0713 12:06:22.167901 207882 QueryInstance.cpp:108] SyntaxError: syntax error near `9779'
E0713 12:08:06.792006 207882 QueryInstance.cpp:108] SyntaxError: syntax error near `9779'
E0713 12:08:25.411557 207882 QueryInstance.cpp:108] SyntaxError: syntax error near `9779'
I0713 12:10:12.805965 207764 GraphDaemon.cpp:203] Signal 15(Terminated) received, stopping this server
I0713 12:10:12.809145 207826 Acceptor.cpp:467] Dropping all connections from Acceptor=0x7f1e26233200 in thread 0x7f1e25401280
I0713 12:10:12.810890 207764 GraphDaemon.cpp:188] nebula-graphd on 172.16.0.164:9669 has been stopped

I0713 12:10:21.003968 208324 GraphDaemon.cpp:124] Starting Graph HTTP Service
I0713 12:10:21.006345 208328 WebService.cpp:124] Web service started on HTTP[19669], HTTP2[19670]
I0713 12:10:21.006402 208324 GraphDaemon.cpp:138] Number of networking IO threads: 8
I0713 12:10:21.006418 208324 GraphDaemon.cpp:147] Number of worker threads: 8
I0713 12:10:21.006606 208324 MetaClient.cpp:57] Create meta client to "172.16.0.164":9559
I0713 12:10:23.052489 208324 MetaClient.cpp:2994] Load leader of "172.16.0.164":9779 in 0 space
I0713 12:10:23.052516 208324 MetaClient.cpp:3000] Load leader ok
I0713 12:10:23.052587 208324 MetaClient.cpp:117] Register time task for heartbeat!
I0713 12:10:23.067844 208324 GraphDaemon.cpp:180] Starting nebula-graphd on 172.16.0.164:9669
E0713 12:10:49.938441 208484 QueryInstance.cpp:108] SyntaxError: syntax error near `balcance'
E0713 12:10:54.399660 208478 BalanceExecutor.cpp:27] No valid host hold the partition!
E0713 12:10:54.399744 208478 QueryInstance.cpp:108] No valid host hold the partition!
I0713 15:44:13.185906 208484 GraphService.cpp:63] Authenticating user root from [::ffff:223.104.106.61]:4990
E0713 15:45:25.160205 208483 QueryInstance.cpp:108] SemanticError: Space was not chosen.

``` :sob:
是不是这个方案有问题
1 个赞
2

只修改配置还不够，你去 console 里，用 show hosts 命令看一下，应该跟你迁移后的 storage ip 不对应，那样的话就需要手动 drop host 再重新 add host，这样应该就可以了

1 个赞
3

对应的，是新节点的ip

image
image932×449 9.11 KB

4

那就建议你用官方的 snapshot 进行迁移尝试吧

5

另外，你这个 distribution 里面都为空，可能是迁移过程有问题，数据库自动把 storage 里数据都清了，我之前也遇到过这种问题……

6

还没试但是应该也是不行，因为这个data里面就带有老数据库的数据快照，如图。但是数据和snapshot似乎都记录了老节点的IP，换到新数据库的时候IP不一致就没法使用，能不能在哪里删掉这个ip呢
image

7

我看storage里面的数据也都在啊
把快照里的data和wal复制到上层目录里，也还是没变化

image
image949×169 5.02 KB

8

No valid partition 基本可以确定是没数据的，你可以尝试一下submit job stats 然后 show stats 看一下数据统计情况，就知道了。

也许复制拷贝的方式只适合备份和恢复。我之前是不同集群但是内网地址是相同的情况下进行的这些迁移操作。

9

image
image971×517 13.6 KB

这个127.0.0.1其实就是老服务器没有改过配置的ip地址。
所以是不同ip没法迁移数据吗，但是好像社区里有人这样子做啊

10

那就是这个方法了，你试一下吧，先把新集群启动起来，然后关掉，删掉新集群数据，拷贝老的过去，启动，启动meta，用上面的帖子换掉hosts，最后启动其他服务。

这个方法我也只是了解过没有试过。

11

此话题已在最后回复的 7 天后被自动关闭。不再允许新回复。