exchange 2 导入数据提示The leader has changed. Try again later

xrfinbj · 2021 年3 月 30 日 02:30

1 console 批量写入是可以的，github上提供的nba数据我就是在console导入的
2 是这个吗？
batch: 50
partition: 10

还有一个疑惑：

spark-submit --class com.vesoft.nebula.exchange.Exchange --master local /home/xx/nebula/nebula-exchange-2.0.0.jar -c /home/xxx/nebula_application_xxx.conf -h
执行完后（我判断的依据是服务负载和db_dump计算的数据量） 没有退出 一直停留在这里

nicole · 2021 年3 月 30 日 03:56

这个配置导入的速度不快的，你看下storage的日志，写入的时候出现leader change了。

执行完没有退出，感谢提出，我们加一下。

xrfinbj · 2021 年3 月 30 日 04:08

今天早上7点多的日志，这个时候我执行了nebula-exchange导入数据
nebula-storaged.ERROR

机器1
E0330 07:34:26.572551 14620 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.8.17":9780, error code is -5
E0330 07:34:26.709370 14619 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 6] Receive response about askForVote from "172.31.8.17":9780, error code is -5



机器2
E0330 07:34:26.671720 15093 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 4] Receive response about askForVote from "172.31.8.17":9780, error code is -5
E0330 07:34:26.834892 15094 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 1] Receive response about askForVote from "172.31.8.17":9780, error code is -5
E0330 08:35:53.890802 14986 RaftPart.cpp:961] [Port: 9780, Space: 15, Part: 2] processAppendLogResponses failed!


机器3
E0330 07:34:29.145084 14457 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 8] Receive response about askForVote from "172.31.1.93":9780, error code is -6
E0330 07:34:29.145184 14457 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 8] Receive response about askForVote from "172.31.4.183":9780, error code is -6
E0330 07:34:29.145110 14456 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.1.93":9780, error code is -6
E0330 07:34:29.145243 14456 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.4.183":9780, error code is -6
E0330 07:34:29.168062 14456 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 10] Receive response about askForVote from "172.31.1.93":9780, error code is -6
。。。。

写入的时候出现leader change了，为什么呢？集群负载高没有响应？

xrfinbj · 2021 年3 月 30 日 04:12

最开始我用 --master local[4] 后来才调小的 local

nicole · 2021 年3 月 30 日 04:43

这个配置其实很低了
Storage这个日志信息请我们负责存储的同学帮忙看下哈

xrfinbj · 2021 年3 月 30 日 04:50

那我提供的信息有助于你们分析问题吗？

xrfinbj · 2021 年3 月 30 日 05:09

有需要可以随时联系我复现的概率很高

nicole · 2021 年3 月 30 日 06:01

你的自动compact配置有没有关，关掉试下呢
在console配置：
nebula> UPDATE CONFIGS storage:rocksdb_column_family_options = {disable_auto_compactions = true};

critical27 · 2021 年3 月 30 日 06:03

麻烦提供几个东西:

show hosts截图
挑一个有问题的part，比如上面图里的“Space: 15, Part: 3” 对于每个storage机器都只grep下INFO日志，grep “Space: 15, Part: 3” log, 然后上传下

xrfinbj · 2021 年3 月 30 日 06:19

我用exchange写入之前把disable_auto_compactions=true，写入后disable_auto_compactions=false并SUBMIT JOB COMPACT

xrfinbj · 2021 年3 月 30 日 06:29

我上午执行过 balance leader

机器1 ip:172.31.1.93
grep "Space: 15, Part: 3" nebula-storaged.INFO
E0330 07:34:26.572551 14620 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.8.17":9780, error code is -5

机器2 ip:172.31.4.183
grep "Space: 15, Part: 3" /usr/local/nebula/logs/nebula-storaged.INFO
空

机器3 ip:172.31.8.17
grep "Space: 15, Part: 3" /usr/local/nebula/logs/nebula-storaged.INFO
E0330 07:34:29.145110 14456 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.1.93":9780, error code is -6
E0330 07:34:29.145243 14456 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.4.183":9780, error code is -6
E0330 07:34:29.937732 14457 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.1.93":9780, error code is -6
E0330 07:34:29.937772 14457 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.4.183":9780, error code is -6
E0330 07:34:30.465073 14458 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.1.93":9780, error code is -6
E0330 07:34:30.465112 14458 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.4.183":9780, error code is -6
E0330 07:34:31.858142 14457 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.4.183":9780, error code is -6
E0330 07:34:31.858225 14457 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.1.93":9780, error code is -6
E0330 07:34:33.324142 14457 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.1.93":9780, error code is -6
E0330 07:34:33.324193 14457 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.4.183":9780, error code is -6
E0330 07:34:35.132741 14457 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.1.93":9780, error code is -6
E0330 07:34:35.132827 14457 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.4.183":9780, error code is -6
E0330 07:34:35.646097 14457 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.4.183":9780, error code is -6
E0330 07:34:35.646155 14457 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.1.93":9780, error code is -6
E0330 07:34:37.434142 14455 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.1.93":9780, error code is -6
E0330 07:34:37.434173 14455 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.4.183":9780, error code is -6
E0330 07:34:38.785360 14456 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.1.93":9780, error code is -6
E0330 07:34:38.785396 14456 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.4.183":9780, error code is -6
E0330 07:34:40.221325 14455 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.4.183":9780, error code is -6
E0330 07:34:40.221385 14455 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.1.93":9780, error code is -6
E0330 07:34:41.676453 14456 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.4.183":9780, error code is -6
E0330 07:34:41.676517 14456 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.1.93":9780, error code is -6
E0330 07:34:42.226251 14455 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.4.183":9780, error code is -6
E0330 07:34:42.226311 14455 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.1.93":9780, error code is -6
E0330 07:34:43.988873 14455 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.1.93":9780, error code is -6
E0330 07:34:43.988934 14455 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.4.183":9780, error code is -6
E0330 07:34:44.969836 14456 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.1.93":9780, error code is -6
E0330 07:34:44.969894 14456 RaftPart.cpp:1143] [Port: 9780, Space: 15, Part: 3] Receive response about askForVote from "172.31.4.183":9780, error code is -6

critical27 · 2021 年3 月 30 日 06:43

你可能得把日志等级放开点现在只打印了Error级别的 minloglevel改成0

xrfinbj · 2021 年3 月 30 日 08:32

好的晚点试试

试了一下没有复现问题

yangmeng · 2021 年4 月 30 日 02:36

你好, 我这也遇到这样的导入问题, 麻烦指导一下
1, 集群

2. 导入你们Nebula官网的篮球员脚本实例

3 .导入命令

4. 导入打印的日志

5 SUBMIT JOB STATS;后查询结果

补充: 服务肯定是正常的, 一直都没有重启

yangmeng · 2021 年4 月 30 日 02:41

你好, DEV, 我
我也有个导入的问题,麻烦你能指导一下: exchange 2 导入数据提示The leader has changed. Try again later - #18 由 yangmeng
谢谢!

wey · 2021 年4 月 30 日 09:12

@yangmeng
从您console链接的 -port 3699 我理解您的nebula graph 服务端是 1.x 的版本的，但是您的console 却是 2.0 的版本，请注意稳定大版本之间因为 thrift 的变动，两边的通信是对不上的，所以会有奇奇怪怪的结果显示。请用 1.x 的console连您的 nebula graph cluster 哈

yangmeng · 2021 年4 月 30 日 09:31

你好, wey DEV
1, 我使用的nebula 不是1.X的, 至于你说的3699端口是我自己吧yaml文件的端口改了而已

2, 我是按照nebula 2.0的的官网上的这一节来操作的Nebula Graph Database 内核手册
start/3.connect-to-nebula-graph/
谢谢!

wey · 2021 年5 月 3 日 06:22

哈，修改端口是为了绕过端口防火墙的操作么，sorry我当时没有仔细看你 image的名字是v2.0.0的已经。

console对于失败的行不会停下来或者retry，中途的失败只能依赖用户自己看日志再手动retry，抱歉带来麻烦了，如您截图的从ngql文件执行的 console 的log显示，那一时刻 leader has changed
那一个请求被拒绝了。这个情况（空闲的cluster，几十行的ngql执行，leader change，必现）可以被重现么？

zhengshuai1030 · 2021 年7 月 5 日 02:08

问下这个问题怎么解决的啊。我这边导入数据后查询也出现这个错误Storage Error: The leader has changed. Try again later

steam · 2021 年7 月 5 日 02:41

请勿一个问题多处分发浪费他人的回复精力，问题以用spark 导入数据后，查询数据出现 Storage Error: The leader has changed. Try again later 中的回复为主