一个关于nebula exchange 的使用问题

nicole · 2022 年2 月 15 日 07:58

只要第二个版本号是一样的就可以使用的， exchange2.6.x可以应用在nebula 2.6.x上。 readme 中版本对应关系和文档中版本对应关系还没有把 nebula 2.6.2 更新上。
这只是部分Exchange的最开始的运行日志，我没看到有错误信息，不知道是否导入失败，你要看最后的日志。

shanliqiu · 2022 年2 月 15 日 08:08

我们把spark、scala、exchange放在了192.168.0.162地址上，而nebula放在了192.168.0.145上。
以下是最后的日志：

Exception in thread “main” com.facebook.thrift.transport.TTransportException: java.net.NoRouteToHostException: No route to host (Host unreachable)
at com.facebook.thrift.transport.TSocket.open(TSocket.java:206)
at com.vesoft.nebula.client.meta.MetaClient.getClient(MetaClient.java:145)
at com.vesoft.nebula.client.meta.MetaClient.doConnect(MetaClient.java:124)
at com.vesoft.nebula.client.meta.MetaClient.connect(MetaClient.java:113)
at com.vesoft.nebula.exchange.MetaProvider.(MetaProvider.scala:56)
at com.vesoft.nebula.exchange.processor.VerticesProcessor.process(VerticesProcessor.scala:110)
at com.vesoft.nebula.exchange.Exchange$$anonfun$main$2.apply(Exchange.scala:150)
at com.vesoft.nebula.exchange.Exchange$$anonfun$main$2.apply(Exchange.scala:126)
at scala.collection.immutable.List.foreach(List.scala:392)
at com.vesoft.nebula.exchange.Exchange$.main(Exchange.scala:126)
at com.vesoft.nebula.exchange.Exchange.main(Exchange.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.net.NoRouteToHostException: No route to host (Host unreachable)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:476)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:218)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:394)
at java.net.Socket.connect(Socket.java:606)
at com.facebook.thrift.transport.TSocket.open(TSocket.java:201)
… 22 more
22/02/15 15:39:07 INFO SparkContext: Invoking stop() from shutdown hook
22/02/15 15:39:07 INFO SparkUI: Stopped Spark web UI at http://kf2-data-162:4040
22/02/15 15:39:07 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
22/02/15 15:39:07 INFO MemoryStore: MemoryStore cleared
22/02/15 15:39:07 INFO BlockManager: BlockManager stopped
22/02/15 15:39:07 INFO BlockManagerMaster: BlockManagerMaster stopped
22/02/15 15:39:07 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
22/02/15 15:39:07 INFO SparkContext: Successfully stopped SparkContext
22/02/15 15:39:07 INFO ShutdownHookManager: Shutdown hook called
22/02/15 15:39:07 INFO ShutdownHookManager: Deleting directory /tmp/spark-c4566332-a1e5-4867-93e1-40a863b903fa
22/02/15 15:39:07 INFO ShutdownHookManager: Deleting directory /tmp/spark-fd069f69-e49e-4def-9f7c-dd4738f4b73a

您看是不是因为这个ip地址不同，倒是host的问题

nicole · 2022 年2 月 15 日 08:11

你这个问题挺明显的，就是在162这个机器上访问不到145机器，你在162上执行下 telnet 看下能不能连通 145机器，检查下nebula服务状态、防火墙

shanliqiu · 2022 年2 月 15 日 08:31

为了防止不同机器的问题，我把包都放到了145机器上，现在错误换成了这个

Exception in thread “main” com.facebook.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at com.facebook.thrift.transport.TSocket.open(TSocket.java:206)
at com.vesoft.nebula.client.meta.MetaClient.getClient(MetaClient.java:145)
at com.vesoft.nebula.client.meta.MetaClient.doConnect(MetaClient.java:124)
at com.vesoft.nebula.client.meta.MetaClient.connect(MetaClient.java:113)
at com.vesoft.nebula.exchange.MetaProvider.(MetaProvider.scala:56)
at com.vesoft.nebula.exchange.processor.VerticesProcessor.process(VerticesProcessor.scala:110)
at com.vesoft.nebula.exchange.Exchange$$anonfun$main$2.apply(Exchange.scala:150)
at com.vesoft.nebula.exchange.Exchange$$anonfun$main$2.apply(Exchange.scala:126)
at scala.collection.immutable.List.foreach(List.scala:392)
at com.vesoft.nebula.exchange.Exchange$.main(Exchange.scala:126)
at com.vesoft.nebula.exchange.Exchange.main(Exchange.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:476)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:218)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:394)
at java.net.Socket.connect(Socket.java:606)
at com.facebook.thrift.transport.TSocket.open(TSocket.java:201)
… 22 more
22/02/15 03:29:51 INFO SparkContext: Invoking stop() from shutdown hook
22/02/15 03:29:51 INFO ContextCleaner: Cleaned accumulator 21
22/02/15 03:29:51 INFO ContextCleaner: Cleaned accumulator 9
22/02/15 03:29:51 INFO ContextCleaner: Cleaned accumulator 12
22/02/15 03:29:51 INFO SparkUI: Stopped Spark web UI at http://mysql:4040
22/02/15 03:29:51 INFO BlockManagerInfo: Removed broadcast_1_piece0 on mysql:35415 in memory (size: 4.5 KB, free: 366.3 MB)
22/02/15 03:29:51 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
22/02/15 03:29:51 INFO MemoryStore: MemoryStore cleared
22/02/15 03:29:51 INFO BlockManager: BlockManager stopped
22/02/15 03:29:51 INFO BlockManagerMaster: BlockManagerMaster stopped
22/02/15 03:29:51 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
22/02/15 03:29:51 INFO SparkContext: Successfully stopped SparkContext
22/02/15 03:29:51 INFO ShutdownHookManager: Shutdown hook called
22/02/15 03:29:51 INFO ShutdownHookManager: Deleting directory /tmp/spark-c415532d-991f-41c4-9c06-e8397133854f
22/02/15 03:29:51 INFO ShutdownHookManager: Deleting directory /tmp/spark-31de8e67-3f2d-4664-9b88-61035407ae53

nicole · 2022 年2 月 15 日 08:42

网络不通

shanliqiu · 2022 年2 月 15 日 08:53

我把包都放到了145机器上，为什么会出现网络不通的情况呢，大佬有什么猜想吗？

nicole · 2022 年2 月 15 日 09:26

你的nebula 是几个节点？你的spark集群是几个节点，你在每个spark节点上去执行下上面说到的telnet 命令

shanliqiu · 2022 年2 月 18 日 07:58

ok，已经解决了

system · 2022 年2 月 25 日 07:59

此话题已在最后回复的 7 天后被自动关闭。不再允许新回复。