2.5版本algorithm模块运行,spark报错问题

  • nebula 版本:(2.5)

  • 部署方式:单机

  • 安装方式:源码编译

  • 是否为线上版本: N
    -scala版本2.11.8,spark2.4.8

  • 问题的具体描述
    安装好spark环境后,执行algorithm模块,spark报错;通样的版本使用相同方法部署在其他机器上可以正常运行,但是换了台机器就报错了,不知是否机器本身问题,

[root@364587 spark-2.4.8-bin-hadoop2.7]# ./bin/spark-submit  --master "local" --class com.vesoft.nebula.algorithm.Main /home/zhanghao/nebula/nebula-spark-utils/nebula-algorithm/target/nebula-algorithm-2.5.0.jar -p /home/zhanghao/nebula/nebula-spark-utils/nebula-algorithm/src/main/resources/application.conf 
21/12/07 10:49:35 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
log4j:WARN No appenders could be found for logger (com.vesoft.nebula.algorithm.Main$).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/12/07 10:49:35 INFO SparkContext: Running Spark version 2.4.8
21/12/07 10:49:35 INFO SparkContext: Submitted application: LPA
21/12/07 10:49:35 INFO SecurityManager: Changing view acls to: root
21/12/07 10:49:35 INFO SecurityManager: Changing modify acls to: root
21/12/07 10:49:35 INFO SecurityManager: Changing view acls groups to: 
21/12/07 10:49:35 INFO SecurityManager: Changing modify acls groups to: 
21/12/07 10:49:35 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
21/12/07 10:49:35 INFO Utils: Successfully started service 'sparkDriver' on port 34825.
21/12/07 10:49:35 INFO SparkEnv: Registering MapOutputTracker
21/12/07 10:49:35 INFO SparkEnv: Registering BlockManagerMaster
21/12/07 10:49:35 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
21/12/07 10:49:35 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
21/12/07 10:49:35 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-016bff53-5c95-410f-87c8-f315e2cfab97
21/12/07 10:49:35 INFO MemoryStore: MemoryStore started with capacity 366.3 MB
21/12/07 10:49:35 INFO SparkEnv: Registering OutputCommitCoordinator
21/12/07 10:49:35 INFO Utils: Successfully started service 'SparkUI' on port 4040.
21/12/07 10:49:35 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://0.5.144.43:4040
21/12/07 10:49:35 INFO SparkContext: Added JAR file:/home/zhanghao/nebula/nebula-spark-utils/nebula-algorithm/target/nebula-algorithm-2.5.0.jar at spark://0.5.144.43:34825/jars/nebula-algorithm-2.5.0.jar with timestamp 1638845375786
21/12/07 10:49:35 INFO Executor: Starting executor ID driver on host localhost
21/12/07 10:49:35 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 38083.
21/12/07 10:49:35 INFO NettyBlockTransferService: Server created on 0.5.144.43:38083
21/12/07 10:49:35 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
21/12/07 10:49:35 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 0.5.144.43, 38083, None)
21/12/07 10:49:35 INFO BlockManagerMasterEndpoint: Registering block manager 0.5.144.43:38083 with 366.3 MB RAM, BlockManagerId(driver, 0.5.144.43, 38083, None)
21/12/07 10:49:35 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 0.5.144.43, 38083, None)
21/12/07 10:49:35 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 0.5.144.43, 38083, None)
21/12/07 10:49:36 WARN ReadNebulaConfig$: returnCols is empty and your result will contain all properties for serve
21/12/07 10:49:36 INFO ReadNebulaConfig$: NebulaReadConfig={space=basketballplayer,label=serve,returnCols=List(),noColumn=false,partitionNum=100}
21/12/07 10:49:36 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/home/zhanghao/spark-2.4.8-bin-hadoop2.7/spark-warehouse').
21/12/07 10:49:36 INFO SharedState: Warehouse path is 'file:/home/zhanghao/spark-2.4.8-bin-hadoop2.7/spark-warehouse'.
21/12/07 10:49:36 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
21/12/07 10:49:36 INFO NebulaDataSource: create reader
21/12/07 10:49:36 INFO NebulaDataSource: options {spacename=basketballplayer, nocolumn=false, metaaddress=127.0.0.1:9559, label=serve, type=edge, connectionretry=2, timeout=6000, executionretry=1, paths=[], limit=1000, returncols=, partitionnumber=100}
Exception in thread "main" com.facebook.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
	at com.facebook.thrift.transport.TSocket.open(TSocket.java:175)
	at com.vesoft.nebula.client.meta.MetaClient.getClient(MetaClient.java:104)
	at com.vesoft.nebula.client.meta.MetaClient.doConnect(MetaClient.java:99)
	at com.vesoft.nebula.client.meta.MetaClient.connect(MetaClient.java:89)
	at com.vesoft.nebula.connector.nebula.MetaProvider.<init>(MetaProvider.scala:22)
	at com.vesoft.nebula.connector.reader.NebulaSourceReader.getSchema(NebulaSourceReader.scala:45)
	at com.vesoft.nebula.connector.reader.NebulaSourceReader.readSchema(NebulaSourceReader.scala:31)
	at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation$.create(DataSourceV2Relation.scala:175)
	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:223)
	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:186)
	at com.vesoft.nebula.connector.connector.package$NebulaDataFrameReader.loadEdgesToDF(package.scala:146)
	at com.vesoft.nebula.algorithm.reader.NebulaReader$$anonfun$read$1.apply$mcVI$sp(DataReader.scala:53)
	at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
	at com.vesoft.nebula.algorithm.reader.NebulaReader.read(DataReader.scala:39)
	at com.vesoft.nebula.algorithm.Main$.createDataSource(Main.scala:96)
	at com.vesoft.nebula.algorithm.Main$.main(Main.scala:72)
	at com.vesoft.nebula.algorithm.Main.main(Main.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:855)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:930)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:939)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at com.facebook.thrift.transport.TSocket.open(TSocket.java:170)
	... 28 more
21/12/07 10:49:36 INFO SparkContext: Invoking stop() from shutdown hook
21/12/07 10:49:36 INFO SparkUI: Stopped Spark web UI at http://0.5.144.43:4040
21/12/07 10:49:36 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
21/12/07 10:49:36 INFO MemoryStore: MemoryStore cleared
21/12/07 10:49:36 INFO BlockManager: BlockManager stopped
21/12/07 10:49:36 INFO BlockManagerMaster: BlockManagerMaster stopped
21/12/07 10:49:36 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
21/12/07 10:49:36 INFO SparkContext: Successfully stopped SparkContext
21/12/07 10:49:36 INFO ShutdownHookManager: Shutdown hook called
21/12/07 10:49:36 INFO ShutdownHookManager: Deleting directory /tmp/spark-8ec08942-94aa-4456-8fc6-c51221be124d
21/12/07 10:49:36 INFO ShutdownHookManager: Deleting directory /tmp/spark-da7213db-19c8-48d7-911c-3953f222d87e

Hadoop - 彻底解决 WARN util.NativeCodeLoader: Unable to load native-hadoop library... - 瘦风 - 博客园

会不会是这个没有本地加载 Hadoop 组件成功的原因呢

新换的机器无法连接nebula的meta服务。

但是nebula的服务我是开启的,怀疑是spark环境问题,现在我单独运行了下spark的示例./bin/run-example SparkPi ,报下面的错误,不知道啥原因,网上也找不到解决方法,求助:

[root@364587 spark-2.4.8-bin-hadoop2.7]# ./bin/run-example SparkPi 
21/12/07 16:21:02 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/12/07 16:21:02 INFO SparkContext: Running Spark version 2.4.8
21/12/07 16:21:02 INFO SparkContext: Submitted application: Spark Pi
21/12/07 16:21:02 INFO SecurityManager: Changing view acls to: root
21/12/07 16:21:02 INFO SecurityManager: Changing modify acls to: root
21/12/07 16:21:02 INFO SecurityManager: Changing view acls groups to: 
21/12/07 16:21:02 INFO SecurityManager: Changing modify acls groups to: 
21/12/07 16:21:02 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
21/12/07 16:21:02 INFO Utils: Successfully started service 'sparkDriver' on port 36362.
21/12/07 16:21:02 INFO SparkEnv: Registering MapOutputTracker
21/12/07 16:21:02 INFO SparkEnv: Registering BlockManagerMaster
21/12/07 16:21:02 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
21/12/07 16:21:02 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
21/12/07 16:21:02 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-47cf4291-4aad-49ba-8205-9530f45eb34e
21/12/07 16:21:02 INFO MemoryStore: MemoryStore started with capacity 366.3 MB
21/12/07 16:21:02 INFO SparkEnv: Registering OutputCommitCoordinator
21/12/07 16:21:03 INFO Utils: Successfully started service 'SparkUI' on port 4040.
21/12/07 16:21:03 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://0.5.144.43:4040
21/12/07 16:21:03 INFO SparkContext: Added JAR file:///home/zhanghao/spark-2.4.8-bin-hadoop2.7/examples/jars/scopt_2.11-3.7.0.jar at spark://0.5.144.43:36362/jars/scopt_2.11-3.7.0.jar with timestamp 1638865263114
21/12/07 16:21:03 INFO SparkContext: Added JAR file:///home/zhanghao/spark-2.4.8-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.4.8.jar at spark://0.5.144.43:36362/jars/spark-examples_2.11-2.4.8.jar with timestamp 1638865263115
21/12/07 16:21:03 INFO Executor: Starting executor ID driver on host localhost
21/12/07 16:21:03 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 41880.
21/12/07 16:21:03 INFO NettyBlockTransferService: Server created on 0.5.144.43:41880
21/12/07 16:21:03 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
21/12/07 16:21:03 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 0.5.144.43, 41880, None)
21/12/07 16:21:03 INFO BlockManagerMasterEndpoint: Registering block manager 0.5.144.43:41880 with 366.3 MB RAM, BlockManagerId(driver, 0.5.144.43, 41880, None)
21/12/07 16:21:03 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 0.5.144.43, 41880, None)
21/12/07 16:21:03 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 0.5.144.43, 41880, None)
21/12/07 16:21:03 INFO SparkContext: Starting job: reduce at SparkPi.scala:38
21/12/07 16:21:03 INFO DAGScheduler: Got job 0 (reduce at SparkPi.scala:38) with 2 output partitions
21/12/07 16:21:03 INFO DAGScheduler: Final stage: ResultStage 0 (reduce at SparkPi.scala:38)
21/12/07 16:21:03 INFO DAGScheduler: Parents of final stage: List()
21/12/07 16:21:03 INFO DAGScheduler: Missing parents: List()
21/12/07 16:21:03 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has no missing parents
21/12/07 16:21:03 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 2.0 KB, free 366.3 MB)
21/12/07 16:21:03 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1381.0 B, free 366.3 MB)
21/12/07 16:21:03 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 0.5.144.43:41880 (size: 1381.0 B, free: 366.3 MB)
21/12/07 16:21:03 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1184
21/12/07 16:21:03 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34) (first 15 tasks are for partitions Vector(0, 1))
21/12/07 16:21:03 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
21/12/07 16:21:03 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 7866 bytes)
21/12/07 16:21:03 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, executor driver, partition 1, PROCESS_LOCAL, 7866 bytes)
21/12/07 16:21:03 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
21/12/07 16:21:03 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
21/12/07 16:21:03 INFO Executor: Fetching spark://0.5.144.43:36362/jars/spark-examples_2.11-2.4.8.jar with timestamp 1638865263115
21/12/07 16:21:03 ERROR Utils: Aborting task
java.io.IOException: Failed to connect to /0.5.144.43:36362
	at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:245)
	at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:187)
	at org.apache.spark.rpc.netty.NettyRpcEnv.org$apache$spark$rpc$netty$NettyRpcEnv$$downloadClient(NettyRpcEnv.scala:368)
	at org.apache.spark.rpc.netty.NettyRpcEnv$$anonfun$openChannel$1.apply$mcV$sp(NettyRpcEnv.scala:336)
	at org.apache.spark.rpc.netty.NettyRpcEnv$$anonfun$openChannel$1.apply(NettyRpcEnv.scala:335)
	at org.apache.spark.rpc.netty.NettyRpcEnv$$anonfun$openChannel$1.apply(NettyRpcEnv.scala:335)
	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394)
	at org.apache.spark.rpc.netty.NettyRpcEnv.openChannel(NettyRpcEnv.scala:339)
	at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:693)
	at org.apache.spark.util.Utils$.fetchFile(Utils.scala:509)
	at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:819)
	at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:811)
	at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
	at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130)
	at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130)
	at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236)
	at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
	at scala.collection.mutable.HashMap.foreach(HashMap.scala:130)
	at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
	at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$updateDependencies(Executor.scala:811)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:378)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.AbstractChannel$AnnotatedSocketException: Invalid argument: /0.5.144.43:36362
Caused by: java.net.SocketException: Invalid argument
	at sun.nio.ch.Net.connect0(Native Method)
	at sun.nio.ch.Net.connect(Net.java:454)
	at sun.nio.ch.Net.connect(Net.java:446)
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
	at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:91)
	at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:88)
	at java.security.AccessController.doPrivileged(Native Method)
	at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:88)
	at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:315)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:248)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342)
	at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:548)
	at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:533)
	at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:517)
	at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:978)
	at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:253)
	at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:244)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)
21/12/07 16:21:03 WARN Utils: Suppressing exception in catch: null
java.lang.NullPointerException
	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1402)
	at org.apache.spark.rpc.netty.NettyRpcEnv.openChannel(NettyRpcEnv.scala:339)

那你就看看新机器是不是开启防火墙了, telnet 连一下nebula的meta服务: telnet ip meta_port

此话题已在最后回复的 30 天后被自动关闭。不再允许新回复。