Nebula Algorithm通过提交算法包运行报错

  • nebula 版本:v2.6.0
    image

nebula-algorithm 2.5.1

spark-2.4.8-bin-hadoop2.7

  • 部署方式:单机
  • 安装方式:Docker
  • 是否为线上版本:Y
  • 问题的具体描述
    提交代码包后,spark任务报错
DAGScheduler: ShuffleMapStage 1 (mapPartitions at GraphImpl.scala:208) failed in 0.107 s due to Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost, executor driver): java.io.IOException: Failed to connect to /0.0.0.212:43671

你 nebula-algorithm 和 Spark 的版本号补充一下

1 个赞

nebula-algorithm 2.5.1

spark-2.4.8-bin-hadoop2.7

看报错信息是启动spark后建立连接失败

22/04/07 21:25:13 INFO Executor: Fetching spark://0.0.0.212:41535/jars/nebula-algorithm-2.5.1.jar with timestamp 1649337909569
22/04/07 21:25:13 ERROR Utils: Aborting task
java.io.IOException: Failed to connect to /0.0.0.212:41535
	at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:245)
	at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:187)
	at org.apache.spark.rpc.netty.NettyRpcEnv.org$apache$spark$rpc$netty$NettyRpcEnv$$downloadClient(NettyRpcEnv.scala:368)
	at org.apache.spark.rpc.netty.NettyRpcEnv$$anonfun$openChannel$1.apply$mcV$sp(NettyRpcEnv.scala:336)
	at org.apache.spark.rpc.netty.NettyRpcEnv$$anonfun$openChannel$1.apply(NettyRpcEnv.scala:335)
	at org.apache.spark.rpc.netty.NettyRpcEnv$$anonfun$openChannel$1.apply(NettyRpcEnv.scala:335)
	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394)
	at org.apache.spark.rpc.netty.NettyRpcEnv.openChannel(NettyRpcEnv.scala:339)
	at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:693)
	at org.apache.spark.util.Utils$.fetchFile(Utils.scala:509)
	at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:819)
	at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:811)
	at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
	at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130)
	at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130)
	at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236)
	at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
	at scala.collection.mutable.HashMap.foreach(HashMap.scala:130)
	at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
	at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$updateDependencies(Executor.scala:811)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:378)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.AbstractChannel$AnnotatedSocketException: 无效的参数: /0.0.0.212:41535
Caused by: java.net.SocketException: 无效的参数
	at sun.nio.ch.Net.connect0(Native Method)
	at sun.nio.ch.Net.connect(Net.java:459)
	at sun.nio.ch.Net.connect(Net.java:451)
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:646)
	at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:91)
	at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:88)
	at java.security.AccessController.doPrivileged(Native Method)
	at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:88)
	at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:315)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:248)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342)
	at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:548)
	at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:533)
	at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:517)
	at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:978)
	at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:253)
	at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:244)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)
22/04/07 21:25:13 WARN Utils: Suppressing exception in catch: null

辛苦大家帮忙看一下,万分感谢 :smiling_face_with_three_hearts:

配置可能有点问题,把你的配置信息贴一下

此话题已在最后回复的 30 天后被自动关闭。不再允许新回复。