调用ShortestPath算法接口异常

问答 问题
1

  • nebula 版本：2.5.1

  • 部署方式：单机

  • 安装方式：源码编译

  • 是否为线上版本：N

  • 问题的具体描述
    ShortestPath算法接口，报NotSerializableException异常。

Dataset<Row> df = package$.MODULE$.GraphoneDataFrameReader(sparkSession.read())
                .graphone(connectionConfig, readGraphoneConfig).loadEdgesToDF();
List<Object> vertexList=new ArrayList<>();
vertexList.add(1000L);
ShortestPathConfig shortestPathConfig=new ShortestPathConfig(JavaConverters.collectionAsScalaIterableConverter(vertexList).asScala().toSeq());
Dataset<Row> result = ShortestPathAlgo.apply(sparkSession, df,shortestPathConfig,false);
  • 日志信息
org.apache.spark.SparkException: Task not serializable
  at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:416) ~[spark-core_2.11-2.4.8.jar:2.4.8]
  at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:406) ~[spark-core_2.11-2.4.8.jar:2.4.8]
  at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:163) ~[spark-core_2.11-2.4.8.jar:2.4.8]
  at org.apache.spark.SparkContext.clean(SparkContext.scala:2332) ~[spark-core_2.11-2.4.8.jar:2.4.8]

......

Caused by: java.io.NotSerializableException: java.util.ArrayList$Itr
Serialization stack:

  at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40) ~[spark-core_2.11-2.4.8.jar:2.4.8]
  at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:46) ~[spark-core_2.11-2.4.8.jar:2.4.8]
  at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:100) ~[spark-core_2.11-2.4.8.jar:2.4.8]
  at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:413) ~[spark-core_2.11-2.4.8.jar:2.4.8]
  ... 68 common frames omitted
2

你是改过代码么， graphone接口是你们自己定义的？

还有看报错信息是java的list在spark中报了序列化问题，你可以试下采用这种方式把java list 转换成scala list：

import scala.collection.JavaConverters._
vertexList.asScala

[quote="floyd, post:1, topic:6393"]
`Caused by: java.io.NotSerializableException: java.util.ArrayList$Itr`
[/quote]
1 个赞
3

ShortestPathConfig需要一个Seq<Object>
，我构造了List<Object>

List<Object> vertexList=new ArrayList<>();
vertexList.add(1000L);

并通过

JavaConverters.collectionAsScalaIterableConverter(vertexList).asScala().toSeq()

转换成Seq<Object>

按理，是Long类型的list，不应该有未序列化的异常。但似乎，
Caused by: java.io.NotSerializableException: java.util.ArrayList$Itr
异常就是出现在这里。

4

正确的list转Seq的方法应该用 ·JavaConverters.asScalaBufferConverter

完整代码如下：

List<Object> vertexList=new ArrayList<>();
vertexList.add(1000L);
ShortestPathConfig shortestPathConfig=new ShortestPathConfig(JavaConverters.asScalaBufferConverter(vertexList).asScala().toSeq());
2 个赞
5

此话题已在最后回复的 7 天后被自动关闭。不再允许新回复。