nebula-spark-connection操作pyspark报错

ChesterXi · 2023 年9 月 8 日 09:53

nebula 版本：3.6
部署方式：云端
安装方式：RPM
是否上生产环境： N
问题的具体描述
使用spark-connector的pyspark测试时，github的测试代码报错

# read vertex
df = spark.read.format(
  "com.vesoft.nebula.connector.NebulaDataSource").option(
    "type", "vertex").option(
    "spaceName", "demo_basketballplayer").option(
    "label", "player").option(
    "returnCols", "name,age").option(
    "metaAddress", "meta0:9559").option(
    "partitionNumber", 1).load()

报错日志

org.apache.linkis.engineplugin.spark.exception.ExecuteError: errCode: 40003 ,desc: Py4JJavaError: An error occurred while calling o104.load.
: java.util.NoSuchElementException: key not found: operateType
	at scala.collection.MapLike$class.default(MapLike.scala:228)
	at org.apache.spark.sql.catalyst.util.CaseInsensitiveMap.default(CaseInsensitiveMap.scala:28)
	at scala.collection.MapLike$class.apply(MapLike.scala:141)
	at org.apache.spark.sql.catalyst.util.CaseInsensitiveMap.apply(CaseInsensitiveMap.scala:28)
	at com.vesoft.nebula.connector.NebulaOptions.<init>(NebulaOptions.scala:38)
	at com.vesoft.nebula.connector.NebulaDataSource.getNebulaOptions(NebulaDataSource.scala:148)
	at com.vesoft.nebula.connector.NebulaDataSource.createReader(NebulaDataSource.scala:45)
	at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation$SourceHelpers.createReader(DataSourceV2Relation.scala:155)
	at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation$.create(DataSourceV2Relation.scala:172)
	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:223)
	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:186)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.lang.Thread.run(Thread.java:748)