本地运行没有问题,打包提交集群后报错:NoSuchMethodError

  • nebula 版本:3.4.1

  • 部署方式:单机

  • 安装方式:源码编译

  • 是否上生产环境:N

  • 问题的具体描述:在IDEA本地运行没有问题,打包提交到集群后出现下面的错误

  • 相关的 meta / storage / graph info 日志信息

23/08/02 16:00:23 ERROR executor.Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.NoSuchMethodError: com.google.common.net.HostAndPort.getHostText()Ljava/lang/String;
        at com.vesoft.nebula.connector.NebulaOptions$$anonfun$getMetaAddress$1.apply(NebulaOptions.scala:195)
        at com.vesoft.nebula.connector.NebulaOptions$$anonfun$getMetaAddress$1.apply(NebulaOptions.scala:192)
        at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
        at com.vesoft.nebula.connector.NebulaOptions.getMetaAddress(NebulaOptions.scala:192)
        at com.vesoft.nebula.connector.writer.NebulaWriter.<init>(NebulaWriter.scala:23)
        at com.vesoft.nebula.connector.writer.NebulaVertexWriter.<init>(NebulaVertexWriter.scala:23)
        at com.vesoft.nebula.connector.writer.NebulaVertexWriterFactory.createDataWriter(NebulaSourceWriter.scala:28)
        at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:113)
        at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec$$anonfun$doExecute$2.apply(WriteToDataSourceV2Exec.scala:67)
        at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec$$anonfun$doExecute$2.apply(WriteToDataSourceV2Exec.scala:66)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
        at org.apache.spark.scheduler.Task.run(Task.scala:123)
        at org.apache.spark.executor.Executor$TaskRunner$$anonfun$11.apply(Executor.scala:413)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1334)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:419)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

翻了一下论坛,有说是jar包冲突的,但应该如何解决呢

根据@nicole大佬提到的方案实施后

又出现以下报错

Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
        at org.apache.hadoop.conf.Configuration.set(Configuration.java:1380)
        at org.apache.hadoop.conf.Configuration.set(Configuration.java:1361)
        at org.apache.spark.deploy.SparkHadoopUtil$$anonfun$org$apache$spark$deploy$SparkHadoopUtil$$appendSparkHadoopConfigs$3.apply(SparkHadoopUtil.scala:460)
        at org.apache.spark.deploy.SparkHadoopUtil$$anonfun$org$apache$spark$deploy$SparkHadoopUtil$$appendSparkHadoopConfigs$3.apply(SparkHadoopUtil.scala:459)
        at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
        at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
        at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
        at org.apache.spark.deploy.SparkHadoopUtil$.org$apache$spark$deploy$SparkHadoopUtil$$appendSparkHadoopConfigs(SparkHadoopUtil.scala:459)
        at org.apache.spark.deploy.SparkHadoopUtil$.org$apache$spark$deploy$SparkHadoopUtil$$appendS3AndSparkHadoopConfigurations(SparkHadoopUtil.scala:449)
        at org.apache.spark.deploy.SparkHadoopUtil$.newConfiguration(SparkHadoopUtil.scala:423)
        at org.apache.spark.deploy.SparkSubmit$$anonfun$2.apply(SparkSubmit.scala:323)
        at org.apache.spark.deploy.SparkSubmit$$anonfun$2.apply(SparkSubmit.scala:323)
        at scala.Option.getOrElse(Option.scala:121)
        at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:323)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:776)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:922)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:931)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

集群的Guava估计是不能改了,现在我想按这个帖子的方法修改后本地打包

但是又报错了,有没有大佬帮忙看一下啊

PS C:\Users\admin\Desktop\nebula-spark-connector-3.4.0> mvn clean package -Dmaven.test.skip=true -Dgpg.skip -Dmaven.javadoc.skip=true -pl nebula-spark-connector_3.0 -am -Psca
la-2.12 -Pspark-3.0
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for com.vesoft:nebula-spark-connector_2.2:jar:3.4.0
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing. @ line 83, column 21
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-surefire-plugin is missing. @ line 201, column 21
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
[ERROR] [ERROR] Could not find the selected project in the reactor: nebula-spark-connector_3.0 @
[ERROR] Could not find the selected project in the reactor: nebula-spark-connector_3.0 -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MavenExecutionException

把每个spark-home/jars下的guava版本换成14.0的jar包。
或者:用最新的spark-connector SNAPSHOT版本,去掉了代码中对guava包的依赖。 不会有guava包冲突了

spark-connector SNAPSHOT版本打包报错了 :sob:

Scala版本:2.11
Spark版本:2.4.7

执行了下面的打包命令:

$ mvn clean package -Dmaven.test.skip=true -Dgpg.skip -Dmaven.javadoc.skip=true -pl nebula-spark-connector -am -Pscala-2.11 -Pspark-2.4

报错信息:

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for nebula-spark 3.0-SNAPSHOT:
[INFO]
[INFO] nebula-spark ....................................... SUCCESS [  0.051 s]
[INFO] nebula-spark-common ................................ FAILURE [  5.013 s]
[INFO] nebula-spark-connector ............................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  5.626 s
[INFO] Finished at: 2023-08-04T11:27:06+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project nebula-spark-common: Could not resolve dependencies for project com.vesoft:nebula-spark-common:jar:3.0-SNAPSHOT: Failed to collect dependencies at com.vesoft:client:jar:3.0-SNAPSHOT: Failed to read artifact descriptor for com.vesoft:client:jar:3.0-SNAPSHOT: Could not transfer artifact com.vesoft:client:pom:3.0-SNAPSHOT from/to snapshots (https://oss.sonatype.org/content/repositories/snapshots/): transfer failed for https://oss.sonatype.org/content/repositories/snapshots/com/vesoft/client/3.0-SNAPSHOT/client-3.0-SNAPSHOT.pom: Connect to oss.sonatype.org:443 [oss.sonatype.org/18.213.201.83, oss.sonatype.org/34.204.83.22] failed: Connection refused: connect -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :nebula-spark-common

你编译的时候下载nebula client SNAPSHOT版本的jar 下载失败了

有打好的包吗,在哪里可以下载呢?

使用spark-connector 不需要自己打包,可以直接在你工程代码中引入nebula-spark-connector的依赖来使用
https://github.com/vesoft-inc/nebula-spark-connector#how-to-use

换成SNAPSHOT版本后问题解决了,感谢

1 个赞

此话题已在最后回复的 7 天后被自动关闭。不再允许新回复。