nebula 3.5 。 spark 用的是odps-spark 2.4.5 ,scala 版本是2.11.12 。nebula-algorithm 版本 3.1.0 。
2023-11-20 09:55:42,234 ERROR org.apache.spark.deploy.yarn.ApplicationMaster [] - Uncaught exception:
org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:226)) ~[_891a126560492a6f180deaad810cd6c9.jar:?]
at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:469)) [spark-yarn_2.11-2.4.5.jar:2.4.5]
at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:305)) [spark-yarn_2.11-2.4.5.jar:2.4.5]
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply$mcV$sp(ApplicationMaster.scala:245)) [spark-yarn_2.11-2.4.5.jar:2.4.5]
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply(ApplicationMaster.scala:245)) [spark-yarn_2.11-2.4.5.jar:2.4.5]
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply(ApplicationMaster.scala:245)) [spark-yarn_2.11-2.4.5.jar:2.4.5]
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:780)) [spark-yarn_2.11-2.4.5.jar:2.4.5]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_65-AliJVM]
at javax.security.auth.Subject.doAs(Subject.java:422)) [?:1.8.0_65-AliJVM]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)) [_891a126560492a6f180deaad810cd6c9.jar:?]
at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:779)) [spark-yarn_2.11-2.4.5.jar:2.4.5]
at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:244)) [spark-yarn_2.11-2.4.5.jar:2.4.5]
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:804)) [spark-yarn_2.11-2.4.5.jar:2.4.5]
at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala) [spark-yarn_2.11-2.4.5.jar:2.4.5]
Caused by: org.apache.spark.SparkException: Dynamic allocation of executors requires the external shuffle service. You may enable this through spark.shuffle.service.enabled.
at org.apache.spark.ExecutorAllocationManager.validateSettings(ExecutorAllocationManager.scala:216)) ~[_891a126560492a6f180deaad810cd6c9.jar:?]
at org.apache.spark.ExecutorAllocationManager.(ExecutorAllocationManager.scala:129)) ~[_891a126560492a6f180deaad810cd6c9.jar:?]
at org.apache.spark.SparkContext.(SparkContext.scala:536)) ~[_891a126560492a6f180deaad810cd6c9.jar:?]
at Wordcount$.main(Wordcount.scala:8)) ~[_891a126560492a6f180deaad810cd6c9.jar:?]
at Wordcount.main(Wordcount.scala) ~[_891a126560492a6f180deaad810cd6c9.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_65-AliJVM]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)) ~[?:1.8.0_65-AliJVM]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)) ~[?:1.8.0_65-AliJVM]
at java.lang.reflect.Method.invoke(Method.java:497)) ~[?:1.8.0_65-AliJVM]
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:685)) ~[spark-yarn_2.11-2.4.5.jar:2.4.5]
在尝试读取nebula时候，会报如上错误，但是在spark conf配置文件里加上spark.shuffle.service.enabled=true，就会一直运行点点点，最后报错。在idea 上本地启动可以直接连接nebula。 执行jar 包报错