nebula-algorithm-3.0 算法包运行错误

提问参考模版:

  • nebula 版本:v3.5.0, 默认 docker compose 文件方式安装
  • spark版本: spark-2.4.8
  • 部署方式:单机
  • 安装方式:Docker
  • 是否上生产环境:N
  • 硬件信息
    满足要求
  • 问题的具体描述:

application.conf 配置文件摘录如下

  data: {
    # data source. optional of nebula,csv,json
    source: nebula
    # data sink, means the algorithm result will be write into this sink. optional of nebula,csv,text
    sink: csv
    # if your algorithm needs weight
    hasWeight: false
  }
  # Nebula Graph relation config
  nebula: {
    # algo's data source from Nebula. If data.source is nebula, then this nebula.read config can be valid.
    read: {
        # Nebula metad server address, multiple addresses are split by English comma
        metaAddress: "127.0.0.1:49161"
        # Nebula space
        space: test
        # Nebula edge types, multiple labels means that data from multiple edges will union together
        labels: ["send"]
        # Nebula edge property name for each edge type, this property will be as weight col for algorithm.
        # Make sure the weightCols are corresponding to labels.
        weightCols: ["value"]

使用
bin/spark-submit --master “local” --class com.vesoft.nebula.algorithm.Main ./nebula-algorithm-3.0-SNAPSHOT.jar -p ./application.conf 运行后日志如下,中途报错

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
23/09/11 17:03:08 INFO SparkContext: Running Spark version 2.4.8
23/09/11 17:03:09 INFO SparkContext: Submitted application: LPA
23/09/11 17:03:09 INFO SecurityManager: Changing view acls to: node
23/09/11 17:03:09 INFO SecurityManager: Changing modify acls to: node
23/09/11 17:03:09 INFO SecurityManager: Changing view acls groups to: 
23/09/11 17:03:09 INFO SecurityManager: Changing modify acls groups to: 
23/09/11 17:03:09 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(node); groups with view permissions: Set(); users  with modify permissions: Set(node); groups with modify permissions: Set()
23/09/11 17:03:09 INFO Utils: Successfully started service 'sparkDriver' on port 44319.
23/09/11 17:03:09 INFO SparkEnv: Registering MapOutputTracker
23/09/11 17:03:09 INFO SparkEnv: Registering BlockManagerMaster
23/09/11 17:03:09 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
23/09/11 17:03:09 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
23/09/11 17:03:09 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-ec5f2928-f9a4-4689-aa0a-6d56e3fdcb62
23/09/11 17:03:09 INFO MemoryStore: MemoryStore started with capacity 2.5 GB
23/09/11 17:03:09 INFO SparkEnv: Registering OutputCommitCoordinator
23/09/11 17:03:10 INFO Utils: Successfully started service 'SparkUI' on port 4040.
23/09/11 17:03:10 INFO SparkUI: Bound SparkUI to 10.10.10.3, and started at http://master.cluster.local:4040
23/09/11 17:03:10 INFO SparkContext: Added JAR file:/home/node/tanlx/spark/nebula-algorithm-3.0-SNAPSHOT.jar at spark://master.cluster.local:44319/jars/nebula-algorithm-3.0-SNAPSHOT.jar with timestamp 1694422990131
23/09/11 17:03:10 INFO Executor: Starting executor ID driver on host localhost
23/09/11 17:03:10 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 37533.
23/09/11 17:03:10 INFO NettyBlockTransferService: Server created on master.cluster.local:37533
23/09/11 17:03:10 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
23/09/11 17:03:10 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, master.cluster.local, 37533, None)
23/09/11 17:03:10 INFO BlockManagerMasterEndpoint: Registering block manager master.cluster.local:37533 with 2.5 GB RAM, BlockManagerId(driver, master.cluster.local, 37533, None)
23/09/11 17:03:10 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, master.cluster.local, 37533, None)
23/09/11 17:03:10 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, master.cluster.local, 37533, None)
23/09/11 17:03:10 WARN ReadNebulaConfig$: returnCols is empty and your result will contain all properties for send
23/09/11 17:03:10 INFO ReadNebulaConfig$: NebulaReadConfig={space=eth,label=send,returnCols=List(),noColumn=false,partitionNum=100}
23/09/11 17:03:10 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/home/node/tanlx/spark/spark-warehouse').
23/09/11 17:03:10 INFO SharedState: Warehouse path is 'file:/home/node/tanlx/spark/spark-warehouse'.
23/09/11 17:03:11 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
23/09/11 17:03:11 INFO NebulaDataSource: create reader
23/09/11 17:03:11 INFO NebulaDataSource: options {spacename=eth, nocolumn=false, enablestoragessl=false, metaaddress=127.0.0.1:49161, label=send, type=edge, connectionretry=2, timeout=6000, executionretry=1, enablemetassl=false, paths=[], limit=1000, returncols=, operatetype=read, partitionnumber=100}
23/09/11 17:03:11 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:11 INFO NebulaDataSourceEdgeReader: dataset's schema: StructType(StructField(_srcId,StringType,false), StructField(_dstId,StringType,false), StructField(_rank,LongType,false), StructField(value,DoubleType,true))
23/09/11 17:03:13 INFO NebulaDataSource: create reader
23/09/11 17:03:13 INFO NebulaDataSource: options {spacename=eth, nocolumn=false, enablestoragessl=false, metaaddress=127.0.0.1:49161, label=send, type=edge, connectionretry=2, timeout=6000, executionretry=1, enablemetassl=false, paths=[], limit=1000, returncols=, operatetype=read, partitionnumber=100}
23/09/11 17:03:13 INFO DataSourceV2Strategy: 
Pushing operators to class com.vesoft.nebula.connector.NebulaDataSource
Pushed Filters: 
Post-Scan Filters: 
Output: _srcId#0, _dstId#1, _rank#2L, value#3
         
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:13 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO NebulaUtils$: prop name value, type DOUBLE 
23/09/11 17:03:14 INFO CodeGenerator: Code generated in 227.497322 ms
23/09/11 17:03:15 INFO CodeGenerator: Code generated in 15.999932 ms
23/09/11 17:03:15 INFO SparkContext: Starting job: reduce at GraphTriangleCountAlgo.scala:24
23/09/11 17:03:15 INFO DAGScheduler: Registering RDD 9 (mapPartitions at VertexRDD.scala:356) as input to shuffle 5
23/09/11 17:03:15 INFO DAGScheduler: Registering RDD 19 (mapPartitions at VertexRDDImpl.scala:247) as input to shuffle 0
23/09/11 17:03:15 INFO DAGScheduler: Registering RDD 25 (map at GraphOps.scala:306) as input to shuffle 6
23/09/11 17:03:15 INFO DAGScheduler: Registering RDD 39 (mapPartitions at GraphImpl.scala:208) as input to shuffle 2
23/09/11 17:03:15 INFO DAGScheduler: Registering RDD 31 (mapPartitions at VertexRDD.scala:356) as input to shuffle 4
23/09/11 17:03:15 INFO DAGScheduler: Registering RDD 51 (mapPartitions at VertexRDDImpl.scala:247) as input to shuffle 1
23/09/11 17:03:15 INFO DAGScheduler: Registering RDD 55 (mapPartitions at GraphImpl.scala:208) as input to shuffle 3
23/09/11 17:03:15 INFO DAGScheduler: Got job 0 (reduce at GraphTriangleCountAlgo.scala:24) with 100 output partitions
23/09/11 17:03:15 INFO DAGScheduler: Final stage: ResultStage 7 (reduce at GraphTriangleCountAlgo.scala:24)
23/09/11 17:03:15 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0, ShuffleMapStage 6, ShuffleMapStage 3, ShuffleMapStage 4)
23/09/11 17:03:15 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0, ShuffleMapStage 6, ShuffleMapStage 3, ShuffleMapStage 4)
23/09/11 17:03:15 INFO DAGScheduler: Submitting ShuffleMapStage 0 (VertexRDD.createRoutingTables - vid2pid (aggregation) MapPartitionsRDD[9] at mapPartitions at VertexRDD.scala:356), which has no missing parents
23/09/11 17:03:15 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 34.0 KB, free 2.5 GB)
23/09/11 17:03:15 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 9.4 KB, free 2.5 GB)
23/09/11 17:03:15 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on master.cluster.local:37533 (size: 9.4 KB, free: 2.5 GB)
23/09/11 17:03:15 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1184
23/09/11 17:03:15 INFO DAGScheduler: Submitting 100 missing tasks from ShuffleMapStage 0 (VertexRDD.createRoutingTables - vid2pid (aggregation) MapPartitionsRDD[9] at mapPartitions at VertexRDD.scala:356) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
23/09/11 17:03:15 INFO TaskSchedulerImpl: Adding task set 0.0 with 100 tasks
23/09/11 17:03:15 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 11350 bytes)
23/09/11 17:03:15 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
23/09/11 17:03:15 INFO Executor: Fetching spark://master.cluster.local:44319/jars/nebula-algorithm-3.0-SNAPSHOT.jar with timestamp 1694422990131
23/09/11 17:03:15 INFO TransportClientFactory: Successfully created connection to master.cluster.local/10.10.10.3:44319 after 44 ms (0 ms spent in bootstraps)
23/09/11 17:03:15 INFO Utils: Fetching spark://master.cluster.local:44319/jars/nebula-algorithm-3.0-SNAPSHOT.jar to /tmp/spark-40c6ae11-dd8c-45c9-ae9c-54d34c022c02/userFiles-a2814686-9626-4442-958f-197628cb22b7/fetchFileTemp7929544148392749908.tmp
23/09/11 17:03:16 INFO Executor: Adding file:/tmp/spark-40c6ae11-dd8c-45c9-ae9c-54d34c022c02/userFiles-a2814686-9626-4442-958f-197628cb22b7/nebula-algorithm-3.0-SNAPSHOT.jar to class loader
23/09/11 17:03:16 INFO NebulaEdgePartitionReader: partition index: 1, scanParts: List(1)
23/09/11 17:03:16 INFO CodeGenerator: Code generated in 27.066875 ms
23/09/11 17:03:16 ERROR ScanEdgeResultIterator: get storage client error, 
java.util.NoSuchElementException: Unable to activate object
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:400)
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:277)
        at com.vesoft.nebula.client.storage.StorageConnPool.getStorageConnection(StorageConnPool.java:43)
        at com.vesoft.nebula.client.storage.scan.ScanEdgeResultIterator.lambda$next$0(ScanEdgeResultIterator.java:85)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: storaged2: Name or service not known
        at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
        at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
        at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
        at java.net.InetAddress.getAllByName0(InetAddress.java:1277)
        at java.net.InetAddress.getAllByName(InetAddress.java:1193)
        at java.net.InetAddress.getAllByName(InetAddress.java:1127)
        at java.net.InetAddress.getByName(InetAddress.java:1077)
        at com.vesoft.nebula.client.storage.GraphStorageConnection.open(GraphStorageConnection.java:68)
        at com.vesoft.nebula.client.storage.StorageConnPoolFactory.activateObject(StorageConnPoolFactory.java:59)
        at com.vesoft.nebula.client.storage.StorageConnPoolFactory.activateObject(StorageConnPoolFactory.java:16)
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:391)
        ... 8 more
23/09/11 17:03:16 ERROR ScanEdgeResultIterator: get storage client error, 
java.util.NoSuchElementException: Unable to activate object
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:400)
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:277)
        at com.vesoft.nebula.client.storage.StorageConnPool.getStorageConnection(StorageConnPool.java:43)
        at com.vesoft.nebula.client.storage.scan.ScanEdgeResultIterator.lambda$next$0(ScanEdgeResultIterator.java:85)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: storaged2
        at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
        at java.net.InetAddress.getAllByName(InetAddress.java:1193)
        at java.net.InetAddress.getAllByName(InetAddress.java:1127)
        at java.net.InetAddress.getByName(InetAddress.java:1077)
        at com.vesoft.nebula.client.storage.GraphStorageConnection.open(GraphStorageConnection.java:68)
        at com.vesoft.nebula.client.storage.StorageConnPoolFactory.activateObject(StorageConnPoolFactory.java:59)
        at com.vesoft.nebula.client.storage.StorageConnPoolFactory.activateObject(StorageConnPoolFactory.java:16)
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:391)
        ... 8 more
23/09/11 17:03:16 ERROR ScanEdgeResultIterator: get storage client error, 
java.util.NoSuchElementException: Unable to activate object
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:400)
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:277)
        at com.vesoft.nebula.client.storage.StorageConnPool.getStorageConnection(StorageConnPool.java:43)
        at com.vesoft.nebula.client.storage.scan.ScanEdgeResultIterator.lambda$next$0(ScanEdgeResultIterator.java:85)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: storaged2
        at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
        at java.net.InetAddress.getAllByName(InetAddress.java:1193)
        at java.net.InetAddress.getAllByName(InetAddress.java:1127)
        at java.net.InetAddress.getByName(InetAddress.java:1077)
        at com.vesoft.nebula.client.storage.GraphStorageConnection.open(GraphStorageConnection.java:68)
        at com.vesoft.nebula.client.storage.StorageConnPoolFactory.activateObject(StorageConnPoolFactory.java:59)
        at com.vesoft.nebula.client.storage.StorageConnPoolFactory.activateObject(StorageConnPoolFactory.java:16)
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:391)
        ... 8 more
23/09/11 17:03:16 ERROR ScanEdgeResultIterator: get storage client error, 
java.util.NoSuchElementException: Unable to activate object
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:400)
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:277)
        at com.vesoft.nebula.client.storage.StorageConnPool.getStorageConnection(StorageConnPool.java:43)
        at com.vesoft.nebula.client.storage.scan.ScanEdgeResultIterator.lambda$next$0(ScanEdgeResultIterator.java:85)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: storaged2
        at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
        at java.net.InetAddress.getAllByName(InetAddress.java:1193)
        at java.net.InetAddress.getAllByName(InetAddress.java:1127)
        at java.net.InetAddress.getByName(InetAddress.java:1077)
        at com.vesoft.nebula.client.storage.GraphStorageConnection.open(GraphStorageConnection.java:68)
        at com.vesoft.nebula.client.storage.StorageConnPoolFactory.activateObject(StorageConnPoolFactory.java:59)
        at com.vesoft.nebula.client.storage.StorageConnPoolFactory.activateObject(StorageConnPoolFactory.java:16)
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:391)
        ... 8 more

问题定位为无法获取storage

spark 这几个工具的读取数据走得是 storaged ,以来集群内部网络,不建议用 docker 部署,如果用 docker 部署 nebulagraph,请把 spark 也在同一个容器网络里部署,(这种可以直接用 GitHub - wey-gu/nebula-up: One-liner NebulaGraph playground with allllllllll-in-one toolchain integrated! 的全家桶模式,一键部署 spark + nebulagraph用来学习测试)。

否则,如果自己已经有 spark cluster,请用 .deb/.rpm 方式部署,并把 storaged/metad 配置在集群外部可以访问的地址上

参考 为什么我的 Nebula-Spark-Connector、Nebula-Algorithm 连不上 K8s 部署的 Nebula Graph 集群? - siwei.io

问题应该出在nebula-spark-connector库,应该可以修改代码来解决

这个问题有一些历史,问题有点复杂,杂糅了几件事儿。

  • 地址列表从服务发现获取
  • 服务实体和服务地址合并成了一件事儿
  • 默认容器化部署的服务地址是容器内部域名
  • spark storage 客户端只从 meta 获得地址

之前考虑过增加要给可配置的mapper,让 storage service instance list 和可访问地址解耦,并允许自定义对应关系: with storage address by wey-gu · Pull Request #47 · vesoft-inc/nebula-spark-connector · GitHub

有意思的是,其他项目也是做了类似的选择:man_facepalming:

我建议先用非容器化部署集群。

是的,可配置才是应该提倡的。docker部署现在是主流,很难避免这种情况

这还有一个方法,手动暴露:

此话题已在最后回复的 30 天后被自动关闭。不再允许新回复。