使用saprk往nebula中导数据报错

问答 问题
1

版本：3.5

24/02/21 10:25:19 WARN scheduler.TaskSetManager: Lost task 3.0 in stage 1.0 (TID 190, cdh046.dn.tcjf.com, executor 66): com.vesoft.nebula.client.graph.exception.IOErrorException: java.net.SocketTimeoutException: Read timed out
	at com.vesoft.nebula.client.graph.net.SyncConnection.executeWithParameter(SyncConnection.java:191)
	at com.vesoft.nebula.client.graph.net.Session.executeWithParameter(Session.java:117)
	at com.vesoft.nebula.client.graph.net.Session.execute(Session.java:82)
	at com.vesoft.exchange.common.GraphProvider.submit(GraphProvider.scala:78)
	at com.vesoft.exchange.common.writer.NebulaGraphClientWriter.writeEdges(ServerBaseWriter.scala:163)
	at com.vesoft.nebula.exchange.processor.EdgeProcessor$$anonfun$com$vesoft$nebula$exchange$processor$EdgeProcessor$$processEachPartition$1.apply(EdgeProcessor.scala:71)
	at com.vesoft.nebula.exchange.processor.EdgeProcessor$$anonfun$com$vesoft$nebula$exchange$processor$EdgeProcessor$$processEachPartition$1.apply(EdgeProcessor.scala:69)
	at scala.collection.Iterator$class.foreach(Iterator.scala:891)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
	at com.vesoft.nebula.exchange.processor.EdgeProcessor.com$vesoft$nebula$exchange$processor$EdgeProcessor$$processEachPartition(EdgeProcessor.scala:69)
	at com.vesoft.nebula.exchange.processor.EdgeProcessor$$anonfun$process$3.apply(EdgeProcessor.scala:183)
	at com.vesoft.nebula.exchange.processor.EdgeProcessor$$anonfun$process$3.apply(EdgeProcessor.scala:183)
	at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:935)
	at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:935)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2113)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2113)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:121)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$11.apply(Executor.scala:407)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:413)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2

读取超时了，你贴下你做导入操作的 spark connector 的配置代码，参考下面的 insert 插入示例代码：

  val config = NebulaConnectionConfig
    .builder()
    .withMetaAddress("127.0.0.1:9559")
    .withGraphAddress("127.0.0.1:9669")
    .build()
  val nebulaWriteVertexConfig: WriteNebulaVertexConfig = WriteNebulaVertexConfig
    .builder()
    .withSpace("test")
    .withTag("person")
    .withVidField("id")
    .withVidAsProp(true)
    .withBatch(1000)
    .withWriteMode(WriteMode.UPDATE)
    .build()
  df.write.nebula(config, nebulaWriteVertexConfig).writeVertices()
3 
{
  spark: {
    app: {
      name: NEBULA_EXCHANGE_EDGE_c6405c52cb394b26914d561c944c29e3_20240220173151
    }
  }

  nebula: {
    address:{
      graph:[*****]
      meta:[*****]
    }
    user: *****
    pswd: *****
    space: TQG_front_face
    connection: {
      timeout: 60000
      retry: 5
    }
    execution: {
      retry: 5
    }
    error: {
      max: 2048
      output: /data/tmp/nebula/log/errors/
    }
    rate: {
      limit: 2048
      timeout: 60000
    }
  }

  tags: [

  ]

  edges: [

    {
      name: orp
      type: {
        source: HIVE
        sink: NEBULA
      }
      exec:"select src_vid, dst_vid, crttime, upttime from dc_xxx.nebula_edge_orp"
      fields: [crttime, upttime]
      nebula.fields: [crttime,upttime]
      source: {
        field: src_vid
      }
      target: {
        field: dst_vid
      }
      batch: 48
      partition: 24
    }

  ]

}
4

…你都打码了，你看看这两个地址你的 spark connector 所在的机器是否能 ping 通。

6

这个是我故意写成这个的，为了iP不被发出来，这个是可以ping通的。

7

- -，你用的 nebula-exchange 啊。不是 nebula-spark-connector。

你看看这个帖子：使用nebula-exchange导入时报错 java.net.SocketTimeoutException: Read timed out

大概率是通信问题（你如果排除无误的话），那就是导入速率问题。

8

解决了，感谢

9

所以是啥问题？batch 么，还是端口号。

10

都不是,hosts的问题

11

困惑，hosts 不是 metad 和 graphd 的服务 ip 和端口信息构成的么？

12

因为要和hive通信解析，hive的这一块的hosts也要加上去，不然，无法解析

1 个赞
13

学习了，感谢 xinquan :face_holding_back_tears:

14

没事，互相学习嘛

1 个赞
15

此话题已在最后回复的 30 天后被自动关闭。不再允许新回复。