- nebula 版本：3.1.0
- 部署方式：分布式
- 安装方式： RPM
- 是否为线上版本：Y
- 硬件信息
- 磁盘 4T*10 HHD
- CPU 20 ;内存 40GB
- 机器数量 3
- spark: 2.4.5 ；cpu 993 ； 内存4074496
- 问题的具体描述：导入时候大量失败，10亿级别数据能导入1/10就会失败
- 相关的 meta / storage / graph info 日志信息（尽量使用文本形式方便检索）
spark-submit --class xxx.xxx.xxxx \
--conf spark.blacklist.enabled=true \
--conf spark.executor.memoryOverhead=8G \
--conf spark.default.parallelism=400 \
--conf spark.yarn.maxAppAttempts=3 \
--conf spark.driver.maxResultSize=10G \
--conf spark.cores.max=400 \
--driver-memory 10G \
--executor-memory 8G --executor-cores 4 --num-executors 200 \
--master yarn --deploy-mode cluster
nebula 日志
E20220610 14:18:54.775686 21735 QueryInstance.cpp:137] Storage Error: Part 142 raft buffer is full. Please retry later.
E20220610 14:18:54.774785 21732 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 27
E20220610 14:18:54.776554 21732 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 82
E20220610 14:18:54.776618 21732 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 109
E20220610 14:18:54.776666 21732 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 142
E20220610 14:18:54.775686 21731 QueryInstance.cpp:137] Storage Error: Part 142 raft buffer is full. Please retry later.
E20220610 14:18:54.778466 21725 QueryInstance.cpp:137] Storage Error: Part 25 raft buffer is full. Please retry later.
E20220610 14:18:54.778470 21739 QueryInstance.cpp:137] Storage Error: Part 133 raft buffer is full. Please retry later.
E20220610 14:18:54.775511 21737 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 23
E20220610 14:18:54.780179 21737 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 133
E20220610 14:18:54.780264 21737 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 34
E20220610 14:18:54.783157 21735 QueryInstance.cpp:137] Storage Error: Part 133 raft buffer is full. Please retry later.
E20220610 14:18:54.783684 21740 QueryInstance.cpp:137] Storage Error: Part 142 raft buffer is full. Please retry later.
E20220610 14:18:54.786095 21736 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 142
E20220610 14:18:54.786216 21736 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 133
E20220610 14:18:54.787642 21737 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 105
E20220610 14:18:54.787770 21737 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 142
E20220610 14:18:54.787817 21737 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 82
E20220610 14:18:54.787890 21737 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 25
E20220610 14:18:54.787957 21737 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 133
E20220610 14:18:54.787997 21737 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 96
E20220610 14:18:54.792618 21730 QueryInstance.cpp:137] Storage Error: Part 18 raft buffer is full. Please retry later.
E20220610 14:18:54.793426 21735 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 105
E20220610 14:18:54.793516 21735 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 96
E20220610 14:18:54.793562 21735 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 25
E20220610 14:18:54.793601 21735 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 82
E20220610 14:18:54.793637 21735 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 142
E20220610 14:18:54.793674 21735 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 133
E20220610 14:18:54.793747 21735 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 34
E20220610 14:18:54.794473 21737 QueryInstance.cpp:137] Storage Error: Part 82 raft buffer is full. Please retry later.
E20220610 14:18:54.794836 21731 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 82
E20220610 14:18:54.794921 21731 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 96
E20220610 14:18:54.795001 21731 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 105
E20220610 14:18:54.795063 21731 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 25
E20220610 14:18:54.795107 21731 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 142
E20220610 14:18:54.795147 21731 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 133
E20220610 14:18:54.795233 21731 QueryInstance.cpp:137] Storage Error: Part 133 raft buffer is full. Please retry later.
E20220610 14:18:54.795393 21728 QueryInstance.cpp:137] Storage Error: Part 18 raft buffer is full. Please retry later.
E20220610 14:18:54.798167 21731 QueryInstance.cpp:137] Storage Error: Part 133 raft buffer is full. Please retry later.
E20220610 14:18:54.802944 21728 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 133
E20220610 14:18:54.803041 21728 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 142
E20220610 14:18:54.803099 21728 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 82
E20220610 14:18:54.803156 21728 QueryInstance.cpp:137] Storage Error: Part 133 raft buffer is full. Please retry later.
E20220610 14:18:54.803331 21734 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 142
E20220610 14:18:54.803452 21734 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 133
E20220610 14:18:54.803537 21734 QueryInstance.cpp:137] Storage Error: Part 96 raft buffer is full. Please retry later.
E20220610 14:18:54.804867 21731 QueryInstance.cpp:137] Storage Error: Part 105 raft buffer is full. Please retry later.
E20220610 14:18:54.810607 21740 QueryInstance.cpp:137] Storage Error: Part 105 raft buffer is full. Please retry later.
E20220610 14:18:54.812666 21725 QueryInstance.cpp:137] Storage Error: Part 96 raft buffer is full. Please retry later.
E20220610 14:18:54.812927 21738 QueryInstance.cpp:137] Storage Error: Part 96 raft buffer is full. Please retry later.
E20220610 14:18:54.813019 21732 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 18
E20220610 14:18:54.813094 21732 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 25
E20220610 14:18:54.813148 21732 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 23
E20220610 14:18:54.813153 21734 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 27
E20220610 14:18:54.813238 21732 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 34
E20220610 14:18:54.813341 21734 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 25
E20220610 14:18:54.813371 21732 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 133
E20220610 14:18:54.813238 21722 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 133
E20220610 14:18:54.813419 21734 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 142
E20220610 14:18:54.813766 21740 QueryInstance.cpp:137] Storage Error: Part 96 raft buffer is full. Please retry later.
E20220610 14:18:54.813597 21722 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 142
E20220610 14:18:54.813963 21722 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 82
E20220610 14:18:54.814014 21722 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 25
E20220610 14:18:54.814093 21722 QueryInstance.cpp:137] Storage Error: Part 25 raft buffer is full. Please retry later.
E20220610 14:18:54.814435 21727 QueryInstance.cpp:137] Storage Error: Part 109 raft buffer is full. Please retry later.
E20220610 14:18:54.814941 21737 QueryInstance.cpp:137] Storage Error: Part 96 raft buffer is full. Please retry later.
E20220610 14:18:54.813726 21741 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 18
E20220610 14:18:54.818185 21741 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 25
E20220610 14:18:54.818270 21741 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 45
E20220610 14:18:54.818352 21731 QueryInstance.cpp:137] Storage Error: Part 142 raft buffer is full. Please retry later.
E20220610 14:18:54.818432 21739 QueryInstance.cpp:137] Storage Error: Part 96 raft buffer is full. Please retry later.
E20220610 14:18:54.813796 21734 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 82
E20220610 14:18:54.818733 21734 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 34
E20220610 14:18:54.818792 21734 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 133
E20220610 14:18:54.813526 21732 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 27
E20220610 14:18:54.819010 21732 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 82
E20220610 14:18:54.819085 21732 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 109
E20220610 14:18:54.819124 21732 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 142
E20220610 14:18:54.818364 21741 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 23
E20220610 14:18:54.819300 21725 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 25
E20220610 14:18:54.819423 21725 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 142
E20220610 14:18:54.819515 21725 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 109
E20220610 14:18:54.819442 21732 QueryInstance.cpp:137] Storage Error: Part 96 raft buffer is full. Please retry later.
E20220610 14:18:54.819314 21741 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 34
E20220610 14:18:54.819844 21741 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 133
E20220610 14:18:54.819901 21741 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 109
E20220610 14:18:54.819942 21741 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 142
E20220610 14:18:54.819998 21741 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 27
E20220610 14:18:54.820060 21741 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 82
E20220610 14:18:54.820516 21727 QueryInstance.cpp:137] Storage Error: Part 18 raft buffer is full. Please retry later.
E20220610 14:18:54.820971 21724 QueryInstance.cpp:137] Storage Error: Part 133 raft buffer is full. Please retry later.
E20220610 14:18:54.821398 21723 QueryInstance.cpp:137] Storage Error: Part 142 raft buffer is full. Please retry later.
E20220610 14:18:54.821956 21735 QueryInstance.cpp:137] Storage Error: Part 82 raft buffer is full. Please retry later.
E20220610 14:18:54.818185 21729 QueryInstance.cpp:137] Storage Error: Part 25 raft buffer is full. Please retry later.
E20220610 14:18:54.819598 21725 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 18
E20220610 14:18:54.822796 21725 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 133
E20220610 14:18:54.822839 21725 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 34
E20220610 14:18:54.822881 21725 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 23
E20220610 14:18:54.822912 21725 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 105
E20220610 14:18:54.822943 21725 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 96
E20220610 14:18:54.822973 21725 StorageAccessExecutor.h:39] InsertVerticesExecutor failed, error E_RAFT_BUFFER_OVERFLOW, part 82
spark日志
com.vesoft.nebula.client.meta.exception.ExecuteFailedException: Execute failed: Get Space execute failed, errorCode: E_LEADER_LEASE_FAILED +details
com.vesoft.nebula.client.meta.exception.ExecuteFailedException: Execute failed: Get Space execute failed, errorCode: E_LEADER_LEASE_FAILED
at com.vesoft.nebula.client.meta.MetaClient.getSpace(MetaClient.java:244)
at com.vesoft.nebula.connector.nebula.MetaProvider.getVidType(MetaProvider.scala:66)
at com.vesoft.nebula.connector.writer.NebulaWriter.<init>(NebulaWriter.scala:40)
at com.vesoft.nebula.connector.writer.NebulaVertexWriter.<init>(NebulaVertexWriter.scala:18)
at com.vesoft.nebula.connector.writer.NebulaVertexWriterFactory.createDataWriter(NebulaSourceWriter.scala:28)
at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:113)
at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec$$anonfun$doExecute$2.apply(WriteToDataSourceV2Exec.scala:67)
at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec$$anonfun$doExecute$2.apply(WriteToDataSourceV2Exec.scala:66)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:123)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:415)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1403)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:421)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)