导入csv 数据报错 ErrMsg: Storage Error: part: 33, error: E_PART_NOT_FOUND(-16)., ErrCode: -1005

defp · 2021 年8 月 30 日 11:29

部署阿里云k8s上

  image: 'registry.cn-hangzhou.aliyuncs.com/nocode/nebula-storaged:v2.5.1'

磁盘用的nfs

- name: storaged
  nfs:
    path: /storaged

clientSettings:
  retry: 3
  concurrency: 10 # number of graph clients
  channelBufferSize: 128

并发设置10，导入几千万的数据也遇到这个问题 “导入csv数据时显示 ErrMsg: Storage Error: part: 48, error: E_RPC_FAILURE(-3)., ErrCode: -8 ”

ErrMsg: Storage Error: part: 33, error: E_PART_NOT_FOUND(-16)., ErrCode: -1005

客户端版本 2.5.0

nebula-importer-linux-amd64-v2.5.0

服务端 version

[root@nebula5-graphd-58454df749-d49vq nebula]# bin/nebula-graphd --version
nebula-graphd version 422c075, Git: 422c075, Build Time: Aug 23 2021 13:32:27
This source code is licensed under Apache 2.0 License, attached with Common Clause Condition 1.0.

类似问题

steam · 2021 年8 月 31 日 02:04

你的磁盘也是 HDD 吗？

defp · 2021 年8 月 31 日 03:04

I0831 11:03:33.914909 40 RaftPart.cpp:938] [Port: 9780, Space: 1, Part: 30] Leader succeeded in committing the logs 79738 to 79738
I0831 11:03:33.914914 40 RaftPart.cpp:958] [Port: 9780, Space: 1, Part: 30] logs size 0
I0831 11:03:33.914918 40 RaftPart.cpp:985] [Port: 9780, Space: 1, Part: 30] No more log to be replicated
I0831 11:03:34.040216 69 RaftPart.cpp:1287] [Port: 9780, Space: 1, Part: 90] Need to send heartbeat
I0831 11:03:34.040264 69 RaftPart.cpp:1913] [Port: 9780, Space: 1, Part: 90] Send heartbeat
I0831 11:03:34.040273 69 RaftPart.cpp:1962] [Port: 9780, Space: 1, Part: 90] Heartbeat is accepted by quorum
I0831 11:03:34.040279 69 RaftPart.cpp:1297] [Port: 9780, Space: 1, Part: 90] Schedule new task
I0831 11:03:34.040283 40 RaftPart.cpp:644] [Port: 9780, Space: 1, Part: 90] Checking whether buffer overflow
I0831 11:03:34.040308 40 RaftPart.cpp:656] [Port: 9780, Space: 1, Part: 90] Appending logs to the buffer
I0831 11:03:34.040318 40 RaftPart.cpp:676] [Port: 9780, Space: 1, Part: 90] Preparing to send AppendLog request
I0831 11:03:34.040321 40 RaftPart.cpp:709] [Port: 9780, Space: 1, Part: 90] Calling appendLogsInternal()
I0831 11:03:34.040326 40 RaftPart.cpp:735] [Port: 9780, Space: 1, Part: 90] Ready to append logs from id 79517 (Current term is 0)
I0831 11:03:34.040342 40 Part.cpp:378] [Port: 9780, Space: 1, Part: 90] logId 79517, termId 2, clusterId 0
I0831 11:03:34.042127 40 RaftPart.cpp:766] [Port: 9780, Space: 1, Part: 90] Succeeded writing logs [79517, 79517] to WAL
I0831 11:03:34.042140 40 RaftPart.cpp:812] [Port: 9780, Space: 1, Part: 90] About to replicate logs to all peer hosts
I0831 11:03:34.042157 40 RaftPart.cpp:856] [Port: 9780, Space: 1, Part: 90] Received enough response
I0831 11:03:34.042161 40 RaftPart.cpp:898] [Port: 9780, Space: 1, Part: 90] 0 hosts have accepted the logs
I0831 11:03:34.042165 40 AtomicLogBuffer.h:352] current node firstLogId = 79477, the seeking logId = 79517
I0831 11:03:34.042171 40 Part.cpp:207] [Port: 9780, Space: 1, Part: 90] Skip the heartbeat!
I0831 11:03:34.042196 40 AtomicLogBuffer.h:374] Release ref, readers = 1
I0831 11:03:34.042201 40 RaftPart.cpp:938] [Port: 9780, Space: 1, Part: 90] Leader succeeded in committing the logs 79517 to 79517

storeged 的日志，看不出啥来

steam · 2021 年8 月 31 日 03:05

哥，我问你磁盘类型是啥，你把你的硬件信息都贴一下

defp · 2021 年8 月 31 日 03:35

https://cos.ap-beijing.myqcloud.com/dropshare-1252438752/pb-tzGaS7aBph.txt
插入时候的日志

I0831 11:36:28.853662    54 Cpp2Connection.cpp:576] Channel 172.28.7.142:37686 closed: N6apache6thrift9transport19TTransportExceptionE: Channel Closed

defp · 2021 年8 月 31 日 03:39

E0831 11:39:23.607940    43 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 3
E0831 11:39:23.607965    43 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 57
E0831 11:39:23.607975    43 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 51
E0831 11:39:23.607985    43 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 75
E0831 11:39:23.607995    43 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 69
E0831 11:39:23.608003    43 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 21
E0831 11:39:23.608012    43 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 9
E0831 11:39:23.608019    43 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 99
E0831 11:39:23.608040    43 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 33
E0831 11:39:23.608052    43 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 45
E0831 11:39:23.608063    43 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 93
E0831 11:39:23.608070    43 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 27
E0831 11:39:23.608078    43 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 39
E0831 11:39:23.608086    43 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 87
E0831 11:39:23.608094    43 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 63
E0831 11:39:23.608101    43 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 81
E0831 11:39:23.608109    43 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 15
E0831 11:39:23.608120    43 StorageAccessExecutor.h:122] Storage Error: part: 3, error: E_PART_NOT_FOUND(-16).
E0831 11:39:23.608144    41 QueryInstance.cpp:110] Storage Error: part: 3, error: E_PART_NOT_FOUND(-16).

graphd的错误日志

E0831 11:41:25.646860    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 33
E0831 11:41:25.646883    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 21
E0831 11:41:25.646893    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 81
E0831 11:41:25.646901    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 45
E0831 11:41:25.646910    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 87
E0831 11:41:25.646926    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 15
E0831 11:41:25.646939    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 93
E0831 11:41:25.646951    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 27
E0831 11:41:25.646962    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 63
E0831 11:41:25.646972    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 57
E0831 11:41:25.646981    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 9
E0831 11:41:25.646992    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 75
E0831 11:41:25.647001    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 39
E0831 11:41:25.647011    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 3
E0831 11:41:25.647022    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 69
E0831 11:41:25.647032    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 99
E0831 11:41:25.647040    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 51
E0831 11:41:25.647056    42 StorageAccessExecutor.h:122] Storage Error: part: 33, error: E_PART_NOT_FOUND(-16).
E0831 11:41:25.647092    44 QueryInstance.cpp:110] Storage Error: part: 33, error: E_PART_NOT_FOUND(-16).
E0831 11:41:26.696188    40 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 33
E0831 11:41:26.696213    40 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 21
E0831 11:41:26.696223    40 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 81
E0831 11:41:26.696233    40 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 45
E0831 11:41:26.696247    40 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 87
E0831 11:41:26.696255    40 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 15
E0831 11:41:26.696264    40 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 93
E0831 11:41:26.696271    40 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 27
E0831 11:41:26.696280    40 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 63
E0831 11:41:26.696287    40 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 57
E0831 11:41:26.696295    40 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 9
E0831 11:41:26.696302    40 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 75
E0831 11:41:26.696310    40 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 39
E0831 11:41:26.696318    40 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 3
E0831 11:41:26.696326    40 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 69
E0831 11:41:26.696333    40 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 99
E0831 11:41:26.696341    40 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 51
E0831 11:41:26.696352    40 StorageAccessExecutor.h:122] Storage Error: part: 33, error: E_PART_NOT_FOUND(-16).
E0831 11:41:26.696377    42 QueryInstance.cpp:110] Storage Error: part: 33, error: E_PART_NOT_FOUND(-16).
E0831 11:41:27.755615    44 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 33
E0831 11:41:27.755641    44 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 21
E0831 11:41:27.755657    44 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 81
E0831 11:41:27.755669    44 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 45
E0831 11:41:27.755681    44 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 87
E0831 11:41:27.755689    44 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 15
E0831 11:41:27.755697    44 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 93
E0831 11:41:27.755704    44 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 27
E0831 11:41:27.755712    44 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 63
E0831 11:41:27.755720    44 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 57
E0831 11:41:27.755728    44 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 9
E0831 11:41:27.755735    44 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 75
E0831 11:41:27.755743    44 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 39
E0831 11:41:27.755750    44 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 3
E0831 11:41:27.755757    44 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 69
E0831 11:41:27.755765    44 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 99
E0831 11:41:27.755782    44 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 51
E0831 11:41:27.755795    44 StorageAccessExecutor.h:122] Storage Error: part: 33, error: E_PART_NOT_FOUND(-16).
E0831 11:41:27.755818    44 QueryInstance.cpp:110] Storage Error: part: 33, error: E_PART_NOT_FOUND(-16).
E0831 11:41:28.835016    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 63
E0831 11:41:28.835036    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 69
E0831 11:41:28.835044    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 57
E0831 11:41:28.835053    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 51
E0831 11:41:28.835060    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 9
E0831 11:41:28.835069    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 15
E0831 11:41:28.835076    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 3
E0831 11:41:28.835091    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 87
E0831 11:41:28.835103    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 21
E0831 11:41:28.835111    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 81
E0831 11:41:28.835121    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 45
E0831 11:41:28.835131    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 33
E0831 11:41:28.835141    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 99
E0831 11:41:28.835151    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 39
E0831 11:41:28.835162    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 93
E0831 11:41:28.835172    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 27
E0831 11:41:28.835181    42 StorageAccessExecutor.h:42] InsertVerticesExecutor failed, error E_PART_NOT_FOUND, part 75
E0831 11:41:28.835193    42 StorageAccessExecutor.h:122] Storage Error: part: 63, error: E_PART_NOT_FOUND(-16).
E0831 11:41:28.835225    42 QueryInstance.cpp:110] Storage Error: part: 63, error: E_PART_NOT_FOUND(-16).
E0831 11:41:43.401557    16 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E0831 11:41:43.401595    16 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E0831 11:41:43.401603    16 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E0831 11:42:04.899190    20 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E0831 11:42:04.899222    20 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E0831 11:42:04.899232    20 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E0831 11:42:11.760253    23 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E0831 11:42:11.760283    23 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E0831 11:42:11.760293    23 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E0831 11:42:19.967459    15 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E0831 11:42:19.967490    15 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E0831 11:42:19.967499    15 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E0831 11:44:49.797235    13 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E0831 11:44:49.797837    13 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E0831 11:44:49.797858    13 GeneratedCodeHelper.cpp:77] invalid message from client in function process

meta 的日志

Running on machine: nebula5-metad-1
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0831 11:29:37.200731   108 ActiveHostsMan.cpp:256] Get last update time failed, error: E_LEADER_CHANGED
E0831 12:06:13.503139   108 ActiveHostsMan.cpp:256] Get last update time failed, error: E_LEADER_CHANGED
[root@nebula5-metad-0 logs]# cat nebula-metad.WARNING 
Log file created at: 2021/08/31 11:29:37
Running on machine: nebula5-metad-1
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
W0831 11:29:37.186342   108 RaftPart.cpp:365] [Port: 9560, Space: 0, Part: 0] The partition is not a leader
W0831 11:29:37.200695   108 RaftPart.cpp:702] [Port: 9560, Space: 0, Part: 0] Cannot append logs, clean the buffer
E0831 11:29:37.200731   108 ActiveHostsMan.cpp:256] Get last update time failed, error: E_LEADER_CHANGED
E0831 12:06:13.503139   108 ActiveHostsMan.cpp:256] Get last update time failed, error: E_LEADER_CHANGED

steam · 2021 年8 月 31 日 04:01

我把你帖子从原来的移动出来了，避免打扰原来的贴主。然后这个新帖子等会会有人来回复你的

chenguang · 2021 年8 月 31 日 05:48

E_PART_NOT_FOUND 是不是表示没有从 meta 取到信息呢？这个错误搜遍了文档也没看到是怎么产生的？能回复下么

defp · 2021 年8 月 31 日 07:56

https://help.aliyun.com/document_detail/88940.html
用的aliyun 的 NAS

defp · 2021 年9 月 1 日 06:44

换ssd了，速度上去了，也没错误了

min.wu · 2021 年9 月 1 日 08:28

https://docs.nebula-graph.com.cn/site/pdf/NebulaGraph-book.pdf p303

defp · 2021 年9 月 1 日 09:47

看过，以为NAS就是性能低点，然而NAS都不能用

defp · 2021 年9 月 1 日 11:30

ErrMsg: Storage Error: part: 47, error: E_RPC_FAILURE(-3)
换成ssd后，导入数据中又出现这个问题了

3台nebula5-storaged 中的一台，挂过一次，重启了一次