storaged性能问题

nextflow · 2021 年4 月 28 日 04:08

nebula 版本：v2-nightly 版本
部署方式：k8s单机
是否为线上版本：Y
硬件信息
- 磁盘SSD
- CPU、内存信息:16 vCPU 128 GiB

新版本新能好像有问题，之前的版本比较正常的

I0428 11:54:33.404713    25 StorageClientBase.inl:166] Invalidate the leader for [6, 26]
I0428 11:54:33.404776    25 StorageClientBase.inl:166] Invalidate the leader for [6, 6]
I0428 11:54:33.404783    25 StorageClientBase.inl:166] Invalidate the leader for [6, 50]
I0428 11:54:33.404790    25 StorageClientBase.inl:166] Invalidate the leader for [6, 58]
I0428 11:54:33.404796    27 StorageClientBase.inl:166] Invalidate the leader for [6, 47]
I0428 11:54:33.404804    27 StorageClientBase.inl:166] Invalidate the leader for [6, 34]
I0428 11:54:33.404809    27 StorageClientBase.inl:166] Invalidate the leader for [6, 18]
I0428 11:54:33.404814    27 StorageClientBase.inl:166] Invalidate the leader for [6, 46]
I0428 11:54:33.404817    27 StorageClientBase.inl:166] Invalidate the leader for [6, 90]
I0428 11:54:33.404821    27 StorageClientBase.inl:166] Invalidate the leader for [6, 75]
I0428 11:54:33.404826    27 StorageClientBase.inl:166] Invalidate the leader for [6, 83]
I0428 11:54:33.404830    27 StorageClientBase.inl:166] Invalidate the leader for [6, 6]
I0428 11:54:33.404834    27 StorageClientBase.inl:166] Invalidate the leader for [6, 94]
I0428 11:54:33.404839    26 StorageClientBase.inl:166] Invalidate the leader for [6, 30]
I0428 11:54:33.404850    26 StorageClientBase.inl:166] Invalidate the leader for [6, 75]
E0428 11:54:33.404872    44 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 58
E0428 11:54:33.404884    38 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 94
E0428 11:54:33.404947    38 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 6
E0428 11:54:33.404912    44 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 50
E0428 11:54:33.404891    42 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 75
E0428 11:54:33.405053    42 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 30
E0428 11:54:33.404981    38 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 83
E0428 11:54:33.405019    44 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 6
E0428 11:54:33.405083    42 StorageAccessExecutor.h:123] Storage Error: part: 75, error: E_RPC_FAILURE(-3).
E0428 11:54:33.405175    44 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 26
E0428 11:54:33.405205    43 QueryInstance.cpp:107] Storage Error: part: 75, error: E_RPC_FAILURE(-3).
E0428 11:54:33.405117    38 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 75
E0428 11:54:33.405211    44 StorageAccessExecutor.h:123] Storage Error: part: 58, error: E_RPC_FAILURE(-3).
E0428 11:54:33.405263    38 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 90
E0428 11:54:33.405310    38 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 46
E0428 11:54:33.405294    44 QueryInstance.cpp:107] Storage Error: part: 58, error: E_RPC_FAILURE(-3).
E0428 11:54:33.405324    38 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 18
E0428 11:54:33.405376    38 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 34
E0428 11:54:33.405388    38 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 47
E0428 11:54:33.405402    38 StorageAccessExecutor.h:123] Storage Error: part: 94, error: E_RPC_FAILURE(-3).
E0428 11:54:33.405441    38 QueryInstance.cpp:107] Storage Error: part: 94, error: E_RPC_FAILURE(-3).
I0428 11:54:33.406250    13 ThriftClientManager.inl:61] resolve "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 as "192.168.201.71":9779
E0428 11:54:33.408720    28 StorageClientBase.inl:214] Request to "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 failed: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
I0428 11:54:33.408769    28 StorageClientBase.inl:166] Invalidate the leader for [6, 98]
I0428 11:54:33.408776    28 StorageClientBase.inl:166] Invalidate the leader for [6, 6]
I0428 11:54:33.408780    28 StorageClientBase.inl:166] Invalidate the leader for [6, 66]
I0428 11:54:33.408785    28 StorageClientBase.inl:166] Invalidate the leader for [6, 46]
I0428 11:54:33.408789    28 StorageClientBase.inl:166] Invalidate the leader for [6, 38]
I0428 11:54:33.408793    28 StorageClientBase.inl:166] Invalidate the leader for [6, 34]
E0428 11:54:33.408865    42 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 46
E0428 11:54:33.408895    42 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 38
E0428 11:54:33.408905    42 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 66
E0428 11:54:33.408915    42 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 34
E0428 11:54:33.408922    42 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 6
E0428 11:54:33.408931    42 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 98
E0428 11:54:33.408942    42 StorageAccessExecutor.h:123] Storage Error: part: 46, error: E_RPC_FAILURE(-3).
E0428 11:54:33.408972    42 QueryInstance.cpp:107] Storage Error: part: 46, error: E_RPC_FAILURE(-3).
E0428 11:54:33.411437    13 StorageClientBase.inl:214] Request to "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 failed: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
I0428 11:54:33.411499    13 StorageClientBase.inl:166] Invalidate the leader for [6, 14]
I0428 11:54:33.411505    13 StorageClientBase.inl:166] Invalidate the leader for [6, 42]
I0428 11:54:33.411511    13 StorageClientBase.inl:166] Invalidate the leader for [6, 35]
E0428 11:54:33.411581    38 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 42
E0428 11:54:33.411617    38 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 35
E0428 11:54:33.411634    38 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 14
E0428 11:54:33.411659    38 StorageAccessExecutor.h:123] Storage Error: part: 42, error: E_RPC_FAILURE(-3).
E0428 11:54:33.411711    48 QueryInstance.cpp:107] Storage Error: part: 42, error: E_RPC_FAILURE(-3).
I0428 11:54:33.417263    14 ThriftClientManager.inl:61] resolve "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 as "192.168.201.71":9779
I0428 11:54:33.417582    15 ThriftClientManager.inl:61] resolve "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 as "192.168.201.71":9779
I0428 11:54:33.419215    16 ThriftClientManager.inl:61] resolve "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 as "192.168.201.71":9779
E0428 11:54:33.424947    15 StorageClientBase.inl:214] Request to "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 failed: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E0428 11:54:33.424947    16 StorageClientBase.inl:214] Request to "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 failed: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E0428 11:54:33.424948    14 StorageClientBase.inl:214] Request to "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 failed: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
I0428 11:54:33.425021    15 StorageClientBase.inl:166] Invalidate the leader for [6, 34]
I0428 11:54:33.425105    15 StorageClientBase.inl:166] Invalidate the leader for [6, 26]
I0428 11:54:33.425113    16 StorageClientBase.inl:166] Invalidate the leader for [6, 51]
I0428 11:54:33.425120    16 StorageClientBase.inl:166] Invalidate the leader for [6, 25]
I0428 11:54:33.425128    14 StorageClientBase.inl:166] Invalidate the leader for [6, 82]
I0428 11:54:33.425143    14 StorageClientBase.inl:166] Invalidate the leader for [6, 66]
E0428 11:54:33.425173    38 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 26
E0428 11:54:33.425174    42 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 66
E0428 11:54:33.425256    42 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 82
E0428 11:54:33.425304    42 StorageAccessExecutor.h:123] Storage Error: part: 66, error: E_RPC_FAILURE(-3).
E0428 11:54:33.425189    47 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 25
E0428 11:54:33.425220    38 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 34
E0428 11:54:33.425352    47 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 51
E0428 11:54:33.425352    51 QueryInstance.cpp:107] Storage Error: part: 66, error: E_RPC_FAILURE(-3).
E0428 11:54:33.425374    38 StorageAccessExecutor.h:123] Storage Error: part: 26, error: E_RPC_FAILURE(-3).
E0428 11:54:33.425396    47 StorageAccessExecutor.h:123] Storage Error: part: 25, error: E_RPC_FAILURE(-3).
E0428 11:54:33.425473    38 QueryInstance.cpp:107] Storage Error: part: 26, error: E_RPC_FAILURE(-3).
E0428 11:54:33.425487    42 QueryInstance.cpp:107] Storage Error: part: 25, error: E_RPC_FAILURE(-3).
I0428 11:54:33.426252    17 ThriftClientManager.inl:61] resolve "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 as "192.168.201.71":9779
E0428 11:54:33.431264    17 StorageClientBase.inl:214] Request to "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 failed: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
I0428 11:54:33.431315    17 StorageClientBase.inl:166] Invalidate the leader for [6, 74]
I0428 11:54:33.431324    17 StorageClientBase.inl:166] Invalidate the leader for [6, 58]
I0428 11:54:33.431330    17 StorageClientBase.inl:166] Invalidate the leader for [6, 22]
I0428 11:54:33.431336    17 StorageClientBase.inl:166] Invalidate the leader for [6, 10]
I0428 11:54:33.431341    17 StorageClientBase.inl:166] Invalidate the leader for [6, 54]
I0428 11:54:33.431356    17 StorageClientBase.inl:166] Invalidate the leader for [6, 42]
I0428 11:54:33.431363    17 StorageClientBase.inl:166] Invalidate the leader for [6, 86]
I0428 11:54:33.431368    17 StorageClientBase.inl:166] Invalidate the leader for [6, 15]
I0428 11:54:33.431375    17 StorageClientBase.inl:166] Invalidate the leader for [6, 90]
E0428 11:54:33.431455    50 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 90
E0428 11:54:33.431496    50 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 15
E0428 11:54:33.431511    50 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 86
E0428 11:54:33.431524    50 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 42
E0428 11:54:33.431535    50 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 54
E0428 11:54:33.431564    50 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 10
E0428 11:54:33.431577    50 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 22
E0428 11:54:33.431592    50 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 58
E0428 11:54:33.431602    50 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 74
E0428 11:54:33.431619    50 StorageAccessExecutor.h:123] Storage Error: part: 90, error: E_RPC_FAILURE(-3).
E0428 11:54:33.431648    42 QueryInstance.cpp:107] Storage Error: part: 90, error: E_RPC_FAILURE(-3).
I0428 11:54:33.437734    18 ThriftClientManager.inl:61] resolve "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 as "192.168.201.71":9779
E0428 11:54:33.446056    18 StorageClientBase.inl:214] Request to "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 failed: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
I0428 11:54:33.446105    18 StorageClientBase.inl:166] Invalidate the leader for [6, 18]
I0428 11:54:33.446112    18 StorageClientBase.inl:166] Invalidate the leader for [6, 59]
I0428 11:54:33.446117    18 StorageClientBase.inl:166] Invalidate the leader for [6, 54]
I0428 11:54:33.446122    18 StorageClientBase.inl:166] Invalidate the leader for [6, 90]
E0428 11:54:33.446199    49 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 90
E0428 11:54:33.446251    49 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 54
E0428 11:54:33.446265    49 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 59
E0428 11:54:33.446276    49 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 18
E0428 11:54:33.446290    49 StorageAccessExecutor.h:123] Storage Error: part: 90, error: E_RPC_FAILURE(-3).
E0428 11:54:33.446333    50 QueryInstance.cpp:107] Storage Error: part: 90, error: E_RPC_FAILURE(-3).
I0428 11:54:33.446920    19 ThriftClientManager.inl:61] resolve "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 as "192.168.201.71":9779
E0428 11:54:33.455265    19 StorageClientBase.inl:214] Request to "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 failed: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
I0428 11:54:33.455324    19 StorageClientBase.inl:166] Invalidate the leader for [6, 98]
I0428 11:54:33.455332    19 StorageClientBase.inl:166] Invalidate the leader for [6, 10]
I0428 11:54:33.455335    19 StorageClientBase.inl:166] Invalidate the leader for [6, 38]
I0428 11:54:33.455341    19 StorageClientBase.inl:166] Invalidate the leader for [6, 6]
I0428 11:54:33.455345    19 StorageClientBase.inl:166] Invalidate the leader for [6, 30]
I0428 11:54:33.455349    19 StorageClientBase.inl:166] Invalidate the leader for [6, 70]
I0428 11:54:33.455355    19 StorageClientBase.inl:166] Invalidate the leader for [6, 78]
I0428 11:54:33.455384    19 StorageClientBase.inl:166] Invalidate the leader for [6, 42]
I0428 11:54:33.455389    19 StorageClientBase.inl:166] Invalidate the leader for [6, 14]
E0428 11:54:33.455458    38 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 14
E0428 11:54:33.455497    38 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 42
E0428 11:54:33.455507    38 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 78
E0428 11:54:33.455514    38 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 70
E0428 11:54:33.455524    38 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 30
E0428 11:54:33.455534    38 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 6
E0428 11:54:33.455564    38 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 38
E0428 11:54:33.455576    38 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 10
E0428 11:54:33.455590    38 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 98
E0428 11:54:33.455605    38 StorageAccessExecutor.h:123] Storage Error: part: 14, error: E_RPC_FAILURE(-3).
E0428 11:54:33.455642    38 QueryInstance.cpp:107] Storage Error: part: 14, error: E_RPC_FAILURE(-3).
I0428 11:54:33.457495    20 ThriftClientManager.inl:61] resolve "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 as "192.168.201.71":9779
E0428 11:54:33.465749    20 StorageClientBase.inl:214] Request to "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 failed: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
I0428 11:54:33.465806    20 StorageClientBase.inl:166] Invalidate the leader for [6, 42]
I0428 11:54:33.465812    20 StorageClientBase.inl:166] Invalidate the leader for [6, 74]
I0428 11:54:33.465817    20 StorageClientBase.inl:166] Invalidate the leader for [6, 30]
I0428 11:54:33.465822    20 StorageClientBase.inl:166] Invalidate the leader for [6, 66]
I0428 11:54:33.465826    20 StorageClientBase.inl:166] Invalidate the leader for [6, 70]
I0428 11:54:33.465831    20 StorageClientBase.inl:166] Invalidate the leader for [6, 90]
I0428 11:54:33.465835    20 StorageClientBase.inl:166] Invalidate the leader for [6, 54]
I0428 11:54:33.465839    20 StorageClientBase.inl:166] Invalidate the leader for [6, 82]

dingding · 2021 年4 月 28 日 07:11

这个插入失败做下重试就好，这个是因为 partition 对应的 leader 改变了。

nextflow · 2021 年4 月 28 日 08:06

请问有没有性能优化的参数现在单个cpu没有占满内存也达不到1G 但是并发稍微多一些就会有报错

dingding · 2021 年4 月 28 日 08:13

麻烦你上传下所有storage的info日志

nextflow · 2021 年4 月 28 日 10:02

用以前的版本就恢复稳定了，用新的2.0.0或者2.0.1都有问题，请问版本之间有什么差异呢

vesoft/nebula-metad                                                           v2-nightly                    65f7cae029a2        3 weeks ago         290MB
vesoft/nebula-graphd                                                          v2-nightly                    745ee6fac97f        3 months ago        292MB
vesoft/nebula-storaged                                                        v2-nightly                    5f089ee01e71        3 months ago        297MB

nextflow · 2021 年4 月 28 日 10:06

有价值的日志就是上面发的其他信息没有太大的用途而且日志太大了

dingding · 2021 年4 月 28 日 10:07

从2.0.0版本开始，确实改了这块逻辑，所以这块逻辑现在还有问题，我们在完善这块逻辑，你插入失败先做重试。

nextflow · 2021 年4 月 28 日 10:15

修复了麻烦通知一下谢谢啦

dingding · 2021 年4 月 28 日 10:18

修复了会在这里同步下。我在github记录这个问，假如修复了，这个issue会有pr关联，这样你也可以看得到。
https://github.com/vesoft-inc/nebula-common/issues/526

xiaochen · 2021 年7 月 7 日 02:26

@dingding 这个issues啥时候能解决，现在查询经常会出现leader change的问题

dingding · 2021 年7 月 7 日 03:10

提了一个缓解这个问题的pr，会在2.1.0发布

xiaochen · 2021 年7 月 7 日 03:37

缓解？能够缓解到什么程度？现在已经是不可用的状态了

dingding · 2021 年7 月 7 日 10:37

你可以用master的代码试下，然后再提问题。