storaged性能问题

  • nebula 版本:v2-nightly 版本
  • 部署方式:k8s单机
  • 是否为线上版本:Y
  • 硬件信息
    • 磁盘SSD
    • CPU、内存信息:16 vCPU 128 GiB

新版本新能好像有问题,之前的版本比较正常的

I0428 11:54:33.404713    25 StorageClientBase.inl:166] Invalidate the leader for [6, 26]
I0428 11:54:33.404776    25 StorageClientBase.inl:166] Invalidate the leader for [6, 6]
I0428 11:54:33.404783    25 StorageClientBase.inl:166] Invalidate the leader for [6, 50]
I0428 11:54:33.404790    25 StorageClientBase.inl:166] Invalidate the leader for [6, 58]
I0428 11:54:33.404796    27 StorageClientBase.inl:166] Invalidate the leader for [6, 47]
I0428 11:54:33.404804    27 StorageClientBase.inl:166] Invalidate the leader for [6, 34]
I0428 11:54:33.404809    27 StorageClientBase.inl:166] Invalidate the leader for [6, 18]
I0428 11:54:33.404814    27 StorageClientBase.inl:166] Invalidate the leader for [6, 46]
I0428 11:54:33.404817    27 StorageClientBase.inl:166] Invalidate the leader for [6, 90]
I0428 11:54:33.404821    27 StorageClientBase.inl:166] Invalidate the leader for [6, 75]
I0428 11:54:33.404826    27 StorageClientBase.inl:166] Invalidate the leader for [6, 83]
I0428 11:54:33.404830    27 StorageClientBase.inl:166] Invalidate the leader for [6, 6]
I0428 11:54:33.404834    27 StorageClientBase.inl:166] Invalidate the leader for [6, 94]
I0428 11:54:33.404839    26 StorageClientBase.inl:166] Invalidate the leader for [6, 30]
I0428 11:54:33.404850    26 StorageClientBase.inl:166] Invalidate the leader for [6, 75]
E0428 11:54:33.404872    44 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 58
E0428 11:54:33.404884    38 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 94
E0428 11:54:33.404947    38 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 6
E0428 11:54:33.404912    44 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 50
E0428 11:54:33.404891    42 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 75
E0428 11:54:33.405053    42 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 30
E0428 11:54:33.404981    38 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 83
E0428 11:54:33.405019    44 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 6
E0428 11:54:33.405083    42 StorageAccessExecutor.h:123] Storage Error: part: 75, error: E_RPC_FAILURE(-3).
E0428 11:54:33.405175    44 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 26
E0428 11:54:33.405205    43 QueryInstance.cpp:107] Storage Error: part: 75, error: E_RPC_FAILURE(-3).
E0428 11:54:33.405117    38 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 75
E0428 11:54:33.405211    44 StorageAccessExecutor.h:123] Storage Error: part: 58, error: E_RPC_FAILURE(-3).
E0428 11:54:33.405263    38 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 90
E0428 11:54:33.405310    38 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 46
E0428 11:54:33.405294    44 QueryInstance.cpp:107] Storage Error: part: 58, error: E_RPC_FAILURE(-3).
E0428 11:54:33.405324    38 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 18
E0428 11:54:33.405376    38 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 34
E0428 11:54:33.405388    38 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 47
E0428 11:54:33.405402    38 StorageAccessExecutor.h:123] Storage Error: part: 94, error: E_RPC_FAILURE(-3).
E0428 11:54:33.405441    38 QueryInstance.cpp:107] Storage Error: part: 94, error: E_RPC_FAILURE(-3).
I0428 11:54:33.406250    13 ThriftClientManager.inl:61] resolve "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 as "192.168.201.71":9779
E0428 11:54:33.408720    28 StorageClientBase.inl:214] Request to "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 failed: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
I0428 11:54:33.408769    28 StorageClientBase.inl:166] Invalidate the leader for [6, 98]
I0428 11:54:33.408776    28 StorageClientBase.inl:166] Invalidate the leader for [6, 6]
I0428 11:54:33.408780    28 StorageClientBase.inl:166] Invalidate the leader for [6, 66]
I0428 11:54:33.408785    28 StorageClientBase.inl:166] Invalidate the leader for [6, 46]
I0428 11:54:33.408789    28 StorageClientBase.inl:166] Invalidate the leader for [6, 38]
I0428 11:54:33.408793    28 StorageClientBase.inl:166] Invalidate the leader for [6, 34]
E0428 11:54:33.408865    42 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 46
E0428 11:54:33.408895    42 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 38
E0428 11:54:33.408905    42 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 66
E0428 11:54:33.408915    42 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 34
E0428 11:54:33.408922    42 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 6
E0428 11:54:33.408931    42 StorageAccessExecutor.h:43] InsertEdgesExecutor failed, error E_RPC_FAILURE, part 98
E0428 11:54:33.408942    42 StorageAccessExecutor.h:123] Storage Error: part: 46, error: E_RPC_FAILURE(-3).
E0428 11:54:33.408972    42 QueryInstance.cpp:107] Storage Error: part: 46, error: E_RPC_FAILURE(-3).
E0428 11:54:33.411437    13 StorageClientBase.inl:214] Request to "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 failed: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
I0428 11:54:33.411499    13 StorageClientBase.inl:166] Invalidate the leader for [6, 14]
I0428 11:54:33.411505    13 StorageClientBase.inl:166] Invalidate the leader for [6, 42]
I0428 11:54:33.411511    13 StorageClientBase.inl:166] Invalidate the leader for [6, 35]
E0428 11:54:33.411581    38 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 42
E0428 11:54:33.411617    38 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 35
E0428 11:54:33.411634    38 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 14
E0428 11:54:33.411659    38 StorageAccessExecutor.h:123] Storage Error: part: 42, error: E_RPC_FAILURE(-3).
E0428 11:54:33.411711    48 QueryInstance.cpp:107] Storage Error: part: 42, error: E_RPC_FAILURE(-3).
I0428 11:54:33.417263    14 ThriftClientManager.inl:61] resolve "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 as "192.168.201.71":9779
I0428 11:54:33.417582    15 ThriftClientManager.inl:61] resolve "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 as "192.168.201.71":9779
I0428 11:54:33.419215    16 ThriftClientManager.inl:61] resolve "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 as "192.168.201.71":9779
E0428 11:54:33.424947    15 StorageClientBase.inl:214] Request to "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 failed: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E0428 11:54:33.424947    16 StorageClientBase.inl:214] Request to "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 failed: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E0428 11:54:33.424948    14 StorageClientBase.inl:214] Request to "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 failed: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
I0428 11:54:33.425021    15 StorageClientBase.inl:166] Invalidate the leader for [6, 34]
I0428 11:54:33.425105    15 StorageClientBase.inl:166] Invalidate the leader for [6, 26]
I0428 11:54:33.425113    16 StorageClientBase.inl:166] Invalidate the leader for [6, 51]
I0428 11:54:33.425120    16 StorageClientBase.inl:166] Invalidate the leader for [6, 25]
I0428 11:54:33.425128    14 StorageClientBase.inl:166] Invalidate the leader for [6, 82]
I0428 11:54:33.425143    14 StorageClientBase.inl:166] Invalidate the leader for [6, 66]
E0428 11:54:33.425173    38 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 26
E0428 11:54:33.425174    42 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 66
E0428 11:54:33.425256    42 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 82
E0428 11:54:33.425304    42 StorageAccessExecutor.h:123] Storage Error: part: 66, error: E_RPC_FAILURE(-3).
E0428 11:54:33.425189    47 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 25
E0428 11:54:33.425220    38 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 34
E0428 11:54:33.425352    47 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 51
E0428 11:54:33.425352    51 QueryInstance.cpp:107] Storage Error: part: 66, error: E_RPC_FAILURE(-3).
E0428 11:54:33.425374    38 StorageAccessExecutor.h:123] Storage Error: part: 26, error: E_RPC_FAILURE(-3).
E0428 11:54:33.425396    47 StorageAccessExecutor.h:123] Storage Error: part: 25, error: E_RPC_FAILURE(-3).
E0428 11:54:33.425473    38 QueryInstance.cpp:107] Storage Error: part: 26, error: E_RPC_FAILURE(-3).
E0428 11:54:33.425487    42 QueryInstance.cpp:107] Storage Error: part: 25, error: E_RPC_FAILURE(-3).
I0428 11:54:33.426252    17 ThriftClientManager.inl:61] resolve "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 as "192.168.201.71":9779
E0428 11:54:33.431264    17 StorageClientBase.inl:214] Request to "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 failed: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
I0428 11:54:33.431315    17 StorageClientBase.inl:166] Invalidate the leader for [6, 74]
I0428 11:54:33.431324    17 StorageClientBase.inl:166] Invalidate the leader for [6, 58]
I0428 11:54:33.431330    17 StorageClientBase.inl:166] Invalidate the leader for [6, 22]
I0428 11:54:33.431336    17 StorageClientBase.inl:166] Invalidate the leader for [6, 10]
I0428 11:54:33.431341    17 StorageClientBase.inl:166] Invalidate the leader for [6, 54]
I0428 11:54:33.431356    17 StorageClientBase.inl:166] Invalidate the leader for [6, 42]
I0428 11:54:33.431363    17 StorageClientBase.inl:166] Invalidate the leader for [6, 86]
I0428 11:54:33.431368    17 StorageClientBase.inl:166] Invalidate the leader for [6, 15]
I0428 11:54:33.431375    17 StorageClientBase.inl:166] Invalidate the leader for [6, 90]
E0428 11:54:33.431455    50 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 90
E0428 11:54:33.431496    50 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 15
E0428 11:54:33.431511    50 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 86
E0428 11:54:33.431524    50 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 42
E0428 11:54:33.431535    50 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 54
E0428 11:54:33.431564    50 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 10
E0428 11:54:33.431577    50 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 22
E0428 11:54:33.431592    50 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 58
E0428 11:54:33.431602    50 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 74
E0428 11:54:33.431619    50 StorageAccessExecutor.h:123] Storage Error: part: 90, error: E_RPC_FAILURE(-3).
E0428 11:54:33.431648    42 QueryInstance.cpp:107] Storage Error: part: 90, error: E_RPC_FAILURE(-3).
I0428 11:54:33.437734    18 ThriftClientManager.inl:61] resolve "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 as "192.168.201.71":9779
E0428 11:54:33.446056    18 StorageClientBase.inl:214] Request to "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 failed: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
I0428 11:54:33.446105    18 StorageClientBase.inl:166] Invalidate the leader for [6, 18]
I0428 11:54:33.446112    18 StorageClientBase.inl:166] Invalidate the leader for [6, 59]
I0428 11:54:33.446117    18 StorageClientBase.inl:166] Invalidate the leader for [6, 54]
I0428 11:54:33.446122    18 StorageClientBase.inl:166] Invalidate the leader for [6, 90]
E0428 11:54:33.446199    49 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 90
E0428 11:54:33.446251    49 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 54
E0428 11:54:33.446265    49 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 59
E0428 11:54:33.446276    49 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 18
E0428 11:54:33.446290    49 StorageAccessExecutor.h:123] Storage Error: part: 90, error: E_RPC_FAILURE(-3).
E0428 11:54:33.446333    50 QueryInstance.cpp:107] Storage Error: part: 90, error: E_RPC_FAILURE(-3).
I0428 11:54:33.446920    19 ThriftClientManager.inl:61] resolve "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 as "192.168.201.71":9779
E0428 11:54:33.455265    19 StorageClientBase.inl:214] Request to "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 failed: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
I0428 11:54:33.455324    19 StorageClientBase.inl:166] Invalidate the leader for [6, 98]
I0428 11:54:33.455332    19 StorageClientBase.inl:166] Invalidate the leader for [6, 10]
I0428 11:54:33.455335    19 StorageClientBase.inl:166] Invalidate the leader for [6, 38]
I0428 11:54:33.455341    19 StorageClientBase.inl:166] Invalidate the leader for [6, 6]
I0428 11:54:33.455345    19 StorageClientBase.inl:166] Invalidate the leader for [6, 30]
I0428 11:54:33.455349    19 StorageClientBase.inl:166] Invalidate the leader for [6, 70]
I0428 11:54:33.455355    19 StorageClientBase.inl:166] Invalidate the leader for [6, 78]
I0428 11:54:33.455384    19 StorageClientBase.inl:166] Invalidate the leader for [6, 42]
I0428 11:54:33.455389    19 StorageClientBase.inl:166] Invalidate the leader for [6, 14]
E0428 11:54:33.455458    38 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 14
E0428 11:54:33.455497    38 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 42
E0428 11:54:33.455507    38 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 78
E0428 11:54:33.455514    38 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 70
E0428 11:54:33.455524    38 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 30
E0428 11:54:33.455534    38 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 6
E0428 11:54:33.455564    38 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 38
E0428 11:54:33.455576    38 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 10
E0428 11:54:33.455590    38 StorageAccessExecutor.h:43] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 98
E0428 11:54:33.455605    38 StorageAccessExecutor.h:123] Storage Error: part: 14, error: E_RPC_FAILURE(-3).
E0428 11:54:33.455642    38 QueryInstance.cpp:107] Storage Error: part: 14, error: E_RPC_FAILURE(-3).
I0428 11:54:33.457495    20 ThriftClientManager.inl:61] resolve "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 as "192.168.201.71":9779
E0428 11:54:33.465749    20 StorageClientBase.inl:214] Request to "nebula-storaged-app-v1-0.nebula-storaged-app-v1.nebula.svc.cluster.local":9779 failed: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
I0428 11:54:33.465806    20 StorageClientBase.inl:166] Invalidate the leader for [6, 42]
I0428 11:54:33.465812    20 StorageClientBase.inl:166] Invalidate the leader for [6, 74]
I0428 11:54:33.465817    20 StorageClientBase.inl:166] Invalidate the leader for [6, 30]
I0428 11:54:33.465822    20 StorageClientBase.inl:166] Invalidate the leader for [6, 66]
I0428 11:54:33.465826    20 StorageClientBase.inl:166] Invalidate the leader for [6, 70]
I0428 11:54:33.465831    20 StorageClientBase.inl:166] Invalidate the leader for [6, 90]
I0428 11:54:33.465835    20 StorageClientBase.inl:166] Invalidate the leader for [6, 54]
I0428 11:54:33.465839    20 StorageClientBase.inl:166] Invalidate the leader for [6, 82]

这个插入失败做下重试就好,这个是因为 partition 对应的 leader 改变了。

1赞

请问有没有性能优化的参数 现在单个cpu没有占满 内存也达不到1G 但是并发稍微多一些就会有报错

麻烦你上传下所有storage的info日志

用以前的版本就恢复稳定了,用新的2.0.0或者2.0.1都有问题,请问版本之间有什么差异呢

vesoft/nebula-metad                                                           v2-nightly                    65f7cae029a2        3 weeks ago         290MB
vesoft/nebula-graphd                                                          v2-nightly                    745ee6fac97f        3 months ago        292MB
vesoft/nebula-storaged                                                        v2-nightly                    5f089ee01e71        3 months ago        297MB

有价值的日志就是上面发的 其他信息没有太大的用途 而且日志太大了

从2.0.0版本开始,确实改了这块逻辑,所以这块逻辑现在还有问题,我们在完善这块逻辑,你插入失败先做重试。

3赞

修复了麻烦通知一下 谢谢啦

修复了会在这里同步下。我在github记录这个问,假如修复了,这个issue会有pr关联,这样你也可以看得到。

1赞

浙ICP备20010487号