nebula进程总挂掉

  • nebula 版本:2.6.1
  • 部署方式:分布式
  • 安装方式:源码编译
  • 是否为线上版本:Y
  • 硬件信息
    • 磁盘 SSD
    • CPU 64核
    • 内存 265G
  • 问题的具体描述
    不定期的程序会挂。storaged、graphd、metad全会一起挂。
  • 相关的 storage 日志信息
#### 最后挂掉之前storage的log:
E1202 00:05:24.001101 61370 AddVerticesProcessor.cpp:164] The vertex locked : tag 49, vid fa0b830beda02b9cc28f4af3b6939e03092
E1202 00:05:24.491807 61363 AddVerticesProcessor.cpp:164] The vertex locked : tag 49, vid fa0b830beda02b9cc28f4af3b6939e03092
E1202 00:05:49.775272 61390 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1202 00:05:49.775305 61388 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1202 00:05:49.775346 61388 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1202 00:05:49.775358 61388 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1202 00:05:49.775367 61390 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1202 00:05:49.775401 61390 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1202 00:05:49.775405 61391 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1202 00:05:49.775445 61391 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1202 00:05:49.775455 61391 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1202 00:05:49.776181 61375 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1202 00:05:49.776208 61375 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1202 00:05:49.776217 61375 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1202 00:05:54.779211 61387 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1202 00:05:54.779266 61387 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1202 00:05:54.779278 61387 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1202 00:05:54.779286 61390 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1202 00:05:54.779318 61390 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1202 00:05:54.779328 61390 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1202 00:05:54.779470 61389 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1202 00:05:54.779513 61389 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1202 00:05:54.779526 61389 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1202 00:05:54.781086 61374 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1202 00:05:54.781113 61374 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1202 00:05:54.781123 61374 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1202 00:05:59.783205 61388 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 1330664521 (hex 0x4f505449, ascii 'OPTI') (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 48074)
E1202 00:05:59.783265 61375 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 1330664521 (hex 0x4f505449, ascii 'OPTI') (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 59950)
E1202 00:05:59.783290 61388 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1202 00:05:59.783321 61389 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 1330664521 (hex 0x4f505449, ascii 'OPTI') (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 58916)
E1202 00:05:59.783336 61375 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1202 00:05:59.783394 61389 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1202 00:05:59.783483 61392 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 1330664521 (hex 0x4f505449, ascii 'OPTI') (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 41422)
E1202 00:05:59.783556 61392 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1202 00:05:59.783856 61387 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 2147483688 (hex 0x80000028) (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 48092)
E1202 00:05:59.783937 61387 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1202 00:05:59.783939 61374 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 2147483688 (hex 0x80000028) (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 59968)
E1202 00:05:59.784003 61374 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1202 00:05:59.784006 61388 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 2147483688 (hex 0x80000028) (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 58934)
E1202 00:05:59.784065 61388 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1202 00:05:59.784135 61385 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 2147483688 (hex 0x80000028) (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 41440)
E1202 00:05:59.784207 61385 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1202 00:06:01.802224 61366 AddEdgesProcessor.cpp:172] edge locked : src JH174069797R90500000000_1, type 50, rank 1, dst e878aff00df1880bb24588a683dd3cf4092
E1202 00:06:12.292932 61392 PeekingManager.h:122] Received SSL connection on non SSL port
E1202 00:06:12.292968 61393 PeekingManager.h:122] Received SSL connection on non SSL port
E1202 00:06:12.293056 61374 PeekingManager.h:122] Received SSL connection on non SSL port
E1202 00:06:12.293159 61395 PeekingManager.h:122] Received SSL connection on non SSL port
E1202 00:06:12.294559 61385 PeekingManager.h:122] Received SSL connection on non SSL port
E1202 00:06:12.294615 61394 PeekingManager.h:122] Received SSL connection on non SSL port
E1202 00:06:12.294634 61392 PeekingManager.h:122] Received SSL connection on non SSL port
E1202 00:06:12.294756 61396 PeekingManager.h:122] Received SSL connection on non SSL port
E1202 00:06:18.298079 61455 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1202 00:06:43.327188 61457 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1202 00:06:43.333395 61458 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1202 00:06:50.846383 61456 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1202 00:06:50.865031 61457 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1202 00:06:50.877558 61458 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1202 00:06:50.882467 61455 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1202 00:06:55.898000 61393 PeekingManager.h:122] Received SSL connection on non SSL port
E1202 00:06:55.898766 61395 PeekingManager.h:122] Received SSL connection on non SSL port
E1202 00:06:55.899483 61385 PeekingManager.h:122] Received SSL connection on non SSL port
E1202 00:06:55.900235 61397 PeekingManager.h:122] Received SSL connection on non SSL port
E1202 00:07:05.044692 61391 MetaClient.cpp:636] Send request to "172.30.93.44":1559, exceed retry limit
E1202 00:07:05.044764 61401 MetaClient.cpp:139] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:18.057801 61394 MetaClient.cpp:636] Send request to "172.30.93.46":1559, exceed retry limit
E1202 00:07:18.057857 61401 MetaClient.cpp:139] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:31.071550 61387 MetaClient.cpp:636] Send request to "172.30.93.47":1559, exceed retry limit
E1202 00:07:31.071609 61401 MetaClient.cpp:139] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:44.084372 61390 MetaClient.cpp:636] Send request to "172.30.93.44":1559, exceed retry limit
E1202 00:07:44.084448 61401 MetaClient.cpp:139] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused

graph的log

E1202 00:06:20.654259 62481 StorageAccessExecutor.h:41] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 23
E1202 00:06:20.654285 62481 StorageAccessExecutor.h:124] Storage Error: part: 23, error: E_RPC_FAILURE(-3).
E1202 00:06:20.654314 62442 QueryInstance.cpp:108] Storage Error: part: 23, error: E_RPC_FAILURE(-3).
E1202 00:06:20.741328 62388 StorageClientBase-inl.h:143] Request to "172.30.93.47":1779 failed: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E1202 00:06:20.741398 62439 StorageAccessExecutor.h:41] GetNeighborsExecutor failed, error E_RPC_FAILURE, part 27
E1202 00:06:20.741432 62439 StorageAccessExecutor.h:124] Storage Error: part: 27, error: E_RPC_FAILURE(-3).
E1202 00:06:20.741466 62481 QueryInstance.cpp:108] Storage Error: part: 27, error: E_RPC_FAILURE(-3).
E1202 00:06:20.773772 62390 StorageClientBase-inl.h:143] Request to "172.30.93.46":2779 failed: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E1202 00:06:20.773821 62481 StorageAccessExecutor.h:41] GetNeighborsExecutor failed, error E_RPC_FAILURE, part 23
E1202 00:06:20.773844 62481 StorageAccessExecutor.h:124] Storage Error: part: 23, error: E_RPC_FAILURE(-3).
E1202 00:06:20.773875 62456 QueryInstance.cpp:108] Storage Error: part: 23, error: E_RPC_FAILURE(-3).
E1202 00:06:20.794982 62393 StorageClientBase-inl.h:143] Request to "172.30.93.47":1779 failed: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E1202 00:06:20.795032 62442 StorageAccessExecutor.h:41] GetNeighborsExecutor failed, error E_RPC_FAILURE, part 26
E1202 00:06:20.795058 62442 StorageAccessExecutor.h:124] Storage Error: part: 26, error: E_RPC_FAILURE(-3).
E1202 00:06:20.795086 62481 QueryInstance.cpp:108] Storage Error: part: 26, error: E_RPC_FAILURE(-3).
E1202 00:06:20.804795 62397 StorageClientBase-inl.h:143] Request to "172.30.93.47":2779 failed: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E1202 00:06:20.804839 62481 StorageAccessExecutor.h:41] GetNeighborsExecutor failed, error E_RPC_FAILURE, part 24
E1202 00:06:20.804857 62481 StorageAccessExecutor.h:124] Storage Error: part: 24, error: E_RPC_FAILURE(-3).
E1202 00:06:20.804888 62442 QueryInstance.cpp:108] Storage Error: part: 24, error: E_RPC_FAILURE(-3).
E1202 00:06:20.859989 62497 StorageClientBase-inl.h:143] Request to "172.30.93.48":3779 failed: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E1202 00:06:20.860054 62439 StorageAccessExecutor.h:41] GetNeighborsExecutor failed, error E_RPC_FAILURE, part 27
E1202 00:06:20.860083 62439 StorageAccessExecutor.h:124] Storage Error: part: 27, error: E_RPC_FAILURE(-3).
E1202 00:06:20.860112 62456 QueryInstance.cpp:108] Storage Error: part: 27, error: E_RPC_FAILURE(-3).
E1202 00:06:20.861400 62513 StorageClientBase-inl.h:143] Request to "172.30.93.47":1779 failed: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E1202 00:06:20.861443 62481 StorageAccessExecutor.h:41] GetNeighborsExecutor failed, error E_RPC_FAILURE, part 26
E1202 00:06:20.861462 62481 StorageAccessExecutor.h:124] Storage Error: part: 26, error: E_RPC_FAILURE(-3).
E1202 00:06:20.861517 62442 QueryInstance.cpp:108] Storage Error: part: 26, error: E_RPC_FAILURE(-3).
E1202 00:06:20.862392 62508 StorageClientBase-inl.h:143] Request to "172.30.93.47":1779 failed: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E1202 00:06:20.862444 62442 StorageAccessExecutor.h:41] GetNeighborsExecutor failed, error E_RPC_FAILURE, part 26
E1202 00:06:20.862471 62442 StorageAccessExecutor.h:124] Storage Error: part: 26, error: E_RPC_FAILURE(-3).
E1202 00:06:20.862499 62481 QueryInstance.cpp:108] Storage Error: part: 26, error: E_RPC_FAILURE(-3).
E1202 00:06:20.965025 62516 StorageClientBase-inl.h:143] Request to "172.30.93.47":1779 failed: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E1202 00:06:20.965076 62447 StorageAccessExecutor.h:41] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 26
E1202 00:06:20.965101 62519 StorageClientBase-inl.h:143] Request to "172.30.93.46":2779 failed: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E1202 00:06:20.965108 62447 StorageAccessExecutor.h:124] Storage Error: part: 26, error: E_RPC_FAILURE(-3).
E1202 00:06:20.965150 62481 StorageAccessExecutor.h:41] DeleteEdgesExecutor failed, error E_RPC_FAILURE, part 23
E1202 00:06:20.965195 62481 StorageAccessExecutor.h:41] DeleteEdgesExecutor failed, error E_LEADER_CHANGED, part 1
E1202 00:06:20.965209 62481 StorageAccessExecutor.h:41] DeleteEdgesExecutor failed, error E_LEADER_CHANGED, part 24
E1202 00:06:20.965220 62481 StorageAccessExecutor.h:41] DeleteEdgesExecutor failed, error E_LEADER_CHANGED, part 29
E1202 00:06:20.965234 62481 StorageAccessExecutor.h:124] Storage Error: part: 23, error: E_RPC_FAILURE(-3).
E1202 00:06:20.965168 62442 QueryInstance.cpp:108] Storage Error: part: 26, error: E_RPC_FAILURE(-3).
E1202 00:06:20.965258 62439 QueryInstance.cpp:108] Storage Error: part: 23, error: E_RPC_FAILURE(-3).
E1202 00:06:21.084614 62381 StorageClientBase-inl.h:143] Request to "172.30.93.47":3779 failed: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E1202 00:06:21.084667 62481 StorageAccessExecutor.h:41] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 23
E1202 00:06:21.084689 62481 StorageAccessExecutor.h:124] Storage Error: part: 23, error: E_RPC_FAILURE(-3).
E1202 00:06:21.084710 62447 QueryInstance.cpp:108] Storage Error: part: 23, error: E_RPC_FAILURE(-3).
E1202 00:06:21.105823 62382 StorageClientBase-inl.h:143] Request to "172.30.93.48":3779 failed: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E1202 00:06:21.105876 62447 StorageAccessExecutor.h:41] GetNeighborsExecutor failed, error E_RPC_FAILURE, part 26
E1202 00:06:21.105901 62447 StorageAccessExecutor.h:124] Storage Error: part: 26, error: E_RPC_FAILURE(-3).
E1202 00:06:21.105926 62481 QueryInstance.cpp:108] Storage Error: part: 26, error: E_RPC_FAILURE(-3).
E1202 00:06:21.115645 62383 StorageClientBase-inl.h:143] Request to "172.30.93.47":3779 failed: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E1202 00:06:21.115697 62447 StorageAccessExecutor.h:41] GetNeighborsExecutor failed, error E_RPC_FAILURE, part 24
E1202 00:06:21.115720 62447 StorageAccessExecutor.h:124] Storage Error: part: 24, error: E_RPC_FAILURE(-3).
E1202 00:06:21.115743 62447 QueryInstance.cpp:108] Storage Error: part: 24, error: E_RPC_FAILURE(-3).
E1202 00:06:21.170869 62384 StorageClientBase-inl.h:143] Request to "172.30.93.48":3779 failed: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E1202 00:06:21.170925 62469 StorageAccessExecutor.h:41] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 27
E1202 00:06:21.170953 62469 StorageAccessExecutor.h:124] Storage Error: part: 27, error: E_RPC_FAILURE(-3).
E1202 00:06:21.170974 62442 QueryInstance.cpp:108] Storage Error: part: 27, error: E_RPC_FAILURE(-3).
E1202 00:06:21.175278 62389 StorageClientBase-inl.h:143] Request to "172.30.93.48":1779 failed: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E1202 00:06:21.175320 62442 StorageAccessExecutor.h:41] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 28
E1202 00:06:21.175343 62442 StorageAccessExecutor.h:124] Storage Error: part: 28, error: E_RPC_FAILURE(-3).
E1202 00:06:21.175362 62447 QueryInstance.cpp:108] Storage Error: part: 28, error: E_RPC_FAILURE(-3).
E1202 00:06:21.395512 62391 StorageClientBase-inl.h:143] Request to "172.30.93.46":2779 failed: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E1202 00:06:21.395570 62442 StorageAccessExecutor.h:41] GetNeighborsExecutor failed, error E_RPC_FAILURE, part 23
E1202 00:06:21.395601 62442 StorageAccessExecutor.h:124] Storage Error: part: 23, error: E_RPC_FAILURE(-3).
E1202 00:06:21.395637 62439 QueryInstance.cpp:108] Storage Error: part: 23, error: E_RPC_FAILURE(-3).
E1202 00:06:21.416709 62392 StorageClientBase-inl.h:143] Request to "172.30.93.48":3779 failed: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E1202 00:06:21.416764 62439 StorageAccessExecutor.h:41] GetNeighborsExecutor failed, error E_RPC_FAILURE, part 26
E1202 00:06:21.416786 62439 StorageAccessExecutor.h:124] Storage Error: part: 26, error: E_RPC_FAILURE(-3).
E1202 00:06:21.416810 62439 QueryInstance.cpp:108] Storage Error: part: 26, error: E_RPC_FAILURE(-3).
E1202 00:06:21.426476 62393 StorageClientBase-inl.h:143] Request to "172.30.93.47":2779 failed: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E1202 00:06:21.426534 62439 StorageAccessExecutor.h:41] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 24
E1202 00:06:21.426558 62439 StorageAccessExecutor.h:124] Storage Error: part: 24, error: E_RPC_FAILURE(-3).
E1202 00:06:21.426578 62439 QueryInstance.cpp:108] Storage Error: part: 24, error: E_RPC_FAILURE(-3).
E1202 00:06:21.486099 62394 StorageClientBase-inl.h:143] Request to "172.30.93.48":3779 failed: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E1202 00:06:21.486155 62439 StorageAccessExecutor.h:41] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 26
E1202 00:06:21.486178 62439 StorageAccessExecutor.h:124] Storage Error: part: 26, error: E_RPC_FAILURE(-3).
E1202 00:06:21.486207 62439 QueryInstance.cpp:108] Storage Error: part: 26, error: E_RPC_FAILURE(-3).
E1202 00:06:21.799878 62487 StorageClientBase-inl.h:143] Request to "172.30.93.47":2779 failed: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E1202 00:06:21.799934 62447 StorageAccessExecutor.h:41] GetNeighborsExecutor failed, error E_RPC_FAILURE, part 26
E1202 00:06:21.799965 62447 StorageAccessExecutor.h:124] Storage Error: part: 26, error: E_RPC_FAILURE(-3).
E1202 00:06:21.799989 62447 QueryInstance.cpp:108] Storage Error: part: 26, error: E_RPC_FAILURE(-3).
E1202 00:06:22.031592 62490 StorageClientBase-inl.h:143] Request to "172.30.93.47":2779 failed: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E1202 00:06:22.031666 62456 StorageAccessExecutor.h:41] GetNeighborsExecutor failed, error E_RPC_FAILURE, part 26
E1202 00:06:22.031700 62456 StorageAccessExecutor.h:124] Storage Error: part: 26, error: E_RPC_FAILURE(-3).
E1202 00:06:22.031733 62456 QueryInstance.cpp:108] Storage Error: part: 26, error: E_RPC_FAILURE(-3).
E1202 00:06:22.370551 62456 StorageAccessExecutor.h:41] GetNeighborsExecutor failed, error E_LEADER_CHANGED, part 23
E1202 00:06:22.370775 62456 QueryInstance.cpp:108] Storage Error: The leader has changed. Try again later
E1202 00:06:43.327251 62305 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1202 00:06:43.333051 62306 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1202 00:06:50.845225 62304 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1202 00:06:50.863744 62305 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1202 00:06:50.878021 62306 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1202 00:06:50.882834 62303 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1202 00:06:55.897100 62516 PeekingManager.h:122] Received SSL connection on non SSL port
E1202 00:07:05.046905 62499 MetaClient.cpp:636] Send request to "172.30.93.48":1559, exceed retry limit
E1202 00:07:05.047003 62395 MetaClient.cpp:139] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:05.047473 62492 MetaClient.cpp:636] Send request to "172.30.93.46":1559, exceed retry limit
E1202 00:07:05.047487 62495 MetaClient.cpp:636] Send request to "172.30.93.46":1559, exceed retry limit
E1202 00:07:05.047533 62439 SwitchSpaceExecutor.cpp:26] RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:05.047567 62447 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:05.047611 62439 GraphService.cpp:86] Create session for userName: root, ip: 10.171.241.38 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:05.047627 62456 QueryInstance.cpp:108] RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:05.047756 62494 MetaClient.cpp:636] Send request to "172.30.93.46":1559, exceed retry limit
E1202 00:07:05.047796 62456 SwitchSpaceExecutor.cpp:26] RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:05.047827 62439 QueryInstance.cpp:108] RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:05.048182 62469 QueryInstance.cpp:108] SemanticError: Space was not chosen.
E1202 00:07:05.048341 62439 QueryInstance.cpp:108] SemanticError: Space was not chosen.
E1202 00:07:05.048641 62439 QueryInstance.cpp:108] SemanticError: Space was not chosen.
E1202 00:07:05.048913 62439 QueryInstance.cpp:108] SemanticError: Space was not chosen.
E1202 00:07:05.049010 62498 MetaClient.cpp:636] Send request to "172.30.93.47":1559, exceed retry limit
E1202 00:07:05.049048 62498 GraphSessionManager.cpp:205] Update sessions failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:05.049090 62396 GraphSessionManager.cpp:229] Update sessions failed: Update sessions failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:05.049477 62496 MetaClient.cpp:636] Send request to "172.30.93.48":1559, exceed retry limit
E1202 00:07:05.049479 62497 MetaClient.cpp:636] Send request to "172.30.93.48":1559, exceed retry limit
E1202 00:07:05.049521 62447 SwitchSpaceExecutor.cpp:26] RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:05.049546 62439 SwitchSpaceExecutor.cpp:26] RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:05.049577 62469 QueryInstance.cpp:108] RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:05.049592 62456 QueryInstance.cpp:108] RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:05.050154 62456 QueryInstance.cpp:108] SemanticError: Space was not chosen.
E1202 00:07:05.050164 62439 QueryInstance.cpp:108] SemanticError: Space was not chosen.
E1202 00:07:05.050576 62439 QueryInstance.cpp:108] SemanticError: Space was not chosen.
E1202 00:07:05.050648 62439 QueryInstance.cpp:108] SemanticError: Space was not chosen.
E1202 00:07:05.050773 62439 QueryInstance.cpp:108] SemanticError: Space was not chosen.
E1202 00:07:05.051007 62456 QueryInstance.cpp:108] SemanticError: Space was not chosen.
E1202 00:07:05.051151 62456 QueryInstance.cpp:108] SemanticError: Space was not chosen.
E1202 00:07:05.051404 62439 QueryInstance.cpp:108] SemanticError: Space was not chosen.
E1202 00:07:05.051410 62456 QueryInstance.cpp:108] SemanticError: Space was not chosen.
E1202 00:07:05.051656 62456 QueryInstance.cpp:108] SemanticError: Space was not chosen.
E1202 00:07:05.051682 62439 QueryInstance.cpp:108] SemanticError: Space was not chosen.
E1202 00:07:05.051946 62439 QueryInstance.cpp:108] SemanticError: Space was not chosen.
E1202 00:07:05.052141 62456 QueryInstance.cpp:108] SemanticError: Space was not chosen.
E1202 00:07:05.052417 62447 QueryInstance.cpp:108] SemanticError: Space was not chosen.
E1202 00:07:27.766528 62500 MetaClient.cpp:636] Send request to "172.30.93.45":1559, exceed retry limit
E1202 00:07:27.766592 62439 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:27.766644 62439 GraphService.cpp:86] Create session for userName: root, ip: 10.171.229.175 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:27.767508 62501 MetaClient.cpp:636] Send request to "172.30.93.44":1559, exceed retry limit
E1202 00:07:27.767562 62439 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:27.767591 62439 GraphService.cpp:86] Create session for userName: root, ip: 10.172.35.191 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:27.767598 62502 MetaClient.cpp:636] Send request to "172.30.93.44":1559, exceed retry limit
E1202 00:07:27.767663 62439 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:27.767689 62439 GraphService.cpp:86] Create session for userName: root, ip: 10.171.168.33 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:28.767508 62505 MetaClient.cpp:636] Send request to "172.30.93.45":1559, exceed retry limit
E1202 00:07:28.767575 62395 MetaClient.cpp:139] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:28.768496 62503 MetaClient.cpp:636] Send request to "172.30.93.46":1559, exceed retry limit
E1202 00:07:28.768543 62439 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:28.768550 62504 MetaClient.cpp:636] Send request to "172.30.93.46":1559, exceed retry limit
E1202 00:07:28.768577 62439 GraphService.cpp:86] Create session for userName: root, ip: 10.171.122.139 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:28.768632 62447 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:28.768682 62447 GraphService.cpp:86] Create session for userName: root, ip: 10.171.9.34 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:28.768677 62506 MetaClient.cpp:636] Send request to "172.30.93.46":1559, exceed retry limit
E1202 00:07:28.768744 62506 GraphSessionManager.cpp:205] Update sessions failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:28.768787 62396 GraphSessionManager.cpp:229] Update sessions failed: Update sessions failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:41.780252 62507 MetaClient.cpp:636] Send request to "172.30.93.44":1559, exceed retry limit
E1202 00:07:41.780318 62395 MetaClient.cpp:139] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:41.784071 62508 MetaClient.cpp:636] Send request to "172.30.93.46":1559, exceed retry limit
E1202 00:07:41.784104 62508 GraphSessionManager.cpp:205] Update sessions failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:41.784133 62396 GraphSessionManager.cpp:229] Update sessions failed: Update sessions failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:46.821509 62509 MetaClient.cpp:636] Send request to "172.30.93.47":1559, exceed retry limit
E1202 00:07:46.821573 62447 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:46.821635 62447 GraphService.cpp:86] Create session for userName: root, ip: 10.171.110.5 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:46.823532 62510 MetaClient.cpp:636] Send request to "172.30.93.48":1559, exceed retry limit
E1202 00:07:46.823603 62456 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:46.823650 62456 GraphService.cpp:86] Create session for userName: root, ip: 10.171.9.34 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.822583 62511 MetaClient.cpp:636] Send request to "172.30.93.44":1559, exceed retry limit
E1202 00:07:49.822660 62456 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.822727 62456 GraphService.cpp:86] Create session for userName: root, ip: 10.172.202.39 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.823602 62513 MetaClient.cpp:636] Send request to "172.30.93.47":1559, exceed retry limit
E1202 00:07:49.823662 62456 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.823701 62456 GraphService.cpp:86] Create session for userName: root, ip: 10.171.241.38 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.824488 62512 MetaClient.cpp:636] Send request to "172.30.93.48":1559, exceed retry limit
E1202 00:07:49.824537 62456 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.824566 62456 GraphService.cpp:86] Create session for userName: root, ip: 10.172.35.191 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.825480 62516 MetaClient.cpp:636] Send request to "172.30.93.48":1559, exceed retry limit
E1202 00:07:49.825495 62514 MetaClient.cpp:636] Send request to "172.30.93.48":1559, exceed retry limit
E1202 00:07:49.825532 62456 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.825565 62456 GraphService.cpp:86] Create session for userName: root, ip: 10.171.122.139 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.825560 62447 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.825616 62456 GraphService.cpp:86] Create session for userName: root, ip: 10.172.5.251 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.825623 62519 MetaClient.cpp:636] Send request to "172.30.93.45":1559, exceed retry limit
E1202 00:07:49.825678 62456 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.825707 62456 GraphService.cpp:86] Create session for userName: root, ip: 10.171.168.33 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.826478 62515 MetaClient.cpp:636] Send request to "172.30.93.44":1559, exceed retry limit
E1202 00:07:49.826488 62518 MetaClient.cpp:636] Send request to "172.30.93.44":1559, exceed retry limit
E1202 00:07:49.826490 62517 MetaClient.cpp:636] Send request to "172.30.93.44":1559, exceed retry limit
E1202 00:07:49.826524 62456 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.826546 62447 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.826584 62456 GraphService.cpp:86] Create session for userName: root, ip: 10.172.110.203 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.826579 62439 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.826602 62447 GraphService.cpp:86] Create session for userName: root, ip: 10.171.128.237 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.826650 62439 GraphService.cpp:86] Create session for userName: root, ip: 10.172.179.203 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.827462 62520 MetaClient.cpp:636] Send request to "172.30.93.46":1559, exceed retry limit
E1202 00:07:49.827471 62521 MetaClient.cpp:636] Send request to "172.30.93.46":1559, exceed retry limit
E1202 00:07:49.827482 62522 MetaClient.cpp:636] Send request to "172.30.93.46":1559, exceed retry limit
E1202 00:07:49.827507 62439 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.827533 62456 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.827551 62447 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.827572 62439 GraphService.cpp:86] Create session for userName: root, ip: 10.172.239.147 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.827589 62456 GraphService.cpp:86] Create session for userName: root, ip: 10.171.235.44 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.827610 62447 GraphService.cpp:86] Create session for userName: root, ip: 10.172.80.108 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.828459 62523 MetaClient.cpp:636] Send request to "172.30.93.47":1559, exceed retry limit
E1202 00:07:49.828497 62447 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.828526 62447 GraphService.cpp:86] Create session for userName: root, ip: 10.171.46.165 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.829466 62524 MetaClient.cpp:636] Send request to "172.30.93.46":1559, exceed retry limit
E1202 00:07:49.829514 62447 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.829543 62447 GraphService.cpp:86] Create session for userName: root, ip: 10.171.127.163 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.830000 62372 MetaClient.cpp:636] Send request to "172.30.93.48":1559, exceed retry limit
E1202 00:07:49.830046 62447 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.830075 62447 GraphService.cpp:86] Create session for userName: root, ip: 10.172.128.175 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.830485 62309 MetaClient.cpp:636] Send request to "172.30.93.44":1559, exceed retry limit
E1202 00:07:49.830492 62308 MetaClient.cpp:636] Send request to "172.30.93.44":1559, exceed retry limit
E1202 00:07:49.830528 62456 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.830549 62447 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.830567 62456 GraphService.cpp:86] Create session for userName: root, ip: 10.172.99.180 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.830585 62447 GraphService.cpp:86] Create session for userName: root, ip: 10.172.242.225 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.834499 62374 MetaClient.cpp:636] Send request to "172.30.93.44":1559, exceed retry limit
E1202 00:07:49.834547 62447 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.834575 62447 GraphService.cpp:86] Create session for userName: root, ip: 10.172.209.254 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.834933 62375 MetaClient.cpp:636] Send request to "172.30.93.46":1559, exceed retry limit
E1202 00:07:49.834978 62447 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.835004 62447 GraphService.cpp:86] Create session for userName: root, ip: 10.172.71.251 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.838001 62376 MetaClient.cpp:636] Send request to "172.30.93.47":1559, exceed retry limit
E1202 00:07:49.838043 62447 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:49.838071 62447 GraphService.cpp:86] Create session for userName: root, ip: 10.171.129.148 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:54.912139 62379 MetaClient.cpp:636] Send request to "172.30.93.47":1559, exceed retry limit
E1202 00:07:54.912161 62380 MetaClient.cpp:636] Send request to "172.30.93.47":1559, exceed retry limit
E1202 00:07:54.912335 62378 MetaClient.cpp:636] Send request to "172.30.93.47":1559, exceed retry limit
E1202 00:07:54.912532 62377 MetaClient.cpp:636] Send request to "172.30.93.47":1559, exceed retry limit
E1202 00:07:54.912912 62456 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:54.912915 62447 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:54.913012 62439 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:54.913224 62469 GraphSessionManager.cpp:102] Create session failed:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:54.913514 62456 GraphService.cpp:86] Create session for userName: root, ip: 10.171.229.175 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:54.913846 62456 GraphService.cpp:86] Create session for userName: root, ip: 10.171.9.34 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:54.913997 62442 GraphService.cpp:86] Create session for userName: root, ip: 10.171.110.5 failed: Create session failed: RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E1202 00:07:54.914260 62451 GraphService.cpp:86] Create session for userName: root, ip

挂之前meta的log

E1202 00:04:47.149371  6211 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1202 00:04:47.149384  6035 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1202 00:04:47.149425  6211 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1202 00:04:47.149447  6035 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1202 00:04:47.149454  6211 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1202 00:04:47.149468  6035 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1202 00:04:52.278714  6037 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1202 00:04:52.278757  6037 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1202 00:04:52.278769  6037 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1202 00:04:52.280206  6212 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1202 00:04:52.280261  6212 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1202 00:04:52.280273  6212 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1202 00:04:57.526290  6038 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 1330664521 (hex 0x4f505449, ascii 'OPTI') (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 47534)
E1202 00:04:57.526373  6038 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1202 00:04:57.526466  6213 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 1330664521 (hex 0x4f505449, ascii 'OPTI') (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 57170)
E1202 00:04:57.526535  6213 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1202 00:04:57.853184  6039 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 2147483688 (hex 0x80000028) (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 53886)
E1202 00:04:57.853274  6039 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1202 00:04:57.853353  6214 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 2147483688 (hex 0x80000028) (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 35290)
E1202 00:04:57.853421  6214 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1202 00:05:10.969748  6043 PeekingManager.h:122] Received SSL connection on non SSL port
E1202 00:05:10.969938  6217 PeekingManager.h:122] Received SSL connection on non SSL port
E1202 00:05:11.159124  6044 PeekingManager.h:122] Received SSL connection on non SSL port
E1202 00:05:11.159232  6218 PeekingManager.h:122] Received SSL connection on non SSL port
E1202 00:05:11.612587  6080 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1202 00:05:36.902297  6078 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1202 00:05:36.904124  6079 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1202 00:05:44.413158  6077 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1202 00:05:44.415843  6078 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1202 00:05:44.418382  6079 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1202 00:05:44.421169  6080 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1202 00:05:54.970489  6027 PeekingManager.h:122] Received SSL connection on non SSL port
E1202 00:05:54.972021  6164 PeekingManager.h:122] Received SSL connection on non SSL port

所有服务都是同一版本么?是否有服务未升级,或者graph指定了错误的端口访问了你的这个集群呢?

storage timeout然后周期性的话,可能是compaction任务影响了业务,你可以看看是否在进行compaction。还有你的服务是部署在一台机器上的话,可能是内存导致kill,波及三个服务,这些日志暂时有这些可能,有什么新的发现可以更新一下。

所有服务肯定都是统一2.6.1版本
配置肯定都是对的,一直有写入和查询。
他是偶尔会core,有时候几个月都不core,有时候连着2天都会core。
挂的时候cpuload会特别高
比较奇怪的是为什么连meta都会一起挂掉
怎么看系统自动的compaction呢?我手动执行compaction发现很快就能执行完毕。

两个集群都有这个现象:
A:10台机器,每台上面1个meta、2个graphd、2个stored。(内存统一256G)
B:9台机器,每台上面1个meta、2个graphd、2~4个stored。(内存384G的4个,192G2个)

看上去某个query触发了bug
CPU高可能只是dump的时候IO各种大。

我觉得这里存在两个问题,一个是storage client timeout。这个问题普遍是资源不够所致。这个先不说,相比集群挂掉要好办。再说集群挂掉,storage client timeout是06:20之后,然后graphd和storaged连接不上metad是07:05之后,但metad的log只到05:54。如果metad的日志是全的话,只能理解为metad突然挂掉。这时候最后可能的是如@SuperYoko 所说,内存不够被kill了。所以可以看一下07:05左右的资源使用情况吗,尤其是内存?

另外,Nebula的设计初衷,也是和其他数据库不一样的地方就是存算分离,可以更好地做资源隔离。我建议可以充分利用这个特性,将graphd, storaged, metad分开来部署。起码确保metad不受其他影响。


内存挂的时候看着还剩余很多。

超时时间我都改的很短
–session_idle_timeout_secs=600
–storage_client_timeout_ms=300
–raft_rpc_timeout_ms=200

和这个图对应的挂掉的log
第一个挂掉的stored的log 172.30.93.40:

E1208 04:28:55.506508 73383 AddVerticesProcessor.cpp:164] The vertex locked : tag 49, vid fabd16b7ecd55bb1383da2d3460719d2092
E1208 06:13:51.466210 73415 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:13:51.466255 73400 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:13:51.466291 73400 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:13:51.466305 73400 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1208 06:13:51.466305 73415 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:13:51.466351 73416 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:13:51.466364 73415 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1208 06:13:51.466383 73416 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:13:51.466405 73416 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1208 06:13:51.466451 73418 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:13:51.466480 73418 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:13:51.466490 73418 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1208 06:13:56.470073 73414 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:13:56.470117 73414 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:13:56.470129 73414 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1208 06:13:56.470254 73409 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:13:56.470292 73409 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:13:56.470306 73409 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1208 06:13:56.470314 73399 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:13:56.470367 73399 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:13:56.470377 73399 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1208 06:13:56.471369 73419 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:13:56.471390 73419 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:13:56.471400 73419 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1208 06:14:01.474103 73416 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 1330664521 (hex 0x4f505449, ascii 'OPTI') (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 36522)
E1208 06:14:01.474200 73416 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1208 06:14:01.474206 73413 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 1330664521 (hex 0x4f505449, ascii 'OPTI') (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 52834)
E1208 06:14:01.474263 73417 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 1330664521 (hex 0x4f505449, ascii 'OPTI') (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 55476)
E1208 06:14:01.474285 73413 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1208 06:14:01.474329 73417 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1208 06:14:01.474370 73420 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 1330664521 (hex 0x4f505449, ascii 'OPTI') (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 43736)
E1208 06:14:01.474438 73420 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1208 06:14:01.474721 73409 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 2147483688 (hex 0x80000028) (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 36538)
E1208 06:14:01.474786 73409 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1208 06:14:01.475003 73412 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 2147483688 (hex 0x80000028) (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 52854)
E1208 06:14:01.475064 73412 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1208 06:14:01.475083 73418 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 2147483688 (hex 0x80000028) (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 55496)
E1208 06:14:01.475131 73421 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 2147483688 (hex 0x80000028) (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 43756)
E1208 06:14:01.475152 73418 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1208 06:14:01.475209 73421 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1208 06:14:13.982388 73419 PeekingManager.h:122] Received SSL connection on non SSL port
E1208 06:14:13.982514 73399 PeekingManager.h:122] Received SSL connection on non SSL port
E1208 06:14:13.982625 73421 PeekingManager.h:122] Received SSL connection on non SSL port
E1208 06:14:13.982686 73415 PeekingManager.h:122] Received SSL connection on non SSL port
E1208 06:14:13.983042 73420 PeekingManager.h:122] Received SSL connection on non SSL port
E1208 06:14:13.983117 73416 PeekingManager.h:122] Received SSL connection on non SSL port
E1208 06:14:13.983409 73422 PeekingManager.h:122] Received SSL connection on non SSL port
E1208 06:14:13.983455 73414 PeekingManager.h:122] Received SSL connection on non SSL port
E1208 06:14:19.985538 73481 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1208 06:14:45.006240 73479 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1208 06:14:45.009835 73480 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1208 06:14:52.519912 73478 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1208 06:14:52.541100 73479 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1208 06:14:52.550216 73480 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1208 06:14:52.555022 73481 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1208 06:14:57.567833 73421 PeekingManager.h:122] Received SSL connection on non SSL port
E1208 06:14:57.568800 73409 PeekingManager.h:122] Received SSL connection on non SSL port
E1208 06:14:57.569514 73423 PeekingManager.h:122] Received SSL connection on non SSL port
E1208 06:14:57.570173 73413 PeekingManager.h:122] Received SSL connection on non SSL port

另外一个节点172.30.93.44的log
metad

E1207 12:10:00.001413 103548 SessionManagerProcessor.cpp:144] Session id `1670384522284422' not found
E1207 14:20:00.000944 103574 SessionManagerProcessor.cpp:144] Session id `1670386922902925' not found
E1207 15:40:00.000795 103581 SessionManagerProcessor.cpp:144] Session id `1670394724917747' not found
E1207 16:00:00.002447 103538 SessionManagerProcessor.cpp:144] Session id `1670399526156713' not found
E1207 17:00:00.000456 103567 SessionManagerProcessor.cpp:144] Session id `1670400726465995' not found
E1207 17:30:00.000757 103565 SessionManagerProcessor.cpp:144] Session id `1670404327395152' not found
E1207 17:30:39.920847 103552 service_tcc.h:73] N6apache6thrift8protocol18TProtocolExceptionE: don't know what type: 14 in function updateSessions
E1207 20:00:00.000891 103552 SessionManagerProcessor.cpp:144] Session id `1670406127860229' not found
E1207 20:34:10.459998 103584 SessionManagerProcessor.cpp:144] Session id `1670416580909009' not found
E1207 21:20:00.001052 103549 SessionManagerProcessor.cpp:144] Session id `1670415130184821' not found
E1207 21:40:00.002331 103550 SessionManagerProcessor.cpp:144] Session id `1670419931423327' not found
E1207 22:30:00.001082 103561 SessionManagerProcessor.cpp:144] Session id `1670421131733563' not found
E1207 23:46:46.783288 103581 service_tcc.h:73] N6apache6thrift8protocol18TProtocolExceptionE: don't know what type: 14 in function updateSessions
E1207 23:46:48.784020 103581 service_tcc.h:73] N6apache6thrift8protocol18TProtocolExceptionE: don't know what type: 14 in function updateSessions
E1207 23:50:00.001206 103563 SessionManagerProcessor.cpp:144] Session id `1670424132508332' not found
E1208 01:30:00.001410 103584 SessionManagerProcessor.cpp:144] Session id `1670428933748428' not found
E1208 03:10:00.002131 103584 SessionManagerProcessor.cpp:144] Session id `1670434935298070' not found
E1208 05:50:00.003150 103567 SessionManagerProcessor.cpp:144] Session id `1670440936848039' not found
E1208 06:17:11.099279 103661 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:17:11.099330 103661 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:17:11.099341 103661 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1208 06:17:11.100064 103485 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:17:11.100108 103485 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:17:11.100122 103485 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1208 06:17:16.102325 103662 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:17:16.102360 103662 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:17:16.102370 103662 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1208 06:17:16.313211 103486 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:17:16.313244 103486 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:17:16.313258 103486 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1208 06:17:21.105691 103487 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 1330664521 (hex 0x4f505449, ascii 'OPTI') (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 39930)
E1208 06:17:21.105736 103663 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 1330664521 (hex 0x4f505449, ascii 'OPTI') (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 49564)
E1208 06:17:21.105765 103487 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1208 06:17:21.105804 103663 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1208 06:17:21.106542 103488 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 2147483688 (hex 0x80000028) (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 39952)
E1208 06:17:21.106597 103488 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1208 06:17:21.106712 103664 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 2147483688 (hex 0x80000028) (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 49588)
E1208 06:17:21.106786 103664 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1208 06:17:33.612941 103475 PeekingManager.h:122] Received SSL connection on non SSL port
E1208 06:17:33.613035 103667 PeekingManager.h:122] Received SSL connection on non SSL port
E1208 06:17:33.614588 103476 PeekingManager.h:122] Received SSL connection on non SSL port
E1208 06:17:33.614635 103613 PeekingManager.h:122] Received SSL connection on non SSL port
E1208 06:17:39.617975 103528 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1208 06:18:04.649883 103526 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1208 06:18:04.654203 103527 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1208 06:18:12.165925 103529 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1208 06:18:12.178359 103526 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1208 06:18:12.193778 103527 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1208 06:18:12.198302 103528 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1208 06:18:17.214542 103477 PeekingManager.h:122] Received SSL connection on non SSL port
E1208 06:18:17.215193 103614 PeekingManager.h:122] Received SSL connection on non SSL port

stored:

E1208 05:36:09.091665 103786 AddEdgesProcessor.cpp:172] edge locked : src 82fadc807edd00a4aa4c07505385bef2092, type -50, rank 1, dst JI218159332R90500000000_1
E1208 05:57:22.832284 103784 AddVerticesProcessor.cpp:164] The vertex locked : tag 49, vid e5af77c7cd22fdaae460fe3923418fcf092
E1208 06:03:08.163130 103784 AddVerticesProcessor.cpp:164] The vertex locked : tag 49, vid 48d17e4072eea3aa990600376c934dce092
E1208 06:04:10.263921 103794 AddVerticesProcessor.cpp:164] The vertex locked : tag 49, vid aeae1703551bca5d913c4644ec19fd6c092
E1208 06:15:59.425132 103781 AddVerticesProcessor.cpp:318] Error! ret = E_LEADER_CHANGED, spaceId 38
E1208 06:16:01.758222 103784 AddVerticesProcessor.cpp:318] Error! ret = E_LEADER_CHANGED, spaceId 38
E1208 06:17:11.100111 103805 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:17:11.100111 103828 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:17:11.100191 103828 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:17:11.100204 103828 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1208 06:17:11.100212 103805 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:17:11.100229 103805 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1208 06:17:11.100574 103824 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:17:11.100574 103821 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:17:11.100605 103824 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:17:11.100632 103821 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:17:11.100641 103824 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1208 06:17:11.100659 103821 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1208 06:17:16.102633 103829 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:17:16.102692 103829 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:17:16.102706 103829 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1208 06:17:16.103878 103823 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:17:16.103914 103823 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:17:16.103924 103823 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1208 06:17:16.105681 103820 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:17:16.105682 103804 GeneratedCodeHelper.cpp:183] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:17:16.105720 103820 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:17:16.105739 103804 GeneratedCodeHelper.cpp:115] received invalid message from client: No version identifier... old protocol client in strict mode? sz=1195725856
E1208 06:17:16.105749 103820 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1208 06:17:16.105763 103804 GeneratedCodeHelper.cpp:77] invalid message from client in function process
E1208 06:17:21.105804 103830 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 1330664521 (hex 0x4f505449, ascii 'OPTI') (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 34322)
E1208 06:17:21.105906 103830 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1208 06:17:21.105968 103816 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 1330664521 (hex 0x4f505449, ascii 'OPTI') (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 50094)
E1208 06:17:21.106035 103816 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1208 06:17:21.106227 103827 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 1330664521 (hex 0x4f505449, ascii 'OPTI') (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 58542)
E1208 06:17:21.106313 103827 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1208 06:17:21.106850 103822 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 2147483688 (hex 0x80000028) (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 34348)
E1208 06:17:21.106907 103822 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1208 06:17:21.106931 103828 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 2147483688 (hex 0x80000028) (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 50118)
E1208 06:17:21.106993 103828 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1208 06:17:21.107211 103819 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 1330664521 (hex 0x4f505449, ascii 'OPTI') (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 60050)
E1208 06:17:21.107301 103819 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1208 06:17:21.107535 103826 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 2147483688 (hex 0x80000028) (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 58574)
E1208 06:17:21.107612 103826 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1208 06:17:21.108395 103818 HeaderServerChannel.cpp:103] Received invalid request from client: N6apache6thrift9transport19TTransportExceptionE: Header transport frame is too large: 2147483688 (hex 0x80000028) (transport N6apache6thrift36PreReceivedDataAsyncTransportWrapperE, address 10.101.100.20, port 60080)
E1208 06:17:21.108464 103818 PeekingManager.h:274] peekSuccess failed, dropping connection: N6apache6thrift9transport19TTransportExceptionE: Channel is !good()
E1208 06:17:33.613117 103819 PeekingManager.h:122] Received SSL connection on non SSL port
E1208 06:17:33.613173 103822 PeekingManager.h:122] Received SSL connection on non SSL port
E1208 06:17:33.613445 103823 PeekingManager.h:122] Received SSL connection on non SSL port
E1208 06:17:33.613791 103827 PeekingManager.h:122] Received SSL connection on non SSL port
E1208 06:17:33.614684 103818 PeekingManager.h:122] Received SSL connection on non SSL port
E1208 06:17:33.614795 103821 PeekingManager.h:122] Received SSL connection on non SSL port
E1208 06:17:33.615161 103816 PeekingManager.h:122] Received SSL connection on non SSL port
E1208 06:17:33.615478 103826 PeekingManager.h:122] Received SSL connection on non SSL port
E1208 06:17:35.407372 103794 DeleteEdgesProcessor.cpp:97] The edge locked : src 5c93e353d295e0d4a00756d2873f96c2092, type -50, tank 1, dst JR186891512R90000000000_1
E1208 06:17:39.618084 103885 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1208 06:18:04.650099 103887 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1208 06:18:04.655326 103884 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1208 06:18:12.168238 103886 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1208 06:18:12.179790 103887 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1208 06:18:12.194208 103884 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1208 06:18:12.200135 103885 HTTP2Codec.cpp:1766] Connection error PROTOCOL_ERROR with ingress=
E1208 06:18:17.215868 103805 PeekingManager.h:122] Received SSL connection on non SSL port
E1208 06:18:17.216512 103820 PeekingManager.h:122] Received SSL connection on non SSL port
E1208 06:18:17.218540 103828 PeekingManager.h:122] Received SSL connection on non SSL port
E1208 06:18:17.221791 103825 PeekingManager.h:122] Received SSL connection on non SSL port

graphd:

E1208 05:59:14.299856 104997 StorageAccessExecutor.h:41] InsertVerticesExecutor failed, error E_DATA_CONFLICT_ERROR, part 35
E1208 05:59:14.299901 105009 QueryInstance.cpp:108] Storage Error: More than one request trying to add/update/delete one edge/vertex at the same time.
E1208 06:02:45.914824 104997 StorageAccessExecutor.h:41] InsertVerticesExecutor failed, error E_DATA_CONFLICT_ERROR, part 12
E1208 06:02:45.914885 105007 QueryInstance.cpp:108] Storage Error: More than one request trying to add/update/delete one edge/vertex at the same time.
E1208 06:03:08.164443 104982 StorageAccessExecutor.h:41] InsertVerticesExecutor failed, error E_DATA_CONFLICT_ERROR, part 9
E1208 06:03:08.164508 104985 QueryInstance.cpp:108] Storage Error: More than one request trying to add/update/delete one edge/vertex at the same time.
E1208 06:06:21.426864 104986 StorageAccessExecutor.h:41] InsertVerticesExecutor failed, error E_DATA_CONFLICT_ERROR, part 9
E1208 06:06:21.426934 104985 QueryInstance.cpp:108] Storage Error: More than one request trying to add/update/delete one edge/vertex at the same time.
E1208 06:12:58.846946 105007 StorageAccessExecutor.h:41] InsertVerticesExecutor failed, error E_DATA_CONFLICT_ERROR, part 38
E1208 06:12:58.847010 104984 QueryInstance.cpp:108] Storage Error: More than one request trying to add/update/delete one edge/vertex at the same time.
E1208 06:15:13.297294 104996 StorageAccessExecutor.h:41] InsertVerticesExecutor failed, error E_DATA_CONFLICT_ERROR, part 35
E1208 06:15:13.297358 104997 QueryInstance.cpp:108] Storage Error: More than one request trying to add/update/delete one edge/vertex at the same time.
E1208 06:15:45.598011 104905 StorageClientBase-inl.h:143] Request to "172.30.93.40":3779 failed: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E1208 06:15:45.598086 104985 StorageAccessExecutor.h:41] GetNeighborsExecutor failed, error E_RPC_FAILURE, part 1
E1208 06:15:45.598120 104985 StorageAccessExecutor.h:124] Storage Error: part: 1, error: E_RPC_FAILURE(-3).
E1208 06:15:45.598153 105005 QueryInstance.cpp:108] Storage Error: part: 1, error: E_RPC_FAILURE(-3).
E1208 06:15:45.598735 105005 StorageAccessExecutor.h:41] GetNeighborsExecutor failed, error E_LEADER_CHANGED, part 1
E1208 06:15:45.598775 105007 QueryInstance.cpp:108] Storage Error: The leader has changed. Try again later
E1208 06:15:45.911286 104939 StorageClientBase-inl.h:143] Request to "172.30.93.40":3779 failed: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E1208 06:15:45.911366 104997 StorageAccessExecutor.h:41] GetNeighborsExecutor failed, error E_RPC_FAILURE, part 1
E1208 06:15:45.911401 104997 StorageAccessExecutor.h:124] Storage Error: part: 1, error: E_RPC_FAILURE(-3).
E1208 06:15:45.911429 104985 QueryInstance.cpp:108] Storage Error: part: 1, error: E_RPC_FAILURE(-3).
E1208 06:15:45.912557 105007 StorageAccessExecutor.h:41] InsertVerticesExecutor failed, error E_LEADER_CHANGED, part 1
E1208 06:15:45.912598 105005 QueryInstance.cpp:108] Storage Error: The leader has changed. Try again later
E1208 06:15:46.223500 105043 StorageClientBase-inl.h:143] Request to "172.30.93.40":3779 failed: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E1208 06:15:46.223552 105007 StorageAccessExecutor.h:41] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 1
E1208 06:15:46.223578 105007 StorageAccessExecutor.h:124] Storage Error: part: 1, error: E_RPC_FAILURE(-3).
E1208 06:15:46.223599 104985 QueryInstance.cpp:108] Storage Error: part: 1, error: E_RPC_FAILURE(-3).
E1208 06:15:47.270200 104915 StorageClientBase-inl.h:143] Request to "172.30.93.40":3779 failed: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E1208 06:15:47.270260 105007 StorageAccessExecutor.h:41] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 1
E1208 06:15:47.270293 105007 StorageAccessExecutor.h:124] Storage Error: part: 1, error: E_RPC_FAILURE(-3).
E1208 06:15:47.270323 104982 QueryInstance.cpp:108] Storage Error: part: 1, error: E_RPC_FAILURE(-3).
E1208 06:15:47.271174 105007 StorageAccessExecutor.h:41] InsertEdgesExecutor failed, error E_LEADER_CHANGED, part 1
E1208 06:15:47.271206 105005 QueryInstance.cpp:108] Storage Error: The leader has changed. Try again later
E1208 06:15:47.609670 105019 StorageClientBase-inl.h:143] Request to "172.30.93.40":3779 failed: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E1208 06:15:47.609726 105005 StorageAccessExecutor.h:41] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 1
E1208 06:15:47.609753 105005 StorageAccessExecutor.h:124] Storage Error: part: 1, error: E_RPC_FAILURE(-3).
E1208 06:15:47.609774 104982 QueryInstance.cpp:108] Storage Error: part: 1, error: E_RPC_FAILURE(-3).
E1208 06:15:47.610839 105007 StorageAccessExecutor.h:41] DeleteEdgesExecutor failed, error E_LEADER_CHANGED, part 1
E1208 06:15:47.610881 104985 QueryInstance.cpp:108] Storage Error: The leader has changed. Try again later
E1208 06:15:49.445972 104918 StorageClientBase-inl.h:143] Request to "172.30.93.40":3779 failed: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E1208 06:15:49.446043 104982 StorageAccessExecutor.h:41] InsertVerticesExecutor failed, error E_RPC_FAILURE, part 31
E1208 06:15:49.446090 104982 StorageAccessExecutor.h:124] Storage Error: part: 31, error: E_RPC_FAILURE(-3).
E1208 06:15:49.446110 105007 QueryInstance.cpp:108] Storage Error: part: 31, error: E_RPC_FAILURE(-3).
E1208 06:15:50.748759 105015 StorageClientBase-inl.h:143] Request to "172.30.93.40":1779 failed: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E1208 06:15:50.748852 105007 StorageAccessExecutor.h:41] GetNeighborsExecutor failed, error E_RPC_FAILURE, part 34

这似乎是个碰到过的bug,找个存储的帮忙确定下
你也可以换2.6.2的包简单试一下。

我让存储的同学来给你看看哈。

我有两个集群,已经把其中一个升级成2.6.2了。问题还在。

1 个赞

我看到你们日志中有SSL相关的连接请求,怀疑是漏洞扫描。搜索后,你可以看一下这个安全扫描时服务宕机
相关的fix在3.2之后。可能需要你cherry-pick一下。

2 个赞