插入报错：Storaged Error：Not the leader of xxx. Plearse retry later

Berserker · 2024 年4 月 23 日 13:30

nebula 版本：v3.6.0
部署方式：k8s v1.23.9部署nebula集群
安装方式：helm安装，nebula-operator1.8.0，nebula-cluster-1.8.0
是否上生产环境： N
硬件信息
- 磁盘：机械硬盘
- CPU、内存信息：3*32C128G
问题的具体描述
使用go客户端插入数据，极大概率报错，客户端报错信息如下：

{"level":"ERROR","ts":"2024-04-23T19:14:43.054+0800","caller":"/root/dql/BACKEND/internal/app/attackablepath/analysis/rulelib/rulecommon.go:350","msg":"tenantId 85761044 writeGraph error : execute error, errmsg Storage Error: Not the leader of 1. Please retry later., err <nil>"}

这个报错不是百分百报错的，有小概率没问题，执行成功，但是大概率都是失败，报错的，执行失败的语句，拷贝到studio上，可以正常执行

相关的 meta / storage / graph info 日志信息（尽量使用文本形式方便检索）
graph日志如下：

I20240423 12:13:57.550557    54 GraphService.cpp:77] Authenticating user root from 190.166.180.89:42364

I20240423 12:14:01.153820   111 MetaClient.cpp:3263] Load leader of "nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 2 space

I20240423 12:14:01.153873   111 MetaClient.cpp:3263] Load leader of "nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 1 space

I20240423 12:14:01.153882   111 MetaClient.cpp:3263] Load leader of "nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 1 space

I20240423 12:14:01.153911   111 MetaClient.cpp:3269] Load leader ok

I20240423 12:14:11.196813   111 MetaClient.cpp:3263] Load leader of "nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 2 space

I20240423 12:14:11.196871   111 MetaClient.cpp:3263] Load leader of "nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 1 space

I20240423 12:14:11.196879   111 MetaClient.cpp:3263] Load leader of "nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 1 space

I20240423 12:14:11.196885   111 MetaClient.cpp:3269] Load leader ok

I20240423 12:14:17.584969    54 SwitchSpaceExecutor.cpp:45] Graph switched to `topo_85761044_33', space id: 110

E20240423 12:14:17.585829    44 LabelIndexSeek.cpp:56] SemanticError: No valid index for label `accessible'.

I20240423 12:14:17.682090    53 GraphService.cpp:77] Authenticating user root from 190.166.180.89:34916

I20240423 12:14:17.693357    52 SwitchSpaceExecutor.cpp:45] Graph switched to `topo_85761044_33', space id: 110

I20240423 12:14:18.627373    42 SwitchSpaceExecutor.cpp:45] Graph switched to `attackablepath_85761044_tmp', space id: 119

E20240423 12:14:18.632239    48 StorageAccessExecutor.h:47] InsertVerticesExecutor failed, error E_LEADER_CHANGED, part 1

E20240423 12:14:18.632369    53 QueryInstance.cpp:151] Storage Error: Not the leader of 1. Please retry later., query: INSERT VERTEX endpoint(),ecs(name) VALUES "809dadeb-0140-11ef-85b5-723a549ad4ed:ins-84ucjizy":("prod")

I20240423 12:14:21.242715   111 MetaClient.cpp:3263] Load leader of "nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 3 space

I20240423 12:14:21.242769   111 MetaClient.cpp:3263] Load leader of "nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 1 space

I20240423 12:14:21.242780   111 MetaClient.cpp:3263] Load leader of "nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 2 space

I20240423 12:14:21.242785   111 MetaClient.cpp:3269] Load leader ok

I20240423 12:14:30.396080    54 GraphService.cpp:77] Authenticating user root from 190.166.180.89:34928

I20240423 12:14:30.410571    48 SwitchSpaceExecutor.cpp:45] Graph switched to `topo_85761044_33', space id: 110

I20240423 12:14:33.517813    53 GraphService.cpp:77] Authenticating user root from 190.166.180.89:42380

I20240423 12:14:34.909772    48 GraphService.cpp:77] Authenticating user root from 190.166.180.89:34956

I20240423 12:14:34.921873    44 SwitchSpaceExecutor.cpp:45] Graph switched to `topo_85761044_33', space id: 110

I20240423 12:14:38.651492   113 MemoryUtils.cpp:227] sys:461.777MiB/3.000GiB 15.03% usr:49.000MiB/1.771GiB 2.70%

I20240423 12:14:41.298296   111 MetaClient.cpp:3263] Load leader of "nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 2 space

I20240423 12:14:41.298343   111 MetaClient.cpp:3263] Load leader of "nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 1 space

I20240423 12:14:41.298352   111 MetaClient.cpp:3263] Load leader of "nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 1 space

I20240423 12:14:41.298357   111 MetaClient.cpp:3269] Load leader ok

I20240423 12:14:51.339752   111 MetaClient.cpp:3263] Load leader of "nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 2 space

I20240423 12:14:51.339826   111 MetaClient.cpp:3263] Load leader of "nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 1 space

I20240423 12:14:51.339840   111 MetaClient.cpp:3263] Load leader of "nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 1 space

I20240423 12:14:51.339848   111 MetaClient.cpp:3269] Load leader ok

I20240423 12:14:53.553181    44 SwitchSpaceExecutor.cpp:45] Graph switched to `topo_85761044_33', space id: 110

E20240423 12:14:53.554116    52 LabelIndexSeek.cpp:56] SemanticError: No valid index for label `accessible'.

I20240423 12:14:54.579499    48 SwitchSpaceExecutor.cpp:45] Graph switched to `attackablepath_85761044_tmp', space id: 120

E20240423 12:14:54.581219    39 StorageAccessExecutor.h:47] InsertVerticesExecutor failed, error E_LEADER_CHANGED, part 2

E20240423 12:14:54.581362    48 QueryInstance.cpp:151] Storage Error: Not the leader of 2. Please retry later., query: INSERT VERTEX endpoint(),ecs(name) VALUES "04bf3695-fe1f-11ee-82c9-6ec2fe49b834:i-bp1eg8yt9s4uly5fm3hb":("VPC1-SWT1-ECS1-A")

I20240423 12:15:01.381985   111 MetaClient.cpp:3263] Load leader of "nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 3 space

I20240423 12:15:01.382046   111 MetaClient.cpp:3263] Load leader of "nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 1 space

I20240423 12:15:01.382055   111 MetaClient.cpp:3263] Load leader of "nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 2 space

I20240423 12:15:01.382061   111 MetaClient.cpp:3269] Load leader ok

I20240423 12:15:02.206214    39 GraphService.cpp:77] Authenticating user root from 190.166.180.89:34916

I20240423 12:15:02.216775    40 SwitchSpaceExecutor.cpp:45] Graph switched to `topo_85761044_33', space id: 110

I20240423 12:15:07.725406    39 GraphService.cpp:77] Authenticating user root from 190.166.180.89:34928

I20240423 12:15:07.734665    39 SwitchSpaceExecutor.cpp:45] Graph switched to `topo_85761044_33', space id: 110

I20240423 12:15:11.432737    41 GraphService.cpp:77] Authenticating user root from 190.166.180.89:34956

I20240423 12:15:11.445641    42 SwitchSpaceExecutor.cpp:45] Graph switched to `topo_85761044_33', space id: 110

I20240423 12:15:37.959306    39 GraphService.cpp:77] Authenticating user root from 190.166.180.89:34916

I20240423 12:15:37.968745    42 SwitchSpaceExecutor.cpp:45] Graph switched to `topo_85761044_33', space id: 110

I20240423 12:15:37.976917    40 GraphService.cpp:77] Authenticating user root from 190.166.180.89:34928

I20240423 12:15:37.986164    48 SwitchSpaceExecutor.cpp:45] Graph switched to `topo_85761044_33', space id: 110

metad的日志如下：

I20240423 11:42:05.894420   145 ListHostsProcessor.cpp:354] List Hosts Failed, error E_LEADER_CHANGED

I20240423 11:44:03.908651   145 ListHostsProcessor.cpp:354] List Hosts Failed, error E_LEADER_CHANGED

I20240423 11:46:14.400156   145 ListHostsProcessor.cpp:354] List Hosts Failed, error E_LEADER_CHANGED

I20240423 12:43:33.475854   145 ListHostsProcessor.cpp:354] List Hosts Failed, error E_LEADER_CHANGED

I20240423 12:46:27.430919   145 ListHostsProcessor.cpp:354] List Hosts Failed, error E_LEADER_CHANGED

I20240423 12:47:19.038807   145 ListHostsProcessor.cpp:354] List Hosts Failed, error E_LEADER_CHANGED

I20240423 12:49:45.555850   145 ListHostsProcessor.cpp:354] List Hosts Failed, error E_LEADER_CHANGED

storaged日志如下：

I20240423 12:13:31.412161    74 MetaClient.cpp:3263] Load leader of "nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 2 space

I20240423 12:13:31.412240    74 MetaClient.cpp:3263] Load leader of "nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 1 space

I20240423 12:13:31.412256    74 MetaClient.cpp:3263] Load leader of "nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 1 space

I20240423 12:13:31.412266    74 MetaClient.cpp:3269] Load leader ok

I20240423 12:13:31.412768    74 NebulaStore.cpp:417] Create data space 118

I20240423 12:13:31.413497   468 RocksEngineConfig.cpp:371] Emplace rocksdb option max_open_files=20000

I20240423 12:13:31.413772   468 RocksEngineConfig.cpp:371] Emplace rocksdb option max_background_jobs=4

I20240423 12:13:31.413787   468 RocksEngineConfig.cpp:371] Emplace rocksdb option max_subcompactions=4

I20240423 12:13:31.413971   468 RocksEngineConfig.cpp:371] Emplace rocksdb option max_bytes_for_level_base=268435456

I20240423 12:13:31.413986   468 RocksEngineConfig.cpp:371] Emplace rocksdb option max_write_buffer_number=4

I20240423 12:13:31.413996   468 RocksEngineConfig.cpp:371] Emplace rocksdb option write_buffer_size=67108864

I20240423 12:13:31.414166   468 RocksEngineConfig.cpp:371] Emplace rocksdb option cache_index_and_filter_blocks=true

I20240423 12:13:31.414176   468 RocksEngineConfig.cpp:371] Emplace rocksdb option block_size=32768

I20240423 12:13:31.440659   468 RocksEngine.cpp:107] open rocksdb on /usr/local/nebula/data/storage/nebula/118/data

I20240423 12:13:31.441164    74 NebulaStore.cpp:480] Space 118, part 1 has been added, asLearner 0

I20240423 12:13:31.441438    74 NebulaStore.cpp:480] Space 118, part 2 has been added, asLearner 0

I20240423 12:13:31.442857    74 Part.cpp:494] [Port: 9780, Space: 116, Part: 1] Clean rocksdb part data

I20240423 12:13:31.442972    74 NebulaStore.cpp:594] Space 116, part 1 has been removed!

I20240423 12:13:31.443768    74 Part.cpp:494] [Port: 9780, Space: 116, Part: 2] Clean rocksdb part data

I20240423 12:13:31.443853    74 NebulaStore.cpp:594] Space 116, part 2 has been removed!

I20240423 12:13:31.443953    74 RocksEngine.h:247] Release rocksdb on /usr/local/nebula/data/storage/nebula/116

I20240423 12:13:31.444402    74 NebulaStore.cpp:737] Try to remove space directory: /usr/local/nebula/data/storage/nebula/116

I20240423 12:13:31.444947    74 NebulaStore.cpp:739] Space directory removed: /usr/local/nebula/data/storage/nebula/116

I20240423 12:13:31.444967    74 NebulaStore.cpp:555] Data space 116 has been removed!

I20240423 12:13:32.990289    76 Part.h:59] [Port: 9780, Space: 116, Part: 1] ~Part()

I20240423 12:13:40.750355    75 Part.h:59] [Port: 9780, Space: 116, Part: 2] ~Part()

I20240423 12:13:41.525489    74 MetaClient.cpp:3263] Load leader of "nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 2 space

I20240423 12:13:41.525554    74 MetaClient.cpp:3263] Load leader of "nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 1 space

I20240423 12:13:41.525568    74 MetaClient.cpp:3263] Load leader of "nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 2 space

I20240423 12:13:41.525576    74 MetaClient.cpp:3269] Load leader ok

I20240423 12:13:48.705850   203 MemoryUtils.cpp:227] sys:561.934MiB/2.000GiB 27.44% usr:134.982MiB/1.171GiB 11.26%

I20240423 12:13:51.569736    74 MetaClient.cpp:3263] Load leader of "nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 2 space

I20240423 12:13:51.569815    74 MetaClient.cpp:3263] Load leader of "nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 1 space

I20240423 12:13:51.569830    74 MetaClient.cpp:3263] Load leader of "nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 2 space

I20240423 12:13:51.569837    74 MetaClient.cpp:3269] Load leader ok

I20240423 12:14:01.614931    74 MetaClient.cpp:3263] Load leader of "nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 2 space

I20240423 12:14:01.615047    74 MetaClient.cpp:3263] Load leader of "nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 1 space

I20240423 12:14:01.615062    74 MetaClient.cpp:3263] Load leader of "nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 1 space

I20240423 12:14:01.615072    74 MetaClient.cpp:3269] Load leader ok

I20240423 12:14:01.615692    74 NebulaStore.cpp:417] Create data space 119

I20240423 12:14:01.626161   484 RocksEngineConfig.cpp:371] Emplace rocksdb option max_open_files=20000

I20240423 12:14:01.626214   484 RocksEngineConfig.cpp:371] Emplace rocksdb option max_background_jobs=4

I20240423 12:14:01.626221   484 RocksEngineConfig.cpp:371] Emplace rocksdb option max_subcompactions=4

I20240423 12:14:01.626353   484 RocksEngineConfig.cpp:371] Emplace rocksdb option max_bytes_for_level_base=268435456

I20240423 12:14:01.626400   484 RocksEngineConfig.cpp:371] Emplace rocksdb option max_write_buffer_number=4

I20240423 12:14:01.626410   484 RocksEngineConfig.cpp:371] Emplace rocksdb option write_buffer_size=67108864

I20240423 12:14:01.626628   484 RocksEngineConfig.cpp:371] Emplace rocksdb option cache_index_and_filter_blocks=true

I20240423 12:14:01.626643   484 RocksEngineConfig.cpp:371] Emplace rocksdb option block_size=32768

I20240423 12:14:01.711282   484 RocksEngine.cpp:107] open rocksdb on /usr/local/nebula/data/storage/nebula/119/data

I20240423 12:14:01.711809    74 NebulaStore.cpp:480] Space 119, part 1 has been added, asLearner 0

I20240423 12:14:01.712095    74 NebulaStore.cpp:480] Space 119, part 2 has been added, asLearner 0

I20240423 12:14:01.716235    74 Part.cpp:494] [Port: 9780, Space: 118, Part: 1] Clean rocksdb part data

I20240423 12:14:01.716382    74 NebulaStore.cpp:594] Space 118, part 1 has been removed!

I20240423 12:14:01.719175    74 Part.cpp:494] [Port: 9780, Space: 118, Part: 2] Clean rocksdb part data

I20240423 12:14:01.719264    74 NebulaStore.cpp:594] Space 118, part 2 has been removed!

I20240423 12:14:01.719341    74 RocksEngine.h:247] Release rocksdb on /usr/local/nebula/data/storage/nebula/118

I20240423 12:14:01.719710    74 NebulaStore.cpp:737] Try to remove space directory: /usr/local/nebula/data/storage/nebula/118

I20240423 12:14:01.720153    74 NebulaStore.cpp:739] Space directory removed: /usr/local/nebula/data/storage/nebula/118

I20240423 12:14:01.720167    74 NebulaStore.cpp:555] Data space 118 has been removed!

I20240423 12:14:02.803607    77 Part.h:59] [Port: 9780, Space: 118, Part: 2] ~Part()

I20240423 12:14:03.080727    75 Part.h:59] [Port: 9780, Space: 118, Part: 1] ~Part()

I20240423 12:14:11.776309    74 MetaClient.cpp:3263] Load leader of "nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 3 space

I20240423 12:14:11.776371    74 MetaClient.cpp:3263] Load leader of "nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 1 space

I20240423 12:14:11.776382    74 MetaClient.cpp:3263] Load leader of "nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 2 space

I20240423 12:14:11.776391    74 MetaClient.cpp:3269] Load leader ok

I20240423 12:14:21.817728    74 MetaClient.cpp:3263] Load leader of "nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 3 space

I20240423 12:14:21.817845    74 MetaClient.cpp:3263] Load leader of "nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 1 space

I20240423 12:14:21.817862    74 MetaClient.cpp:3263] Load leader of "nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 2 space

I20240423 12:14:21.817869    74 MetaClient.cpp:3269] Load leader ok

I20240423 12:14:41.877460    74 MetaClient.cpp:3263] Load leader of "nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 2 space

I20240423 12:14:41.877538    74 MetaClient.cpp:3263] Load leader of "nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 1 space

I20240423 12:14:41.877549    74 MetaClient.cpp:3263] Load leader of "nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 1 space

I20240423 12:14:41.877580    74 MetaClient.cpp:3269] Load leader ok

I20240423 12:14:41.878026    74 NebulaStore.cpp:417] Create data space 120

I20240423 12:14:41.878655   500 RocksEngineConfig.cpp:371] Emplace rocksdb option max_open_files=20000

I20240423 12:14:41.878692   500 RocksEngineConfig.cpp:371] Emplace rocksdb option max_background_jobs=4

I20240423 12:14:41.878700   500 RocksEngineConfig.cpp:371] Emplace rocksdb option max_subcompactions=4

I20240423 12:14:41.878834   500 RocksEngineConfig.cpp:371] Emplace rocksdb option max_bytes_for_level_base=268435456

I20240423 12:14:41.879701   500 RocksEngineConfig.cpp:371] Emplace rocksdb option max_write_buffer_number=4

I20240423 12:14:41.879714   500 RocksEngineConfig.cpp:371] Emplace rocksdb option write_buffer_size=67108864

I20240423 12:14:41.879974   500 RocksEngineConfig.cpp:371] Emplace rocksdb option cache_index_and_filter_blocks=true

I20240423 12:14:41.879990   500 RocksEngineConfig.cpp:371] Emplace rocksdb option block_size=32768

I20240423 12:14:41.907958   500 RocksEngine.cpp:107] open rocksdb on /usr/local/nebula/data/storage/nebula/120/data

I20240423 12:14:41.908371    74 NebulaStore.cpp:480] Space 120, part 1 has been added, asLearner 0

I20240423 12:14:41.908528    74 NebulaStore.cpp:480] Space 120, part 2 has been added, asLearner 0

I20240423 12:14:41.908802    74 Part.cpp:494] [Port: 9780, Space: 119, Part: 1] Clean rocksdb part data

I20240423 12:14:41.908881    74 NebulaStore.cpp:594] Space 119, part 1 has been removed!

I20240423 12:14:41.909071    74 Part.cpp:494] [Port: 9780, Space: 119, Part: 2] Clean rocksdb part data

I20240423 12:14:41.909111    74 NebulaStore.cpp:594] Space 119, part 2 has been removed!

I20240423 12:14:41.909170    74 RocksEngine.h:247] Release rocksdb on /usr/local/nebula/data/storage/nebula/119

I20240423 12:14:41.909519    74 NebulaStore.cpp:737] Try to remove space directory: /usr/local/nebula/data/storage/nebula/119

I20240423 12:14:41.909982    74 NebulaStore.cpp:739] Space directory removed: /usr/local/nebula/data/storage/nebula/119

I20240423 12:14:41.910009    74 NebulaStore.cpp:555] Data space 119 has been removed!

I20240423 12:14:42.881053    77 Part.h:59] [Port: 9780, Space: 119, Part: 1] ~Part()

I20240423 12:14:43.028348    78 Part.h:59] [Port: 9780, Space: 119, Part: 2] ~Part()

I20240423 12:14:48.706548   203 MemoryUtils.cpp:227] sys:562.500MiB/2.000GiB 27.47% usr:136.979MiB/1.171GiB 11.43%

I20240423 12:14:51.953059    74 MetaClient.cpp:3263] Load leader of "nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 3 space

I20240423 12:14:51.953146    74 MetaClient.cpp:3263] Load leader of "nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 1 space

I20240423 12:14:51.953163    74 MetaClient.cpp:3263] Load leader of "nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 2 space

I20240423 12:14:51.953174    74 MetaClient.cpp:3269] Load leader ok

I20240423 12:15:01.997784    74 MetaClient.cpp:3263] Load leader of "nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 3 space

I20240423 12:15:01.997857    74 MetaClient.cpp:3263] Load leader of "nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 1 space

I20240423 12:15:01.997876    74 MetaClient.cpp:3263] Load leader of "nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 2 space

I20240423 12:15:01.997887    74 MetaClient.cpp:3269] Load leader ok

I20240423 12:15:48.705739   203 MemoryUtils.cpp:227] sys:562.539MiB/2.000GiB 27.47% usr:135.979MiB/1.171GiB 11.34%

I20240423 12:16:48.705987   203 MemoryUtils.cpp:227] sys:562.457MiB/2.000GiB 27.46% usr:135.979MiB/1.171GiB 11.34%

I20240423 12:17:42.298864    74 MetaClient.cpp:3263] Load leader of "nebula-storaged-0.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 2 space

I20240423 12:17:42.298974    74 MetaClient.cpp:3263] Load leader of "nebula-storaged-1.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 1 space

I20240423 12:17:42.299018    74 MetaClient.cpp:3263] Load leader of "nebula-storaged-2.nebula-storaged-headless.nebula.svc.cluster.local":9779 in 1 space

I20240423 12:17:42.299027    74 MetaClient.cpp:3269] Load leader ok

I20240423 12:17:42.299407    74 NebulaStore.cpp:417] Create data space 121

江一旺 · 2024 年4 月 24 日 06:01

看起来像是你的storage连接的meta节点并非leader节点导致的，你可以尝试nebula-storaged.conf调整meta_address配置中的metad地址试试

Berserker · 2024 年4 月 24 日 06:42

这个怎么调，我通过k8s部署的，这个貌似是operator自动化调整的吧？
当前我的storaged的meta_server_addrs配置是这样的，这个我没配，应当是operator自动配上的：

meta_server_addrs="nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559,nebula-metad-1.nebula-metad-headless.nebula.svc.cluster.local:9559,nebula-metad-2.nebula-metad-headless.nebula.svc.cluster.local:9559"

JuniorG · 2024 年4 月 25 日 08:43

遇到过类似问题，提供一个思路：
1、你是否有多个meta节点，通过LB或者类似的代理中转后链接meta，或者在链接时配置了所有meta节点的IP地址
2、如果是，我遇到这种情况是发生在客户端连接到meta的从节点，然后从节点会返回主节点的地址，但这个地址是主节点的域名。此时如果在客户端与nebula不在同一集群，无法解析主节点域名或者解析域名错误，则会在重试后最终报这个错
3、解决方法：保证域名解析正确，或者在链接时只配置meta的主节点地址

Berserker · 2024 年4 月 25 日 08:47

确实有3个meta节点，但是这是通过operator在k8s上部署的，里面关于集群通信的一些基本配置，都是自动化的，所以应当不是你说的这种情况

kevin.qiao · 2024 年4 月 25 日 09:20

看起来是插入数据时partition 1的leader信息不一致导致的，server的日志都挺正常的，可以在客户端加个重试逻辑，遇到 leader changed之类的就重试

Berserker · 2024 年4 月 25 日 09:25

目前就是通过重试规避的，但是，这个报错，要是频率低的话，我还不会来发帖，主要是目前，报错频率甚至还要高于执行成功，感觉是不是哪里有坑？就这边新项目触发一次插入单条数据，5次有3次报这个错

江一旺 · 2024 年4 月 26 日 05:59

我猜想是你的storage集群负载比较高，part的leader一直在变化导致的，可以考虑把raft_hearbeat_timeout的值调大一些

Berserker · 2024 年4 月 26 日 06:41

大佬，你说的这个配置是哪个服务的呢？你看下我的配置呗，然后这个该调大多少呢？

然后就是，当前我们正在基于nebula做开发联调，里面就100多条数据，我不太认为是storaged负载过高导致的，因为集群内3个服务都是3节点的，内存storaged给的2G，3个节点都只占用了500Mi作用，cpu占用仅0.001个cpu。
然后就是我们这边业务会比较频繁的创建/删除图空间，举个例子，有个daemon，每小时同步一次数据到nebula中，然后这1个小时内，就只存在读操作了，之后再次同步新的数据，会写入一个新的图空间，旧的图空间会被删除。
我在想，会不会是我们这个使用方式不对，导致的？
大佬可以麻烦帮忙分析下吗

MuYi-方扬 · 2024 年5 月 6 日 15:38

100 多条数据的话，读和写的 qps 具体是多少？读是怎样的读？

Berserker · 2024 年5 月 7 日 01:23

没有读qps一说，页面点一下，触发一次查询，插入，平时读写qps=0

system · 2024 年6 月 6 日 01:23

此话题已在最后回复的 30 天后被自动关闭。不再允许新回复。