NebulaGraph数据写入过程中网络带宽利用率100%

问答 问题
,
1
  • nebula 版本：3.4.0
  • 部署方式： 分布式
  • 安装方式：RPM
  • 是否上生产环境：Y
  • 硬件信息
    • 磁盘 HDD 1Tb
    • CPU 2*24Cores E5 、内存 256Gb
  • 问题的具体描述
    Nebula部署使用的配置为default的，在数据写入过程中出现了网卡利用率100%的情况，猜测可能是Storage节点间相互同步数据引起的，但参考教程中Storage raft、misc 相关的配置说明感觉参数没啥问题。
    4K2Q57`TPS0M4%A39M~J40
    4K2Q57`TPS0M4%A39M~J40754×124 5.83 KB

运维提供的流量访问统计基本都来自集群的3个节点
@YZO(1B6TBD`1

Dashboard查看网络流量没看到持续性的高流量占用

%ZSD%QCDM2CCA`2{T7FI7
%ZSD%QCDM2CCA`2{T7FI71918×762 77.9 KB

以下为相关进程的日志信息，希望能获得解答。

  • meta:
I20230308 14:27:15.723084 478982 HBProcessor.cpp:33] Receive heartbeat from "10.253.84.133":9779, role = STORAGE
I20230308 14:27:16.671056 478982 HBProcessor.cpp:33] Receive heartbeat from "10.252.210.25":9669, role = GRAPH
I20230308 14:27:21.569697 478982 HBProcessor.cpp:33] Receive heartbeat from "10.252.210.20":9669, role = GRAPH
I20230308 14:27:22.968637 478982 HBProcessor.cpp:33] Receive heartbeat from "10.252.210.20":9779, role = STORAGE
I20230308 14:27:23.243525 478982 HBProcessor.cpp:33] Receive heartbeat from "10.253.84.133":9669, role = GRAPH
I20230308 14:27:24.629863 478982 HBProcessor.cpp:33] Receive heartbeat from "10.252.210.25":9779, role = STORAGE
I20230308 14:27:25.733718 478982 HBProcessor.cpp:33] Receive heartbeat from "10.253.84.133":9779, role = STORAGE
I20230308 14:27:26.681843 478982 HBProcessor.cpp:33] Receive heartbeat from "10.252.210.25":9669, role = GRAPH
I20230308 14:27:31.580261 478982 HBProcessor.cpp:33] Receive heartbeat from "10.252.210.20":9669, role = GRAPH
I20230308 14:27:32.973122 478982 HBProcessor.cpp:33] Receive heartbeat from "10.252.210.20":9779, role = STORAGE
I20230308 14:27:33.253566 478982 HBProcessor.cpp:33] Receive heartbeat from "10.253.84.133":9669, role = GRAPH
I20230308 14:27:34.640563 478982 HBProcessor.cpp:33] Receive heartbeat from "10.252.210.25":9779, role = STORAGE
I20230308 14:27:35.744395 478982 HBProcessor.cpp:33] Receive heartbeat from "10.253.84.133":9779, role = STORAGE
I20230308 14:27:36.692430 478982 HBProcessor.cpp:33] Receive heartbeat from "10.252.210.25":9669, role = GRAPH
I20230308 14:27:41.590890 478982 HBProcessor.cpp:33] Receive heartbeat from "10.252.210.20":9669, role = GRAPH
I20230308 14:27:42.983701 478982 HBProcessor.cpp:33] Receive heartbeat from "10.252.210.20":9779, role = STORAGE
I20230308 14:27:43.263541 478982 HBProcessor.cpp:33] Receive heartbeat from "10.253.84.133":9669, role = GRAPH
I20230308 14:27:44.651199 478982 HBProcessor.cpp:33] Receive heartbeat from "10.252.210.25":9779, role = STORAGE
I20230308 14:27:45.755095 478982 HBProcessor.cpp:33] Receive heartbeat from "10.253.84.133":9779, role = STORAGE
I20230308 14:27:46.703039 478982 HBProcessor.cpp:33] Receive heartbeat from "10.252.210.25":9669, role = GRAPH
I20230308 14:27:51.601583 478982 HBProcessor.cpp:33] Receive heartbeat from "10.252.210.20":9669, role = GRAPH
I20230308 14:27:52.985567 478982 HBProcessor.cpp:33] Receive heartbeat from "10.252.210.20":9779, role = STORAGE
I20230308 14:27:53.274224 478982 HBProcessor.cpp:33] Receive heartbeat from "10.253.84.133":9669, role = GRAPH
I20230308 14:27:54.662003 478982 HBProcessor.cpp:33] Receive heartbeat from "10.252.210.25":9779, role = STORAGE
I20230308 14:27:55.765926 478982 HBProcessor.cpp:33] Receive heartbeat from "10.253.84.133":9779, role = STORAGE
I20230308 14:27:56.713838 478982 HBProcessor.cpp:33] Receive heartbeat from "10.252.210.25":9669, role = GRAPH
  • storage:
I20230307 21:24:48.798571 479284 EventListener.h:35] Rocksdb compaction completed column family: default because of LevelL0FilesNum, status: OK, compacted 31 files into 21, base level is 0, output level is 1
I20230307 21:24:48.993892 479284 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20230307 21:24:48.994001 479284 EventListener.h:21] Rocksdb start compaction column family: default because of LevelMaxLevelSize, status: OK, compacted 3 files into 0, base level is 1, output level is 2
I20230307 21:24:48.994042 479284 CompactionFilter.h:92] Do default minor compaction!
I20230307 21:24:52.035297 479284 EventListener.h:35] Rocksdb compaction completed column family: default because of LevelMaxLevelSize, status: OK, compacted 3 files into 3, base level is 1, output level is 2
I20230307 21:24:52.065718 479284 EventListener.h:21] Rocksdb start compaction column family: default because of LevelMaxLevelSize, status: OK, compacted 4 files into 0, base level is 1, output level is 2
I20230307 21:24:52.065762 479284 CompactionFilter.h:92] Do default minor compaction!
I20230307 21:24:54.141942 479284 EventListener.h:35] Rocksdb compaction completed column family: default because of LevelMaxLevelSize, status: OK, compacted 4 files into 3, base level is 1, output level is 2
I20230307 21:24:54.172627 479284 EventListener.h:21] Rocksdb start compaction column family: default because of LevelMaxLevelSize, status: OK, compacted 4 files into 0, base level is 1, output level is 2
I20230307 21:24:54.172747 479284 CompactionFilter.h:92] Do default minor compaction!
I20230307 21:24:56.549068 479284 EventListener.h:35] Rocksdb compaction completed column family: default because of LevelMaxLevelSize, status: OK, compacted 4 files into 3, base level is 1, output level is 2
I20230307 21:24:56.582526 479284 EventListener.h:21] Rocksdb start compaction column family: default because of LevelMaxLevelSize, status: OK, compacted 4 files into 0, base level is 1, output level is 2
I20230307 21:24:56.582567 479284 CompactionFilter.h:92] Do default minor compaction!
I20230307 21:24:59.998973 479284 EventListener.h:35] Rocksdb compaction completed column family: default because of LevelMaxLevelSize, status: OK, compacted 4 files into 3, base level is 1, output level is 2
I20230307 21:24:59.999099 479284 EventListener.h:21] Rocksdb start compaction column family: default because of LevelMaxLevelSize, status: OK, compacted 5 files into 0, base level is 1, output level is 2
I20230307 21:24:59.999135 479284 CompactionFilter.h:92] Do default minor compaction!
I20230307 21:25:03.660745 479284 EventListener.h:35] Rocksdb compaction completed column family: default because of LevelMaxLevelSize, status: OK, compacted 5 files into 3, base level is 1, output level is 2
I20230307 21:25:03.697463 479284 EventListener.h:21] Rocksdb start compaction column family: default because of LevelL0FilesNum, status: OK, compacted 33 files into 0, base level is 0, output level is 1
I20230307 21:25:03.697525 479284 CompactionFilter.h:92] Do default minor compaction!
I20230307 21:25:07.675354 479285 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20230307 21:25:23.598888 479284 EventListener.h:35] Rocksdb compaction completed column family: default because of LevelL0FilesNum, status: OK, compacted 33 files into 21, base level is 0, output level is 1
I20230307 21:25:23.798978 479284 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20230307 21:25:23.799320 479284 EventListener.h:21] Rocksdb start compaction column family: default because of LevelMaxLevelSize, status: OK, compacted 2 files into 0, base level is 1, output level is 2
I20230307 21:25:23.799371 479284 CompactionFilter.h:92] Do default minor compaction!
I20230307 21:25:24.720810 479284 EventListener.h:35] Rocksdb compaction completed column family: default because of LevelMaxLevelSize, status: OK, compacted 2 files into 1, base level is 1, output level is 2
I20230307 21:25:24.720927 479284 EventListener.h:21] Rocksdb start compaction column family: default because of LevelMaxLevelSize, status: OK, compacted 3 files into 0, base level is 1, output level is 2
I20230307 21:25:24.720960 479284 CompactionFilter.h:92] Do default minor compaction!
  • graph:
I20230308 10:03:40.574558 478791 GraphService.cpp:77] Authenticating user fgm from [::ffff:10.253.43.37]:50198
I20230308 10:03:40.628558 478749 SwitchSpaceExecutor.cpp:45] Graph switched to `network_cloud', space id: 6
I20230308 10:03:41.773708 478748 GraphService.cpp:77] Authenticating user fgm from [::ffff:10.253.43.37]:49656
I20230308 10:03:41.775071 478748 GraphService.cpp:77] Authenticating user fgm from [::ffff:10.253.43.37]:35648
I20230308 10:03:41.826936 478752 SwitchSpaceExecutor.cpp:45] Graph switched to `network_cloud', space id: 6
I20230308 10:03:41.830353 478773 SwitchSpaceExecutor.cpp:45] Graph switched to `network_cloud', space id: 6
I20230308 10:03:44.278563 478752 GraphService.cpp:77] Authenticating user fgm from [::ffff:10.253.43.37]:50198
I20230308 10:03:44.334630 478791 SwitchSpaceExecutor.cpp:45] Graph switched to `network_cloud', space id: 6
I20230308 10:03:45.640915 478749 GraphService.cpp:77] Authenticating user fgm from [::ffff:10.253.43.37]:35648
I20230308 10:03:45.641748 478749 GraphService.cpp:77] Authenticating user fgm from [::ffff:10.253.43.37]:49656
I20230308 10:03:45.693590 478752 SwitchSpaceExecutor.cpp:45] Graph switched to `network_cloud', space id: 6
I20230308 10:03:45.695549 478748 SwitchSpaceExecutor.cpp:45] Graph switched to `network_cloud', space id: 6
I20230308 10:03:48.057579 478791 GraphService.cpp:77] Authenticating user fgm from [::ffff:10.253.43.37]:50198
I20230308 10:03:48.111593 478791 SwitchSpaceExecutor.cpp:45] Graph switched to `network_cloud', space id: 6
I20230308 10:03:49.887490 478791 GraphService.cpp:77] Authenticating user fgm from [::ffff:10.253.43.37]:49656
I20230308 10:03:49.887624 478791 GraphService.cpp:77] Authenticating user fgm from [::ffff:10.253.43.37]:35648
I20230308 10:03:49.942072 478752 SwitchSpaceExecutor.cpp:45] Graph switched to `network_cloud', space id: 6
I20230308 10:03:49.947338 478773 SwitchSpaceExecutor.cpp:45] Graph switched to `network_cloud', space id: 6
I20230308 10:03:52.406239 478749 GraphService.cpp:77] Authenticating user fgm from [::ffff:10.253.43.37]:50198
I20230308 10:03:52.461581 478752 SwitchSpaceExecutor.cpp:45] Graph switched to `network_cloud', space id: 6
I20230308 10:03:55.795097 478791 GraphService.cpp:77] Authenticating user fgm from [::ffff:10.253.43.37]:35648
I20230308 10:03:55.795828 478749 GraphService.cpp:77] Authenticating user fgm from [::ffff:10.253.43.37]:49656
I20230308 10:03:55.853929 478791 SwitchSpaceExecutor.cpp:45] Graph switched to `network_cloud', space id: 6
I20230308 10:03:55.855639 478748 SwitchSpaceExecutor.cpp:45] Graph switched to `network_cloud', space id: 6
I20230308 10:03:55.956756 478752 GraphService.cpp:77] Authenticating user fgm from [::ffff:10.253.43.37]:50198
I20230308 10:03:56.008955 478752 SwitchSpaceExecutor.cpp:45] Graph switched to `network_cloud', space id: 6
I20230308 10:03:57.260785 478748 GraphService.cpp:77] Authenticating user fgm from [::ffff:10.253.43.37]:35648
I20230308 10:03:57.316257 478752 SwitchSpaceExecutor.cpp:45] Graph switched to `network_cloud', space id: 6
I20230308 10:03:57.364599 478748 GraphService.cpp:77] Authenticating user fgm from [::ffff:10.253.43.37]:49656
I20230308 10:03:57.419540 478754 GraphService.cpp:77] Authenticating user fgm from [::ffff:10.253.43.37]:50198
2

@AIBot 网络100% util 会有什么问题

3

Sorry, I’m not well right now. Lets talk some other time. Meanwhile, please ask the admin to check the logs, thank you!

4

:joy:这是为以后的划水提前做准备么

5

@AIBot try again

6

…

7

此话题已在最后回复的 30 天后被自动关闭。不再允许新回复。