nebula-Storage日志不停的刷start send snapshot of commitLogId 日志 导致磁盘占满 进程异常

问答 问题
,
1
  • nebula 版本：3.6.0
  • 部署方式：分布式
  • 安装方式：源码编译
  • 是否上生产环境：Y
  • 硬件信息
    • 磁盘（ 推荐使用 SSD）
    • CPU、内存信息
      64U 128G
  • 问题的具体描述
    storaged日志 打了几百万行直到磁盘被占满
I20241005 23:15:43.510989 46879 NebulaSnapshotManager.cpp:67] Space 1 Part 6 start send snapshot of commitLogId 3239187 commitLogTerm 45, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.514302 46884 NebulaSnapshotManager.cpp:67] Space 1 Part 4 start send snapshot of commitLogId 3227556 commitLogTerm 48, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.519387 46888 NebulaSnapshotManager.cpp:67] Space 1 Part 9 start send snapshot of commitLogId 3231023 commitLogTerm 53, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.522881 46877 NebulaSnapshotManager.cpp:67] Space 1 Part 2 start send snapshot of commitLogId 3214951 commitLogTerm 61, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.525214 46879 NebulaSnapshotManager.cpp:67] Space 1 Part 6 start send snapshot of commitLogId 3239187 commitLogTerm 45, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.525755 46884 NebulaSnapshotManager.cpp:67] Space 1 Part 7 start send snapshot of commitLogId 3231809 commitLogTerm 49, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.528735 46888 NebulaSnapshotManager.cpp:67] Space 1 Part 4 start send snapshot of commitLogId 3227556 commitLogTerm 48, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.533113 46877 NebulaSnapshotManager.cpp:67] Space 1 Part 9 start send snapshot of commitLogId 3231023 commitLogTerm 53, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.535230 46879 NebulaSnapshotManager.cpp:67] Space 1 Part 2 start send snapshot of commitLogId 3214951 commitLogTerm 61, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.539093 46884 NebulaSnapshotManager.cpp:67] Space 1 Part 6 start send snapshot of commitLogId 3239187 commitLogTerm 45, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.543133 46888 NebulaSnapshotManager.cpp:67] Space 1 Part 4 start send snapshot of commitLogId 3227556 commitLogTerm 48, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.546491 46877 NebulaSnapshotManager.cpp:67] Space 1 Part 9 start send snapshot of commitLogId 3231023 commitLogTerm 53, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.548683 46879 NebulaSnapshotManager.cpp:67] Space 1 Part 2 start send snapshot of commitLogId 3214951 commitLogTerm 61, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.553301 46884 NebulaSnapshotManager.cpp:67] Space 1 Part 6 start send snapshot of commitLogId 3239187 commitLogTerm 45, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.558199 46888 NebulaSnapshotManager.cpp:67] Space 1 Part 4 start send snapshot of commitLogId 3227556 commitLogTerm 48, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.558609 46877 NebulaSnapshotManager.cpp:67] Space 1 Part 9 start send snapshot of commitLogId 3231023 commitLogTerm 53, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.562649 46879 NebulaSnapshotManager.cpp:67] Space 1 Part 2 start send snapshot of commitLogId 3214951 commitLogTerm 61, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.570513 46884 NebulaSnapshotManager.cpp:67] Space 1 Part 6 start send snapshot of commitLogId 3239187 commitLogTerm 45, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.570672 46888 NebulaSnapshotManager.cpp:67] Space 1 Part 9 start send snapshot of commitLogId 3231023 commitLogTerm 53, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.573602 46877 NebulaSnapshotManager.cpp:67] Space 1 Part 4 start send snapshot of commitLogId 3227556 commitLogTerm 48, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.576656 46879 NebulaSnapshotManager.cpp:67] Space 1 Part 2 start send snapshot of commitLogId 3214951 commitLogTerm 61, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.583012 46884 NebulaSnapshotManager.cpp:67] Space 1 Part 9 start send snapshot of commitLogId 3231023 commitLogTerm 53, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.587556 46888 NebulaSnapshotManager.cpp:67] Space 1 Part 6 start send snapshot of commitLogId 3239187 commitLogTerm 45, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.588948 46877 NebulaSnapshotManager.cpp:67] Space 1 Part 4 start send snapshot of commitLogId 3227556 commitLogTerm 48, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.591085 46879 NebulaSnapshotManager.cpp:67] Space 1 Part 2 start send snapshot of commitLogId 3214951 commitLogTerm 61, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.596204 46884 NebulaSnapshotManager.cpp:67] Space 1 Part 9 start send snapshot of commitLogId 3231023 commitLogTerm 53, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.602216 46888 NebulaSnapshotManager.cpp:67] Space 1 Part 2 start send snapshot of commitLogId 3214951 commitLogTerm 61, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.604555 46877 NebulaSnapshotManager.cpp:67] Space 1 Part 6 start send snapshot of commitLogId 3239187 commitLogTerm 45, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.605329 46879 NebulaSnapshotManager.cpp:67] Space 1 Part 4 start send snapshot of commitLogId 3227556 commitLogTerm 48, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.609090 46884 NebulaSnapshotManager.cpp:67] Space 1 Part 9 start send snapshot of commitLogId 3231023 commitLogTerm 53, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.614218 46888 NebulaSnapshotManager.cpp:67] Space 1 Part 2 start send snapshot of commitLogId 3214951 commitLogTerm 61, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.618738 46877 NebulaSnapshotManager.cpp:67] Space 1 Part 3 start send snapshot of commitLogId 3248867 commitLogTerm 61, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.619733 46879 NebulaSnapshotManager.cpp:67] Space 1 Part 6 start send snapshot of commitLogId 3239187 commitLogTerm 45, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.621390 46888 NebulaSnapshotManager.cpp:67] Space 1 Part 9 start send snapshot of commitLogId 3231023 commitLogTerm 53, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.621801 46884 NebulaSnapshotManager.cpp:67] Space 1 Part 4 start send snapshot of commitLogId 3227556 commitLogTerm 48, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.626835 46877 NebulaSnapshotManager.cpp:67] Space 1 Part 7 start send snapshot of commitLogId 3231809 commitLogTerm 49, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.627533 46879 NebulaSnapshotManager.cpp:67] Space 1 Part 2 start send snapshot of commitLogId 3214951 commitLogTerm 61, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.631117 46884 NebulaSnapshotManager.cpp:67] Space 1 Part 8 start send snapshot of commitLogId 3191246 commitLogTerm 44, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.633548 46888 NebulaSnapshotManager.cpp:67] Space 1 Part 9 start send snapshot of commitLogId 3231023 commitLogTerm 53, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.635064 46877 NebulaSnapshotManager.cpp:67] Space 1 Part 4 start send snapshot of commitLogId 3227556 commitLogTerm 48, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.642053 46879 NebulaSnapshotManager.cpp:67] Space 1 Part 2 start send snapshot of commitLogId 3214951 commitLogTerm 61, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.644279 46884 NebulaSnapshotManager.cpp:67] Space 1 Part 6 start send snapshot of commitLogId 3239187 commitLogTerm 45, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.646096 46888 NebulaSnapshotManager.cpp:67] Space 1 Part 9 start send snapshot of commitLogId 3231023 commitLogTerm 53, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.648857 46877 NebulaSnapshotManager.cpp:67] Space 1 Part 4 start send snapshot of commitLogId 3227556 commitLogTerm 48, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.650887 46879 NebulaSnapshotManager.cpp:67] Space 1 Part 5 start send snapshot of commitLogId 3277636 commitLogTerm 50, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.654443 46884 NebulaSnapshotManager.cpp:67] Space 1 Part 2 start send snapshot of commitLogId 3214951 commitLogTerm 61, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.659210 46888 NebulaSnapshotManager.cpp:67] Space 1 Part 6 start send snapshot of commitLogId 3239187 commitLogTerm 45, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.661824 46877 NebulaSnapshotManager.cpp:67] Space 1 Part 9 start send snapshot of commitLogId 3231023 commitLogTerm 53, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.662525 46879 NebulaSnapshotManager.cpp:67] Space 1 Part 4 start send snapshot of commitLogId 3227556 commitLogTerm 48, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.667660 46884 NebulaSnapshotManager.cpp:67] Space 1 Part 2 start send snapshot of commitLogId 3214951 commitLogTerm 61, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.674212 46888 NebulaSnapshotManager.cpp:67] Space 1 Part 6 start send snapshot of commitLogId 3239187 commitLogTerm 45, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.674412 46877 NebulaSnapshotManager.cpp:67] Space 1 Part 9 start send snapshot of commitLogId 3231023 commitLogTerm 53, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.676085 46879 NebulaSnapshotManager.cpp:67] Space 1 Part 4 start send snapshot of commitLogId 3227556 commitLogTerm 48, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.680836 46884 NebulaSnapshotManager.cpp:67] Space 1 Part 2 start send snapshot of commitLogId 3214951 commitLogTerm 61, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.688467 46888 NebulaSnapshotManager.cpp:67] Space 1 Part 9 start send snapshot of commitLogId 3231023 commitLogTerm 53, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.688835 46877 NebulaSnapshotManager.cpp:67] Space 1 Part 6 start send snapshot of commitLogId 3239187 commitLogTerm 45, rate limited to 10485760, batch size is 1048576
I20241005 23:15:43.688853 46879 NebulaSnapshotManager.cpp:67] Space 1 Part 4 start send snapshot of commitLogId 3227556 commitLogTerm 48, rate limited to 10485760, batch size is 1048576

请问这是什么情况？

2

出现日志是因为有机器日志进度落后leader太多了（比如机器挂了一段时间） 只能通过拉快照的方式。
你盘多大，有没有可能扩下盘？

3

盘40个g 但是这个平均每分钟打了几十万行日志 这正常吗？

4

而且里面有很多重复的id

5

感觉是也不太正常。之前的操作信息有没

6

应该没什么其他操作 只有国庆期间设备下电了
然后重新上电 就这样了

7

有大佬知道这种情况吗？

8

此话题已在最后回复的 30 天后被自动关闭。不再允许新回复。