storaged节点扩容时一直报wal写异常

yechen89 · 2021 年10 月 21 日 05:53

nebula 版本：2.5.0
部署方式：分布式
安装方式：Docker
是否为线上版本：N
硬件信息
- 磁盘（aws gp3 200G）
- CPU、内存信息：8C32G
问题的具体描述
storaged新节点扩容时一直报wal写异常
storage日志信息

E1021 12:41:21.286585    29 FileBasedWal.cpp:505] [Port: 9780, Space: 35, Part: 4] There is a gap in the log id. The last log id is 6146790, and the id being appended is 6146790
E1021 12:41:21.286648    29 FileBasedWal.cpp:593] [Port: 9780, Space: 35, Part: 4] Failed to append log for logId 6146790
E1021 12:41:21.575647    22 FileBasedWal.cpp:505] [Port: 9780, Space: 35, Part: 2] There is a gap in the log id. The last log id is 6146872, and the id being appended is 6146872
E1021 12:41:21.575723    22 FileBasedWal.cpp:593] [Port: 9780, Space: 35, Part: 2] Failed to append log for logId 6146872
E1021 12:41:26.811825    29 FileBasedWal.cpp:505] [Port: 9780, Space: 35, Part: 6] There is a gap in the log id. The last log id is 6147014, and the id being appended is 6147014
E1021 12:41:26.811866    29 FileBasedWal.cpp:593] [Port: 9780, Space: 35, Part: 6] Failed to append log for logId 6147014

critical27 · 2021 年10 月 21 日 09:05

麻烦贴下对应的INFO日志最近在梳理相关的问题目前只能先重启下对应的storage解决下问题

yechen89 · 2021 年10 月 21 日 11:25

INFO日志

E1021 12:41:19.752522    43 FileBasedWal.cpp:505] [Port: 9780, Space: 35, Part: 5] There is a gap in the log id. The last log id is 6146946, and the id being appended is 6146946
E1021 12:41:19.752591    43 FileBasedWal.cpp:593] [Port: 9780, Space: 35, Part: 5] Failed to append log for logId 6146946
E1021 12:41:20.735356    25 FileBasedWal.cpp:505] [Port: 9780, Space: 35, Part: 9] There is a gap in the log id. The last log id is 6147085, and the id being appended is 6147085
E1021 12:41:20.735419    25 FileBasedWal.cpp:593] [Port: 9780, Space: 35, Part: 9] Failed to append log for logId 6147085
I1021 12:41:21.131036    71 EventListener.h:30] Rocksdb compaction completed column family: default because of LevelL0FilesNum, status: OK, compacted 25 files into 20, base level is 0, output level is 1
E1021 12:41:21.223389    22 FileBasedWal.cpp:505] [Port: 9780, Space: 35, Part: 1] There is a gap in the log id. The last log id is 6146738, and the id being appended is 6146738
E1021 12:41:21.223501    22 FileBasedWal.cpp:593] [Port: 9780, Space: 35, Part: 1] Failed to append log for logId 6146738
I1021 12:41:21.269237    71 EventListener.h:112] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I1021 12:41:21.269358    71 EventListener.h:18] Rocksdb start compaction column family: default because of LevelMaxLevelSize, status: OK, compacted 1 files into 0, base level is 1, output level is 2
I1021 12:41:21.272891    71 EventListener.h:30] Rocksdb compaction completed column family: default because of LevelMaxLevelSize, status: OK, compacted 1 files into 1, base level is 1, output level is 2
I1021 12:41:21.273044    71 EventListener.h:18] Rocksdb start compaction column family: default because of LevelMaxLevelSize, status: OK, compacted 1 files into 0, base level is 1, output level is 2
I1021 12:41:21.276046    71 EventListener.h:30] Rocksdb compaction completed column family: default because of LevelMaxLevelSize, status: OK, compacted 1 files into 1, base level is 1, output level is 2
I1021 12:41:21.276317    71 EventListener.h:18] Rocksdb start compaction column family: default because of LevelMaxLevelSize, status: OK, compacted 1 files into 0, base level is 1, output level is 2
I1021 12:41:21.279381    71 EventListener.h:30] Rocksdb compaction completed column family: default because of LevelMaxLevelSize, status: OK, compacted 1 files into 1, base level is 1, output level is 2
I1021 12:41:21.279578    71 EventListener.h:18] Rocksdb start compaction column family: default because of LevelMaxLevelSize, status: OK, compacted 1 files into 0, base level is 1, output level is 2
E1021 12:41:21.286585    29 FileBasedWal.cpp:505] [Port: 9780, Space: 35, Part: 4] There is a gap in the log id. The last log id is 6146790, and the id being appended is 6146790
E1021 12:41:21.286648    29 FileBasedWal.cpp:593] [Port: 9780, Space: 35, Part: 4] Failed to append log for logId 6146790
I1021 12:41:21.286823    71 EventListener.h:30] Rocksdb compaction completed column family: default because of LevelMaxLevelSize, status: OK, compacted 1 files into 1, base level is 1, output level is 2
I1021 12:41:21.287125    71 EventListener.h:18] Rocksdb start compaction column family: default because of LevelMaxLevelSize, status: OK, compacted 2 files into 0, base level is 1, output level is 2
I1021 12:41:21.287176    71 CompactionFilter.h:66] Do default minor compaction!
E1021 12:41:21.575647    22 FileBasedWal.cpp:505] [Port: 9780, Space: 35, Part: 2] There is a gap in the log id. The last log id is 6146872, and the id being appended is 6146872
E1021 12:41:21.575723    22 FileBasedWal.cpp:593] [Port: 9780, Space: 35, Part: 2] Failed to append log for logId 6146872
I1021 12:41:21.765717    73 EventListener.h:112] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I1021 12:41:21.786036    29 EventListener.h:112] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I1021 12:41:22.572880    73 EventListener.h:112] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I1021 12:41:22.596000    22 EventListener.h:112] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I1021 12:41:23.447573    73 EventListener.h:112] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I1021 12:41:23.473985    29 EventListener.h:112] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I1021 12:41:24.281247    43 EventListener.h:112] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I1021 12:41:25.000108    71 EventListener.h:30] Rocksdb compaction completed column family: default because of LevelMaxLevelSize, status: OK, compacted 2 files into 2, base level is 1, output level is 2
I1021 12:41:25.000413    71 EventListener.h:18] Rocksdb start compaction column family: default because of LevelMaxLevelSize, status: OK, compacted 2 files into 0, base level is 1, output level is 2
I1021 12:41:25.000473    71 CompactionFilter.h:66] Do default minor compaction!
I1021 12:41:25.024926    70 EventListener.h:30] Rocksdb compaction completed column family: default because of LevelMaxLevelSize, status: OK, compacted 4 files into 3, base level is 2, output level is 3
I1021 12:41:25.040447    73 EventListener.h:112] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I1021 12:41:25.061655    70 EventListener.h:18] Rocksdb start compaction column family: default because of LevelMaxLevelSize, status: OK, compacted 3 files into 0, base level is 1, output level is 2
I1021 12:41:25.061728    70 CompactionFilter.h:66] Do default minor compaction!
I1021 12:41:25.225937    72 EventListener.h:30] Rocksdb compaction completed column family: default because of LevelMaxLevelSize, status: OK, compacted 6 files into 3, base level is 2, output level is 3
I1021 12:41:25.468792    72 EventListener.h:18] Rocksdb start compaction column family: default because of LevelL0FilesNum, status: OK, compacted 4 files into 0, base level is 0, output level is 0
I1021 12:41:25.468849    72 CompactionFilter.h:66] Do default minor compaction!

system · 2021 年11 月 20 日 11:26

此话题已在最后回复的 30 天后被自动关闭。不再允许新回复。