storaged服务内存占用很高,然后退出,集群不可用

如何处理compaction占用内存的问题。
按照下面做了配置,也无效。不能将compaction问题解决。

这个问题,大佬们有知道怎么处理吗

 Rocksdb start compaction column family: default because of LevelL0FilesNum, status: OK, compacted 4 files into 0, base level is 0, output level is 1
I20240522 11:47:30.555838 13495 CompactionFilter.h:81] Do full/manual compaction!
I20240522 11:47:32.151760 13363 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:47:32.777109 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:47:34.702911 13359 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:47:37.315721 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:47:40.910625 13359 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:47:41.835584 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:47:42.705633 13357 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:47:45.302251 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:47:47.752426 13362 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:47:48.840039 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:47:49.587426 13357 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:47:52.796077 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:47:56.172631 13359 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:47:56.235337 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:47:57.266538 13357 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:47:59.873075 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:48:00.073182 13495 EventListener.h:35] Rocksdb compaction completed column family: default because of LevelL0FilesNum, status: OK, compacted 4 files into 1, base level is 0, output level is 1
I20240522 11:48:00.073324 13495 EventListener.h:21] Rocksdb start compaction column family: default because of LevelL0FilesNum, status: OK, compacted 12 files into 0, base level is 0, output level is 1
I20240522 11:48:00.073359 13495 CompactionFilter.h:81] Do full/manual compaction!
I20240522 11:48:02.815263 13362 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:48:03.243304 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:48:04.348729 13360 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:48:07.060519 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:48:10.114420 13354 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:48:10.233290 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:48:11.323235 13363 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:48:13.867271 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:48:17.194983 13357 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:48:17.248914 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:48:18.377730 13359 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:48:56.020211 13495 EventListener.h:35] Rocksdb compaction completed column family: default because of LevelL0FilesNum, status: OK, compacted 12 files into 1, base level is 0, output level is 1
I20240522 11:48:56.217064 13495 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:48:56.217272 13495 EventListener.h:21] Rocksdb start compaction column family: default because of LevelL0FilesNum, status: OK, compacted 12 files into 0, base level is 0, output level is 1
I20240522 11:48:56.217312 13495 CompactionFilter.h:81] Do full/manual compaction!
I20240522 11:49:06.042227 13363 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:49:06.342064 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:49:07.478976 13362 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:49:09.883534 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:49:13.279233 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:50:18.350471 13495 EventListener.h:35] Rocksdb compaction completed column family: default because of LevelL0FilesNum, status: OK, compacted 12 files into 1, base level is 0, output level is 1
I20240522 11:50:18.685401 13495 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:50:18.685576 13495 EventListener.h:21] Rocksdb start compaction column family: default because of LevelL0FilesNum, status: OK, compacted 12 files into 0, base level is 0, output level is 1
I20240522 11:50:18.685622 13495 CompactionFilter.h:81] Do full/manual compaction!
I20240522 11:50:24.880283 13363 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:50:25.034950 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:50:26.239526 13363 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:50:28.971491 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:50:35.707932 13360 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:52:09.605711 13495 EventListener.h:35] Rocksdb compaction completed column family: default because of LevelL0FilesNum, status: OK, compacted 12 files into 1, base level is 0, output level is 1
I20240522 11:52:10.025617 13495 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:52:10.025676 13495 EventListener.h:21] Rocksdb start compaction column family: default because of LevelMaxLevelSize, status: OK, compacted 1 files into 0, base level is 1, output level is 2
I20240522 11:52:10.026181 13495 EventListener.h:35] Rocksdb compaction completed column family: default because of LevelMaxLevelSize, status: OK, compacted 1 files into 1, base level is 1, output level is 2
I20240522 11:52:10.026298 13495 EventListener.h:21] Rocksdb start compaction column family: default because of LevelL0FilesNum, status: OK, compacted 11 files into 0, base level is 0, output level is 1
I20240522 11:52:10.026330 13495 CompactionFilter.h:85] Do automatic or periodic compaction!
I20240522 11:52:14.740123 13363 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:52:14.953940 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:52:16.076925 13358 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:52:19.099285 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:52:23.157472 13359 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:52:23.233814 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:52:24.425187 13360 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:52:27.015311 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:52:30.422747 13355 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:52:51.726053 13495 EventListener.h:35] Rocksdb compaction completed column family: default because of LevelL0FilesNum, status: OK, compacted 11 files into 1, base level is 0, output level is 1
I20240522 11:52:51.858341 13495 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:52:51.858489 13495 EventListener.h:21] Rocksdb start compaction column family: default because of LevelL0FilesNum, status: OK, compacted 11 files into 0, base level is 0, output level is 1
I20240522 11:52:51.858525 13495 CompactionFilter.h:85] Do automatic or periodic compaction!
I20240522 11:52:57.192984 13363 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:52:57.212968 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:52:58.814530 13356 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:53:01.163130 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:53:07.564675 13357 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:53:07.767256 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:53:09.051259 13355 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:53:11.898145 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:53:13.586043 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:54:00.618898 13495 EventListener.h:35] Rocksdb compaction completed column family: default because of LevelL0FilesNum, status: OK, compacted 11 files into 1, base level is 0, output level is 1
I20240522 11:54:00.852922 13495 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:54:00.853080 13495 EventListener.h:21] Rocksdb start compaction column family: default because of LevelL0FilesNum, status: OK, compacted 13 files into 0, base level is 0, output level is 1
I20240522 11:54:00.853117 13495 CompactionFilter.h:85] Do automatic or periodic compaction!
I20240522 11:54:07.387518 13353 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:54:07.549255 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:54:08.914732 13361 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:54:11.359279 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:54:16.047145 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:55:39.788129 13495 EventListener.h:35] Rocksdb compaction completed column family: default because of LevelL0FilesNum, status: OK, compacted 13 files into 1, base level is 0, output level is 1
I20240522 11:55:40.164017 13495 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:55:40.164214 13495 EventListener.h:21] Rocksdb start compaction column family: default because of LevelL0FilesNum, status: OK, compacted 12 files into 0, base level is 0, output level is 1
I20240522 11:55:40.164254 13495 CompactionFilter.h:85] Do automatic or periodic compaction!
I20240522 11:55:45.046943 13361 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:55:45.483469 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:55:46.651008 13363 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:55:49.504902 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:55:57.913717 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
W20240522 11:56:37.720750 13557 MemoryUtils.cpp:133] Memory usage has hit the high watermark of system, available: 2.8861e+09 vs. total: 17179869184 in bytes.
I20240522 11:57:47.705003 13495 EventListener.h:35] Rocksdb compaction completed column family: default because of LevelL0FilesNum, status: OK, compacted 12 files into 1, base level is 0, output level is 1
I20240522 11:57:48.079031 13495 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:57:48.079133 13495 EventListener.h:21] Rocksdb start compaction column family: default because of LevelMaxLevelSize, status: OK, compacted 2 files into 0, base level is 1, output level is 2
I20240522 11:57:48.079167 13495 CompactionFilter.h:85] Do automatic or periodic compaction!
I20240522 11:57:56.226176 13359 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:57:56.394716 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:57:57.708518 13355 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
I20240522 11:58:00.209956 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Normal, previous condition: Delayed
I20240522 11:58:03.634306 13496 EventListener.h:158] Stall conditions changed column family: default, current condition: Delayed, previous condition: Normal
W20240522 11:58:17.708501 13557 MemoryUtils.cpp:133] Memory usage has hit the high watermark of system, available: 1.19461e+09 vs. total: 17179869184 in bytes.

这个方案可以

2 个赞

此话题已在最后回复的 7 天后被自动关闭。不再允许新回复。