寻求解决方案:生产环境,从不同graphd节点查询,集群有一半节点leader count均为0,目前集群可正常工作,需要尽快处理。

nebula 版本:v1.2.0

部署方式(分布式 ):

是否为线上版本:Y

硬件信息

服务器:30台
磁盘: 8块SSD硬盘。
CPU: 24core,Intel® Xeon® CPU E5-2630 v2 @ 2.60GHz
内存:128G

问题的具体描述:
寻求解决方案:生产环境,27个graphd节点,目前发现从不同graphd节点登录集群,,有一半节点show hosts后发现leader count全为0. 查看最近历史日志未见明显报错,现在集群还可正常工作,未发现其他异常。之前提交贴子问题复现(可参考之前故障帖子https://discuss.nebula-graph.com.cn/t/topic/3844)。
storaged日志打开级别及其他部分参数如下:

(root@nebula) [(none)]> show configs;
=============================================================================================================================================================================================================================
| module  | name                              | type   | mode    | value                                                                                                                                                    |
=============================================================================================================================================================================================================================
| GRAPH   | v                                 | INT64  | MUTABLE | 0                                                                                                                                                        |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| GRAPH   | minloglevel                       | INT64  | MUTABLE | 1                                                                                                                                                        |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| GRAPH   | slow_op_threshhold_ms             | INT64  | MUTABLE | 50                                                                                                                                                       |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| GRAPH   | heartbeat_interval_secs           | INT64  | MUTABLE | 10                                                                                                                                                       |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| GRAPH   | meta_client_retry_times           | INT64  | MUTABLE | 3                                                                                                                                                        |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| STORAGE | v                                 | INT64  | MUTABLE | 0                                                                                                                                                        |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| STORAGE | wal_ttl                           | INT64  | MUTABLE | 14400                                                                                                                                                    |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| STORAGE | minloglevel                       | INT64  | MUTABLE | 0                                                                                                                                                        |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| STORAGE | rocksdb_db_options                | NESTED | MUTABLE | {
  "max_background_jobs": "4",
  "max_subcompactions": "4"
}                                                                                            |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| STORAGE | enable_multi_versions             | BOOL   | MUTABLE | False                                                                                                                                                    |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| STORAGE | slow_op_threshhold_ms             | INT64  | MUTABLE | 50                                                                                                                                                       |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| STORAGE | clean_wal_interval_secs           | INT64  | MUTABLE | 600                                                                                                                                                      |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| STORAGE | heartbeat_interval_secs           | INT64  | MUTABLE | 10                                                                                                                                                       |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| STORAGE | meta_client_retry_times           | INT64  | MUTABLE | 3                                                                                                                                                        |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| STORAGE | enable_reservoir_sampling         | BOOL   | MUTABLE | False                                                                                                                                                    |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| STORAGE | custom_filter_interval_secs       | INT64  | MUTABLE | 86400                                                                                                                                                    |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| STORAGE | max_edge_returned_per_vertex      | INT64  | MUTABLE | 100000                                                                                                                                                   |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| STORAGE | rocksdb_column_family_options     | NESTED | MUTABLE | {
  "max_write_buffer_number": "4",
  "max_bytes_for_level_base": "268435456",
  "write_buffer_size": "67108864",
  "disable_auto_compactions": "true"
} |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| STORAGE | rocksdb_block_based_table_options | NESTED | MUTABLE | {
  "block_size": "8192"
}                       

其他相关的 日志信息
storaged log: 大量类似的log如下:

I0615 07:56:09.665710 10494 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 21 files into 21, base level is 1, output level is 1
I0615 07:56:09.774513 10493 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 24 files into 24, base level is 1, output level is 1
I0615 07:56:10.074739 10492 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 24 files into 0, base level is 1, output level is 1
I0615 07:56:10.074988 10492 CompactionFilter.h:57] Do full/manual compaction!
I0615 07:56:10.075052 22874 CompactionFilter.h:57] Do full/manual compaction!
I0615 07:56:10.075063 22875 CompactionFilter.h:57] Do full/manual compaction!
I0615 07:56:10.075084 22873 CompactionFilter.h:57] Do full/manual compaction!
I0615 07:56:38.445175 10492 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 24 files into 24, base level is 1, output level is 1
I0615 07:56:38.727484 10494 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 22 files into 0, base level is 1, output level is 1
I0615 07:56:38.727707 10494 CompactionFilter.h:57] Do full/manual compaction!
I0615 07:56:38.727752 22942 CompactionFilter.h:57] Do full/manual compaction!
I0615 07:56:38.727852 22943 CompactionFilter.h:57] Do full/manual compaction!
I0615 07:56:38.727852 22944 CompactionFilter.h:57] Do full/manual compaction!
I0615 07:57:04.338635 10494 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 22 files into 22, base level is 1, output level is 1
I0615 07:57:04.608956 11054 NebulaStore.cpp:711] Space 1 compaction done.
I0615 07:57:25.302142 10435 InMemoryLogBuffer.h:27] [Port: 44501, Space: 1, Part: 563] InMemoryLogBuffer dtor, firstLogId 286406350
I0615 07:57:25.372519 10435 InMemoryLogBuffer.h:23] [Port: 44501, Space: 1, Part: 563] InMemoryLogBuffer ctor, firstLogId 286531717
I0615 08:01:23.597173 10458 InMemoryLogBuffer.h:27] [Port: 44501, Space: 1, Part: 323] InMemoryLogBuffer dtor, firstLogId 286908811
I0615 08:01:23.639088 10458 InMemoryLogBuffer.h:23] [Port: 44501, Space: 1, Part: 323] InMemoryLogBuffer ctor, firstLogId 287035492
I0615 08:02:36.305224 10451 InMemoryLogBuffer.h:27] [Port: 44501, Space: 154, Part: 593] InMemoryLogBuffer dtor, firstLogId 40854652
I0615 08:02:36.326629 10451 InMemoryLogBuffer.h:23] [Port: 44501, Space: 154, Part: 593] InMemoryLogBuffer ctor, firstLogId 40910823
I0615 08:06:11.981632 10462 InMemoryLogBuffer.h:27] [Port: 44501, Space: 1, Part: 384] InMemoryLogBuffer dtor, firstLogId 286760255
I0615 08:06:12.029466 10462 InMemoryLogBuffer.h:23] [Port: 44501, Space: 1, Part: 384] InMemoryLogBuffer ctor, firstLogId 286886585
I0615 08:10:31.804358 10437 InMemoryLogBuffer.h:27] [Port: 44501, Space: 1, Part: 264] InMemoryLogBuffer dtor, firstLogId 287705641
I0615 08:10:31.836766 10437 InMemoryLogBuffer.h:23] [Port: 44501, Space: 1, Part: 264] InMemoryLogBuffer ctor, firstLogId 287831910
I0615 08:10:46.645437 10440 InMemoryLogBuffer.h:27] [Port: 44501, Space: 1, Part: 414] InMemoryLogBuffer dtor, firstLogId 286772543
I0615 08:10:46.669179 10440 InMemoryLogBuffer.h:23] [Port: 44501, Space: 1, Part: 414] InMemoryLogBuffer ctor, firstLogId 286897705
I0615 08:18:49.805006 10458 InMemoryLogBuffer.h:27] [Port: 44501, Space: 1, Part: 533] InMemoryLogBuffer dtor, firstLogId 286286668
I0615 08:18:49.834666 10458 InMemoryLogBuffer.h:23] [Port: 44501, Space: 1, Part: 533] InMemoryLogBuffer ctor, firstLogId 286413249
I0615 08:21:33.552639 10437 InMemoryLogBuffer.h:27] [Port: 44501, Space: 1, Part: 1164] InMemoryLogBuffer dtor, firstLogId 285784172
I0615 08:21:33.595099 10437 InMemoryLogBuffer.h:23] [Port: 44501, Space: 1, Part: 1164] InMemoryLogBuffer ctor, firstLogId 285908989
I0615 08:23:20.623356 10489 FileBasedWal.cpp:760] [Port: 44501, Space: 154, Part: 443] Clean wals number 1
I0615 08:23:20.623881 10489 FileBasedWal.cpp:760] [Port: 44501, Space: 154, Part: 54] Clean wals number 1
I0615 08:26:02.856120 10458 InMemoryLogBuffer.h:27] [Port: 44501, Space: 1, Part: 804] InMemoryLogBuffer dtor, firstLogId 287309782
I0615 08:26:02.881983 10458 InMemoryLogBuffer.h:23] [Port: 44501, Space: 1, Part: 804] InMemoryLogBuffer ctor, firstLogId 287420611
I0615 08:29:52.147449 10437 InMemoryLogBuffer.h:27] [Port: 44501, Space: 154, Part: 892] InMemoryLogBuffer dtor, firstLogId 40764022
I0615 08:29:52.174314 10437 InMemoryLogBuffer.h:23] [Port: 44501, Space: 154, Part: 892] InMemoryLogBuffer ctor, firstLogId 40820814
I0615 08:31:04.247539 10439 InMemoryLogBuffer.h:27] [Port: 44501, Space: 1, Part: 413] InMemoryLogBuffer dtor, firstLogId 290588703
I0615 08:31:04.280656 10439 InMemoryLogBuffer.h:23] [Port: 44501, Space: 1, Part: 413] InMemoryLogBuffer ctor, firstLogId 290711398
I0615 08:31:09.703428 10439 InMemoryLogBuffer.h:27] [Port: 44501, Space: 154, Part: 412] InMemoryLogBuffer dtor, firstLogId 40882291
I0615 08:31:09.723649 10439 InMemoryLogBuffer.h:23] [Port: 44501, Space: 154, Part: 412] InMemoryLogBuffer ctor, firstLogId 40938341
I0615 08:33:25.251148 10448 InMemoryLogBuffer.h:27] [Port: 44501, Space: 1, Part: 623] InMemoryLogBuffer dtor, firstLogId 288711932
I0615 08:33:25.277424 10448 InMemoryLogBuffer.h:23] [Port: 44501, Space: 1, Part: 623] InMemoryLogBuffer ctor, firstLogId 288834126
I0615 08:37:42.336659 10444 InMemoryLogBuffer.h:27] [Port: 44501, Space: 154, Part: 743] InMemoryLogBuffer dtor, firstLogId 40851515
I0615 08:37:42.352147 10444 InMemoryLogBuffer.h:23] [Port: 44501, Space: 154, Part: 743] InMemoryLogBuffer ctor, firstLogId 40908760
I0615 08:40:35.902006 10460 InMemoryLogBuffer.h:27] [Port: 44501, Space: 154, Part: 52] InMemoryLogBuffer dtor, firstLogId 40907242
I0615 08:40:35.913347 10460 InMemoryLogBuffer.h:23] [Port: 44501, Space: 154, Part: 52] InMemoryLogBuffer ctor, firstLogId 40964447
I0615 08:45:35.332094 10457 InMemoryLogBuffer.h:27] [Port: 44501, Space: 1, Part: 82] InMemoryLogBuffer dtor, firstLogId 287578152
I0615 08:45:35.368476 10457 InMemoryLogBuffer.h:23] [Port: 44501, Space: 1, Part: 82] InMemoryLogBuffer ctor, firstLogId 287701930
I0615 08:46:35.484318 11054 StorageHttpAdminHandler.cpp:55] do compact at space=prod_v1
I0615 08:46:35.484592 11054 NebulaStore.cpp:697] Space 154 start compaction.
I0615 08:46:35.667284 10493 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 131 files into 0, base level is 0, output level is 1
I0615 08:46:35.668447 10493 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:46:35.668591 31483 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:46:35.669028 31481 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:46:35.671063 31482 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:46:35.761973 10492 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 131 files into 0, base level is 0, output level is 1
I0615 08:46:35.762558 10492 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:46:35.763984 31486 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:46:35.763924 31484 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:46:35.763659 31485 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:46:35.906479 10494 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 131 files into 0, base level is 0, output level is 1
I0615 08:46:35.907110 10494 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:46:35.907276 31489 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:46:35.907439 31488 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:46:35.908329 31487 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:49:03.581486 10439 InMemoryLogBuffer.h:27] [Port: 44501, Space: 154, Part: 864] InMemoryLogBuffer dtor, firstLogId 40562346
I0615 08:49:03.598278 10439 InMemoryLogBuffer.h:23] [Port: 44501, Space: 154, Part: 864] InMemoryLogBuffer ctor, firstLogId 40619161
I0615 08:50:56.708673 10492 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 131 files into 129, base level is 0, output level is 1
I0615 08:50:57.366384 10492 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 24 files into 0, base level is 1, output level is 1
I0615 08:50:57.366670 10492 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:50:57.366722 32289 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:50:57.366719 32291 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:50:57.366734 32290 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:50:57.768632 10494 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 131 files into 129, base level is 0, output level is 1
I0615 08:50:57.969389 10493 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 131 files into 129, base level is 0, output level is 1
I0615 08:50:58.485078 10494 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 131 files into 0, base level is 0, output level is 1
I0615 08:50:58.485425 10494 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:50:58.485479 32292 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:50:58.485507 32294 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:50:58.485589 32293 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:50:58.693581 10493 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 131 files into 0, base level is 0, output level is 1
I0615 08:50:58.694039 32295 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:50:58.694120 10493 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:50:58.694170 32297 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:50:58.694126 32296 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:51:50.339958 10492 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 24 files into 24, base level is 1, output level is 1
I0615 08:51:50.677843 10492 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 131 files into 0, base level is 0, output level is 1
I0615 08:51:50.678220 10492 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:51:50.678300 32393 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:51:50.678323 32391 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:51:50.678369 32392 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:52:04.723979 10448 InMemoryLogBuffer.h:27] [Port: 44501, Space: 154, Part: 533] InMemoryLogBuffer dtor, firstLogId 40878529
I0615 08:52:04.739688 10448 InMemoryLogBuffer.h:23] [Port: 44501, Space: 154, Part: 533] InMemoryLogBuffer ctor, firstLogId 40936045
I0615 08:53:18.010444 10450 InMemoryLogBuffer.h:27] [Port: 44501, Space: 154, Part: 683] InMemoryLogBuffer dtor, firstLogId 40780202
I0615 08:53:18.025288 10450 InMemoryLogBuffer.h:23] [Port: 44501, Space: 154, Part: 683] InMemoryLogBuffer ctor, firstLogId 40838231
I0615 08:53:20.640722 10488 FileBasedWal.cpp:760] [Port: 44501, Space: 154, Part: 262] Clean wals number 1
I0615 08:53:39.716028 10457 InMemoryLogBuffer.h:27] [Port: 44501, Space: 1, Part: 1192] InMemoryLogBuffer dtor, firstLogId 292860750
I0615 08:53:39.778265 10457 InMemoryLogBuffer.h:23] [Port: 44501, Space: 1, Part: 1192] InMemoryLogBuffer ctor, firstLogId 292983496
I0615 08:53:39.778632 10457 SlowOpTracker.h:33] [Port: 44501, Space: 1, Part: 1192] , total time:63ms, Write WAL, total 2
I0615 08:55:20.016427 10494 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 131 files into 129, base level is 0, output level is 1
I0615 08:55:20.529700 10493 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 131 files into 129, base level is 0, output level is 1
I0615 08:55:20.735741 10494 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 131 files into 0, base level is 0, output level is 1
I0615 08:55:20.736289 10494 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:55:20.736403   378 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:55:20.736459   380 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:55:20.736408   379 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:55:21.224133 10493 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 131 files into 0, base level is 0, output level is 1
I0615 08:55:21.224839   387 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:55:21.224890   388 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:55:21.224881 10493 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:55:21.224980   389 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:56:08.393524 10492 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 131 files into 129, base level is 0, output level is 1
I0615 08:56:09.232105 10492 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 24 files into 0, base level is 1, output level is 1
I0615 08:56:09.232543 10492 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:56:09.232636   760 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:56:09.232650   759 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:56:09.232672   761 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:57:04.061316 10492 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 24 files into 24, base level is 1, output level is 1
I0615 08:57:04.384204 10492 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 24 files into 0, base level is 1, output level is 1
I0615 08:57:04.384514 10492 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:57:04.384589   869 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:57:04.384609   868 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:57:04.384619   867 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:57:58.989131 10492 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 24 files into 24, base level is 1, output level is 1
I0615 08:57:59.332940 10492 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 25 files into 0, base level is 1, output level is 1
I0615 08:57:59.333204 10492 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:57:59.333272  1047 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:57:59.333284  1049 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:57:59.333273  1048 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:58:02.632593 10458 InMemoryLogBuffer.h:27] [Port: 44501, Space: 1, Part: 622] InMemoryLogBuffer dtor, firstLogId 285474629
I0615 08:58:02.686664 10458 InMemoryLogBuffer.h:23] [Port: 44501, Space: 1, Part: 622] InMemoryLogBuffer ctor, firstLogId 285599997
I0615 08:58:54.850769 10492 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 25 files into 25, base level is 1, output level is 1
I0615 08:58:55.178196 10492 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 24 files into 0, base level is 1, output level is 1
I0615 08:58:55.178501 10492 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:58:55.178550  1191 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:58:55.178565  1193 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:58:55.178570  1192 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:59:31.642540 10494 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 131 files into 129, base level is 0, output level is 1
I0615 08:59:32.424746 10494 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 24 files into 0, base level is 1, output level is 1
I0615 08:59:32.425253 10494 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:59:32.425371  1284 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:59:32.425384  1285 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:59:32.425606  1286 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:59:33.082051 10493 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 131 files into 129, base level is 0, output level is 1
I0615 08:59:33.881603 10493 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 24 files into 0, base level is 1, output level is 1
I0615 08:59:33.881927 10493 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:59:33.881973  1287 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:59:33.881991  1288 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:59:33.882067  1289 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:59:42.258080 10492 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 24 files into 24, base level is 1, output level is 1
I0615 08:59:42.559800 10492 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 25 files into 0, base level is 1, output level is 1
I0615 08:59:42.560101 10492 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:59:42.560154  1301 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:59:42.560137  1300 CompactionFilter.h:57] Do full/manual compaction!
I0615 08:59:42.560148  1302 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:00:04.541036 10448 InMemoryLogBuffer.h:27] [Port: 44501, Space: 1, Part: 832] InMemoryLogBuffer dtor, firstLogId 286009004
I0615 09:00:04.584981 10448 InMemoryLogBuffer.h:23] [Port: 44501, Space: 1, Part: 832] InMemoryLogBuffer ctor, firstLogId 286133365
I0615 09:00:21.590850 10494 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 24 files into 24, base level is 1, output level is 1
I0615 09:00:21.879097 10494 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 25 files into 0, base level is 1, output level is 1
I0615 09:00:21.879343 10494 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:00:21.879420  1463 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:00:21.879433  1464 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:00:21.879451  1465 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:00:23.093670 10493 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 24 files into 24, base level is 1, output level is 1
I0615 09:00:23.378649 10493 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 25 files into 0, base level is 1, output level is 1
I0615 09:00:23.378911 10493 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:00:23.378959  1469 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:00:23.378988  1471 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:00:23.378993  1470 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:00:32.532649 10492 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 25 files into 25, base level is 1, output level is 1
I0615 09:00:32.839020 10492 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 24 files into 0, base level is 1, output level is 1
I0615 09:00:32.839298 10492 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:00:32.839367  1570 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:00:32.839371  1569 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:00:32.839371  1568 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:01:11.449059 10494 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 25 files into 25, base level is 1, output level is 1
I0615 09:01:11.735059 10494 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 24 files into 0, base level is 1, output level is 1
I0615 09:01:11.735307 10494 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:01:11.735378  1863 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:01:11.735394  1864 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:01:11.735406  1862 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:01:12.629180 10493 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 25 files into 25, base level is 1, output level is 1
I0615 09:01:12.927486 10493 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 25 files into 0, base level is 1, output level is 1
I0615 09:01:12.927800 10493 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:01:12.927824  1869 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:01:12.927836  1871 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:01:12.927829  1870 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:01:22.649324 10492 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 24 files into 24, base level is 1, output level is 1
I0615 09:01:22.966346 10492 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 25 files into 0, base level is 1, output level is 1
I0615 09:01:22.966667 10492 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:01:22.966699  1883 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:01:22.966706  1885 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:01:22.966732  1884 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:02:00.636485 10494 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 24 files into 24, base level is 1, output level is 1
I0615 09:02:00.971241 10494 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 25 files into 0, base level is 1, output level is 1
I0615 09:02:00.971511 10494 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:02:00.971629  1914 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:02:00.971630  1912 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:02:00.971653  1913 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:02:03.811925 10493 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 25 files into 25, base level is 1, output level is 1
I0615 09:02:04.100033 10493 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 25 files into 0, base level is 1, output level is 1
I0615 09:02:04.100278 10493 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:02:04.100337  1956 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:02:04.100353  1958 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:02:04.100353  1957 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:02:11.623625 10492 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 25 files into 25, base level is 1, output level is 1
I0615 09:02:12.009634 10492 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 25 files into 0, base level is 1, output level is 1
I0615 09:02:12.009941 10492 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:02:12.009985  2012 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:02:12.009989  2014 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:02:12.010004  2013 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:02:51.549801 10494 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 25 files into 25, base level is 1, output level is 1
I0615 09:02:51.849128 10494 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 24 files into 0, base level is 1, output level is 1
I0615 09:02:51.849398 10494 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:02:51.849450  2051 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:02:51.849470  2052 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:02:51.849474  2053 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:02:52.977278 10493 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 25 files into 25, base level is 1, output level is 1
I0615 09:02:53.271276 10493 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 25 files into 0, base level is 1, output level is 1
I0615 09:02:53.271517 10493 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:02:53.271574  2057 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:02:53.271595  2058 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:02:53.271602  2059 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:03:00.795794 10492 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 25 files into 25, base level is 1, output level is 1
I0615 09:03:01.144528 10492 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 25 files into 0, base level is 1, output level is 1
I0615 09:03:01.144850 10492 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:03:01.144909  2071 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:03:01.144932  2072 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:03:01.145035  2070 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:03:40.795811 10494 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 24 files into 24, base level is 1, output level is 1
I0615 09:03:41.080008 10494 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 25 files into 0, base level is 1, output level is 1
I0615 09:03:41.080308 10494 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:03:41.080404  2160 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:03:41.080430  2158 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:03:41.080444  2159 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:03:43.841426 10493 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 25 files into 25, base level is 1, output level is 1
I0615 09:03:44.123925 10493 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 25 files into 0, base level is 1, output level is 1
I0615 09:03:44.124168 10493 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:03:44.124208  2164 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:03:44.124218  2165 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:03:44.124248  2166 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:03:49.875860 10492 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 25 files into 25, base level is 1, output level is 1
I0615 09:03:50.180970 10492 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 25 files into 0, base level is 1, output level is 1
I0615 09:03:50.181412 10492 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:03:50.181495  2172 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:03:50.181504  2173 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:03:50.181504  2174 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:04:30.570276 10494 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 25 files into 25, base level is 1, output level is 1
I0615 09:04:30.883929 10494 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 24 files into 0, base level is 1, output level is 1
I0615 09:04:30.884217 10494 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:04:30.884274  2284 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:04:30.884284  2282 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:04:30.884299  2283 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:04:34.446360 10493 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 25 files into 25, base level is 1, output level is 1
I0615 09:04:34.806552 10493 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 24 files into 0, base level is 1, output level is 1
I0615 09:04:34.806921 10493 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:04:34.807030  2294 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:04:34.806999  2295 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:04:34.806999  2296 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:04:39.232472 10492 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 25 files into 25, base level is 1, output level is 1
I0615 09:04:39.532572 10492 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 25 files into 0, base level is 1, output level is 1
I0615 09:04:39.532838 10492 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:04:39.532903  2301 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:04:39.532904  2300 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:04:39.532930  2302 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:05:20.886731 10494 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 24 files into 24, base level is 1, output level is 1
I0615 09:05:21.184911 10494 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 25 files into 0, base level is 1, output level is 1
I0615 09:05:21.185158 10494 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:05:21.185247  2430 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:05:21.185288  2429 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:05:21.185379  2431 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:05:24.219347 10493 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 24 files into 24, base level is 1, output level is 1
I0615 09:05:24.561851 10493 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 25 files into 0, base level is 1, output level is 1
I0615 09:05:24.562178 10493 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:05:24.562261  2462 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:05:24.562292  2463 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:05:24.562263  2461 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:05:28.989837 10492 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 25 files into 25, base level is 1, output level is 1
I0615 09:05:29.384451 10492 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 24 files into 0, base level is 1, output level is 1
I0615 09:05:29.384811 10492 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:05:29.384812  2502 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:05:29.384835  2503 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:05:29.384867  2504 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:06:11.096186 10494 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 25 files into 25, base level is 1, output level is 1
I0615 09:06:11.392951 10494 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 24 files into 0, base level is 1, output level is 1
I0615 09:06:11.393199 10494 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:06:11.393256  2828 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:06:11.393268  2826 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:06:11.393287  2827 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:06:13.474530 10493 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 25 files into 25, base level is 1, output level is 1
I0615 09:06:13.755354 10493 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 24 files into 0, base level is 1, output level is 1
I0615 09:06:13.755658 10493 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:06:13.755715  2836 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:06:13.755715  2835 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:06:13.755748  2834 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:06:18.573688 10492 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 24 files into 24, base level is 1, output level is 1
I0615 09:06:18.862663 10492 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 25 files into 0, base level is 1, output level is 1
I0615 09:06:18.862964 10492 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:06:18.863003  2841 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:06:18.863008  2840 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:06:18.863025  2842 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:07:01.112530 10494 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 24 files into 24, base level is 1, output level is 1
I0615 09:07:01.407899 10494 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 25 files into 0, base level is 1, output level is 1
I0615 09:07:01.408145 10494 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:07:01.408196  2884 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:07:01.408206  2886 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:07:01.408208  2885 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:07:04.107234 10493 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 24 files into 24, base level is 1, output level is 1
I0615 09:07:04.401783 10493 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 6 files into 0, base level is 1, output level is 1
I0615 09:07:04.402029 10493 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:07:04.402082  2901 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:07:04.402083  2900 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:07:07.464378 10492 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 25 files into 25, base level is 1, output level is 1
I0615 09:07:07.761152 10492 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 24 files into 0, base level is 1, output level is 1
I0615 09:07:07.761433 10492 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:07:07.761490  2906 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:07:07.761502  2905 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:07:07.761512  2907 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:07:18.173573 10493 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 6 files into 6, base level is 1, output level is 1
I0615 09:07:18.248889 10493 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 24 files into 0, base level is 1, output level is 1
I0615 09:07:18.249130 10493 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:07:18.249217  2961 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:07:18.249253  2962 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:07:18.249706  2960 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:07:50.219064 10494 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 25 files into 25, base level is 1, output level is 1
I0615 09:07:50.536432 10494 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 25 files into 0, base level is 1, output level is 1
I0615 09:07:50.536805 10494 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:07:50.536883  2992 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:07:50.536896  2990 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:07:50.536886  2991 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:07:57.777242 10492 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 24 files into 24, base level is 1, output level is 1
I0615 09:07:58.049168 10492 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 25 files into 0, base level is 1, output level is 1
I0615 09:07:58.049434 10492 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:07:58.049495  3004 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:07:58.049506  3002 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:07:58.049521  3003 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:08:07.197494 10493 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 24 files into 24, base level is 1, output level is 1
I0615 09:08:07.570525 10493 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 25 files into 0, base level is 1, output level is 1
I0615 09:08:07.570804 10493 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:08:07.570842  3048 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:08:07.570858  3050 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:08:07.570991  3049 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:08:19.161962 10461 InMemoryLogBuffer.h:27] [Port: 44501, Space: 1, Part: 1193] InMemoryLogBuffer dtor, firstLogId 288181126
I0615 09:08:19.200693 10461 InMemoryLogBuffer.h:23] [Port: 44501, Space: 1, Part: 1193] InMemoryLogBuffer ctor, firstLogId 288305842
I0615 09:08:39.949709 10494 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 25 files into 25, base level is 1, output level is 1
I0615 09:08:40.228348 10494 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 6 files into 0, base level is 1, output level is 1
I0615 09:08:40.228590 10494 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:08:40.228642  3119 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:08:40.228674  3120 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:08:46.790678 10492 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 25 files into 25, base level is 1, output level is 1
I0615 09:08:47.064477 10492 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 6 files into 0, base level is 1, output level is 1
I0615 09:08:47.064719 10492 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:08:47.064783  3130 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:08:47.064817  3131 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:08:51.490653 10494 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 6 files into 6, base level is 1, output level is 1
I0615 09:08:51.560477 10494 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 25 files into 0, base level is 1, output level is 1
I0615 09:08:51.560750 10494 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:08:51.560889  3143 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:08:51.560895  3142 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:08:51.560907  3144 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:08:53.668856 10493 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 25 files into 25, base level is 1, output level is 1
I0615 09:08:53.957299 10493 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 25 files into 0, base level is 1, output level is 1
I0615 09:08:53.957598 10493 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:08:53.957651  3150 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:08:53.957667  3149 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:08:53.957677  3148 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:08:59.591961 10492 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 6 files into 6, base level is 1, output level is 1
I0615 09:08:59.714759 10492 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 6 files into 0, base level is 1, output level is 1
I0615 09:08:59.715042 10492 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:08:59.715111  3154 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:08:59.715248  3153 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:09:14.432552 10492 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 6 files into 6, base level is 1, output level is 1
I0615 09:09:14.526918 10492 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 6 files into 0, base level is 1, output level is 1
I0615 09:09:14.527187 10492 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:09:14.527256  3226 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:09:14.528221  3227 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:09:27.537202 10492 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 6 files into 6, base level is 1, output level is 1
I0615 09:09:27.603796 10492 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 6 files into 0, base level is 1, output level is 1
I0615 09:09:27.604003 10492 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:09:27.604068  3237 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:09:27.604077  3238 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:09:37.423409 10494 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 25 files into 25, base level is 1, output level is 1
I0615 09:09:37.707473 10494 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 6 files into 0, base level is 1, output level is 1
I0615 09:09:37.707679 10494 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:09:37.707747  3250 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:09:37.707784  3249 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:09:38.525876 10492 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 6 files into 6, base level is 1, output level is 1
I0615 09:09:39.272238 10493 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 25 files into 25, base level is 1, output level is 1
I0615 09:09:39.588248 10492 EventListner.h:18] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 6 files into 0, base level is 1, output level is 1
I0615 09:09:39.588567 10492 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:09:39.588745  3256 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:09:39.588825  3257 CompactionFilter.h:57] Do full/manual compaction!
I0615 09:09:39.752158 10448 InMemoryLogBuffer.h:27] [Port: 44501, Space: 154, Part: 1013] InMemoryLogBuffer dtor, firstLogId 40706896
I0615 09:09:39.768815 10448 InMemoryLogBuffer.h:23] [Port: 44501, Space: 154, Part: 1013] InMemoryLogBuffer ctor, firstLogId 40764464
I0615 09:09:46.282706 10494 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 6 files into 6, base level is 1, output level is 1
I0615 09:09:47.924736 10492 EventListner.h:30] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 6 files into 6, base level is 1, output level is 1
I0615 09:09:47.991506 11054 NebulaStore.cpp:711] Space 154 compaction done.
I0615 09:16:19.232841 10442 InMemoryLogBuffer.h:27] [Port: 44501, Space: 1, Part: 893] InMemoryLogBuffer dtor, firstLogId 286842837
I0615 09:16:19.278848 10442 InMemoryLogBuffer.h:23] [Port: 44501, Space: 1, Part: 893] InMemoryLogBuffer ctor, firstLogId 286958966
I0615 09:18:33.635221 10453 InMemoryLogBuffer.h:27] [Port: 44501, Space: 1, Part: 983] InMemoryLogBuffer dtor, firstLogId 288630869
I0615 09:18:33.673673 10453 InMemoryLogBuffer.h:23] [Port: 44501, Space: 1, Part: 983] InMemoryLogBuffer ctor, firstLogId 288755410
I0615 09:21:19.092784 10439 InMemoryLogBuffer.h:27] [Port: 44501, Space: 154, Part: 474] InMemoryLogBuffer dtor, firstLogId 40635534
I0615 09:21:19.110185 10439 InMemoryLogBuffer.h:23] [Port: 44501, Space: 154, Part: 474] InMemoryLogBuffer ctor, firstLogId 40693945
I0615 09:21:35.342262 10450 InMemoryLogBuffer.h:27] [Port: 44501, Space: 1, Part: 1162] InMemoryLogBuffer dtor, firstLogId 285791259
I0615 09:21:35.373193 10450 InMemoryLogBuffer.h:23] [Port: 44501, Space: 1, Part: 1162] InMemoryLogBuffer ctor, firstLogId 285913759
I0615 09:24:03.398370 10434 InMemoryLogBuffer.h:27] [Port: 44501, Space: 1, Part: 742] InMemoryLogBuffer dtor, firstLogId 286498552
I0615 09:24:03.451337 10434 InMemoryLogBuffer.h:23] [Port: 44501, Space: 1, Part: 742] InMemoryLogBuffer ctor, firstLogId 286623402
I0615 09:24:03.451709 10434 SlowOpTracker.h:33] [Port: 44501, Space: 1, Part: 742] , total time:53ms, Write WAL, total 2
I0615 09:24:10.409507 10458 InMemoryLogBuffer.h:27] [Port: 44501, Space: 1, Part: 412] InMemoryLogBuffer dtor, firstLogId 287151013
I0615 09:24:10.453003 10458 InMemoryLogBuffer.h:23] [Port: 44501, Space: 1, Part: 412] InMemoryLogBuffer ctor, firstLogId 287271450
I0615 09:27:06.724117 10453 InMemoryLogBuffer.h:27] [Port: 44501, Space: 154, Part: 204] InMemoryLogBuffer dtor, firstLogId 40745713
I0615 09:27:06.736843 10453 InMemoryLogBuffer.h:23] [Port: 44501, Space: 154, Part: 204] InMemoryLogBuffer ctor, firstLogId 40804529
I0615 09:29:23.535810 10448 InMemoryLogBuffer.h:27] [Port: 44501, Space: 1, Part: 444] InMemoryLogBuffer dtor, firstLogId 288418040
I0615 09:29:23.583572 10448 InMemoryLogBuffer.h:23] [Port: 44501, Space: 1, Part: 444] InMemoryLogBuffer ctor, firstLogId 288540991
I0615 09:29:23.638648 10444 SlowOpTracker.h:33] [Port: 44501, Space: 1, Part: 444] , total time:55ms, Total send logs: 2
I0615 09:29:44.573834 10439 InMemoryLogBuffer.h:27] [Port: 44501, Space: 1, Part: 534] InMemoryLogBuffer dtor, firstLogId 286699514
I0615 09:29:44.597038 10439 InMemoryLogBuffer.h:23] [Port: 44501, Space: 1, Part: 534] InMemoryLogBuffer ctor, firstLogId 286822365
I0615 09:30:09.464900 10445 InMemoryLogBuffer.h:27] [Port: 44501, Space: 154, Part: 174] InMemoryLogBuffer dtor, firstLogId 40753881
I0615 09:30:09.478113 10445 InMemoryLogBuffer.h:23] [Port: 44501, Space: 154, Part: 174] InMemoryLogBuffer ctor, firstLogId 40812664
I0615 09:33:20.657176 10488 FileBasedWal.cpp:760] [Port: 44501, Space: 154, Part: 24] Clean wals number 1
I0615 09:42:03.368108 10441 InMemoryLogBuffer.h:27] [Port: 44501, Space: 1, Part: 503] InMemoryLogBuffer dtor, firstLogId 286796991
I0615 09:42:03.428016 10441 InMemoryLogBuffer.h:23] [Port: 44501, Space: 1, Part: 503] InMemoryLogBuffer ctor, firstLogId 286919993
I0615 09:42:32.757472 10444 InMemoryLogBuffer.h:27] [Port: 44501, Space: 1, Part: 592] InMemoryLogBuffer dtor, firstLogId 285696903
I0615 09:42:32.789788 10444 InMemoryLogBuffer.h:23] [Port: 44501, Space: 1, Part: 592] InMemoryLogBuffer ctor, firstLogId 285814060
I0615 09:44:45.472395 10461 InMemoryLogBuffer.h:27] [Port: 44501, Space: 154, Part: 382] InMemoryLogBuffer dtor, firstLogId 40610274
I0615 09:44:45.486384 10461 InMemoryLogBuffer.h:23] [Port: 44501, Space: 154, Part: 382] InMemoryLogBuffer ctor, firstLogId 40669890
I0615 09:50:20.296826 10454 InMemoryLogBuffer.h:27] [Port: 44501, Space: 1, Part: 924] InMemoryLogBuffer dtor, firstLogId 292515224
I0615 09:50:20.366132 10454 InMemoryLogBuffer.h:23] [Port: 44501, Space: 1, Part: 924] InMemoryLogBuffer ctor, firstLogId 292632916
I0615 09:50:20.366351 10454 SlowOpTracker.h:33] [Port: 44501, Space: 1, Part: 924] , total time:70ms, Write WAL, total 2
I0615 09:51:48.682073 10454 InMemoryLogBuffer.h:27] [Port: 44501, Space: 1, Part: 863] InMemoryLogBuffer dtor, firstLogId 296018339
I0615 09:51:48.717883 10454 InMemoryLogBuffer.h:23] [Port: 44501, Space: 1, Part: 863] InMemoryLogBuffer ctor, firstLogId 296139109
I0615 09:52:14.904310 10454 InMemoryLogBuffer.h:27] [Port: 44501, Space: 154, Part: 504] InMemoryLogBuffer dtor, firstLogId 40778666
I0615 09:52:14.918805 10454 InMemoryLogBuffer.h:23] [Port: 44501, Space: 154, Part: 504] InMemoryLogBuffer ctor, firstLogId 40838643
I0615 09:53:20.662688 10486 FileBasedWal.cpp:760] [Port: 44501, Space: 1, Part: 594] Clean wals number 1
1 个赞

参考这个呢:Storage Balance - Nebula Graph Database Manual

leader分布不均衡而已, 正常

nebula 分为leader balance 和data balance 两种操作

上一次出现故障时,已尝试。现在尝试如下,不起作用:

(root@nebula) [(none)]> show hosts;
=====================================================================================================================
| Ip            | Port  | Status | Leader count | Leader distribution | Partition distribution                      |
=====================================================================================================================
| 10.X.X.X | 44500 | online | 0            |                     | offline_X: 40, prod: 120, prod_v1: 120     |
---------------------------------------------------------------------------------------------------------------------
| 10.X.X.X | 44500 | online | 0            |                     | offline_X: 40, prod: 120, prod_v1: 120     |
---------------------------------------------------------------------------------------------------------------------
| 10.X.X.X  | 44500 | online | 0            |                     | offline_X: 40, prod: 120, prod_v1: 120     |
---------------------------------------------------------------------------------------------------------------------
| 10.X.X.X | 44500 | online | 0            |                     | offline_X: 40, prod: 120, prod_v1: 120     |
---------------------------------------------------------------------------------------------------------------------
| 10.X.X.X  | 44500 | online | 0            |                     | offline_X: 40, prod: 120, prod_v1: 120     |
---------------------------------------------------------------------------------------------------------------------
| 10.X.X.X  | 44500 | online | 0            |                     | offline_X: 40, prod: 120, prod_v1: 120     |
---------------------------------------------------------------------------------------------------------------------
| 10.X.X.X  | 44500 | online | 0            |                     | offline_X: 40, prod: 120, prod_v1: 120     |
---------------------------------------------------------------------------------------------------------------------
| 10.X.X.X | 44500 | online | 0            |                     | offline_X: 40, prod: 120, prod_v1: 120     |
---------------------------------------------------------------------------------------------------------------------
| 10.X.X.X | 44500 | online | 0            |                     | offline_X: 40, prod: 120, prod_v1: 120     |
---------------------------------------------------------------------------------------------------------------------
| 10.X.X.X | 44500 | online | 0            |                     | offline_X: 40, prod: 120, prod_v1: 120     |
---------------------------------------------------------------------------------------------------------------------
| 10.X.X.X | 44500 | online | 0            |                     | offline_X: 40, prod: 120, prod_v1: 120     |
---------------------------------------------------------------------------------------------------------------------
| 10.X.X.X | 44500 | online | 0            |                     | offline_X: 40, prod: 120, prod_v1: 120     |
---------------------------------------------------------------------------------------------------------------------
| 10.X.X.X | 44500 | online | 0            |                     | offline_X: 40, prod: 120, prod_v1: 120     |
---------------------------------------------------------------------------------------------------------------------
| 10.X.X.X | 44500 | online | 0            |                     | offline_X: 40, prod: 120, prod_v1: 120     |
---------------------------------------------------------------------------------------------------------------------
| 10.X.X.X | 44500 | online | 0            |                     | offline_X: 40, prod: 120, prod_v1: 120     |
---------------------------------------------------------------------------------------------------------------------
| 10.X.X.X | 44500 | online | 0            |                     | offline_X: 40, prod: 120, prod_v1: 120     |
---------------------------------------------------------------------------------------------------------------------
| 10.X.X.X | 44500 | online | 0            |                     | offline_X: 40, prod: 120, prod_v1: 120     |
---------------------------------------------------------------------------------------------------------------------
| 10.X.X.X | 44500 | online | 0            |                     | offline_X: 40, prod: 120, prod_v1: 120     |
---------------------------------------------------------------------------------------------------------------------
| 10.X.X.X | 44500 | online | 0            |                     | offline_X: 40, prod: 120, prod_v1: 120     |
---------------------------------------------------------------------------------------------------------------------
| 10.X.X.X | 44500 | online | 0            |                     | offline_X: 40, prod: 120, prod_v1: 120     |
---------------------------------------------------------------------------------------------------------------------
| 10.X.X.X | 44500 | online | 0            |                     | offline_X: 40, prod: 120, prod_v1: 120     |
---------------------------------------------------------------------------------------------------------------------
| 10.X.X.X | 44500 | online | 0            |                     | offline_X: 40, prod: 120, prod_v1: 120     |
---------------------------------------------------------------------------------------------------------------------
| 10.X.X.X | 44500 | online | 0            |                     | offline_X: 40, prod: 120, prod_v1: 120     |
---------------------------------------------------------------------------------------------------------------------
| 10.X.X.X | 44500 | online | 0            |                     | offline_X: 40, prod: 120, prod_v1: 120     |
---------------------------------------------------------------------------------------------------------------------
| 10.X.X.X | 44500 | online | 0            |                     | offline_X: 40, prod: 120, prod_v1: 120     |
---------------------------------------------------------------------------------------------------------------------
| 10.X.X.X | 44500 | online | 0            |                     | offline_X: 40, prod: 120, prod_v1: 120     |
---------------------------------------------------------------------------------------------------------------------
| 10.X.X.X | 44500 | online | 0            |                     | offline_X: 40, prod: 120, prod_v1: 120     |
---------------------------------------------------------------------------------------------------------------------
| 10.X.X.X | 44500 | online | 0            |                     | offline_X: 40, prod: 120, prod_v1: 120     |
---------------------------------------------------------------------------------------------------------------------
| 10.X.X.X | 44500 | online | 0            |                     | offline_X: 40, prod: 120, prod_v1: 120     |
---------------------------------------------------------------------------------------------------------------------
| 10.X.X.X | 44500 | online | 0            |                     | offline_X: 40, prod: 120, prod_v1: 120     |
---------------------------------------------------------------------------------------------------------------------
| Total         |       |        | 0            |                     | offline_X: 1200, prod: 3600, prod_v1: 3600 |
---------------------------------------------------------------------------------------------------------------------
Got 31 rows (Time spent: 5.442/6.853 ms)

Wed Jun 16 10:06:22 2021

(root@nebula) [(none)]> balance leader;
Execution succeeded (Time spent: 7.807/8.903 ms)

Wed Jun 16 10:06:31 2021

(root@nebula) [(none)]> balance data;
[ERROR (-8)]: The cluster is balanced!
Wed Jun 16 10:06:36 2021

设置的定时compact工作是正常的。

(root@nebula) [(none)]> show jobs;
===================================================================================
| Job Id | Command             | Status   | Start Time        | Stop Time         |
===================================================================================
| 270    | compact prod_v1     | finished | 06/16/21 08:00:01 | 06/16/21 09:10:03 |
-----------------------------------------------------------------------------------
| 269    | compact prod        | finished | 06/16/21 01:00:01 | 06/16/21 07:57:18 |
-----------------------------------------------------------------------------------
| 268    | compact prod_v1     | finished | 06/15/21 08:00:01 | 06/15/21 09:09:58 |
-----------------------------------------------------------------------------------
| 267    | compact prod        | finished | 06/15/21 01:00:02 | 06/15/21 07:58:27 |
-----------------------------------------------------------------------------------
| 266    | compact prod_v1     | finished | 06/14/21 08:00:01 | 06/14/21 09:09:55 |
-----------------------------------------------------------------------------------
| 265    | compact prod        | finished | 06/14/21 01:00:01 | 06/14/21 07:57:21 |

关键是连接部分graphd 所有节点的leader都为0,上一次(https://discuss.nebula-graph.com.cn/t/topic/3844 )就是后台报错no leader,最后导致业务一个请求都是延时秒级的,最后重启所有服务问题就自己恢复了。现在刚出现leader为0,还没影响业务。就怕出现上一次故障。问题还是没有根治呀。

leader count显示为0只是个显示问题,你们要么把compaction频率调高,要么开auto compaction,再不行就加机器吧,compaction的时间太长了。

等了几天,显示为0的问题还是没有恢复,显示为0的问题就不用管了是吗?暂时还没法加机器,每台机器8块ssd磁盘空间才占用百分之十几,开通auto compation 会影响我们灌数的速度,数据compaction一次处理时间过长的问题后面会有什么影响吗?

@critical27

你灌数据的时候,关闭auto compation,灌完数据的时候,将auto compation打开。

另外可以将max_background_jobs 和max_subcompactions 调大点。

好的,谢谢。

该话题在最后一个回复创建后30天后自动关闭。不再允许新的回复。