storage进程cpu飙升，服务不可用

fisherman · 2022 年1 月 11 日 02:52

nebula 版本：v2.6.1
部署方式：分布式
安装方式： RPM
是否为线上版本：Y
硬件信息
- 磁盘 SSD
- CPU、内存信息 128G 16core
问题的具体描述：从昨晚九点半之后storage进程占满cpu，服务不可用。

image1710×769 379 KB

当前集群三副本六节点；6个graphd、storaged节点，3个metad节点。
一开始把所有的storaged进程重启，过了一段时间（大约五分钟）后storaged节点又会把cpu耗尽。然后把三台带有metad节点服务全部重启，cpu就恢复了。如下图

观察业务方日志，昨晚开始业务请求量并没有变高，请问一下是那些原因导致storaged耗尽cpu。

pandasheeps · 2022 年1 月 11 日 03:10

猜测是不是在做compact，你确认下
然后看看storage log，那段时间的log 贴一下
用perf 工具，把信息贴一下

fisherman · 2022 年1 月 11 日 05:56

storage日志
nebula_storaged.log (244.4 KB)


I0110 20:17:01.427897 27076 EventListener.h:18] Rocksdb start compaction column family: default because of LevelMaxLevelSize, status: OK, compacted 3 files into 0, base level is 1, output level is 2
I0110 20:17:01.427939 27076 CompactionFilter.h:62] Do default minor compaction!
I0110 20:17:02.693636 27076 EventListener.h:28] Rocksdb compaction completed column family: default because of LevelMaxLevelSize, status: OK, compacted 3 files into 2, base level is 1, output level is 2
I0110 20:17:02.714396 27076 EventListener.h:18] Rocksdb start compaction column family: default because of LevelMaxLevelSize, status: OK, compacted 3 files into 0, base level is 1, output level is 2
I0110 20:17:02.714452 27076 CompactionFilter.h:62] Do default minor compaction!
I0110 20:17:05.029330 27076 EventListener.h:28] Rocksdb compaction completed column family: default because of LevelMaxLevelSize, status: OK, compacted 3 files into 3, base level is 1, output level is 2
I0110 20:18:37.245857 27077 EventListener.h:18] Rocksdb start compaction column family: default because of LevelL0FilesNum, status: OK, compacted 10 files into 0, base level is 0, output level is 1
I0110 20:18:37.245967 27077 CompactionFilter.h:62] Do default minor compaction!
I0110 20:18:37.246016 29909 CompactionFilter.h:62] Do default minor compaction!
I0110 20:18:37.246018 29910 CompactionFilter.h:62] Do default minor compaction!
I0110 20:18:37.246026 29908 CompactionFilter.h:62] Do default minor compaction!
I0110 20:18:38.536311 27077 EventListener.h:28] Rocksdb compaction completed column family: default because of LevelL0FilesNum, status: OK, compacted 10 files into 7, base level is 0, output level is 1
I0110 20:20:26.790475 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 30] Clean wals number 1
I0110 20:20:26.794176 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 22] Clean wals number 1
I0110 20:20:26.798060 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 24] Clean wals number 1
I0110 20:20:26.801790 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 28] Clean wals number 1
I0110 20:20:26.805979 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 10] Clean wals number 1
I0110 20:20:26.809820 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 17] Clean wals number 1
I0110 20:20:26.813616 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 12] Clean wals number 1
I0110 20:20:26.817337 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 11] Clean wals number 1
I0110 20:20:26.821915 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 18] Clean wals number 1
I0110 20:30:26.826905 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 22] Clean wals number 1
I0110 20:30:26.830835 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 4] Clean wals number 1
I0110 20:30:26.834817 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 5] Clean wals number 1
I0110 20:30:26.839340 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 16] Clean wals number 1
I0110 20:30:26.843796 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 29] Clean wals number 1
I0110 20:30:26.848343 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 6] Clean wals number 1
I0110 20:30:26.852223 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 23] Clean wals number 1
I0110 20:30:26.855896 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 11] Clean wals number 1
I0110 20:34:13.373889 27078 EventListener.h:18] Rocksdb start compaction column family: default because of LevelL0FilesNum, status: OK, compacted 8 files into 0, base level is 0, output level is 1
I0110 20:34:13.373993 27078 CompactionFilter.h:62] Do default minor compaction!
I0110 20:34:13.374035 31136 CompactionFilter.h:62] Do default minor compaction!
I0110 20:34:13.374081 31137 CompactionFilter.h:62] Do default minor compaction!
I0110 20:34:15.533607 27078 EventListener.h:28] Rocksdb compaction completed column family: default because of LevelL0FilesNum, status: OK, compacted 8 files into 6, base level is 0, output level is 1
I0110 20:34:15.601110 27078 EventListener.h:18] Rocksdb start compaction column family: default because of LevelMaxLevelSize, status: OK, compacted 5 files into 0, base level is 1, output level is 2
I0110 20:34:15.601150 27078 CompactionFilter.h:62] Do default minor compaction!
I0110 20:34:18.814826 27078 EventListener.h:28] Rocksdb compaction completed column family: default because of LevelMaxLevelSize, status: OK, compacted 5 files into 5, base level is 1, output level is 2
I0110 20:40:26.866989 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 30] Clean wals number 1
I0110 20:40:26.870901 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 22] Clean wals number 1
I0110 20:40:26.875536 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 24] Clean wals number 1
I0110 20:40:26.880343 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 28] Clean wals number 1
I0110 20:40:26.884420 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 5] Clean wals number 1
I0110 20:40:26.888624 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 16] Clean wals number 1
I0110 20:40:26.892277 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 10] Clean wals number 1
I0110 20:40:26.896291 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 17] Clean wals number 1
I0110 20:40:26.900338 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 29] Clean wals number 1
I0110 20:40:26.904345 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 6] Clean wals number 1
I0110 20:40:26.908278 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 12] Clean wals number 1
I0110 20:40:26.912108 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 18] Clean wals number 1
I0110 20:47:03.150735 27076 EventListener.h:18] Rocksdb start compaction column family: default because of LevelL0FilesNum, status: OK, compacted 11 files into 0, base level is 0, output level is 1
I0110 20:47:03.150856 27076 CompactionFilter.h:62] Do default minor compaction!
I0110 20:47:03.150892 32133 CompactionFilter.h:62] Do default minor compaction!
I0110 20:47:03.150913 32131 CompactionFilter.h:62] Do default minor compaction!
I0110 20:47:03.150916 32132 CompactionFilter.h:62] Do default minor compaction!
I0110 20:47:04.335124 27076 EventListener.h:28] Rocksdb compaction completed column family: default because of LevelL0FilesNum, status: OK, compacted 11 files into 7, base level is 0, output level is 1
I0110 20:50:26.922490 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 30] Clean wals number 1
I0110 20:50:26.926566 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 22] Clean wals number 1
I0110 20:50:26.930704 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 24] Clean wals number 1
I0110 20:50:26.934584 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 4] Clean wals number 1
I0110 20:50:26.938930 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 28] Clean wals number 1
I0110 20:50:26.943338 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 5] Clean wals number 1
I0110 20:50:26.947293 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 16] Clean wals number 1
I0110 20:50:26.951210 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 10] Clean wals number 1
I0110 20:50:26.955236 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 17] Clean wals number 1
I0110 20:50:26.959555 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 6] Clean wals number 1
I0110 20:50:26.963706 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 12] Clean wals number 1
I0110 20:50:26.967597 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 23] Clean wals number 1
I0110 20:50:26.971724 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 11] Clean wals number 1
I0110 20:50:26.975533 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 18] Clean wals number 1
I0110 20:51:21.325153 27077 EventListener.h:18] Rocksdb start compaction column family: default because of LevelL0FilesNum, status: OK, compacted 8 files into 0, base level is 0, output level is 1
I0110 20:51:21.325259 27077 CompactionFilter.h:62] Do default minor compaction!
I0110 20:51:21.325309 32464 CompactionFilter.h:62] Do default minor compaction!
I0110 20:51:21.325327 32465 CompactionFilter.h:62] Do default minor compaction!
I0110 20:51:23.401340 27077 EventListener.h:28] Rocksdb compaction completed column family: default because of LevelL0FilesNum, status: OK, compacted 8 files into 7, base level is 0, output level is 1
I0110 20:51:23.478075 27077 EventListener.h:18] Rocksdb start compaction column family: default because of LevelMaxLevelSize, status: OK, compacted 3 files into 0, base level is 1, output level is 2
I0110 20:51:23.478116 27077 CompactionFilter.h:62] Do default minor compaction!
I0110 20:51:25.743608 27077 EventListener.h:28] Rocksdb compaction completed column family: default because of LevelMaxLevelSize, status: OK, compacted 3 files into 3, base level is 1, output level is 2
I0110 20:51:25.784193 27076 EventListener.h:18] Rocksdb start compaction column family: default because of LevelMaxLevelSize, status: OK, compacted 4 files into 0, base level is 1, output level is 2
I0110 20:51:25.784246 27076 CompactionFilter.h:62] Do default minor compaction!
I0110 20:51:28.273079 27076 EventListener.h:28] Rocksdb compaction completed column family: default because of LevelMaxLevelSize, status: OK, compacted 4 files into 4, base level is 1, output level is 2
I0110 21:00:26.983835 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 30] Clean wals number 1
I0110 21:00:26.988056 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 4] Clean wals number 1
I0110 21:00:26.992081 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 28] Clean wals number 1
I0110 21:00:26.996225 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 10] Clean wals number 1
I0110 21:00:27.000244 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 29] Clean wals number 1
I0110 21:00:27.004801 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 23] Clean wals number 1
I0110 21:00:27.008657 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 11] Clean wals number 1
I0110 21:10:27.020871 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 29] Clean wals number 1
I0110 21:10:27.024663 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 22] Clean wals number 1
I0110 21:10:27.028409 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 24] Clean wals number 1
I0110 21:10:27.032162 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 5] Clean wals number 1
I0110 21:10:27.035876 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 16] Clean wals number 1
I0110 21:10:27.039532 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 17] Clean wals number 1
I0110 21:10:27.043632 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 6] Clean wals number 1
I0110 21:10:27.047341 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 12] Clean wals number 1
I0110 21:10:27.051187 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 18] Clean wals number 1
I0110 21:20:27.064985 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 3, Part: 29] Clean wals number 1
I0110 21:20:27.069018 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 3, Part: 6] Clean wals number 1
I0110 21:20:27.072839 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 30] Clean wals number 1
I0110 21:20:27.076539 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 22] Clean wals number 1
I0110 21:20:27.080560 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 24] Clean wals number 1
I0110 21:20:27.084378 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 4] Clean wals number 1
I0110 21:20:27.088174 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 28] Clean wals number 1
I0110 21:20:27.091805 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 10] Clean wals number 1
I0110 21:20:27.095500 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 29] Clean wals number 1
I0110 21:20:27.099591 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 12] Clean wals number 1
I0110 21:20:27.103188 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 23] Clean wals number 1
I0110 21:20:27.106809 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 11] Clean wals number 1
I0110 21:30:27.117800 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 6] Clean wals number 1
I0110 21:30:27.122043 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 22] Clean wals number 1
I0110 21:30:27.126186 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 4] Clean wals number 1
I0110 21:30:27.130015 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 5] Clean wals number 1
I0110 21:30:27.133726 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 16] Clean wals number 1
I0110 21:30:27.137423 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 17] Clean wals number 1
I0110 21:30:27.141054 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 29] Clean wals number 1
I0110 21:30:27.144682 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 6] Clean wals number 1
I0110 21:30:27.148341 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 11] Clean wals number 1
I0110 21:30:27.152477 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 18] Clean wals number 1
I0110 21:32:34.994552 27029 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:54ms, Total send logs: 2
I0110 21:32:35.178468 27038 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:53ms, Total commit: 1
I0110 21:32:47.185264 27028 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:53ms, Total send logs: 2
I0110 21:32:47.487548 27012 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:51ms, Total send logs: 7
I0110 21:32:55.625613 27010 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:51ms, Total send logs: 2
I0110 21:38:36.284466 27009 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:66ms, Total send logs: 2
I0110 21:38:41.102483 27030 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:78ms, Total send logs: 2
I0110 21:38:46.590557 27022 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:56ms, Total send logs: 2
I0110 21:40:27.163301 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 28] Clean wals number 1
I0110 21:40:27.167124 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 30] Clean wals number 1
I0110 21:40:27.170823 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 24] Clean wals number 1
I0110 21:40:27.175422 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 28] Clean wals number 1
I0110 21:40:27.179042 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 16] Clean wals number 1
I0110 21:40:27.182632 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 10] Clean wals number 1
I0110 21:40:27.186266 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 12] Clean wals number 1
I0110 21:40:27.189963 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 23] Clean wals number 1
I0110 21:41:52.334508 27008 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:54ms, Total send logs: 2
I0110 21:48:47.962481 27016 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:63ms, Total send logs: 4
I0110 21:48:52.680554 27025 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:60ms, Total send logs: 4
I0110 21:49:18.117583 27016 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:75ms, Total send logs: 2
I0110 21:49:23.598723 27027 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:51ms, Total send logs: 2
I0110 21:50:27.202286 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 4] Clean wals number 1
I0110 21:50:27.206073 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 11] Clean wals number 1
I0110 21:50:27.209795 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 22] Clean wals number 1
I0110 21:50:27.213452 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 5] Clean wals number 1
I0110 21:50:27.217037 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 17] Clean wals number 1
I0110 21:50:27.220623 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 6] Clean wals number 1
I0110 21:50:27.224165 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 11] Clean wals number 1
I0110 21:50:27.227757 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 18] Clean wals number 1
I0110 21:57:00.573130 27028 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:64ms, Total send logs: 4
I0110 21:57:04.468838 27016 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:72ms, Total send logs: 2
I0110 21:57:07.334523 27017 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:52ms, Total send logs: 6
I0110 21:57:07.340544 27017 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:67ms, Total send logs: 2
I0110 21:57:08.027483 27014 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:73ms, Total send logs: 2
I0110 21:57:08.035552 27035 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:81ms, Total send logs: 2
I0110 21:57:08.723482 27033 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:52ms, Total send logs: 2
I0110 21:57:11.995463 27016 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:51ms, Total send logs: 2
I0110 21:57:11.997555 27021 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:53ms, Total send logs: 2
I0110 21:57:12.476505 27012 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:52ms, Total send logs: 2
I0110 21:57:15.101459 27033 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:60ms, Total send logs: 3
I0110 21:57:21.784878 27028 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:52ms, Total send logs: 2
I0110 21:57:21.969542 27014 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:66ms, Total send logs: 4
I0110 21:57:21.999475 27027 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:60ms, Total send logs: 3
I0110 21:57:38.315606 27017 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:77ms, Total send logs: 3
I0110 21:57:38.360600 27021 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:54ms, Total send logs: 2
I0110 21:57:58.598546 27014 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:59ms, Total send logs: 2
I0110 21:58:18.187201 27015 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:59ms, Total send logs: 4
I0110 21:58:18.256567 27008 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:54ms, Total send logs: 3
I0110 22:00:27.240439 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 30] Clean wals number 1
I0110 22:00:27.244125 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 24] Clean wals number 1
I0110 22:00:27.247786 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 4] Clean wals number 1
I0110 22:00:27.251457 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 28] Clean wals number 1
I0110 22:00:27.255137 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 16] Clean wals number 1
I0110 22:00:27.258803 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 10] Clean wals number 1
I0110 22:00:27.262791 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 17] Clean wals number 1
I0110 22:00:27.266628 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 29] Clean wals number 1
I0110 22:00:27.270352 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 12] Clean wals number 1
I0110 22:00:27.274377 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 23] Clean wals number 1
I0110 22:00:27.278093 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 11] Clean wals number 1
I0110 22:00:27.281875 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 18] Clean wals number 1
I0110 22:01:15.757920 27038 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:53ms, Total send logs: 3
I0110 22:01:23.978458 27038 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:61ms, Total send logs: 2
I0110 22:01:26.628458 27016 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:56ms, Total send logs: 2
I0110 22:01:26.661463 27016 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:55ms, Total send logs: 2
I0110 22:01:26.669461 27016 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:63ms, Total send logs: 2
I0110 22:01:33.362536 27030 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:66ms, Total send logs: 2
I0110 22:01:33.416599 27020 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:60ms, Total send logs: 2
I0110 22:01:35.560485 27010 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:59ms, Total send logs: 2
I0110 22:01:37.262460 27020 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:51ms, Total send logs: 2
I0110 22:01:50.173485 27016 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:51ms, Total send logs: 2
I0110 22:02:00.141490 27016 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:52ms, Total send logs: 2
I0110 22:02:00.141808 27013 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:53ms, Total send logs: 2
I0110 22:02:00.200667 27013 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:59ms, Total send logs: 3
I0110 22:02:36.855532 27024 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:65ms, Total send logs: 2
I0110 22:02:59.388525 27011 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:74ms, Total send logs: 2
I0110 22:03:00.572530 27018 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:54ms, Total send logs: 2
I0110 22:03:08.310473 27020 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:59ms, Total send logs: 3
I0110 22:03:28.214494 27022 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:54ms, Total send logs: 4
I0110 22:03:36.313469 27026 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:52ms, Total send logs: 2
I0110 22:03:46.468573 27014 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:57ms, Total send logs: 4
I0110 22:04:00.540479 27033 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:55ms, Total send logs: 2
I0110 22:04:00.543484 27020 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:58ms, Total send logs: 2
I0110 22:04:00.623461 27018 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:56ms, Total send logs: 2
I0110 22:04:13.299537 27012 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:61ms, Total send logs: 2
I0110 22:08:26.735245 27028 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:58ms, Total send logs: 2
I0110 22:08:28.048524 27033 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:55ms, Total send logs: 3
I0110 22:08:30.109556 27034 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:62ms, Total send logs: 2
I0110 22:08:45.122531 27026 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:59ms, Total send logs: 2
I0110 22:08:45.252763 27017 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:57ms, Total send logs: 2
I0110 22:08:45.404589 27022 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:63ms, Total send logs: 2
I0110 22:08:51.575834 27025 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:53ms, Total send logs: 17
I0110 22:08:51.589465 27011 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:67ms, Total send logs: 16
I0110 22:08:54.514526 27026 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:56ms, Total send logs: 2
I0110 22:08:54.593050 27012 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:62ms, Total send logs: 2
I0110 22:08:57.545648 27029 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:52ms, Total send logs: 2
I0110 22:09:01.220604 27011 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:58ms, Total send logs: 2
I0110 22:09:26.764156 27038 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:63ms, Total send logs: 2
I0110 22:09:31.430476 27025 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:54ms, Total send logs: 2
I0110 22:09:31.515491 27038 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:53ms, Total send logs: 2
I0110 22:09:35.086527 27015 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:51ms, Total commit: 1
I0110 22:09:36.667501 27009 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:52ms, Total send logs: 2
I0110 22:10:05.194494 27013 SlowOpTracker.h:31] [Port: 9780, Space: 42, Part: 12] total time:51ms, Total send logs: 2
I0110 22:10:19.062675 27008 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:57ms, Total send logs: 2
I0110 22:10:27.296587 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 18] Clean wals number 1
I0110 22:10:27.319499 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 22] Clean wals number 1
I0110 22:10:27.333036 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 4] Clean wals number 1
I0110 22:10:27.338393 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 5] Clean wals number 1
I0110 22:10:27.358924 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 29] Clean wals number 1
I0110 22:10:27.364171 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 6] Clean wals number 1
I0110 22:10:45.361624 27012 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:52ms, Total send logs: 4
I0110 22:13:30.617846 27077 EventListener.h:18] Rocksdb start compaction column family: default because of LevelL0FilesNum, status: OK, compacted 11 files into 0, base level is 0, output level is 1
I0110 22:13:30.617985 27077 CompactionFilter.h:62] Do default minor compaction!
I0110 22:13:30.618029  6474 CompactionFilter.h:62] Do default minor compaction!
I0110 22:13:30.618052  6473 CompactionFilter.h:62] Do default minor compaction!
I0110 22:13:30.618057  6472 CompactionFilter.h:62] Do default minor compaction!
I0110 22:13:32.233918 27077 EventListener.h:28] Rocksdb compaction completed column family: default because of LevelL0FilesNum, status: OK, compacted 11 files into 7, base level is 0, output level is 1
I0110 22:13:32.292079 27076 EventListener.h:18] Rocksdb start compaction column family: default because of LevelMaxLevelSize, status: OK, compacted 11 files into 0, base level is 1, output level is 2
I0110 22:13:32.292152 27076 CompactionFilter.h:62] Do default minor compaction!
I0110 22:13:38.409534 27076 EventListener.h:28] Rocksdb compaction completed column family: default because of LevelMaxLevelSize, status: OK, compacted 11 files into 9, base level is 1, output level is 2
I0110 22:14:08.113538 27019 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:61ms, Total send logs: 2
I0110 22:14:08.142493 27038 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:68ms, Total send logs: 2
I0110 22:14:11.963482 27017 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:64ms, Total send logs: 2
I0110 22:14:16.419466 27009 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:52ms, Total send logs: 2
I0110 22:14:16.788641 27010 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:53ms, Total send logs: 2
I0110 22:14:16.807708 27010 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:60ms, Total send logs: 4
I0110 22:14:34.926535 27022 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:60ms, Total send logs: 2
I0110 22:14:34.935313 27020 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:58ms, Total send logs: 2
I0110 22:18:54.804512 27029 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:51ms, Total send logs: 2
I0110 22:19:02.921582 27021 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:57ms, Total send logs: 2
I0110 22:19:14.545627 27020 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:54ms, Total send logs: 4
I0110 22:19:14.621630 27009 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:52ms, Total send logs: 2
I0110 22:19:14.700476 27021 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:52ms, Total send logs: 2
I0110 22:19:17.423481 27018 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:52ms, Total send logs: 2
I0110 22:19:17.438395 27018 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:61ms, Total send logs: 2
I0110 22:19:18.107570 27035 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:58ms, Total send logs: 2
I0110 22:19:20.344632 27018 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:54ms, Total send logs: 2
I0110 22:19:20.884675 27035 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:52ms, Total send logs: 2
I0110 22:19:24.320904 27022 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:60ms, Total send logs: 2
I0110 22:19:26.751479 27012 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:61ms, Total send logs: 4
I0110 22:19:27.815624 27032 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:55ms, Total send logs: 3
I0110 22:19:35.298497 27019 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:58ms, Total send logs: 2
I0110 22:19:35.349491 27031 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:51ms, Total send logs: 2
I0110 22:19:38.157475 27021 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:53ms, Total send logs: 2
I0110 22:19:54.894471 27021 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:56ms, Total send logs: 3
I0110 22:20:27.387974 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 12] Clean wals number 1
I0110 22:20:27.393190 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 23] Clean wals number 1
I0110 22:20:27.401522 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 22] Clean wals number 1
I0110 22:20:27.406888 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 16] Clean wals number 1
I0110 22:20:27.424319 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 5] Clean wals number 1
I0110 22:20:27.441725 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 17] Clean wals number 1
I0110 22:20:27.498540 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 30] Clean wals number 1
I0110 22:20:27.504062 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 24] Clean wals number 1
I0110 22:20:27.515393 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 30] Clean wals number 1
I0110 22:20:27.526610 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 28] Clean wals number 1
I0110 22:20:27.535867 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 16] Clean wals number 1
I0110 22:20:27.547029 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 17] Clean wals number 1
I0110 22:20:27.556190 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 12] Clean wals number 1
I0110 22:20:35.599798 27017 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:54ms, Total send logs: 3
I0110 22:20:38.569664 27012 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:74ms, Total send logs: 3
I0110 22:20:39.503130 27012 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:60ms, Total send logs: 2
I0110 22:20:39.611418 27032 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:59ms, Total send logs: 2
I0110 22:20:42.332919 27025 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:59ms, Total send logs: 2
I0110 22:20:45.774296 27032 SlowOpTracker.h:31] [Port: 9780, Space: 3, Part: 23] total time:55ms, Total send logs: 2
I0110 22:21:07.348821 27009 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:55ms, Total commit: 6
I0110 22:25:57.434490 27033 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:61ms, Total send logs: 3
I0110 22:25:57.434657 27033 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:54ms, Total send logs: 4
I0110 22:25:57.436367 27016 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:63ms, Total send logs: 4
I0110 22:25:57.628546 27022 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:94ms, Total send logs: 2
I0110 22:25:57.636054 27034 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:64ms, Total send logs: 2
I0110 22:25:57.659492 27038 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:88ms, Total send logs: 2
I0110 22:26:02.054504 27032 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:61ms, Total send logs: 4
I0110 22:26:02.253486 27031 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:53ms, Total send logs: 4
I0110 22:26:09.014631 27035 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:53ms, Total send logs: 2
I0110 22:26:12.142521 27033 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:58ms, Total send logs: 2
I0110 22:26:22.417533 27018 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:62ms, Total send logs: 2
I0110 22:26:34.603220 27022 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:55ms, Total send logs: 2
I0110 22:26:34.628465 27016 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:67ms, Total send logs: 2
I0110 22:26:34.654510 27037 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:51ms, Total send logs: 4
I0110 22:26:34.667636 27022 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:75ms, Total send logs: 3
I0110 22:26:34.667840 27022 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:56ms, Total send logs: 4
I0110 22:26:34.673624 27027 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:62ms, Total send logs: 3
I0110 22:26:34.775485 27037 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:62ms, Total send logs: 2
I0110 22:26:34.813390 27016 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:51ms, Total send logs: 3
I0110 22:26:35.846534 27020 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:55ms, Total send logs: 3
I0110 22:26:41.119549 27023 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:54ms, Total send logs: 6
I0110 22:26:41.125097 27030 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:76ms, Total send logs: 2
I0110 22:26:41.173491 27023 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:61ms, Total send logs: 3
I0110 22:26:41.174474 27037 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:55ms, Total send logs: 5
I0110 22:26:46.333995 27016 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:54ms, Total send logs: 2
I0110 22:26:54.453547 27029 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:54ms, Total send logs: 2
I0110 22:26:54.595278 27037 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:54ms, Total send logs: 5
I0110 22:26:56.541532 27022 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:69ms, Total send logs: 2
I0110 22:27:09.222476 27027 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:73ms, Total send logs: 2
I0110 22:27:15.492520 27011 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:56ms, Total send logs: 4
I0110 22:27:15.492609 27033 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:56ms, Total send logs: 4
I0110 22:27:45.720531 27012 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:51ms, Total send logs: 2
I0110 22:28:21.150622 27031 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:64ms, Total send logs: 3
I0110 22:28:21.169075 27013 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:66ms, Total send logs: 3
I0110 22:28:38.630478 27024 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:59ms, Total send logs: 2
I0110 22:28:42.996794 27027 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:61ms, Total send logs: 2
I0110 22:28:45.826454 27033 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:67ms, Total send logs: 2
I0110 22:29:05.722592 27011 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:56ms, Total send logs: 2
I0110 22:29:54.302553 27033 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:51ms, Total send logs: 2
I0110 22:29:54.648633 27027 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:75ms, Total send logs: 3
I0110 22:30:03.041610 27013 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:52ms, Total send logs: 4
I0110 22:30:27.568888 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 22] Clean wals number 1
I0110 22:30:27.574671 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 18] Clean wals number 1
I0110 22:30:27.579941 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 4] Clean wals number 1
I0110 22:30:27.585040 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 11] Clean wals number 1
I0110 22:30:27.590162 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 10] Clean wals number 1
I0110 22:30:27.595178 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 5] Clean wals number 1
I0110 22:30:27.600349 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 28] Clean wals number 1
I0110 22:30:27.605526 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 30] Clean wals number 1
I0110 22:30:27.610800 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 24] Clean wals number 1
I0110 22:30:27.616576 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 6] Clean wals number 1
I0110 22:30:27.621739 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 29] Clean wals number 1
I0110 22:30:27.626881 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 10] Clean wals number 1
I0110 22:30:27.631944 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 23] Clean wals number 1
I0110 22:32:47.673169 27078 EventListener.h:18] Rocksdb start compaction column family: default because of LevelL0FilesNum, status: OK, compacted 9 files into 0, base level is 0, output level is 1
I0110 22:32:47.673281 27078 CompactionFilter.h:62] Do default minor compaction!
I0110 22:32:47.673332  7957 CompactionFilter.h:62] Do default minor compaction!
I0110 22:32:47.673344  7958 CompactionFilter.h:62] Do default minor compaction!
I0110 22:32:48.854627 27078 EventListener.h:28] Rocksdb compaction completed column family: default because of LevelL0FilesNum, status: OK, compacted 9 files into 6, base level is 0, output level is 1
I0110 22:33:43.297523 27029 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:54ms, Total send logs: 4
I0110 22:33:43.371560 27035 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:51ms, Total send logs: 2
I0110 22:33:48.245460 27015 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:57ms, Total send logs: 3
I0110 22:33:48.304522 27029 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:70ms, Total send logs: 5
I0110 22:33:48.306068 27033 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:61ms, Total send logs: 6
I0110 22:33:54.936480 27011 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:60ms, Total send logs: 2
I0110 22:33:55.031509 27015 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:52ms, Total send logs: 2
I0110 22:33:55.055577 27013 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:59ms, Total send logs: 2
I0110 22:33:56.066462 27011 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:51ms, Total send logs: 2
I0110 22:34:16.226620 27014 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:61ms, Total send logs: 2
I0110 22:34:29.278671 27033 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:66ms, Total send logs: 3
I0110 22:34:38.617466 27033 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:55ms, Total send logs: 4
I0110 22:34:38.631508 27019 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:70ms, Total send logs: 2
I0110 22:34:52.245548 27035 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:58ms, Total send logs: 3
I0110 22:40:00.632607 27032 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:53ms, Total send logs: 3
I0110 22:40:00.640506 27038 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:54ms, Total send logs: 2
I0110 22:40:00.685488 27028 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:55ms, Total send logs: 2
I0110 22:40:00.692241 27011 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:60ms, Total send logs: 2
I0110 22:40:01.099866 27020 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:90ms, Total send logs: 3
I0110 22:40:01.147625 27019 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:136ms, Total send logs: 4
I0110 22:40:01.180478 27019 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:62ms, Total send logs: 2
I0110 22:40:13.505544 27017 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:56ms, Total send logs: 2
I0110 22:40:13.509709 27017 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:65ms, Total send logs: 3
I0110 22:40:13.578279 27017 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:69ms, Total send logs: 4
I0110 22:40:13.578285 27031 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:73ms, Total send logs: 3
I0110 22:40:13.843479 27038 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:57ms, Total send logs: 2
I0110 22:40:14.375479 27016 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:61ms, Total send logs: 2
I0110 22:40:14.375599 27016 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:55ms, Total send logs: 2
I0110 22:40:17.095541 27014 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:55ms, Total send logs: 5
I0110 22:40:17.098453 27017 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:90ms, Total send logs: 2
I0110 22:40:17.580516 27022 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:56ms, Total send logs: 2
I0110 22:40:24.108566 27017 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:55ms, Total send logs: 2
I0110 22:40:24.203462 27017 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:51ms, Total send logs: 5
I0110 22:40:27.649798 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 12] Clean wals number 1
I0110 22:40:27.655056 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 23] Clean wals number 1
I0110 22:40:27.670405 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 18] Clean wals number 1
I0110 22:40:27.682860 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 4] Clean wals number 1
I0110 22:40:27.694211 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 16] Clean wals number 1
I0110 22:40:27.705552 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 11] Clean wals number 1
I0110 22:40:27.714977 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 10] Clean wals number 1
I0110 22:40:27.726418 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 28] Clean wals number 1
I0110 22:40:27.737844 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 17] Clean wals number 1
I0110 22:40:27.749209 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 6] Clean wals number 1
I0110 22:40:27.760499 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 29] Clean wals number 1
I0110 22:40:27.771809 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 24] Clean wals number 1
I0110 22:40:27.782094 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 29] Clean wals number 1
I0110 22:40:28.195560 27029 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:66ms, Total send logs: 2
I0110 22:40:34.157701 27012 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:59ms, Total send logs: 3
I0110 22:40:34.188542 27029 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:69ms, Total send logs: 2
I0110 22:40:34.191689 27031 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:68ms, Total send logs: 2
I0110 22:40:37.548563 27023 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:57ms, Total send logs: 2
I0110 22:40:37.553517 27022 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:62ms, Total send logs: 2
I0110 22:40:39.610498 27020 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:82ms, Total send logs: 3
I0110 22:40:40.188470 27029 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:51ms, Total send logs: 2
I0110 22:40:40.217468 27029 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:52ms, Total send logs: 2
I0110 22:40:41.645475 27027 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:57ms, Total send logs: 2
I0110 22:40:41.645951 27020 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:57ms, Total send logs: 2
I0110 22:40:41.703080 27020 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:57ms, Total send logs: 2
I0110 22:40:41.703096 27009 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:58ms, Total send logs: 2
I0110 22:40:43.706458 27023 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:66ms, Total send logs: 2
I0110 22:40:44.340551 27022 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:60ms, Total send logs: 2
I0110 22:40:47.043478 27025 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:59ms, Total send logs: 2
I0110 22:40:47.355463 27025 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:52ms, Total send logs: 2
I0110 22:40:49.754570 27027 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:63ms, Total send logs: 2
I0110 22:40:50.742591 27022 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:69ms, Total send logs: 7
I0110 22:40:50.802253 27025 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:59ms, Total send logs: 2
I0110 22:40:51.352491 27012 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:55ms, Total send logs: 2
I0110 22:40:51.363632 27012 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:58ms, Total send logs: 2
I0110 22:40:56.251555 27026 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:51ms, Total send logs: 2
I0110 22:40:58.939260 27022 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:51ms, Total send logs: 2
I0110 22:41:03.918486 27022 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:55ms, Total send logs: 3
I0110 22:41:03.969966 27028 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:60ms, Total send logs: 3
I0110 22:41:04.181573 27025 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:59ms, Total commit: 1
I0110 22:41:24.193470 27027 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:52ms, Total send logs: 2
I0110 22:41:25.005472 27030 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:55ms, Total send logs: 2
I0110 22:41:25.420480 27033 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:52ms, Total send logs: 2
I0110 22:41:26.206499 27029 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:66ms, Total send logs: 2
I0110 22:41:36.084465 27014 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:57ms, Total send logs: 2
I0110 22:41:48.293582 27010 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:68ms, Total send logs: 2
I0110 22:41:48.334539 27010 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:54ms, Total send logs: 2
I0110 22:41:51.986460 27034 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:71ms, Total send logs: 2
I0110 22:41:55.917476 27027 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:61ms, Total send logs: 2
I0110 22:41:56.711455 27027 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:54ms, Total send logs: 2
I0110 22:41:57.053457 27027 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:55ms, Total send logs: 2
I0110 22:41:58.385624 27039 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:64ms, Total send logs: 2
I0110 22:49:06.170516 27030 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:53ms, Total send logs: 2
I0110 22:50:27.793072 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 12] Clean wals number 1
I0110 22:50:27.796875 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 22] Clean wals number 1
I0110 22:50:27.800595 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 5] Clean wals number 1
I0110 22:50:27.804261 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 17] Clean wals number 1
I0110 22:50:27.807920 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 30] Clean wals number 1
I0110 22:50:27.811575 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 24] Clean wals number 1
I0110 22:50:27.815886 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 3, Part: 23] Clean wals number 1
I0110 22:50:27.819667 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 22] Clean wals number 1
I0110 22:50:27.823386 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 11] Clean wals number 1
I0110 22:50:27.827005 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 18] Clean wals number 1
I0110 22:52:23.085515 27037 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:55ms, Total send logs: 4
I0110 22:52:23.220481 27018 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:57ms, Total send logs: 3
I0110 22:52:37.337481 27035 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:61ms, Total send logs: 3
I0110 22:52:41.195564 27039 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:53ms, Total send logs: 2
I0110 22:52:47.341715 27012 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:61ms, Total send logs: 2
I0110 22:52:57.835579 27011 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:51ms, Total send logs: 2
I0110 22:52:57.891464 27036 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:56ms, Total send logs: 3
I0110 22:54:50.197561 27033 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:54ms, Total send logs: 3
I0110 22:58:27.067374 27010 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:51ms, Total send logs: 6
I0110 22:58:35.955328 27018 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:51ms, Total send logs: 5
I0110 22:58:35.960561 27030 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:58ms, Total send logs: 4
I0110 23:00:27.832541 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 23] Clean wals number 1
I0110 23:00:27.836341 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 18] Clean wals number 1
I0110 23:00:27.840142 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 4] Clean wals number 1
I0110 23:00:27.843865 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 16] Clean wals number 1
I0110 23:00:27.847579 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 11] Clean wals number 1
I0110 23:00:27.851277 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 10] Clean wals number 1
I0110 23:00:27.855142 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 5] Clean wals number 1
I0110 23:00:27.859153 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 28] Clean wals number 1
I0110 23:00:27.862991 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 30] Clean wals number 1
I0110 23:00:27.867059 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 6] Clean wals number 1
I0110 23:00:27.870760 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 29] Clean wals number 1
I0110 23:00:27.874547 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 30] Clean wals number 1
I0110 23:00:27.878975 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 4] Clean wals number 1
I0110 23:00:27.882647 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 28] Clean wals number 1
I0110 23:00:27.886348 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 5] Clean wals number 1
I0110 23:00:27.890503 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 17] Clean wals number 1
I0110 23:00:27.894162 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 29] Clean wals number 1
I0110 23:00:27.897826 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 6] Clean wals number 1
I0110 23:01:06.505457 27017 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:54ms, Total send logs: 3
I0110 23:02:47.783537 27032 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:59ms, Total send logs: 12
I0110 23:02:47.809514 27023 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:63ms, Total send logs: 2
I0110 23:02:47.818539 27017 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:57ms, Total send logs: 2
I0110 23:02:47.859463 27025 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:64ms, Total send logs: 4
I0110 23:02:52.264562 27038 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:52ms, Total send logs: 2
I0110 23:02:52.431485 27036 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:59ms, Total send logs: 2
I0110 23:02:52.452083 27036 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:79ms, Total send logs: 2
I0110 23:02:55.010718 27023 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:67ms, Total send logs: 3
I0110 23:02:55.017539 27012 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:54ms, Total send logs: 3
I0110 23:02:55.828483 27013 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:51ms, Total send logs: 2
I0110 23:02:58.625597 27018 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:51ms, Total send logs: 3
I0110 23:02:59.138460 27038 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:54ms, Total send logs: 2
I0110 23:03:00.090550 27038 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:61ms, Total send logs: 2
I0110 23:03:03.278560 27016 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:66ms, Total send logs: 4
I0110 23:03:04.311137 27028 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:63ms, Total send logs: 2
I0110 23:03:04.409184 27027 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:51ms, Total send logs: 2
I0110 23:03:04.409304 27027 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:52ms, Total send logs: 2
I0110 23:03:04.417460 27016 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:60ms, Total send logs: 2
I0110 23:03:04.417760 27032 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:60ms, Total send logs: 2
I0110 23:03:04.583462 27028 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:68ms, Total send logs: 2
I0110 23:03:04.679539 27027 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:59ms, Total send logs: 2
I0110 23:03:08.035894 27037 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:57ms, Total send logs: 2
I0110 23:03:08.143661 27029 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:56ms, Total send logs: 5
I0110 23:03:08.146926 27035 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:59ms, Total send logs: 6
I0110 23:03:20.821528 27030 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:54ms, Total send logs: 2
I0110 23:03:20.857472 27030 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:90ms, Total send logs: 2
I0110 23:03:20.881520 27031 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:60ms, Total send logs: 2
I0110 23:03:20.905473 27030 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:90ms, Total send logs: 2
I0110 23:03:20.911537 27032 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:54ms, Total send logs: 2
I0110 23:03:20.941542 27022 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:54ms, Total send logs: 2
I0110 23:03:21.060469 27029 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:77ms, Total send logs: 2
I0110 23:03:22.945688 27032 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:74ms, Total send logs: 2
I0110 23:03:23.568572 27033 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:63ms, Total send logs: 2
I0110 23:03:23.574528 27033 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:77ms, Total send logs: 2
I0110 23:03:23.580474 27033 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:83ms, Total send logs: 2
I0110 23:03:28.490486 27039 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:58ms, Total commit: 1
I0110 23:03:47.348524 27029 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:55ms, Total send logs: 2
I0110 23:08:44.460458 27009 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:54ms, Total send logs: 2
I0110 23:08:44.750617 27019 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:58ms, Total send logs: 4
I0110 23:08:44.794479 27022 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:102ms, Total send logs: 4
I0110 23:08:44.805519 27022 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:57ms, Total send logs: 2
I0110 23:08:57.731561 27019 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:65ms, Total send logs: 4
I0110 23:08:59.019600 27010 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:53ms, Total send logs: 3
I0110 23:09:11.994134 27027 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:60ms, Total send logs: 2
I0110 23:09:14.732486 27010 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:51ms, Total send logs: 4
I0110 23:09:14.761462 27010 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:57ms, Total send logs: 3
I0110 23:09:18.448477 27018 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:58ms, Total send logs: 8
I0110 23:09:19.773550 27038 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:71ms, Total send logs: 2
I0110 23:09:21.656682 27018 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:52ms, Total send logs: 3
I0110 23:09:24.852479 27021 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:55ms, Total send logs: 2
I0110 23:09:26.273501 27021 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:56ms, Total send logs: 3
I0110 23:09:26.335641 27016 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:52ms, Total send logs: 2
I0110 23:09:26.436455 27010 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:53ms, Total send logs: 2
I0110 23:09:26.770535 27009 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:52ms, Total send logs: 2
I0110 23:09:26.816550 27019 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:85ms, Total send logs: 2
I0110 23:09:26.821673 27018 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:55ms, Total send logs: 2
I0110 23:09:26.827518 27009 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:57ms, Total send logs: 2
I0110 23:09:34.101457 27010 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:100ms, Total send logs: 2
I0110 23:09:34.101604 27010 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:55ms, Total send logs: 2
I0110 23:09:34.672576 27014 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:65ms, Total send logs: 2
I0110 23:09:34.677621 27014 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:52ms, Total send logs: 2
I0110 23:09:34.759480 27015 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:54ms, Total send logs: 2
I0110 23:09:36.443534 27033 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:51ms, Total send logs: 2
I0110 23:09:36.527518 27027 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:53ms, Total send logs: 3
I0110 23:09:37.473493 27015 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:61ms, Total send logs: 2
I0110 23:09:37.515605 27010 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:68ms, Total send logs: 2
I0110 23:09:43.422499 27012 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:60ms, Total send logs: 6
I0110 23:09:45.516481 27030 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:52ms, Total send logs: 2
I0110 23:09:45.527458 27030 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:64ms, Total send logs: 2
I0110 23:09:50.637476 27016 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:54ms, Total send logs: 2
I0110 23:09:51.416529 27035 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:59ms, Total send logs: 2
I0110 23:09:52.818519 27014 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:51ms, Total send logs: 2
I0110 23:09:54.374480 27019 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:64ms, Total send logs: 2
I0110 23:09:55.618595 27010 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:59ms, Total send logs: 2
I0110 23:09:55.698467 27036 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:51ms, Total send logs: 2
I0110 23:09:55.779001 27036 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:55ms, Total send logs: 2
I0110 23:09:59.714547 27010 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:54ms, Total send logs: 2
I0110 23:09:59.725534 27020 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:52ms, Total send logs: 2
I0110 23:10:00.437595 27019 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:64ms, Total send logs: 2
I0110 23:10:00.449543 27019 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:76ms, Total send logs: 2
I0110 23:10:00.459195 27019 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:54ms, Total send logs: 2
I0110 23:10:02.180090 27009 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:53ms, Total send logs: 3
I0110 23:10:02.304921 27019 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:61ms, Total send logs: 2
I0110 23:10:02.317993 27019 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:74ms, Total send logs: 2
I0110 23:10:17.998590 27020 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:66ms, Total send logs: 2
I0110 23:10:18.028478 27030 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:79ms, Total send logs: 5
I0110 23:10:18.035583 27037 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:77ms, Total send logs: 5
I0110 23:10:18.039451 27030 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:74ms, Total send logs: 5
I0110 23:10:20.492473 27037 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:55ms, Total send logs: 2
I0110 23:10:21.337527 27014 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:51ms, Total send logs: 2
I0110 23:10:27.913671 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 12] Clean wals number 1
I0110 23:10:27.919622 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 23] Clean wals number 1
I0110 23:10:27.938459 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 22] Clean wals number 1
I0110 23:10:27.943574 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 18] Clean wals number 1
I0110 23:10:27.959857 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 16] Clean wals number 1
I0110 23:10:27.965137 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 11] Clean wals number 1
I0110 23:10:27.970278 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 10] Clean wals number 1
I0110 23:10:27.989297 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 17] Clean wals number 1
I0110 23:10:27.994446 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 24] Clean wals number 1
I0110 23:10:27.999565 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 6] Clean wals number 1
I0110 23:10:28.004732 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 29] Clean wals number 1
I0110 23:10:28.029563 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 3, Part: 30] Clean wals number 1
I0110 23:10:28.619599 27016 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:53ms, Total send logs: 2
I0110 23:10:32.049929 27012 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:60ms, Total send logs: 4
I0110 23:10:32.070574 27039 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:77ms, Total send logs: 5
I0110 23:10:32.105482 27016 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:51ms, Total send logs: 2
I0110 23:10:41.329474 27025 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:52ms, Total send logs: 2
I0110 23:10:43.407770 27013 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:70ms, Total send logs: 4
I0110 23:10:44.081573 27025 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:51ms, Total send logs: 2
I0110 23:10:44.143461 27016 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:61ms, Total send logs: 2
I0110 23:10:44.165457 27016 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:58ms, Total send logs: 2
I0110 23:10:44.398532 27013 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:65ms, Total send logs: 2
I0110 23:10:46.058460 27019 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:53ms, Total send logs: 2
I0110 23:10:46.094461 27016 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:51ms, Total send logs: 3
I0110 23:10:46.105515 27016 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:57ms, Total send logs: 4
I0110 23:10:46.865469 27010 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:52ms, Total send logs: 2
I0110 23:10:49.585480 27037 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:60ms, Total send logs: 2
I0110 23:10:58.103536 27034 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:81ms, Total send logs: 2
I0110 23:10:58.109401 27034 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:88ms, Total send logs: 2
I0110 23:11:03.497642 27010 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:68ms, Total send logs: 8
I0110 23:11:03.531543 27035 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:72ms, Total send logs: 2
I0110 23:11:03.559589 27018 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:54ms, Total send logs: 6
I0110 23:11:03.571480 27035 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:56ms, Total send logs: 7
I0110 23:11:03.573151 27035 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:69ms, Total send logs: 5
I0110 23:11:03.586525 27035 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:54ms, Total send logs: 9
I0110 23:11:05.039527 27016 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:57ms, Total send logs: 2
I0110 23:11:13.223487 27031 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:52ms, Total send logs: 2
I0110 23:11:14.183557 27017 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:60ms, Total send logs: 3
I0110 23:11:14.244979 27022 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:53ms, Total send logs: 2
I0110 23:11:15.306525 27031 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:60ms, Total send logs: 2
I0110 23:11:15.354485 27029 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:55ms, Total send logs: 5
I0110 23:11:15.368537 27029 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:57ms, Total send logs: 6
I0110 23:11:16.566454 27029 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:51ms, Total commit: 1
I0110 23:11:17.943523 27030 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:52ms, Total commit: 1
I0110 23:11:17.945536 27009 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:56ms, Total commit: 1
I0110 23:11:17.953501 27016 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:53ms, Total commit: 1
I0110 23:11:18.834554 27034 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:56ms, Total send logs: 3
I0110 23:11:19.780874 27031 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:69ms, Total send logs: 3
I0110 23:11:19.881520 27027 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:77ms, Total send logs: 3
I0110 23:11:23.278903 27029 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:60ms, Total send logs: 3
I0110 23:11:24.134085 27029 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:60ms, Total send logs: 3
I0110 23:11:24.148499 27029 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:90ms, Total send logs: 3
I0110 23:11:24.207475 27027 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:59ms, Total send logs: 5
I0110 23:11:24.633458 27035 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:72ms, Total send logs: 2
I0110 23:11:24.961129 27013 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:51ms, Total send logs: 2
I0110 23:11:25.450094 27013 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:61ms, Total send logs: 3
I0110 23:11:25.452077 27013 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:53ms, Total send logs: 2
I0110 23:11:25.508059 27013 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:56ms, Total send logs: 2
I0110 23:11:25.513054 27013 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:63ms, Total send logs: 2
I0110 23:11:26.473910 27034 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:51ms, Total send logs: 2
I0110 23:11:26.860174 27018 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:55ms, Total send logs: 3
I0110 23:11:26.863291 27018 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:62ms, Total send logs: 3
I0110 23:11:27.410889 27013 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:60ms, Total send logs: 2
I0110 23:11:27.459147 27013 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:51ms, Total send logs: 2
I0110 23:11:28.498929 27018 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:56ms, Total send logs: 2
I0110 23:11:28.581171 27034 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:55ms, Total send logs: 2
I0110 23:20:28.039469 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 22] Clean wals number 1
I0110 23:20:28.043247 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 4] Clean wals number 1
I0110 23:20:28.047010 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 5] Clean wals number 1
I0110 23:20:28.050747 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 28] Clean wals number 1
I0110 23:20:28.054505 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 17] Clean wals number 1
I0110 23:20:28.058204 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 30] Clean wals number 1
I0110 23:20:28.062248 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 24] Clean wals number 1
I0110 23:20:28.066066 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 24] Clean wals number 1
I0110 23:20:28.069931 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 16] Clean wals number 1
I0110 23:20:28.073648 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 10] Clean wals number 1
I0110 23:20:28.077489 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 12] Clean wals number 1
I0110 23:20:28.081605 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 23] Clean wals number 1
I0110 23:20:50.344552 27008 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:55ms, Total send logs: 2
I0110 23:20:50.368543 27025 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:59ms, Total send logs: 3
I0110 23:20:50.429534 27010 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:62ms, Total send logs: 2
I0110 23:20:50.429658 27010 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:61ms, Total send logs: 2
I0110 23:20:55.962460 27025 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:67ms, Total send logs: 2
I0110 23:20:55.971458 27025 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:76ms, Total send logs: 2
I0110 23:21:09.339498 27034 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:79ms, Total send logs: 2
I0110 23:21:09.386504 27015 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:65ms, Total send logs: 2
I0110 23:21:09.388473 27029 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:68ms, Total send logs: 2
I0110 23:21:09.972486 27020 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:62ms, Total send logs: 2
I0110 23:21:09.980476 27020 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:53ms, Total send logs: 2
I0110 23:21:12.122500 27008 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:55ms, Total send logs: 2
I0110 23:21:12.131453 27008 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:63ms, Total send logs: 2
I0110 23:21:12.737468 27034 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:66ms, Total send logs: 2
I0110 23:21:12.773480 27038 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:54ms, Total send logs: 2
I0110 23:21:12.791457 27038 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:79ms, Total send logs: 2
I0110 23:21:12.800463 27038 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:63ms, Total send logs: 2
I0110 23:21:13.950476 27035 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:52ms, Total send logs: 5
I0110 23:21:17.705464 27038 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:62ms, Total send logs: 2
I0110 23:21:18.021462 27015 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:67ms, Total send logs: 2
I0110 23:21:18.872153 27034 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:59ms, Total send logs: 2
I0110 23:21:20.741479 27036 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:59ms, Total send logs: 2
I0110 23:21:27.375672 27009 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:58ms, Total send logs: 2
I0110 23:21:28.034524 27009 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:53ms, Total send logs: 2
I0110 23:21:28.034540 27020 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:53ms, Total send logs: 2
I0110 23:21:29.792537 27039 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:57ms, Total send logs: 2
I0110 23:21:31.260502 27023 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:68ms, Total send logs: 2
I0110 23:21:44.943466 27028 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:51ms, Total send logs: 2
I0110 23:21:55.992470 27031 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:54ms, Total send logs: 2
I0110 23:22:23.654145 27020 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:51ms, Total commit: 2
I0110 23:22:41.342967 27027 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:53ms, Total commit: 1
I0110 23:22:41.354722 27013 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:54ms, Total commit: 1
I0110 23:27:48.711534 27037 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:54ms, Total send logs: 2
I0110 23:30:28.095558 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 12] Clean wals number 1
I0110 23:30:28.099526 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 23] Clean wals number 1
I0110 23:30:28.103543 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 18] Clean wals number 1
I0110 23:30:28.107462 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 4] Clean wals number 1
I0110 23:30:28.111373 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 16] Clean wals number 1
I0110 23:30:28.115257 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 11] Clean wals number 1
I0110 23:30:28.119051 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 10] Clean wals number 1
I0110 23:30:28.122822 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 5] Clean wals number 1
I0110 23:30:28.126585 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 6] Clean wals number 1
I0110 23:30:28.130362 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 29] Clean wals number 1
I0110 23:30:28.135056 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 22] Clean wals number 1
I0110 23:30:28.139173 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 4] Clean wals number 1
I0110 23:30:28.142937 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 28] Clean wals number 1
I0110 23:30:28.146903 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 5] Clean wals number 1
I0110 23:30:28.150771 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 11] Clean wals number 1
I0110 23:30:28.154716 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 18] Clean wals number 1
I0110 23:33:55.339596 27026 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:54ms, Total send logs: 2
I0110 23:33:55.442685 27019 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:54ms, Total send logs: 2
I0110 23:33:55.442798 27019 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:66ms, Total send logs: 2
I0110 23:38:08.568483 27013 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:80ms, Total send logs: 2
I0110 23:38:08.637531 27013 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:59ms, Total send logs: 2
I0110 23:38:08.669489 27017 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:57ms, Total send logs: 2
I0110 23:38:13.797513 27013 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:54ms, Total send logs: 2
I0110 23:38:20.312492 27027 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:54ms, Total send logs: 2
I0110 23:40:28.170323 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 22] Clean wals number 1
I0110 23:40:28.175784 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 28] Clean wals number 1
I0110 23:40:28.180982 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 17] Clean wals number 1
I0110 23:40:28.195363 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 30] Clean wals number 1
I0110 23:40:28.213536 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 24] Clean wals number 1
I0110 23:40:28.218904 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 30] Clean wals number 1
I0110 23:40:28.225309 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 29] Clean wals number 1
I0110 23:40:28.230537 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 6] Clean wals number 1
I0110 23:44:23.230495 27024 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:54ms, Total send logs: 3
I0110 23:44:23.279474 27038 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:78ms, Total send logs: 2
I0110 23:44:23.526645 27010 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:55ms, Total send logs: 2
I0110 23:44:23.557482 27016 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:53ms, Total send logs: 2
I0110 23:44:23.664459 27012 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:100ms, Total send logs: 2
I0110 23:44:45.213591 27009 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:53ms, Total send logs: 2
I0110 23:44:46.658481 27026 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:66ms, Total send logs: 2
I0110 23:44:48.331471 27038 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:60ms, Total send logs: 2
I0110 23:44:48.348598 27008 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:75ms, Total send logs: 3
I0110 23:44:48.406462 27015 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:57ms, Total send logs: 2
I0110 23:44:52.714569 27031 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:57ms, Total send logs: 2
I0110 23:44:56.613546 27014 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:54ms, Total send logs: 2
I0110 23:44:56.646545 27032 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:57ms, Total send logs: 2
I0110 23:44:56.664644 27032 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:54ms, Total send logs: 2
I0110 23:44:56.664691 27033 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:54ms, Total send logs: 2
I0110 23:44:56.664748 27032 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:51ms, Total send logs: 4
I0110 23:44:56.773458 27038 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:61ms, Total send logs: 3
I0110 23:44:56.773609 27038 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:62ms, Total send logs: 4
I0110 23:44:56.793702 27022 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:82ms, Total send logs: 4
I0110 23:44:57.734740 27036 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:69ms, Total send logs: 2
I0110 23:44:57.738577 27037 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:51ms, Total send logs: 5
I0110 23:44:57.807484 27023 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:58ms, Total send logs: 3
I0110 23:50:12.082490 27024 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:72ms, Total send logs: 2
I0110 23:50:12.084218 27039 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:74ms, Total send logs: 2
I0110 23:50:12.298882 27039 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:52ms, Total send logs: 4
I0110 23:50:12.374565 27029 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:56ms, Total send logs: 2
I0110 23:50:12.436511 27029 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:52ms, Total send logs: 2
I0110 23:50:22.030246 27024 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:57ms, Total send logs: 2
I0110 23:50:22.038486 27038 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:66ms, Total send logs: 2
I0110 23:50:22.123535 27038 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:85ms, Total send logs: 8
I0110 23:50:26.945735 27019 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:51ms, Total send logs: 2
I0110 23:50:26.947469 27019 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:52ms, Total send logs: 2
I0110 23:50:28.249176 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 12] Clean wals number 1
I0110 23:50:28.254637 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 23] Clean wals number 1
I0110 23:50:28.259773 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 18] Clean wals number 1
I0110 23:50:28.276858 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 4] Clean wals number 1
I0110 23:50:28.283690 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 16] Clean wals number 1
I0110 23:50:28.293056 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 11] Clean wals number 1
I0110 23:50:28.345961 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 10] Clean wals number 1
I0110 23:50:28.357466 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 5] Clean wals number 1
I0110 23:50:28.363656 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 28] Clean wals number 1
I0110 23:50:28.372274 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 6] Clean wals number 1
I0110 23:50:28.388489 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 53, Part: 29] Clean wals number 1
I0110 23:50:28.443109 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 24] Clean wals number 1
I0110 23:50:28.448801 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 16] Clean wals number 1
I0110 23:50:28.463176 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 10] Clean wals number 1
I0110 23:50:28.485682 27073 FileBasedWal.cpp:672] [Port: 9780, Space: 30, Part: 17] Clean wals number 1
I0110 23:50:36.157974 27028 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:52ms, Total send logs: 3
I0110 23:50:36.177594 27028 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:77ms, Total send logs: 2
I0110 23:50:47.013458 27008 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:56ms, Total send logs: 2
I0110 23:50:47.039477 27016 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:57ms, Total send logs: 9
I0110 23:51:51.638468 27027 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:52ms, Total send logs: 2
I0110 23:51:51.694559 27010 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:51ms, Total send logs: 2
I0110 23:51:51.694671 27010 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:56ms, Total send logs: 2
I0110 23:51:53.028704 27010 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:61ms, Total send logs: 3
I0110 23:51:53.037305 27010 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:72ms, Total send logs: 2
I0110 23:51:55.589496 27029 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:54ms, Total send logs: 2
I0110 23:51:55.776535 27029 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:69ms, Total send logs: 2
I0110 23:52:09.898515 27013 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:58ms, Total send logs: 2
I0110 23:52:11.634474 27028 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:70ms, Total send logs: 2
I0110 23:52:12.049070 27034 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:77ms, Total send logs: 2
I0110 23:52:16.518527 27012 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:54ms, Total send logs: 2
I0110 23:52:16.529541 27011 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:53ms, Total send logs: 2
I0110 23:52:20.017488 27021 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:53ms, Total send logs: 4
I0110 23:52:29.447531 27023 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:65ms, Total send logs: 2
I0110 23:52:51.402501 27027 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:57ms, Total send logs: 2
I0110 23:53:01.271553 27035 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:53ms, Total send logs: 2
I0110 23:53:46.526803 27039 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:51ms, Total send logs: 4
I0110 23:53:46.536326 27011 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:66ms, Total send logs: 4
I0110 23:53:46.599480 27017 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:62ms, Total send logs: 4
I0110 23:53:48.470472 27024 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:52ms, Total send logs: 2
I0110 23:53:48.479455 27024 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:61ms, Total send logs: 2
I0110 23:53:48.501461 27024 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:82ms, Total send logs: 2
I0110 23:53:50.129534 27039 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:59ms, Total send logs: 2
I0110 23:53:52.200457 27024 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:58ms, Total send logs: 2
I0110 23:53:53.215641 27025 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:64ms, Total send logs: 2
I0110 23:53:58.390488 27038 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:73ms, Total send logs: 3
I0110 23:53:58.406596 27010 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:54ms, Total send logs: 2
I0110 23:53:58.573571 27030 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:65ms, Total send logs: 2
I0110 23:53:58.626502 27026 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:53ms, Total send logs: 2
I0110 23:53:58.677462 27021 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:51ms, Total send logs: 2
I0110 23:53:59.576491 27021 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:79ms, Total send logs: 2
I0110 23:53:59.582540 27039 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:70ms, Total send logs: 2
I0110 23:54:03.018465 27037 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:76ms, Total send logs: 7
I0110 23:54:07.848594 27019 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:52ms, Total send logs: 2
I0110 23:54:20.450007 27017 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:55ms, Total send logs: 2
I0110 23:55:38.480460 27024 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:52ms, Total send logs: 3
I0110 23:58:22.198475 27014 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:78ms, Total send logs: 2
I0110 23:58:22.207458 27014 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:87ms, Total send logs: 2
I0110 23:58:35.970958 27031 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:79ms, Total send logs: 2
I0110 23:58:36.109463 27025 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:52ms, Total send logs: 2
I0110 23:58:36.113507 27025 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:64ms, Total send logs: 2
I0110 23:58:51.614477 27034 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 10] total time:52ms, Total send logs: 3
I0110 23:58:55.794553 27010 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 12] total time:56ms, Total send logs: 2
I0110 23:58:55.795641 27017 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:57ms, Total send logs: 2
I0110 23:58:55.876457 27010 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 24] total time:52ms, Total send logs: 2
I0110 23:58:55.887493 27012 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 22] total time:72ms, Total send logs: 2
I0110 23:59:00.970533 27018 SlowOpTracker.h:31] [Port: 9780, Space: 30, Part: 18] total time:51ms, Total commit: 1

猜测是不是在做compact，你确认下

然后看看storage log，那段时间的log 贴一下

日志中有compact操作

用perf 工具，把信息贴一下

fisherman · 2022 年1 月 11 日 06:10

perf

不过现在cpu正常，不确定现在能否看出来进程中哪些调用cpu耗时比较久

fisherman · 2022 年1 月 12 日 02:25

ping @pandasheeps

liuyu85cn · 2022 年1 月 14 日 03:16

稳稳的在做 compaction 啊 , 看你 log 都是 raft 在做 insert. inset 完之后肯定 rocksdb 会做 compaction 的.

fisherman · 2022 年1 月 14 日 03:28

每天都会有数据更新，所以每天都会进行自动compact，但是就那天晚上cpu使用率直接100%，所以有疑惑

system · 2022 年2 月 13 日 03:28

此话题已在最后回复的 30 天后被自动关闭。不再允许新回复。