nebula storage 一启动就oom

liuqian1990 · 2022 年3 月 15 日 02:08

nebula 2.6.2

--daemonize=true
--pid_file=pids/nebula-storaged.pid
--local_config=true

--log_dir=logs
--minloglevel=0
--v=0
--logbufsecs=0
--redirect_stdout=true
--stdout_log_file=storaged-stdout.log
--stderr_log_file=storaged-stderr.log
--stderrthreshold=2

--meta_server_addrs=
--local_ip=
--port=9779
--ws_ip=0.0.0.0
--ws_http_port=19779
--ws_h2_port=19780
--heartbeat_interval_secs=10

--raft_heartbeat_interval_secs=30
--raft_rpc_timeout_ms=500
--wal_ttl=600

--data_path=/data/nebula/data/storage

--minimum_reserved_bytes=268435456

--rocksdb_batch_size=4096
--rocksdb_block_cache=4

--rocksdb_compression=lz4

--rocksdb_compression_per_level=

--rocksdb_db_options={"max_subcompactions":"4","max_background_jobs":"4"}
--rocksdb_column_family_options={"disable_auto_compactions":"false","write_buffer_size":"67108864","max_write_buffer_number":"4","max_bytes_for_level_base":"268435456"}
--rocksdb_block_based_table_options={"block_size":"8192"}

--enable_rocksdb_statistics=false

--rocksdb_stats_level=kExceptHistogramOrTimers

--enable_rocksdb_prefix_filtering=true
--enable_rocksdb_whole_key_filtering=false
--enable_partitioned_index_filter=true

--snapshot_part_rate_limit=8388608
--snapshot_batch_size=1048576
--rebuild_index_part_rate_limit=4194304
--rebuild_index_batch_size=1048576
[s-user@iZbp1702pfcdtieotkgp50Z logs]$ cat ../etc/nebula-storaged.conf | grep -v "^#" | grep -v "^"
[s-user@iZbp1702pfcdtieotkgp50Z logs]$ cat ../etc/nebula-storaged.conf | grep -v "^#"
--daemonize=true
--pid_file=pids/nebula-storaged.pid
--local_config=true

--log_dir=logs
--minloglevel=0
--v=0
--logbufsecs=0
--redirect_stdout=true
--stdout_log_file=storaged-stdout.log
--stderr_log_file=storaged-stderr.log
--stderrthreshold=2

--meta_server_addrs=
--local_ip=
--port=9779
--ws_ip=0.0.0.0
--ws_http_port=19779
--ws_h2_port=19780
--heartbeat_interval_secs=10

--raft_heartbeat_interval_secs=30
--raft_rpc_timeout_ms=500
--wal_ttl=600

--data_path=/data/nebula/data/storage

--minimum_reserved_bytes=268435456

--rocksdb_batch_size=4096
--rocksdb_block_cache=4

--rocksdb_compression=lz4

--rocksdb_compression_per_level=

--rocksdb_db_options={"max_subcompactions":"4","max_background_jobs":"4"}
--rocksdb_column_family_options={"disable_auto_compactions":"false","write_buffer_size":"67108864","max_write_buffer_number":"4","max_bytes_for_level_base":"268435456"}
--rocksdb_block_based_table_options={"block_size":"8192"}

--enable_rocksdb_statistics=false

--rocksdb_stats_level=kExceptHistogramOrTimers

--enable_rocksdb_prefix_filtering=true
--enable_rocksdb_whole_key_filtering=false
--enable_partitioned_index_filter=true

--snapshot_part_rate_limit=8388608
--snapshot_batch_size=1048576
--rebuild_index_part_rate_limit=4194304
--rebuild_index_batch_size=1048576

steam · 2022 年3 月 15 日 02:10

机器配置贴一下

liuqian1990 · 2022 年3 月 15 日 03:37

16核 64G 单个节点数据 60多G

liuqian1990 · 2022 年3 月 15 日 03:38

看了相关日志流程是在load数据

steam · 2022 年3 月 15 日 03:41

grep -i ‘killed process’ /var/log/messages

执行下，看下 OOM 时候的记录。

liuqian1990 · 2022 年3 月 15 日 03:43

Mar 15 10:41:08 iZbp1702pfcdtieotkgp4zZ kernel: Killed process 27937 (nebula-storaged), UID 0, total-vm:78942468kB, anon-rss:63664528kB, file-rss:0kB, shmem-rss:0kB

critical27 · 2022 年3 月 15 日 03:48

数据量很大吗？在storage里面加个参数enable_partitioned_index_filter=true

liuqian1990 · 2022 年3 月 15 日 03:48

这个加过了，没有用

liuqian1990 · 2022 年3 月 15 日 03:50

du -sh /data/nebula/data/storage/nebula/*
95M	/data/nebula/data/storage/nebula/0
133M	/data/nebula/data/storage/nebula/1
59G	/data/nebula/data/storage/nebula/2
158G	/data/nebula/data/storage/nebula/3

du -sh /data/nebula/data/storage/nebula/3/*
54G	/data/nebula/data/storage/nebula/3/data
104G	/data/nebula/data/storage/nebula/3/wal

du -sh /data/nebula/data/storage/nebula/2/*
59G	/data/nebula/data/storage/nebula/2/data
1.5M	/data/nebula/data/storage/nebula/2/wal

liuqian1990 · 2022 年3 月 15 日 03:53

加了这个enable_partitioned_index_filter=true参数，一样启动内存就持续增长

[root@iZbp1702pfcdtieotkgp4zZ nebula]# free -g
              total        used        free      shared  buff/cache   available
Mem:             62          28          21           0          11          33
Swap:             0           0           0
[root@iZbp1702pfcdtieotkgp4zZ nebula]# free -g
              total        used        free      shared  buff/cache   available
Mem:             62          40           5           0          16          21
Swap:             0           0           0
[root@iZbp1702pfcdtieotkgp4zZ nebula]# free -g
              total        used        free      shared  buff/cache   available
Mem:             62          44           0           0          17          17
Swap:             0           0           0
[root@iZbp1702pfcdtieotkgp4zZ nebula]# free -g
              total        used        free      shared  buff/cache   available
Mem:             62          49           0           0          13          12
Swap:             0           0           0
[root@iZbp1702pfcdtieotkgp4zZ nebula]# free -g
              total        used        free      shared  buff/cache   available
Mem:             62          53           0           0           9           9
Swap:             0           0           0```

critical27 · 2022 年3 月 15 日 04:02

不好说大概率是rocksdb的compaction 你可以看看storage日志和rocksdb日志或者都传上来最好再用perf记录下启动时候的在干啥

liuqian1990 · 2022 年3 月 15 日 04:32

nebula-storge

Log file created at: 2022/03/15 12:30:07
Running on machine: iZbp1702pfcdtieotkgp4yZ
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0315 12:30:07.160270 49321 StorageDaemon.cpp:125] localhost = "10.226.14.134":9779
I0315 12:30:07.197847 49321 MetaClient.cpp:57] Create meta client to "10.226.14.134":9559
I0315 12:30:07.202709 49321 FileBasedClusterIdMan.cpp:53] Get clusterId: 1416130955739936920
I0315 12:30:07.373330 49321 MetaClient.cpp:2994] Load leader of "10.226.14.132":9779 in 0 space
I0315 12:30:07.373371 49321 MetaClient.cpp:2994] Load leader of "10.226.14.133":9779 in 0 space
I0315 12:30:07.373378 49321 MetaClient.cpp:2994] Load leader of "10.226.14.134":9779 in 2 space
I0315 12:30:07.373383 49321 MetaClient.cpp:3000] Load leader ok
I0315 12:30:07.374137 49321 MetaClient.cpp:117] Register time task for heartbeat!
I0315 12:30:07.374156 49321 StorageServer.cpp:167] Init schema manager
I0315 12:30:07.374163 49321 StorageServer.cpp:170] Init index manager
I0315 12:30:07.374168 49321 StorageServer.cpp:173] Init kvstore
I0315 12:30:07.374217 49321 NebulaStore.cpp:52] Start the raft service...
I0315 12:30:07.379730 49321 NebulaSnapshotManager.cpp:25] Send snapshot is rate limited to 8388608 for each part by default
I0315 12:30:07.379899 49321 RaftexService.cpp:62] Init thrift server for raft service, port: 9780
I0315 12:30:07.381749 49420 RaftexService.cpp:93] Starting the Raftex Service
I0315 12:30:07.394148 49420 RaftexService.cpp:83] Starting the Raftex Service on 9780
I0315 12:30:07.394172 49420 RaftexService.cpp:103] Start the Raftex Service successfully
I0315 12:30:07.394310 49321 NebulaStore.cpp:84] Scan the local path, and init the spaces_
I0315 12:30:07.394510 49321 NebulaStore.cpp:90] Scan path "/data/nebula/data/storage/nebula/0"
I0315 12:30:07.394522 49321 NebulaStore.cpp:90] Scan path "/data/nebula/data/storage/nebula/3"
I0315 12:30:07.394659 49321 RocksEngineConfig.cpp:299] Emplace rocksdb option max_background_jobs=0
I0315 12:30:07.394673 49321 RocksEngineConfig.cpp:299] Emplace rocksdb option max_subcompactions=0
I0315 12:30:07.394801 49321 RocksEngineConfig.cpp:299] Emplace rocksdb option max_bytes_for_level_base=268435456
I0315 12:30:07.394809 49321 RocksEngineConfig.cpp:299] Emplace rocksdb option max_write_buffer_number=4
I0315 12:30:07.394814 49321 RocksEngineConfig.cpp:299] Emplace rocksdb option write_buffer_size=67108864
I0315 12:30:07.394819 49321 RocksEngineConfig.cpp:299] Emplace rocksdb option disable_auto_compactions=false
I0315 12:30:07.394893 49321 RocksEngineConfig.cpp:299] Emplace rocksdb option block_size=8192
I0315 12:30:18.748127 49422 EventListener.h:18] Rocksdb start compaction column family: default because of LevelMaxLevelSize, status: OK, compacted 4 files into 0, base level is 2, output level is 3
I0315 12:30:18.748214 49422 CompactionFilter.h:58] Do custom minor compaction!
I0315 12:30:18.907646 49321 RocksEngine.cpp:128] open rocksdb on /data/nebula/data/storage/nebula/3/data
I0315 12:30:18.907677 49321 NebulaStore.cpp:114] Load space 3 from disk
I0315 12:30:19.060806 49321 NebulaStore.cpp:142] Need to open 5 parts of space 3
I0315 12:30:19.113682 49415 FileBasedWal.cpp:66] [Port: 9780, Space: 3, Part: 2] lastLogId in wal is 4309538, lastLogTerm is 200, path is /data/nebula/data/storage/nebula/3/wal/2/0000000000004308523.wal
I0315 12:30:19.113870 49415 RaftPart.cpp:278] [Port: 9780, Space: 3, Part: 2] There are 0 peer hosts, and total 1 copies. The quorum is 1, as learner 0, lastLogId 4309538, lastLogTerm 200, committedLogId 4309538, term 200
I0315 12:30:19.113979 49415 NebulaStore.cpp:146] Load part 3, 2 from disk
I0315 12:30:19.120131 49417 FileBasedWal.cpp:66] [Port: 9780, Space: 3, Part: 8] lastLogId in wal is 4311499, lastLogTerm is 200, path is /data/nebula/data/storage/nebula/3/wal/8/0000000000004310121.wal
I0315 12:30:19.120308 49417 RaftPart.cpp:278] [Port: 9780, Space: 3, Part: 8] There are 0 peer hosts, and total 1 copies. The quorum is 1, as learner 0, lastLogId 4311499, lastLogTerm 200, committedLogId 4311499, term 200
I0315 12:30:19.120420 49417 NebulaStore.cpp:146] Load part 3, 8 from disk
I0315 12:30:19.123147 49418 FileBasedWal.cpp:66] [Port: 9780, Space: 3, Part: 11] lastLogId in wal is 4377355, lastLogTerm is 200, path is /data/nebula/data/storage/nebula/3/wal/11/0000000000004376747.wal
I0315 12:30:19.123265 49418 RaftPart.cpp:278] [Port: 9780, Space: 3, Part: 11] There are 0 peer hosts, and total 1 copies. The quorum is 1, as learner 0, lastLogId 4377355, lastLogTerm 200, committedLogId 4377355, term 200
I0315 12:30:19.123369 49418 NebulaStore.cpp:146] Load part 3, 11 from disk
I0315 12:30:19.130182 49416 FileBasedWal.cpp:66] [Port: 9780, Space: 3, Part: 5] lastLogId in wal is 4291429, lastLogTerm is 200, path is /data/nebula/data/storage/nebula/3/wal/5/0000000000004288043.wal
I0315 12:30:19.130318 49416 RaftPart.cpp:278] [Port: 9780, Space: 3, Part: 5] There are 0 peer hosts, and total 1 copies. The quorum is 1, as learner 0, lastLogId 4291429, lastLogTerm 200, committedLogId 4291429, term 200
I0315 12:30:19.130424 49416 NebulaStore.cpp:146] Load part 3, 5 from disk
I0315 12:30:19.164666 49415 FileBasedWal.cpp:66] [Port: 9780, Space: 3, Part: 14] lastLogId in wal is 4354597, lastLogTerm is 200, path is /data/nebula/data/storage/nebula/3/wal/14/0000000000004354152.wal
I0315 12:30:19.164772 49415 RaftPart.cpp:278] [Port: 9780, Space: 3, Part: 14] There are 0 peer hosts, and total 1 copies. The quorum is 1, as learner 0, lastLogId 4354597, lastLogTerm 200, committedLogId 4354597, term 200
I0315 12:30:19.164825 49415 NebulaStore.cpp:146] Load part 3, 14 from disk
I0315 12:30:19.164861 49321 NebulaStore.cpp:161] Load space 3 complete
I0315 12:30:19.164882 49321 NebulaStore.cpp:90] Scan path "/data/nebula/data/storage/nebula/1"
I0315 12:30:19.164928 49321 RocksEngineConfig.cpp:299] Emplace rocksdb option max_background_jobs=0
I0315 12:30:19.164937 49321 RocksEngineConfig.cpp:299] Emplace rocksdb option max_subcompactions=0
I0315 12:30:19.164976 49321 RocksEngineConfig.cpp:299] Emplace rocksdb option max_bytes_for_level_base=268435456
I0315 12:30:19.164981 49321 RocksEngineConfig.cpp:299] Emplace rocksdb option max_write_buffer_number=4
I0315 12:30:19.164986 49321 RocksEngineConfig.cpp:299] Emplace rocksdb option write_buffer_size=67108864
I0315 12:30:19.164991 49321 RocksEngineConfig.cpp:299] Emplace rocksdb option disable_auto_compactions=false
I0315 12:30:19.165027 49321 RocksEngineConfig.cpp:299] Emplace rocksdb option block_size=8192
I0315 12:30:19.172685 49321 RocksEngine.cpp:128] open rocksdb on /data/nebula/data/storage/nebula/1/data
I0315 12:30:19.172715 49321 NebulaStore.cpp:114] Load space 1 from disk
I0315 12:30:19.172775 49321 NebulaStore.cpp:142] Need to open 5 parts of space 1
I0315 12:30:19.480192 49415 FileBasedWal.cpp:66] [Port: 9780, Space: 1, Part: 8] lastLogId in wal is 106609, lastLogTerm is 201, path is /data/nebula/data/storage/nebula/1/wal/8/0000000000000000001.wal
I0315 12:30:19.480324 49415 RaftPart.cpp:278] [Port: 9780, Space: 1, Part: 8] There are 0 peer hosts, and total 1 copies. The quorum is 1, as learner 0, lastLogId 106609, lastLogTerm 201, committedLogId 106609, term 201
I0315 12:30:19.480376 49415 NebulaStore.cpp:146] Load part 1, 8 from disk
I0315 12:30:19.487277 49416 FileBasedWal.cpp:66] [Port: 9780, Space: 1, Part: 11] lastLogId in wal is 106609, lastLogTerm is 201, path is /data/nebula/data/storage/nebula/1/wal/11/0000000000000000001.wal
I0315 12:30:19.487390 49416 RaftPart.cpp:278] [Port: 9780, Space: 1, Part: 11] There are 0 peer hosts, and total 1 copies. The quorum is 1, as learner 0, lastLogId 106609, lastLogTerm 201, committedLogId 106609, term 201
I0315 12:30:19.487448 49416 NebulaStore.cpp:146] Load part 1, 11 from disk
I0315 12:30:19.497045 49418 FileBasedWal.cpp:66] [Port: 9780, Space: 1, Part: 5] lastLogId in wal is 106609, lastLogTerm is 201, path is /data/nebula/data/storage/nebula/1/wal/5/0000000000000000001.wal
I0315 12:30:19.497148 49418 RaftPart.cpp:278] [Port: 9780, Space: 1, Part: 5] There are 0 peer hosts, and total 1 copies. The quorum is 1, as learner 0, lastLogId 106609, lastLogTerm 201, committedLogId 106609, term 201
I0315 12:30:19.497161 49417 FileBasedWal.cpp:66] [Port: 9780, Space: 1, Part: 2] lastLogId in wal is 106609, lastLogTerm is 201, path is /data/nebula/data/storage/nebula/1/wal/2/0000000000000000001.wal
I0315 12:30:19.497216 49418 NebulaStore.cpp:146] Load part 1, 5 from disk
I0315 12:30:19.497220 49416 RaftPart.cpp:957] [Port: 9780, Space: 3, Part: 14] Start leader election, reason: lastMsgDur 383, term 200
I0315 12:30:19.497239 49417 RaftPart.cpp:278] [Port: 9780, Space: 1, Part: 2] There are 0 peer hosts, and total 1 copies. The quorum is 1, as learner 0, lastLogId 106609, lastLogTerm 201, committedLogId 106609, term 201
I0315 12:30:19.497252 49416 RaftPart.cpp:1095] [Port: 9780, Space: 3, Part: 14] Sending out an election request (space = 3, part = 14, term = 201, lastLogId = 4354597, lastLogTerm = 200, candidateIP = 10.226.14.134, candidatePort = 9780)
I0315 12:30:19.497267 49416 RaftPart.cpp:1059] [Port: 9780, Space: 3, Part: 14] Partition is elected as the new leader for term 201
I0315 12:30:19.497272 49416 RaftPart.cpp:1137] [Port: 9780, Space: 3, Part: 14] The partition is elected as the leader
I0315 12:30:19.497283 49417 NebulaStore.cpp:146] Load part 1, 2 from disk
I0315 12:30:19.805028 49417 FileBasedWal.cpp:66] [Port: 9780, Space: 1, Part: 14] lastLogId in wal is 106609, lastLogTerm is 201, path is /data/nebula/data/storage/nebula/1/wal/14/0000000000000000001.wal
I0315 12:30:19.805156 49417 RaftPart.cpp:278] [Port: 9780, Space: 1, Part: 14] There are 0 peer hosts, and total 1 copies. The quorum is 1, as learner 0, lastLogId 106609, lastLogTerm 201, committedLogId 106609, term 201
I0315 12:30:19.805205 49417 NebulaStore.cpp:146] Load part 1, 14 from disk
I0315 12:30:19.805233 49417 RaftPart.cpp:957] [Port: 9780, Space: 1, Part: 2] Start leader election, reason: lastMsgDur 633, term 201
I0315 12:30:19.805239 49417 RaftPart.cpp:1095] [Port: 9780, Space: 1, Part: 2] Sending out an election request (space = 1, part = 2, term = 202, lastLogId = 106609, lastLogTerm = 201, candidateIP = 10.226.14.134, candidatePort = 9780)
I0315 12:30:19.805243 49321 NebulaStore.cpp:161] Load space 1 complete
I0315 12:30:19.805291 49321 NebulaStore.cpp:90] Scan path "/data/nebula/data/storage/nebula/2"
I0315 12:30:19.805245 49417 RaftPart.cpp:1059] [Port: 9780, Space: 1, Part: 2] Partition is elected as the new leader for term 202
I0315 12:30:19.805325 49417 RaftPart.cpp:1137] [Port: 9780, Space: 1, Part: 2] The partition is elected as the leader
I0315 12:30:19.805310 49321 NebulaStore.cpp:170] Init data from partManager for "10.226.14.134":9779
I0315 12:30:19.805356 49321 NebulaStore.cpp:262] Data space 1 has existed!
I0315 12:30:19.805366 49321 NebulaStore.cpp:302] [Space: 1, Part: 2] has existed!
I0315 12:30:19.805371 49321 NebulaStore.cpp:302] [Space: 1, Part: 5] has existed!
I0315 12:30:19.805377 49321 NebulaStore.cpp:302] [Space: 1, Part: 8] has existed!
I0315 12:30:19.805380 49321 NebulaStore.cpp:302] [Space: 1, Part: 11] has existed!
I0315 12:30:19.805385 49321 NebulaStore.cpp:302] [Space: 1, Part: 14] has existed!
I0315 12:30:19.805390 49321 NebulaStore.cpp:262] Data space 3 has existed!
I0315 12:30:19.805395 49321 NebulaStore.cpp:302] [Space: 3, Part: 2] has existed!
I0315 12:30:19.805400 49321 NebulaStore.cpp:302] [Space: 3, Part: 5] has existed!
I0315 12:30:19.805405 49321 NebulaStore.cpp:302] [Space: 3, Part: 8] has existed!
I0315 12:30:19.805410 49321 NebulaStore.cpp:302] [Space: 3, Part: 11] has existed!
I0315 12:30:19.805414 49321 NebulaStore.cpp:302] [Space: 3, Part: 14] has existed!
I0315 12:30:19.805418 49417 RaftPart.cpp:957] [Port: 9780, Space: 3, Part: 8] Start leader election, reason: lastMsgDur 745, term 200
I0315 12:30:19.805428 49417 RaftPart.cpp:1095] [Port: 9780, Space: 3, Part: 8] Sending out an election request (space = 3, part = 8, term = 201, lastLogId = 4311499, lastLogTerm = 200, candidateIP = 10.226.14.134, candidatePort = 9780)
I0315 12:30:19.805433 49417 RaftPart.cpp:1059] [Port: 9780, Space: 3, Part: 8] Partition is elected as the new leader for term 201
I0315 12:30:19.805438 49321 NebulaStore.cpp:77] Register handler...
I0315 12:30:19.805477 49321 StorageServer.cpp:85] Starting Storage HTTP Service
I0315 12:30:19.805438 49417 RaftPart.cpp:1137] [Port: 9780, Space: 3, Part: 8] The partition is elected as the leader
I0315 12:30:19.805670 49321 StorageServer.cpp:89] Http Thread Pool started
I0315 12:30:19.807617 49544 WebService.cpp:124] Web service started on HTTP[19779], HTTP2[19780]
I0315 12:30:19.807679 49321 TransactionManager.cpp:25] TransactionManager ctor()
I0315 12:30:19.807713 49321 TransactionManager.cpp:153] space=1, part=14 added
I0315 12:30:19.807729 49321 TransactionManager.cpp:153] space=1, part=2 added
I0315 12:30:19.807734 49321 TransactionManager.cpp:153] space=1, part=5 added
I0315 12:30:19.807740 49321 TransactionManager.cpp:153] space=1, part=8 added
I0315 12:30:19.807745 49321 TransactionManager.cpp:153] space=1, part=11 added
I0315 12:30:19.807750 49321 TransactionManager.cpp:153] space=3, part=14 added
I0315 12:30:19.807756 49321 TransactionManager.cpp:153] space=3, part=5 added
I0315 12:30:19.807762 49321 TransactionManager.cpp:153] space=3, part=11 added
I0315 12:30:19.807767 49321 TransactionManager.cpp:153] space=3, part=2 added
I0315 12:30:19.807772 49321 TransactionManager.cpp:153] space=3, part=8 added
I0315 12:30:19.807801 49321 TransactionManager.cpp:178] scanPrimes(), spaceId=1, partId=14
I0315 12:30:19.807816 49321 TransactionManager.cpp:178] scanPrimes(), spaceId=1, partId=2
I0315 12:30:19.807890 49321 TransactionManager.cpp:227] insert space=1, part=2, into white list suc=1
I0315 12:30:19.807898 49321 TransactionManager.cpp:178] scanPrimes(), spaceId=1, partId=5
I0315 12:30:19.807904 49321 TransactionManager.cpp:178] scanPrimes(), spaceId=1, partId=8
I0315 12:30:19.807909 49321 TransactionManager.cpp:178] scanPrimes(), spaceId=1, partId=11
I0315 12:30:19.807914 49321 TransactionManager.cpp:178] scanPrimes(), spaceId=3, partId=14
I0315 12:30:19.807935 49321 TransactionManager.cpp:227] insert space=3, part=14, into white list suc=1
I0315 12:30:19.807941 49321 TransactionManager.cpp:178] scanPrimes(), spaceId=3, partId=5
I0315 12:30:19.807946 49321 TransactionManager.cpp:178] scanPrimes(), spaceId=3, partId=11
I0315 12:30:19.807951 49321 TransactionManager.cpp:178] scanPrimes(), spaceId=3, partId=2
I0315 12:30:19.807956 49321 TransactionManager.cpp:178] scanPrimes(), spaceId=3, partId=8
I0315 12:30:19.807967 49321 TransactionManager.cpp:227] insert space=3, part=8, into white list suc=1
I0315 12:30:19.808064 49321 RocksEngineConfig.cpp:299] Emplace rocksdb option max_background_jobs=0
I0315 12:30:19.808079 49321 RocksEngineConfig.cpp:299] Emplace rocksdb option max_subcompactions=0
I0315 12:30:19.808125 49321 RocksEngineConfig.cpp:299] Emplace rocksdb option max_bytes_for_level_base=268435456
I0315 12:30:19.808130 49321 RocksEngineConfig.cpp:299] Emplace rocksdb option max_write_buffer_number=4
I0315 12:30:19.808135 49321 RocksEngineConfig.cpp:299] Emplace rocksdb option write_buffer_size=67108864
I0315 12:30:19.808140 49321 RocksEngineConfig.cpp:299] Emplace rocksdb option disable_auto_compactions=false
I0315 12:30:19.808177 49321 RocksEngineConfig.cpp:299] Emplace rocksdb option block_size=8192
I0315 12:30:19.816300 49321 RocksEngine.cpp:128] open rocksdb on /data/nebula/data/storage/nebula/0/data
I0315 12:30:19.816509 49321 AdminTaskManager.cpp:22] max concurrenct subtasks: 10
I0315 12:30:19.816620 49321 AdminTaskManager.cpp:35] exit AdminTaskManager::init()
I0315 12:30:19.816666 49566 AdminTaskManager.cpp:231] waiting for incoming task
I0315 12:30:19.816900 49569 StorageServer.cpp:261] The admin service start on "10.226.14.134":9778
I0315 12:30:19.816953 49570 StorageServer.cpp:286] The internal storage service start(same with admin) on "10.226.14.134":9777
I0315 12:30:19.821233 49568 StorageServer.cpp:232] The storage service start on "10.226.14.134":9779
I0315 12:30:19.849813 49418 RaftPart.cpp:957] [Port: 9780, Space: 1, Part: 8] Start leader election, reason: lastMsgDur 677, term 201
I0315 12:30:19.849846 49418 RaftPart.cpp:1095] [Port: 9780, Space: 1, Part: 8] Sending out an election request (space = 1, part = 8, term = 202, lastLogId = 106609, lastLogTerm = 201, candidateIP = 10.226.14.134, candidatePort = 9780)
I0315 12:30:19.849864 49418 RaftPart.cpp:1059] [Port: 9780, Space: 1, Part: 8] Partition is elected as the new leader for term 202
I0315 12:30:19.849871 49418 RaftPart.cpp:1137] [Port: 9780, Space: 1, Part: 8] The partition is elected as the leader
I0315 12:30:19.850021 49418 TransactionManager.cpp:172] leader get do scanPrimes space=1, part=8, term=202
I0315 12:30:19.850036 49418 TransactionManager.cpp:178] scanPrimes(), spaceId=1, partId=8
I0315 12:30:19.850114 49418 TransactionManager.cpp:227] insert space=1, part=8, into white list suc=1
I0315 12:30:19.943043 49416 RaftPart.cpp:957] [Port: 9780, Space: 3, Part: 2] Start leader election, reason: lastMsgDur 882, term 200
I0315 12:30:19.943074 49416 RaftPart.cpp:1095] [Port: 9780, Space: 3, Part: 2] Sending out an election request (space = 3, part = 2, term = 201, lastLogId = 4309538, lastLogTerm = 200, candidateIP = 10.226.14.134, candidatePort = 9780)
I0315 12:30:19.943084 49416 RaftPart.cpp:1059] [Port: 9780, Space: 3, Part: 2] Partition is elected as the new leader for term 201
I0315 12:30:19.943089 49416 RaftPart.cpp:1137] [Port: 9780, Space: 3, Part: 2] The partition is elected as the leader
I0315 12:30:19.943217 49417 TransactionManager.cpp:172] leader get do scanPrimes space=3, part=2, term=201
I0315 12:30:19.943245 49417 TransactionManager.cpp:178] scanPrimes(), spaceId=3, partId=2
I0315 12:30:19.943318 49417 TransactionManager.cpp:227] insert space=3, part=2, into white list suc=1
I0315 12:30:19.979287 49418 RaftPart.cpp:957] [Port: 9780, Space: 3, Part: 11] Start leader election, reason: lastMsgDur 919, term 200
I0315 12:30:19.979317 49418 RaftPart.cpp:1095] [Port: 9780, Space: 3, Part: 11] Sending out an election request (space = 3, part = 11, term = 201, lastLogId = 4377355, lastLogTerm = 200, candidateIP = 10.226.14.134, candidatePort = 9780)
I0315 12:30:19.979326 49418 RaftPart.cpp:1059] [Port: 9780, Space: 3, Part: 11] Partition is elected as the new leader for term 201
I0315 12:30:19.979331 49418 RaftPart.cpp:1137] [Port: 9780, Space: 3, Part: 11] The partition is elected as the leader
I0315 12:30:19.979537 49416 TransactionManager.cpp:172] leader get do scanPrimes space=3, part=11, term=201
I0315 12:30:19.979565 49416 TransactionManager.cpp:178] scanPrimes(), spaceId=3, partId=11
I0315 12:30:19.979650 49416 TransactionManager.cpp:227] insert space=3, part=11, into white list suc=1
I0315 12:30:20.082526 49415 RaftPart.cpp:957] [Port: 9780, Space: 3, Part: 5] Start leader election, reason: lastMsgDur 1022, term 200
I0315 12:30:20.082551 49415 RaftPart.cpp:1095] [Port: 9780, Space: 3, Part: 5] Sending out an election request (space = 3, part = 5, term = 201, lastLogId = 4291429, lastLogTerm = 200, candidateIP = 10.226.14.134, candidatePort = 9780)
I0315 12:30:20.082558 49415 RaftPart.cpp:1059] [Port: 9780, Space: 3, Part: 5] Partition is elected as the new leader for term 201
I0315 12:30:20.082563 49415 RaftPart.cpp:1137] [Port: 9780, Space: 3, Part: 5] The partition is elected as the leader
I0315 12:30:20.082721 49415 TransactionManager.cpp:172] leader get do scanPrimes space=3, part=5, term=201
I0315 12:30:20.082732 49415 TransactionManager.cpp:178] scanPrimes(), spaceId=3, partId=5
I0315 12:30:20.082780 49415 TransactionManager.cpp:227] insert space=3, part=5, into white list suc=1
I0315 12:30:20.247951 49416 RaftPart.cpp:957] [Port: 9780, Space: 1, Part: 5] Start leader election, reason: lastMsgDur 1076, term 201
I0315 12:30:20.247982 49416 RaftPart.cpp:1095] [Port: 9780, Space: 1, Part: 5] Sending out an election request (space = 1, part = 5, term = 202, lastLogId = 106609, lastLogTerm = 201, candidateIP = 10.226.14.134, candidatePort = 9780)
I0315 12:30:20.247990 49416 RaftPart.cpp:1059] [Port: 9780, Space: 1, Part: 5] Partition is elected as the new leader for term 202
I0315 12:30:20.247995 49416 RaftPart.cpp:1137] [Port: 9780, Space: 1, Part: 5] The partition is elected as the leader
I0315 12:30:20.248132 49418 TransactionManager.cpp:172] leader get do scanPrimes space=1, part=5, term=202
I0315 12:30:20.248153 49418 TransactionManager.cpp:178] scanPrimes(), spaceId=1, partId=5
I0315 12:30:20.248214 49418 TransactionManager.cpp:227] insert space=1, part=5, into white list suc=1
I0315 12:30:20.265142 49417 RaftPart.cpp:957] [Port: 9780, Space: 1, Part: 14] Start leader election, reason: lastMsgDur 768, term 201
I0315 12:30:20.265172 49417 RaftPart.cpp:1095] [Port: 9780, Space: 1, Part: 14] Sending out an election request (space = 1, part = 14, term = 202, lastLogId = 106609, lastLogTerm = 201, candidateIP = 10.226.14.134, candidatePort = 9780)
I0315 12:30:20.265179 49417 RaftPart.cpp:1059] [Port: 9780, Space: 1, Part: 14] Partition is elected as the new leader for term 202
I0315 12:30:20.265184 49417 RaftPart.cpp:1137] [Port: 9780, Space: 1, Part: 14] The partition is elected as the leader
I0315 12:30:20.265323 49417 TransactionManager.cpp:172] leader get do scanPrimes space=1, part=14, term=202
I0315 12:30:20.265336 49417 TransactionManager.cpp:178] scanPrimes(), spaceId=1, partId=14
I0315 12:30:20.265412 49417 TransactionManager.cpp:227] insert space=1, part=14, into white list suc=1
I0315 12:30:20.416399 49415 RaftPart.cpp:957] [Port: 9780, Space: 1, Part: 11] Start leader election, reason: lastMsgDur 1244, term 201
I0315 12:30:20.416427 49415 RaftPart.cpp:1095] [Port: 9780, Space: 1, Part: 11] Sending out an election request (space = 1, part = 11, term = 202, lastLogId = 106609, lastLogTerm = 201, candidateIP = 10.226.14.134, candidatePort = 9780)
I0315 12:30:20.416435 49415 RaftPart.cpp:1059] [Port: 9780, Space: 1, Part: 11] Partition is elected as the new leader for term 202
I0315 12:30:20.416440 49415 RaftPart.cpp:1137] [Port: 9780, Space: 1, Part: 11] The partition is elected as the leader
I0315 12:30:20.416569 49416 TransactionManager.cpp:172] leader get do scanPrimes space=1, part=11, term=202
I0315 12:30:20.416581 49416 TransactionManager.cpp:178] scanPrimes(), spaceId=1, partId=11
I0315 12:30:20.416661 49416 TransactionManager.cpp:227] insert space=1, part=11, into white list suc=1
E0315 12:30:20.984055 49607 UpdateNode.h:475] edge conflict 3:8:2835414:92:0:2022011319365182aa2539123aee4a528ddd1bc4818ecd01622fcab788724c
E0315 12:30:21.502187 49606 UpdateNode.h:475] edge conflict 3:5:32137405:92:0:2020111319313141d3f00d3fc2cade0d5cc0e562035f9001f16741848c165e

liuqian1990 · 2022 年3 月 15 日 04:32

请问下这个rocksdb的compaction 怎么解决呢

liuqian1990 · 2022 年3 月 15 日 04:34

Rocksdb 日志

[root@iZbp1702pfcdtieotkgp4yZ data]# cat LOG
2022/03/15-12:30:07.396114 2b4fd26cc740 RocksDB version: 6.15.5
2022/03/15-12:30:07.396166 2b4fd26cc740 Git sha rocksdb_build_git_sha:@0@
2022/03/15-12:30:07.396169 2b4fd26cc740 Compile date Aug 23 2021
2022/03/15-12:30:07.396240 2b4fd26cc740 DB SUMMARY
2022/03/15-12:30:07.396244 2b4fd26cc740 DB Session ID:  1050EACY3GPNN8YYPQSC
2022/03/15-12:30:07.396884 2b4fd26cc740 CURRENT file:  CURRENT
2022/03/15-12:30:07.396887 2b4fd26cc740 IDENTITY file:  IDENTITY
2022/03/15-12:30:07.396899 2b4fd26cc740 MANIFEST file:  MANIFEST-093680 size: 6411524 Bytes
2022/03/15-12:30:07.396901 2b4fd26cc740 SST files in /data/nebula/data/storage/nebula/3/data dir, Total Num: 578, files: 035303.sst 036198.sst 036240.sst 036263.sst 036285.sst 036401.sst 036425.sst 036444.sst 036463.sst
2022/03/15-12:30:07.396903 2b4fd26cc740 Write Ahead Log file in /data/nebula/data/storage/nebula/3/data: 093681.log size: 12158980 ;
2022/03/15-12:30:07.396908 2b4fd26cc740                         Options.error_if_exists: 0
2022/03/15-12:30:07.396910 2b4fd26cc740                       Options.create_if_missing: 1
2022/03/15-12:30:07.396911 2b4fd26cc740                         Options.paranoid_checks: 1
2022/03/15-12:30:07.396912 2b4fd26cc740                               Options.track_and_verify_wals_in_manifest: 0
2022/03/15-12:30:07.396913 2b4fd26cc740                                     Options.env: 0x2d3ac20
2022/03/15-12:30:07.396915 2b4fd26cc740                                      Options.fs: Posix File System
2022/03/15-12:30:07.396916 2b4fd26cc740                                Options.info_log: 0x2b4fd386c670
2022/03/15-12:30:07.396917 2b4fd26cc740                Options.max_file_opening_threads: 16
2022/03/15-12:30:07.396920 2b4fd26cc740                              Options.statistics: (nil)
2022/03/15-12:30:07.396922 2b4fd26cc740                               Options.use_fsync: 0
2022/03/15-12:30:07.396923 2b4fd26cc740                       Options.max_log_file_size: 0
2022/03/15-12:30:07.396924 2b4fd26cc740                  Options.max_manifest_file_size: 1073741824
2022/03/15-12:30:07.396925 2b4fd26cc740                   Options.log_file_time_to_roll: 0
2022/03/15-12:30:07.396926 2b4fd26cc740                       Options.keep_log_file_num: 1000
2022/03/15-12:30:07.396927 2b4fd26cc740                    Options.recycle_log_file_num: 0
2022/03/15-12:30:07.396928 2b4fd26cc740                         Options.allow_fallocate: 1
2022/03/15-12:30:07.396929 2b4fd26cc740                        Options.allow_mmap_reads: 0
2022/03/15-12:30:07.396930 2b4fd26cc740                       Options.allow_mmap_writes: 0
2022/03/15-12:30:07.396931 2b4fd26cc740                        Options.use_direct_reads: 0
2022/03/15-12:30:07.396932 2b4fd26cc740                        Options.use_direct_io_for_flush_and_compaction: 0
2022/03/15-12:30:07.396933 2b4fd26cc740          Options.create_missing_column_families: 0
2022/03/15-12:30:07.396934 2b4fd26cc740                              Options.db_log_dir:
2022/03/15-12:30:07.396936 2b4fd26cc740                                 Options.wal_dir: /data/nebula/data/storage/nebula/3/data
2022/03/15-12:30:07.396937 2b4fd26cc740                Options.table_cache_numshardbits: 6
2022/03/15-12:30:07.396938 2b4fd26cc740                         Options.WAL_ttl_seconds: 0
2022/03/15-12:30:07.396939 2b4fd26cc740                       Options.WAL_size_limit_MB: 0
2022/03/15-12:30:07.396940 2b4fd26cc740                        Options.max_write_batch_group_size_bytes: 1048576
2022/03/15-12:30:07.396941 2b4fd26cc740             Options.manifest_preallocation_size: 4194304
2022/03/15-12:30:07.396942 2b4fd26cc740                     Options.is_fd_close_on_exec: 1
2022/03/15-12:30:07.396943 2b4fd26cc740                   Options.advise_random_on_open: 1
2022/03/15-12:30:07.396944 2b4fd26cc740                    Options.db_write_buffer_size: 0
2022/03/15-12:30:07.396945 2b4fd26cc740                    Options.write_buffer_manager: 0x2b4fdd57e1b0
2022/03/15-12:30:07.396946 2b4fd26cc740         Options.access_hint_on_compaction_start: 1
2022/03/15-12:30:07.396947 2b4fd26cc740  Options.new_table_reader_for_compaction_inputs: 0
2022/03/15-12:30:07.396958 2b4fd26cc740           Options.random_access_max_buffer_size: 1048576
2022/03/15-12:30:07.396959 2b4fd26cc740                      Options.use_adaptive_mutex: 0
2022/03/15-12:30:07.396960 2b4fd26cc740                            Options.rate_limiter: (nil)
2022/03/15-12:30:07.396964 2b4fd26cc740     Options.sst_file_manager.rate_bytes_per_sec: 0
2022/03/15-12:30:07.396965 2b4fd26cc740                       Options.wal_recovery_mode: 2
2022/03/15-12:30:07.396966 2b4fd26cc740                  Options.enable_thread_tracking: 0
2022/03/15-12:30:07.396967 2b4fd26cc740                  Options.enable_pipelined_write: 0
2022/03/15-12:30:07.396968 2b4fd26cc740                  Options.unordered_write: 0
2022/03/15-12:30:07.396969 2b4fd26cc740         Options.allow_concurrent_memtable_write: 1
2022/03/15-12:30:07.396970 2b4fd26cc740      Options.enable_write_thread_adaptive_yield: 1
2022/03/15-12:30:07.396971 2b4fd26cc740             Options.write_thread_max_yield_usec: 100
2022/03/15-12:30:07.396972 2b4fd26cc740            Options.write_thread_slow_yield_usec: 3
2022/03/15-12:30:07.396973 2b4fd26cc740                               Options.row_cache: 16000000
2022/03/15-12:30:07.396975 2b4fd26cc740                              Options.wal_filter: None
2022/03/15-12:30:07.396978 2b4fd26cc740             Options.avoid_flush_during_recovery: 0
2022/03/15-12:30:07.396979 2b4fd26cc740             Options.allow_ingest_behind: 0
2022/03/15-12:30:07.396980 2b4fd26cc740             Options.preserve_deletes: 0
2022/03/15-12:30:07.396981 2b4fd26cc740             Options.two_write_queues: 0
2022/03/15-12:30:07.396982 2b4fd26cc740             Options.manual_wal_flush: 0
2022/03/15-12:30:07.396983 2b4fd26cc740             Options.atomic_flush: 0
2022/03/15-12:30:07.396984 2b4fd26cc740             Options.avoid_unnecessary_blocking_io: 0
2022/03/15-12:30:07.396985 2b4fd26cc740                 Options.persist_stats_to_disk: 0
2022/03/15-12:30:07.396986 2b4fd26cc740                 Options.write_dbid_to_manifest: 0
2022/03/15-12:30:07.396987 2b4fd26cc740                 Options.log_readahead_size: 0
2022/03/15-12:30:07.396988 2b4fd26cc740                 Options.file_checksum_gen_factory: Unknown
2022/03/15-12:30:07.396991 2b4fd26cc740                 Options.best_efforts_recovery: 0
2022/03/15-12:30:07.396992 2b4fd26cc740                Options.max_bgerror_resume_count: 2147483647
2022/03/15-12:30:07.396994 2b4fd26cc740            Options.bgerror_resume_retry_interval: 1000000
2022/03/15-12:30:07.396995 2b4fd26cc740             Options.allow_data_in_errors: 0
2022/03/15-12:30:07.396996 2b4fd26cc740             Options.db_host_id: __hostname__
2022/03/15-12:30:07.396997 2b4fd26cc740             Options.max_background_jobs: 0
2022/03/15-12:30:07.396998 2b4fd26cc740             Options.max_background_compactions: -1
2022/03/15-12:30:07.396999 2b4fd26cc740             Options.max_subcompactions: 0
2022/03/15-12:30:07.397000 2b4fd26cc740             Options.avoid_flush_during_shutdown: 0
2022/03/15-12:30:07.397001 2b4fd26cc740           Options.writable_file_max_buffer_size: 1048576
2022/03/15-12:30:07.397002 2b4fd26cc740             Options.delayed_write_rate : 16777216
2022/03/15-12:30:07.397003 2b4fd26cc740             Options.max_total_wal_size: 0
2022/03/15-12:30:07.397004 2b4fd26cc740             Options.delete_obsolete_files_period_micros: 21600000000
2022/03/15-12:30:07.397006 2b4fd26cc740                   Options.stats_dump_period_sec: 600
2022/03/15-12:30:07.397007 2b4fd26cc740                 Options.stats_persist_period_sec: 600
2022/03/15-12:30:07.397008 2b4fd26cc740                 Options.stats_history_buffer_size: 1048576
2022/03/15-12:30:07.397009 2b4fd26cc740                          Options.max_open_files: -1
2022/03/15-12:30:07.397010 2b4fd26cc740                          Options.bytes_per_sync: 0
2022/03/15-12:30:07.397011 2b4fd26cc740                      Options.wal_bytes_per_sync: 0
2022/03/15-12:30:07.397012 2b4fd26cc740                   Options.strict_bytes_per_sync: 0
2022/03/15-12:30:07.397013 2b4fd26cc740       Options.compaction_readahead_size: 0
2022/03/15-12:30:07.397021 2b4fd26cc740                  Options.max_background_flushes: -1
2022/03/15-12:30:07.397022 2b4fd26cc740 Compression algorithms supported:
2022/03/15-12:30:07.397025 2b4fd26cc740 	kZSTDNotFinalCompression supported: 1
2022/03/15-12:30:07.397029 2b4fd26cc740 	kZSTD supported: 1
2022/03/15-12:30:07.397031 2b4fd26cc740 	kXpressCompression supported: 0
2022/03/15-12:30:07.397032 2b4fd26cc740 	kLZ4HCCompression supported: 1
2022/03/15-12:30:07.397033 2b4fd26cc740 	kLZ4Compression supported: 1
2022/03/15-12:30:07.397035 2b4fd26cc740 	kBZip2Compression supported: 1
2022/03/15-12:30:07.397036 2b4fd26cc740 	kZlibCompression supported: 1
2022/03/15-12:30:07.397037 2b4fd26cc740 	kSnappyCompression supported: 1
2022/03/15-12:30:07.397042 2b4fd26cc740 Fast CRC32 supported: Supported on x86
2022/03/15-12:30:07.397272 2b4fd26cc740 [db/version_set.cc:4724] Recovering from manifest file: /data/nebula/data/storage/nebula/3/data/MANIFEST-093680
2022/03/15-12:30:07.397490 2b4fd26cc740 [db/column_family.cc:595] --------------- Options for column family [default]:
2022/03/15-12:30:07.397497 2b4fd26cc740               Options.comparator: leveldb.BytewiseComparator
2022/03/15-12:30:07.397499 2b4fd26cc740           Options.merge_operator: None
2022/03/15-12:30:07.397500 2b4fd26cc740        Options.compaction_filter: None
2022/03/15-12:30:07.397502 2b4fd26cc740        Options.compaction_filter_factory: StorageCompactionFilterFactory
2022/03/15-12:30:07.397503 2b4fd26cc740  Options.sst_partitioner_factory: None
2022/03/15-12:30:07.397504 2b4fd26cc740         Options.memtable_factory: SkipListFactory
2022/03/15-12:30:07.397505 2b4fd26cc740            Options.table_factory: BlockBasedTable
2022/03/15-12:30:07.397543 2b4fd26cc740            table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x2b4fd3808098)
  cache_index_and_filter_blocks: 1
  cache_index_and_filter_blocks_with_high_priority: 1
  pin_l0_filter_and_index_blocks_in_cache: 1
  pin_top_level_index_and_filter: 1
  index_type: 2
  data_block_index_type: 0
  index_shortening: 1
  data_block_hash_table_util_ratio: 0.750000
  hash_index_allow_collision: 1
  checksum: 1
  no_block_cache: 0
  block_cache: 0x2b4fdd42ed10
  block_cache_name: LRUCache
  block_cache_options:
    capacity : 8589934592
    num_shard_bits : 8
    strict_capacity_limit : 0
    memory_allocator : None
    high_pri_pool_ratio: 0.500
  block_cache_compressed: (nil)
  persistent_cache: (nil)
  block_size: 8192
  block_size_deviation: 10
  block_restart_interval: 16
  index_block_restart_interval: 1
  metadata_block_size: 4096
  partition_filters: 1
  use_delta_encoding: 1
  filter_policy: rocksdb.BuiltinBloomFilter
  whole_key_filtering: 0
  verify_compression: 0
  read_amp_bytes_per_bit: 0
  format_version: 4
  enable_index_compression: 1
  block_align: 0
2022/03/15-12:30:07.397554 2b4fd26cc740        Options.write_buffer_size: 67108864
2022/03/15-12:30:07.397555 2b4fd26cc740  Options.max_write_buffer_number: 4
2022/03/15-12:30:07.397557 2b4fd26cc740          Options.compression: LZ4
2022/03/15-12:30:07.397558 2b4fd26cc740                  Options.bottommost_compression: Disabled
2022/03/15-12:30:07.397561 2b4fd26cc740       Options.prefix_extractor: rocksdb.CappedPrefix.3004
2022/03/15-12:30:07.397563 2b4fd26cc740   Options.memtable_insert_with_hint_prefix_extractor: nullptr
2022/03/15-12:30:07.397566 2b4fd26cc740             Options.num_levels: 7
2022/03/15-12:30:07.397567 2b4fd26cc740        Options.min_write_buffer_number_to_merge: 1
2022/03/15-12:30:07.397568 2b4fd26cc740     Options.max_write_buffer_number_to_maintain: 0
2022/03/15-12:30:07.397569 2b4fd26cc740     Options.max_write_buffer_size_to_maintain: 0
2022/03/15-12:30:07.397571 2b4fd26cc740            Options.bottommost_compression_opts.window_bits: -14
2022/03/15-12:30:07.397572 2b4fd26cc740                  Options.bottommost_compression_opts.level: 32767
2022/03/15-12:30:07.397573 2b4fd26cc740               Options.bottommost_compression_opts.strategy: 0
2022/03/15-12:30:07.397584 2b4fd26cc740         Options.bottommost_compression_opts.max_dict_bytes: 0
2022/03/15-12:30:07.397585 2b4fd26cc740         Options.bottommost_compression_opts.zstd_max_train_bytes: 0
2022/03/15-12:30:07.397586 2b4fd26cc740         Options.bottommost_compression_opts.parallel_threads: 1
2022/03/15-12:30:07.397587 2b4fd26cc740                  Options.bottommost_compression_opts.enabled: false
2022/03/15-12:30:07.397588 2b4fd26cc740            Options.compression_opts.window_bits: -14
2022/03/15-12:30:07.397589 2b4fd26cc740                  Options.compression_opts.level: 32767
2022/03/15-12:30:07.397590 2b4fd26cc740               Options.compression_opts.strategy: 0
2022/03/15-12:30:07.397591 2b4fd26cc740         Options.compression_opts.max_dict_bytes: 0
2022/03/15-12:30:07.397592 2b4fd26cc740         Options.compression_opts.zstd_max_train_bytes: 0
2022/03/15-12:30:07.397593 2b4fd26cc740         Options.compression_opts.parallel_threads: 1
2022/03/15-12:30:07.397594 2b4fd26cc740                  Options.compression_opts.enabled: false
2022/03/15-12:30:07.397596 2b4fd26cc740      Options.level0_file_num_compaction_trigger: 4
2022/03/15-12:30:07.397597 2b4fd26cc740          Options.level0_slowdown_writes_trigger: 20
2022/03/15-12:30:07.397598 2b4fd26cc740              Options.level0_stop_writes_trigger: 36
2022/03/15-12:30:07.397599 2b4fd26cc740                   Options.target_file_size_base: 67108864
2022/03/15-12:30:07.397601 2b4fd26cc740             Options.target_file_size_multiplier: 1
2022/03/15-12:30:07.397602 2b4fd26cc740                Options.max_bytes_for_level_base: 268435456
2022/03/15-12:30:07.397603 2b4fd26cc740 Options.level_compaction_dynamic_level_bytes: 0
2022/03/15-12:30:07.397604 2b4fd26cc740          Options.max_bytes_for_level_multiplier: 10.000000
2022/03/15-12:30:07.397607 2b4fd26cc740 Options.max_bytes_for_level_multiplier_addtl[0]: 1
2022/03/15-12:30:07.397608 2b4fd26cc740 Options.max_bytes_for_level_multiplier_addtl[1]: 1
2022/03/15-12:30:07.397609 2b4fd26cc740 Options.max_bytes_for_level_multiplier_addtl[2]: 1
2022/03/15-12:30:07.397610 2b4fd26cc740 Options.max_bytes_for_level_multiplier_addtl[3]: 1
2022/03/15-12:30:07.397611 2b4fd26cc740 Options.max_bytes_for_level_multiplier_addtl[4]: 1
2022/03/15-12:30:07.397612 2b4fd26cc740 Options.max_bytes_for_level_multiplier_addtl[5]: 1
2022/03/15-12:30:07.397614 2b4fd26cc740 Options.max_bytes_for_level_multiplier_addtl[6]: 1
2022/03/15-12:30:07.397615 2b4fd26cc740       Options.max_sequential_skip_in_iterations: 8
2022/03/15-12:30:07.397616 2b4fd26cc740                    Options.max_compaction_bytes: 1677721600
2022/03/15-12:30:07.397617 2b4fd26cc740                        Options.arena_block_size: 8388608
2022/03/15-12:30:07.397618 2b4fd26cc740   Options.soft_pending_compaction_bytes_limit: 68719476736
2022/03/15-12:30:07.397619 2b4fd26cc740   Options.hard_pending_compaction_bytes_limit: 274877906944
2022/03/15-12:30:07.397620 2b4fd26cc740       Options.rate_limit_delay_max_milliseconds: 100
2022/03/15-12:30:07.397621 2b4fd26cc740                Options.disable_auto_compactions: 0
2022/03/15-12:30:07.397623 2b4fd26cc740                        Options.compaction_style: kCompactionStyleLevel
2022/03/15-12:30:07.397625 2b4fd26cc740                          Options.compaction_pri: kMinOverlappingRatio
2022/03/15-12:30:07.397626 2b4fd26cc740 Options.compaction_options_universal.size_ratio: 1
2022/03/15-12:30:07.397627 2b4fd26cc740 Options.compaction_options_universal.min_merge_width: 2
2022/03/15-12:30:07.397628 2b4fd26cc740 Options.compaction_options_universal.max_merge_width: 4294967295
2022/03/15-12:30:07.397629 2b4fd26cc740 Options.compaction_options_universal.max_size_amplification_percent: 200
2022/03/15-12:30:07.397631 2b4fd26cc740 Options.compaction_options_universal.compression_size_percent: -1
2022/03/15-12:30:07.397632 2b4fd26cc740 Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize
2022/03/15-12:30:07.397633 2b4fd26cc740 Options.compaction_options_fifo.max_table_files_size: 1073741824
2022/03/15-12:30:07.397641 2b4fd26cc740 Options.compaction_options_fifo.allow_compaction: 0
2022/03/15-12:30:07.397644 2b4fd26cc740                   Options.table_properties_collectors:
2022/03/15-12:30:07.397645 2b4fd26cc740                   Options.inplace_update_support: 0
2022/03/15-12:30:07.397646 2b4fd26cc740                 Options.inplace_update_num_locks: 10000
2022/03/15-12:30:07.397647 2b4fd26cc740               Options.memtable_prefix_bloom_size_ratio: 0.000000
2022/03/15-12:30:07.397649 2b4fd26cc740               Options.memtable_whole_key_filtering: 0
2022/03/15-12:30:07.397650 2b4fd26cc740   Options.memtable_huge_page_size: 0
2022/03/15-12:30:07.397651 2b4fd26cc740                           Options.bloom_locality: 0
2022/03/15-12:30:07.397652 2b4fd26cc740                    Options.max_successive_merges: 0
2022/03/15-12:30:07.397653 2b4fd26cc740                Options.optimize_filters_for_hits: 0
2022/03/15-12:30:07.397654 2b4fd26cc740                Options.paranoid_file_checks: 0
2022/03/15-12:30:07.397655 2b4fd26cc740                Options.force_consistency_checks: 1
2022/03/15-12:30:07.397657 2b4fd26cc740                Options.report_bg_io_stats: 0
2022/03/15-12:30:07.397658 2b4fd26cc740                               Options.ttl: 2592000
2022/03/15-12:30:07.397659 2b4fd26cc740          Options.periodic_compaction_seconds: 2592000
2022/03/15-12:30:07.397660 2b4fd26cc740                    Options.enable_blob_files: false
2022/03/15-12:30:07.397661 2b4fd26cc740                        Options.min_blob_size: 0
2022/03/15-12:30:07.397662 2b4fd26cc740                       Options.blob_file_size: 268435456
2022/03/15-12:30:07.397664 2b4fd26cc740                Options.blob_compression_type: NoCompression
2022/03/15-12:30:07.397665 2b4fd26cc740       Options.enable_blob_garbage_collection: false
2022/03/15-12:30:07.397666 2b4fd26cc740   Options.blob_garbage_collection_age_cutoff: 0.250000
2022/03/15-12:30:18.632322 2b4fd26cc740 [db/version_set.cc:4764] Recovered from manifest file:/data/nebula/data/storage/nebula/3/data/MANIFEST-093680 succeeded,manifest_file_number is 93680, next_file_number is 93689, last_sequence is 438908905, log_number is 93675,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0
2022/03/15-12:30:18.632349 2b4fd26cc740 [db/version_set.cc:4779] Column family [default] (ID 0), log number is 93675
2022/03/15-12:30:18.633531 2b4fd26cc740 [db/version_set.cc:4082] Creating manifest 93689
2022/03/15-12:30:18.676646 2b4fd26cc740 EVENT_LOG_v1 {"time_micros": 1647318618676636, "job": 1, "event": "recovery_started", "wal_files": [93681]}
2022/03/15-12:30:18.676654 2b4fd26cc740 [db/db_impl/db_impl_open.cc:845] Recovering log #93681 mode 2
2022/03/15-12:30:18.708904 2b4fd26cc740 EVENT_LOG_v1 {"time_micros": 1647318618708871, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 93690, "file_size": 237449, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 204705, "index_size": 294465, "index_partitions": 34, "top_level_index_size": 93582, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 16986, "raw_key_size": 8652240, "raw_average_key_size": 4534, "raw_value_size": 65282, "raw_average_value_size": 34, "num_data_blocks": 796, "num_entries": 1908, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "rocksdb.CappedPrefix.3004", "property_collectors": "[]", "compression": "LZ4", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1647318618, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "d1e283e9-e61a-4d27-aa42-6f5489ea0441", "db_session_id": "1050EACY3GPNN8YYPQSC"}}
2022/03/15-12:30:18.710356 2b4fd26cc740 [db/version_set.cc:4082] Creating manifest 93691
2022/03/15-12:30:18.738696 2b4fd26cc740 EVENT_LOG_v1 {"time_micros": 1647318618738691, "job": 1, "event": "recovery_finished"}
2022/03/15-12:30:18.743511 2b4fd26cc740 [file/delete_scheduler.cc:69] Deleted file /data/nebula/data/storage/nebula/3/data/093681.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000
2022/03/15-12:30:18.745357 2b4fd26cc740 [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x2b4fe745d380
2022/03/15-12:30:18.748167 2b4fefa01700 [db/compaction/compaction_job.cc:1881] [default] [JOB 3] Compacting 1@2 + 3@3 files to L3, score 1.04
2022/03/15-12:30:18.748178 2b4fefa01700 [db/compaction/compaction_job.cc:1887] [default] Compaction start summary: Base version 3 Base level 2, inputs: [88919(82MB)], [65289(94MB) 86862(170MB) 86893(123MB)]
2022/03/15-12:30:18.748202 2b4fefa01700 EVENT_LOG_v1 {"time_micros": 1647318618748188, "job": 3, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L2": [88919], "files_L3": [65289, 86862, 86893], "score": 1.04147, "input_data_size": 493136927}
2022/03/15-12:30:18.748777 2b4fd26cc740 DB pointer 0x2b4fe7b40800
2022/03/15-12:30:18.749197 2b4ffc010700 [db/db_impl/db_impl.cc:901] ------- DUMPING STATS -------
2022/03/15-12:30:18.749243 2b4ffc010700 [db/db_impl/db_impl.cc:903]
** DB Stats **
Uptime(secs): 11.4 total, 11.4 interval
Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s
Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s
Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s
Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s
Interval stall: 00:00:0.000 H:M:S, 0.0 percent

** Compaction Stats [default] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      1/0   231.88 KB   0.2      0.0     0.0      0.0       0.0      0.0       0.0   1.0      0.0     18.6      0.01              0.00         1    0.012       0      0
  L1      4/0   214.65 MB   0.8      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0
  L2     48/1    2.60 GB   1.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0
  L3    289/3   24.99 GB   1.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0
  L4    237/0   26.78 GB   0.1      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0
 Sum    579/4   54.59 GB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   1.0      0.0     18.6      0.01              0.00         1    0.012       0      0
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   1.0      0.0     18.6      0.01              0.00         1    0.012       0      0

** Compaction Stats [default] **
Priority    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
User      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0     18.6      0.01              0.00         1    0.012       0      0
Uptime(secs): 11.4 total, 11.4 interval
Flush(GB): cumulative 0.000, interval 0.000
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.00 GB write, 0.02 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Interval compaction: 0.00 GB write, 0.02 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count

** File Read Latency Histogram By Level [default] **

** Compaction Stats [default] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      1/0   231.88 KB   0.2      0.0     0.0      0.0       0.0      0.0       0.0   1.0      0.0     18.6      0.01              0.00         1    0.012       0      0
  L1      4/0   214.65 MB   0.8      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0
  L2     48/1    2.60 GB   1.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0
  L3    289/3   24.99 GB   1.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0
  L4    237/0   26.78 GB   0.1      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0
 Sum    579/4   54.59 GB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   1.0      0.0     18.6      0.01              0.00         1    0.012       0      0
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0

** Compaction Stats [default] **
Priority    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
User      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0     18.6      0.01              0.00         1    0.012       0      0
Uptime(secs): 11.4 total, 0.0 interval
Flush(GB): cumulative 0.000, interval 0.000
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.00 GB write, 0.02 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count

** File Read Latency Histogram By Level [default] **```

system · 2022 年4 月 14 日 04:35

此话题已在最后回复的 30 天后被自动关闭。不再允许新回复。