1.1 导入过程中的storagerr.log日志
2 我的配置
使用curl查看配置, 其他的基本都是默认配置, (其实原因已经很清楚了, 就是限制速度了, 但是该如何配置才能最大化利用机器资源呢???)
在客户端使用show stroage config查看(这个客户端配置是我最开始在客户端update的, 但是重启失败后, 加了local_config=1后再配置文件中配置的是上面的curl的配置截图, 但是现在客户端也没有和curl的配置保持一致, 这里我很疑惑…)
4, 监控
看一眼对应rocksdb的日志？
可以导入时候先关闭一下auto_compact
我是说和sst文件在一起的rocksdb的自己的LOG文件
改配置是修改 storaged.conf文件（头部加上–local_config=true），里面对应的rocksdb的各个项。然后重启进程。
是不是默认l0的sst文件数量太多了，触发了stall write。
不要管show config能看到啥，看rocksdb LOG文件里面的显示
吴老师, 这个我看了下, 除了max_open_files不是我配置的之外, 其他都是已经配置生效了, 在启动参数中了
2021/07/16-15:59:19.471120 7f03d4caa980 RocksDB version: 6.7.3
2021/07/16-15:59:19.471228 7f03d4caa980 Git sha rocksdb_build_git_sha:@0@
2021/07/16-15:59:19.471234 7f03d4caa980 Compile date Jul 8 2020
2021/07/16-15:59:19.471241 7f03d4caa980 DB SUMMARY
2021/07/16-15:59:19.483432 7f03d4caa980 CURRENT file: CURRENT
2021/07/16-15:59:19.483449 7f03d4caa980 MANIFEST file: MANIFEST-000008 size: 714453220 Bytes
2021/07/16-15:59:19.483454 7f03d4caa980 SST files in /data/storage/nebula/1/data dir, Total Num: 8960, files: 540587.sst 550502.sst 550503.sst 550504.sst 550505.sst 550506.sst 550507.sst 550508.sst 550509.sst
2021/07/16-15:59:19.483457 7f03d4caa980 Write Ahead Log file in /data/storage/nebula/1/data:
2021/07/16-15:59:19.483474 7f03d4caa980 Options.error_if_exists: 0
2021/07/16-15:59:19.483476 7f03d4caa980 Options.create_if_missing: 1
2021/07/16-15:59:19.483478 7f03d4caa980 Options.paranoid_checks: 1
2021/07/16-15:59:19.483480 7f03d4caa980 Options.env: 0x2c0b9a0
2021/07/16-15:59:19.483487 7f03d4caa980 Options.fs: Posix File System
2021/07/16-15:59:19.483490 7f03d4caa980 Options.info_log: 0x7f03d388b290
2021/07/16-15:59:19.483492 7f03d4caa980 Options.max_file_opening_threads: 16
2021/07/16-15:59:19.483494 7f03d4caa980 Options.statistics: (nil)
2021/07/16-15:59:19.483496 7f03d4caa980 Options.use_fsync: 0
2021/07/16-15:59:19.483498 7f03d4caa980 Options.max_log_file_size: 0
2021/07/16-15:59:19.483500 7f03d4caa980 Options.max_manifest_file_size: 1073741824
2021/07/16-15:59:19.483502 7f03d4caa980 Options.log_file_time_to_roll: 0
2021/07/16-15:59:19.483503 7f03d4caa980 Options.keep_log_file_num: 1000
2021/07/16-15:59:19.483505 7f03d4caa980 Options.recycle_log_file_num: 0
2021/07/16-15:59:19.483507 7f03d4caa980 Options.allow_fallocate: 1
2021/07/16-15:59:19.483509 7f03d4caa980 Options.allow_mmap_reads: 0
2021/07/16-15:59:19.483511 7f03d4caa980 Options.allow_mmap_writes: 0
2021/07/16-15:59:19.483512 7f03d4caa980 Options.use_direct_reads: 0
2021/07/16-15:59:19.483517 7f03d4caa980 Options.use_direct_io_for_flush_and_compaction: 0
2021/07/16-15:59:19.483519 7f03d4caa980 Options.create_missing_column_families: 0
2021/07/16-15:59:19.483521 7f03d4caa980 Options.db_log_dir:
2021/07/16-15:59:19.483523 7f03d4caa980 Options.wal_dir: /data/storage/nebula/1/data
2021/07/16-15:59:19.483525 7f03d4caa980 Options.table_cache_numshardbits: 6
2021/07/16-15:59:19.483527 7f03d4caa980 Options.max_subcompactions: 32
2021/07/16-15:59:19.483529 7f03d4caa980 Options.max_background_flushes: 32
2021/07/16-15:59:19.483531 7f03d4caa980 Options.WAL_ttl_seconds: 0
2021/07/16-15:59:19.483533 7f03d4caa980 Options.WAL_size_limit_MB: 0
2021/07/16-15:59:19.483534 7f03d4caa980 Options.max_write_batch_group_size_bytes: 1048576
2021/07/16-15:59:19.483537 7f03d4caa980 Options.manifest_preallocation_size: 4194304
2021/07/16-15:59:19.483538 7f03d4caa980 Options.is_fd_close_on_exec: 1
2021/07/16-15:59:19.483540 7f03d4caa980 Options.advise_random_on_open: 1
2021/07/16-15:59:19.483542 7f03d4caa980 Options.db_write_buffer_size: 0
2021/07/16-15:59:19.483544 7f03d4caa980 Options.write_buffer_manager: 0x7f03be417480
2021/07/16-15:59:19.483545 7f03d4caa980 Options.access_hint_on_compaction_start: 1
2021/07/16-15:59:19.483547 7f03d4caa980 Options.new_table_reader_for_compaction_inputs: 0
2021/07/16-15:59:19.483549 7f03d4caa980 Options.random_access_max_buffer_size: 1048576
2021/07/16-15:59:19.483551 7f03d4caa980 Options.use_adaptive_mutex: 0
2021/07/16-15:59:19.483571 7f03d4caa980 Options.rate_limiter: (nil)
2021/07/16-15:59:19.483577 7f03d4caa980 Options.sst_file_manager.rate_bytes_per_sec: 0
2021/07/16-15:59:19.483579 7f03d4caa980 Options.wal_recovery_mode: 2
2021/07/16-15:59:19.483580 7f03d4caa980 Options.enable_thread_tracking: 0
2021/07/16-15:59:19.483582 7f03d4caa980 Options.enable_pipelined_write: 0
2021/07/16-15:59:19.483584 7f03d4caa980 Options.unordered_write: 0
2021/07/16-15:59:19.483586 7f03d4caa980 Options.allow_concurrent_memtable_write: 1
2021/07/16-15:59:19.483587 7f03d4caa980 Options.enable_write_thread_adaptive_yield: 1
2021/07/16-15:59:19.483589 7f03d4caa980 Options.write_thread_max_yield_usec: 100
2021/07/16-15:59:19.483591 7f03d4caa980 Options.write_thread_slow_yield_usec: 3
2021/07/16-15:59:19.483593 7f03d4caa980 Options.row_cache: None
2021/07/16-15:59:19.483595 7f03d4caa980 Options.wal_filter: None
2021/07/16-15:59:19.483597 7f03d4caa980 Options.avoid_flush_during_recovery: 0
2021/07/16-15:59:19.483599 7f03d4caa980 Options.allow_ingest_behind: 0
2021/07/16-15:59:19.483601 7f03d4caa980 Options.preserve_deletes: 0
2021/07/16-15:59:19.483603 7f03d4caa980 Options.two_write_queues: 0
2021/07/16-15:59:19.483605 7f03d4caa980 Options.manual_wal_flush: 0
2021/07/16-15:59:19.483606 7f03d4caa980 Options.atomic_flush: 0
2021/07/16-15:59:19.483608 7f03d4caa980 Options.avoid_unnecessary_blocking_io: 0
2021/07/16-15:59:19.483610 7f03d4caa980 Options.persist_stats_to_disk: 0
2021/07/16-15:59:19.483612 7f03d4caa980 Options.write_dbid_to_manifest: 0
2021/07/16-15:59:19.483614 7f03d4caa980 Options.log_readahead_size: 0
2021/07/16-15:59:19.483616 7f03d4caa980 Options.max_background_jobs: 64
2021/07/16-15:59:19.483618 7f03d4caa980 Options.max_background_compactions: -1
2021/07/16-15:59:19.483620 7f03d4caa980 Options.avoid_flush_during_shutdown: 0
2021/07/16-15:59:19.483621 7f03d4caa980 Options.writable_file_max_buffer_size: 1048576
2021/07/16-15:59:19.483623 7f03d4caa980 Options.delayed_write_rate : 16777216
2021/07/16-15:59:19.483625 7f03d4caa980 Options.max_total_wal_size: 0
2021/07/16-15:59:19.483627 7f03d4caa980 Options.delete_obsolete_files_period_micros: 21600000000
2021/07/16-15:59:19.483629 7f03d4caa980 Options.stats_dump_period_sec: 600
2021/07/16-15:59:19.483631 7f03d4caa980 Options.stats_persist_period_sec: 600
2021/07/16-15:59:19.483633 7f03d4caa980 Options.stats_history_buffer_size: 1048576
2021/07/16-15:59:19.483635 7f03d4caa980 Options.max_open_files: 65536
2021/07/16-15:59:19.483637 7f03d4caa980 Options.bytes_per_sync: 0
2021/07/16-15:59:19.483639 7f03d4caa980 Options.wal_bytes_per_sync: 0
2021/07/16-15:59:19.483640 7f03d4caa980 Options.strict_bytes_per_sync: 0
2021/07/16-15:59:19.483642 7f03d4caa980 Options.compaction_readahead_size: 0
2021/07/16-15:59:19.483644 7f03d4caa980 Compression algorithms supported:
2021/07/16-15:59:19.483646 7f03d4caa980 kZSTDNotFinalCompression supported: 1
2021/07/16-15:59:19.485598 7f03d4caa980 kZSTD supported: 1
2021/07/16-15:59:19.485601 7f03d4caa980 kXpressCompression supported: 0
2021/07/16-15:59:19.485603 7f03d4caa980 kLZ4HCCompression supported: 1
2021/07/16-15:59:19.485606 7f03d4caa980 kLZ4Compression supported: 1
2021/07/16-15:59:19.485608 7f03d4caa980 kBZip2Compression supported: 1
2021/07/16-15:59:19.485610 7f03d4caa980 kZlibCompression supported: 1
2021/07/16-15:59:19.485612 7f03d4caa980 kSnappyCompression supported: 1
2021/07/16-15:59:19.485622 7f03d4caa980 Fast CRC32 supported: Supported on x86
2021/07/16-15:59:19.486597 7f03d4caa980 [version_set.cc:4401] Recovering from manifest file: /data/storage/nebula/1/data/MANIFEST-000008
2021/07/16-15:59:19.487208 7f03d4caa980 [column_family.cc:550] --------------- Options for column family [default]:
2021/07/16-15:59:19.487218 7f03d4caa980 Options.comparator: leveldb.BytewiseComparator
2021/07/16-15:59:19.487221 7f03d4caa980 Options.merge_operator: None
2021/07/16-15:59:19.487223 7f03d4caa980 Options.compaction_filter: None
2021/07/16-15:59:19.487225 7f03d4caa980 Options.compaction_filter_factory: StorageCompactionFilterFactory
2021/07/16-15:59:19.487227 7f03d4caa980 Options.memtable_factory: SkipListFactory
2021/07/16-15:59:19.487229 7f03d4caa980 Options.table_factory: BlockBasedTable
2021/07/16-15:59:19.487296 7f03d4caa980 table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x7f03d38080b0)
cache_index_and_filter_blocks: 0
cache_index_and_filter_blocks_with_high_priority: 1
pin_l0_filter_and_index_blocks_in_cache: 0
pin_top_level_index_and_filter: 1
index_type: 0
data_block_index_type: 0
index_shortening: 1
data_block_hash_table_util_ratio: 0.750000
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x7f03be433a90
block_cache_name: LRUCache
block_cache_options:
capacity : 8388608
num_shard_bits : 8
strict_capacity_limit : 0
memory_allocator : None
high_pri_pool_ratio: 0.500
block_cache_compressed: (nil)
persistent_cache: (nil)
block_size: 8192
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
metadata_block_size: 4096
partition_filters: 0
use_delta_encoding: 1
filter_policy: rocksdb.BuiltinBloomFilter
whole_key_filtering: 1
verify_compression: 0
read_amp_bytes_per_bit: 0
format_version: 2
enable_index_compression: 1
block_align: 0
2021/07/16-15:59:19.487303 7f03d4caa980 Options.write_buffer_size: 134217728
2021/07/16-15:59:19.487305 7f03d4caa980 Options.max_write_buffer_number: 10
2021/07/16-15:59:19.487311 7f03d4caa980 Options.compression: LZ4
2021/07/16-15:59:19.487313 7f03d4caa980 Options.bottommost_compression: Disabled
2021/07/16-15:59:19.487315 7f03d4caa980 Options.prefix_extractor: nullptr
2021/07/16-15:59:19.487317 7f03d4caa980 Options.memtable_insert_with_hint_prefix_extractor: nullptr
2021/07/16-15:59:19.487319 7f03d4caa980 Options.num_levels: 7
2021/07/16-15:59:19.487321 7f03d4caa980 Options.min_write_buffer_number_to_merge: 4
2021/07/16-15:59:19.487323 7f03d4caa980 Options.max_write_buffer_number_to_maintain: 0
2021/07/16-15:59:19.487325 7f03d4caa980 Options.max_write_buffer_size_to_maintain: 0
2021/07/16-15:59:19.487327 7f03d4caa980 Options.bottommost_compression_opts.window_bits: -14
2021/07/16-15:59:19.487329 7f03d4caa980 Options.bottommost_compression_opts.level: 32767
2021/07/16-15:59:19.487331 7f03d4caa980 Options.bottommost_compression_opts.strategy: 0
2021/07/16-15:59:19.487333 7f03d4caa980 Options.bottommost_compression_opts.max_dict_bytes: 0
2021/07/16-15:59:19.487335 7f03d4caa980 Options.bottommost_compression_opts.zstd_max_train_bytes: 0
2021/07/16-15:59:19.487337 7f03d4caa980 Options.bottommost_compression_opts.enabled: false
2021/07/16-15:59:19.487339 7f03d4caa980 Options.compression_opts.window_bits: -14
2021/07/16-15:59:19.487340 7f03d4caa980 Options.compression_opts.level: 32767
2021/07/16-15:59:19.487342 7f03d4caa980 Options.compression_opts.strategy: 0
2021/07/16-15:59:19.487344 7f03d4caa980 Options.compression_opts.max_dict_bytes: 0
2021/07/16-15:59:19.487346 7f03d4caa980 Options.compression_opts.zstd_max_train_bytes: 0
2021/07/16-15:59:19.487348 7f03d4caa980 Options.compression_opts.enabled: false
2021/07/16-15:59:19.487350 7f03d4caa980 Options.level0_file_num_compaction_trigger: 4
2021/07/16-15:59:19.487366 7f03d4caa980 Options.level0_slowdown_writes_trigger: 20
2021/07/16-15:59:19.487368 7f03d4caa980 Options.level0_stop_writes_trigger: 36
2021/07/16-15:59:19.487370 7f03d4caa980 Options.target_file_size_base: 134217728
2021/07/16-15:59:19.487372 7f03d4caa980 Options.target_file_size_multiplier: 1
2021/07/16-15:59:19.487374 7f03d4caa980 Options.max_bytes_for_level_base: 536870912
2021/07/16-15:59:19.487376 7f03d4caa980 Options.level_compaction_dynamic_level_bytes: 0
2021/07/16-15:59:19.487378 7f03d4caa980 Options.max_bytes_for_level_multiplier: 10.000000
2021/07/16-15:59:19.487382 7f03d4caa980 Options.max_bytes_for_level_multiplier_addtl[0]: 1
2021/07/16-15:59:19.487384 7f03d4caa980 Options.max_bytes_for_level_multiplier_addtl[1]: 1
2021/07/16-15:59:19.487386 7f03d4caa980 Options.max_bytes_for_level_multiplier_addtl[2]: 1
2021/07/16-15:59:19.487388 7f03d4caa980 Options.max_bytes_for_level_multiplier_addtl[3]: 1
2021/07/16-15:59:19.487389 7f03d4caa980 Options.max_bytes_for_level_multiplier_addtl[4]: 1
2021/07/16-15:59:19.487391 7f03d4caa980 Options.max_bytes_for_level_multiplier_addtl[5]: 1
2021/07/16-15:59:19.487393 7f03d4caa980 Options.max_bytes_for_level_multiplier_addtl[6]: 1
2021/07/16-15:59:19.487395 7f03d4caa980 Options.max_sequential_skip_in_iterations: 8
2021/07/16-15:59:19.487397 7f03d4caa980 Options.max_compaction_bytes: 3355443200
2021/07/16-15:59:19.487399 7f03d4caa980 Options.arena_block_size: 16777216
2021/07/16-15:59:19.487401 7f03d4caa980 Options.soft_pending_compaction_bytes_limit: 68719476736
2021/07/16-15:59:19.487403 7f03d4caa980 Options.hard_pending_compaction_bytes_limit: 274877906944
2021/07/16-15:59:19.487405 7f03d4caa980 Options.rate_limit_delay_max_milliseconds: 100
2021/07/16-15:59:19.487406 7f03d4caa980 Options.disable_auto_compactions: 0
2021/07/16-15:59:19.487409 7f03d4caa980 Options.compaction_style: kCompactionStyleLevel
2021/07/16-15:59:19.487412 7f03d4caa980 Options.compaction_pri: kMinOverlappingRatio
2021/07/16-15:59:19.487414 7f03d4caa980 Options.compaction_options_universal.size_ratio: 1
2021/07/16-15:59:19.487416 7f03d4caa980 Options.compaction_options_universal.min_merge_width: 2
2021/07/16-15:59:19.487418 7f03d4caa980 Options.compaction_options_universal.max_merge_width: 4294967295
2021/07/16-15:59:19.487420 7f03d4caa980 Options.compaction_options_universal.max_size_amplification_percent: 200
2021/07/16-15:59:19.487422 7f03d4caa980 Options.compaction_options_universal.compression_size_percent: -1
2021/07/16-15:59:19.487425 7f03d4caa980 Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize
2021/07/16-15:59:19.487427 7f03d4caa980 Options.compaction_options_fifo.max_table_files_size: 1073741824
2021/07/16-15:59:19.487429 7f03d4caa980 Options.compaction_options_fifo.allow_compaction: 0
2021/07/16-15:59:19.487431 7f03d4caa980 Options.table_properties_collectors:
2021/07/16-15:59:19.487433 7f03d4caa980 Options.inplace_update_support: 0
2021/07/16-15:59:19.487435 7f03d4caa980 Options.inplace_update_num_locks: 10000
2021/07/16-15:59:19.487437 7f03d4caa980 Options.memtable_prefix_bloom_size_ratio: 0.000000
2021/07/16-15:59:19.487440 7f03d4caa980 Options.memtable_whole_key_filtering: 0
2021/07/16-15:59:19.487441 7f03d4caa980 Options.memtable_huge_page_size: 0
2021/07/16-15:59:19.487443 7f03d4caa980 Options.bloom_locality: 0
2021/07/16-15:59:19.487445 7f03d4caa980 Options.max_successive_merges: 0
2021/07/16-15:59:19.487447 7f03d4caa980 Options.optimize_filters_for_hits: 0
2021/07/16-15:59:19.487449 7f03d4caa980 Options.paranoid_file_checks: 0
2021/07/16-15:59:19.487451 7f03d4caa980 Options.force_consistency_checks: 0
2021/07/16-15:59:19.487453 7f03d4caa980 Options.report_bg_io_stats: 0
2021/07/16-15:59:19.487462 7f03d4caa980 Options.ttl: 2592000
2021/07/16-15:59:19.487465 7f03d4caa980 Options.periodic_compaction_seconds: 2592000
2021/07/16-15:59:20.552839 7f03d4caa980 [version_set.cc:4544] Recovered from manifest file:/data/storage/nebula/1/data/MANIFEST-000008 succeeded,manifest_file_number is 8, next_file_number is 559463, last_sequence is 16270486869, log_number is 559460,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0
2021/07/16-15:59:20.552860 7f03d4caa980 [version_set.cc:4560] Column family [default] (ID 0), log number is 559460
2021/07/16-15:59:20.719806 7f03d4caa980 DB pointer 0x7f03d3860800
2021/07/16-15:59:20.721101 7f0387fff700 [db_impl/db_impl.cc:849] ------- DUMPING STATS -------
2021/07/16-15:59:20.721184 7f0387fff700 [db_impl/db_impl.cc:851]
** DB Stats **
Uptime(secs): 1.2 total, 1.2 interval
Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s
Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s
Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s
Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s
Interval stall: 00:00:0.000 H:M:S, 0.0 percent
** Compaction Stats [default] **
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
L0 2/0 2.33 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
L5 8958/0 1.14 TB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Sum 8960/0 1.14 TB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
** Compaction Stats [default] **
Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Uptime(secs): 1.2 total, 1.2 interval
Flush(GB): cumulative 0.000, interval 0.000
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
** File Read Latency Histogram By Level [default] **
** Compaction Stats [default] **
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
L0 2/0 2.33 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
L5 8958/0 1.14 TB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Sum 8960/0 1.14 TB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
** Compaction Stats [default] **
Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Uptime(secs): 1.2 total, 0.0 interval
Flush(GB): cumulative 0.000, interval 0.000
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
** File Read Latency Histogram By Level [default] **
2021/07/16-16:09:20.722256 7f0387fff700 [db_impl/db_impl.cc:849] ------- DUMPING STATS -------
2021/07/16-16:09:20.722381 7f0387fff700 [db_impl/db_impl.cc:851]
** DB Stats **
Uptime(secs): 601.2 total, 600.0 interval
Cumulative writes: 450 writes, 450 keys, 329 commit groups, 1.4 writes per commit group, ingest: 0.00 GB, 0.00 MB/s
Cumulative WAL: 450 writes, 0 syncs, 450.00 writes per sync, written: 0.00 GB, 0.00 MB/s
Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
Interval writes: 450 writes, 450 keys, 329 commit groups, 1.4 writes per commit group, ingest: 0.02 MB, 0.00 MB/s
Interval WAL: 450 writes, 0 syncs, 450.00 writes per sync, written: 0.00 MB, 0.00 MB/s
Interval stall: 00:00:0.000 H:M:S, 0.0 percent
** Compaction Stats [default] **
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
L0 2/0 2.33 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
L5 8958/0 1.14 TB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Sum 8960/0 1.14 TB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
** Compaction Stats [default] **
Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Uptime(secs): 601.2 total, 600.0 interval
Flush(GB): cumulative 0.000, interval 0.000
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
** File Read Latency Histogram By Level [default] **
** Compaction Stats [default] **
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------已经按照你这个步骤启动好多次了, 都还是不行
吴老师, 这个我看了下, 除了max_open_files不是我配置的之外, 其他都是已经配置生效了, 在启动参数中了
disable_auto_compact = true 再试试吧。
瞬间压力应该是flush memfile吧
false前提下, 我使用了下面这2句更新命令, 后来某些原因需要重启的时候就报错了, 报错大概日志是启动的时候会加载上次我在客户端update的配置(就下面这2句话)
UPDATE CONFIGS storage:rocksdb_db_options = {max_subcompactions: 32, max_background_jobs: 64,max_open_files: 100000000, max_background_compaction: 32, num_compaction_threads: 32, max_background_flushes: 32, rate_limit: 536870912}
UPDATE CONFIGS storage:rocksdb_column_family_options = {max_bytes_for_level_base: 536870912, max_write_buffer_number: 10, write_buffer_size: 134217728, min_write_buffer_number_to_merge: 4, level0_file_num_compaction_trigger: 4, target_file_size_base: 134217728, target_file_size_multiplier: 1}
true的时候, 我的配置如下, 少了max_background_compaction和num_compaction_threads这2个配置, 为啥不加呢? 因为加上任何一个都起不来,…还是加载配置出错, 对了, 有方法吧之前在客户端更新的配置缓存清除吗???
–rocksdb_column_family_options={“max_bytes_for_level_base”:“536870912”,“max_write_buffer_number”:“10”,“write_buffer_size”:“134217728”,“min_write_buffer_number_to_merge”:“4”,“level0_file_num_compaction_trigger”:“4”,“target_file_size_base”:“134217728”,“target_file_size_multiplier”:“1”}
--rocksdb_db_options={"max_subcompactions":"32","max_background_jobs":"64","max_open_files":"100000000","max_background_flushes":"32"}
disable_auto_compact = true这个能消除stall condition…吗? 我现在已经导入了50亿数据, 停的话还得重新开始, 有办法能让他在local_conifg=true的情况下不重启升生效吗???
我觉得就是周期性的flush而已，flush时候可能会略微慢一点。上层应用重试一下好了。
大批量写入的时候没有warning也不太可能的。
我应该知道为啥我在客户端修改了之后启动不了了, 在客户端update的时候不会校验参数的正确性, 随便杀参数都可以设置成功, 一旦设置了, 就可以用show config看到, 但是在重启的时候会加载之前在客户端配置的参数重新更新, 我加的max_background_compaction和num_compaction_threads这2个参数报错的原因是: 第一个少了…s
num_compaction_threads这我没在rocksdb的log中找到该参数, 估计没有这参数(个人理解)
那么现在问题来了, 怎么在不使用lcoal_conifg=1的情况下怎么清除客户端的参数设置缓存? 重新更新么???
不要启动storage 只启动graph和meta，把正确的update configs执行了再启动storage
