FLink导入速度慢, 日志有stall condition....

  1. 大佬好, 我们使用Flink导入大量数据, 发现刚开始几分钟速度还行, 但是后续将会越来越慢… 监控图感觉一卡一卡的, 日志中还有stall condition…日志, 这个日志其实还可以理解是限制速度, 但是这个导入速度真的是不高, 高的话限速可以理解, 但是这不高怎么能限制速度了?? (最开始我是没有使用local_config=true的, 所有的rocksdb就是在客户端配置的, 但是发现后续重启会失败, 包更新配置检查失败, 加了这个local_cofig才可以启动)

1.1 导入过程中的storagerr.log日志

2 我的配置
使用curl查看配置, 其他的基本都是默认配置, (其实原因已经很清楚了, 就是限制速度了, 但是该如何配置才能最大化利用机器资源呢???)

在客户端使用show stroage config查看(这个客户端配置是我最开始在客户端update的, 但是重启失败后, 加了local_config=1后再配置文件中配置的是上面的curl的配置截图, 但是现在客户端也没有和curl的配置保持一致, 这里我很疑惑…)


4, 监控
image

image

1 个赞

看一眼对应rocksdb的日志?
可以导入时候先关闭一下auto_compact

wu老师, 就是这个storage-stderr.log

我是说和sst文件在一起的rocksdb的自己的LOG文件

改配置是修改 storaged.conf文件(头部加上–local_config=true),里面对应的rocksdb的各个项。然后重启进程。

是不是默认l0的sst文件数量太多了,触发了stall write。

不要管show config能看到啥,看rocksdb LOG文件里面的显示

1 个赞
吴老师, 这个我看了下, 除了max_open_files不是我配置的之外, 其他都是已经配置生效了, 在启动参数中了  

  2021/07/16-15:59:19.471120 7f03d4caa980 RocksDB version: 6.7.3
    2021/07/16-15:59:19.471228 7f03d4caa980 Git sha rocksdb_build_git_sha:@0@
    2021/07/16-15:59:19.471234 7f03d4caa980 Compile date Jul  8 2020
    2021/07/16-15:59:19.471241 7f03d4caa980 DB SUMMARY
    2021/07/16-15:59:19.483432 7f03d4caa980 CURRENT file:  CURRENT
    2021/07/16-15:59:19.483449 7f03d4caa980 MANIFEST file:  MANIFEST-000008 size: 714453220 Bytes
    2021/07/16-15:59:19.483454 7f03d4caa980 SST files in /data/storage/nebula/1/data dir, Total Num: 8960, files: 540587.sst 550502.sst 550503.sst 550504.sst 550505.sst 550506.sst 550507.sst 550508.sst 550509.sst 
    2021/07/16-15:59:19.483457 7f03d4caa980 Write Ahead Log file in /data/storage/nebula/1/data: 
    2021/07/16-15:59:19.483474 7f03d4caa980                         Options.error_if_exists: 0
    2021/07/16-15:59:19.483476 7f03d4caa980                       Options.create_if_missing: 1
    2021/07/16-15:59:19.483478 7f03d4caa980                         Options.paranoid_checks: 1
    2021/07/16-15:59:19.483480 7f03d4caa980                                     Options.env: 0x2c0b9a0
    2021/07/16-15:59:19.483487 7f03d4caa980                                      Options.fs: Posix File System
    2021/07/16-15:59:19.483490 7f03d4caa980                                Options.info_log: 0x7f03d388b290
    2021/07/16-15:59:19.483492 7f03d4caa980                Options.max_file_opening_threads: 16
    2021/07/16-15:59:19.483494 7f03d4caa980                              Options.statistics: (nil)
    2021/07/16-15:59:19.483496 7f03d4caa980                               Options.use_fsync: 0
    2021/07/16-15:59:19.483498 7f03d4caa980                       Options.max_log_file_size: 0
    2021/07/16-15:59:19.483500 7f03d4caa980                  Options.max_manifest_file_size: 1073741824
    2021/07/16-15:59:19.483502 7f03d4caa980                   Options.log_file_time_to_roll: 0
    2021/07/16-15:59:19.483503 7f03d4caa980                       Options.keep_log_file_num: 1000
    2021/07/16-15:59:19.483505 7f03d4caa980                    Options.recycle_log_file_num: 0
    2021/07/16-15:59:19.483507 7f03d4caa980                         Options.allow_fallocate: 1
    2021/07/16-15:59:19.483509 7f03d4caa980                        Options.allow_mmap_reads: 0
    2021/07/16-15:59:19.483511 7f03d4caa980                       Options.allow_mmap_writes: 0
    2021/07/16-15:59:19.483512 7f03d4caa980                        Options.use_direct_reads: 0
    2021/07/16-15:59:19.483517 7f03d4caa980                        Options.use_direct_io_for_flush_and_compaction: 0
    2021/07/16-15:59:19.483519 7f03d4caa980          Options.create_missing_column_families: 0
    2021/07/16-15:59:19.483521 7f03d4caa980                              Options.db_log_dir: 
    2021/07/16-15:59:19.483523 7f03d4caa980                                 Options.wal_dir: /data/storage/nebula/1/data
    2021/07/16-15:59:19.483525 7f03d4caa980                Options.table_cache_numshardbits: 6
    2021/07/16-15:59:19.483527 7f03d4caa980                      Options.max_subcompactions: 32
    2021/07/16-15:59:19.483529 7f03d4caa980                  Options.max_background_flushes: 32
    2021/07/16-15:59:19.483531 7f03d4caa980                         Options.WAL_ttl_seconds: 0
    2021/07/16-15:59:19.483533 7f03d4caa980                       Options.WAL_size_limit_MB: 0
    2021/07/16-15:59:19.483534 7f03d4caa980                        Options.max_write_batch_group_size_bytes: 1048576
    2021/07/16-15:59:19.483537 7f03d4caa980             Options.manifest_preallocation_size: 4194304
    2021/07/16-15:59:19.483538 7f03d4caa980                     Options.is_fd_close_on_exec: 1
    2021/07/16-15:59:19.483540 7f03d4caa980                   Options.advise_random_on_open: 1
    2021/07/16-15:59:19.483542 7f03d4caa980                    Options.db_write_buffer_size: 0
    2021/07/16-15:59:19.483544 7f03d4caa980                    Options.write_buffer_manager: 0x7f03be417480
    2021/07/16-15:59:19.483545 7f03d4caa980         Options.access_hint_on_compaction_start: 1
    2021/07/16-15:59:19.483547 7f03d4caa980  Options.new_table_reader_for_compaction_inputs: 0
    2021/07/16-15:59:19.483549 7f03d4caa980           Options.random_access_max_buffer_size: 1048576
    2021/07/16-15:59:19.483551 7f03d4caa980                      Options.use_adaptive_mutex: 0
    2021/07/16-15:59:19.483571 7f03d4caa980                            Options.rate_limiter: (nil)
    2021/07/16-15:59:19.483577 7f03d4caa980     Options.sst_file_manager.rate_bytes_per_sec: 0
    2021/07/16-15:59:19.483579 7f03d4caa980                       Options.wal_recovery_mode: 2
    2021/07/16-15:59:19.483580 7f03d4caa980                  Options.enable_thread_tracking: 0
    2021/07/16-15:59:19.483582 7f03d4caa980                  Options.enable_pipelined_write: 0
    2021/07/16-15:59:19.483584 7f03d4caa980                  Options.unordered_write: 0
    2021/07/16-15:59:19.483586 7f03d4caa980         Options.allow_concurrent_memtable_write: 1
    2021/07/16-15:59:19.483587 7f03d4caa980      Options.enable_write_thread_adaptive_yield: 1
    2021/07/16-15:59:19.483589 7f03d4caa980             Options.write_thread_max_yield_usec: 100
    2021/07/16-15:59:19.483591 7f03d4caa980            Options.write_thread_slow_yield_usec: 3
    2021/07/16-15:59:19.483593 7f03d4caa980                               Options.row_cache: None
    2021/07/16-15:59:19.483595 7f03d4caa980                              Options.wal_filter: None
    2021/07/16-15:59:19.483597 7f03d4caa980             Options.avoid_flush_during_recovery: 0
    2021/07/16-15:59:19.483599 7f03d4caa980             Options.allow_ingest_behind: 0
    2021/07/16-15:59:19.483601 7f03d4caa980             Options.preserve_deletes: 0
    2021/07/16-15:59:19.483603 7f03d4caa980             Options.two_write_queues: 0
    2021/07/16-15:59:19.483605 7f03d4caa980             Options.manual_wal_flush: 0
    2021/07/16-15:59:19.483606 7f03d4caa980             Options.atomic_flush: 0
    2021/07/16-15:59:19.483608 7f03d4caa980             Options.avoid_unnecessary_blocking_io: 0
    2021/07/16-15:59:19.483610 7f03d4caa980                 Options.persist_stats_to_disk: 0
    2021/07/16-15:59:19.483612 7f03d4caa980                 Options.write_dbid_to_manifest: 0
    2021/07/16-15:59:19.483614 7f03d4caa980                 Options.log_readahead_size: 0
    2021/07/16-15:59:19.483616 7f03d4caa980             Options.max_background_jobs: 64
    2021/07/16-15:59:19.483618 7f03d4caa980             Options.max_background_compactions: -1
    2021/07/16-15:59:19.483620 7f03d4caa980             Options.avoid_flush_during_shutdown: 0
    2021/07/16-15:59:19.483621 7f03d4caa980           Options.writable_file_max_buffer_size: 1048576
    2021/07/16-15:59:19.483623 7f03d4caa980             Options.delayed_write_rate : 16777216
    2021/07/16-15:59:19.483625 7f03d4caa980             Options.max_total_wal_size: 0
    2021/07/16-15:59:19.483627 7f03d4caa980             Options.delete_obsolete_files_period_micros: 21600000000
    2021/07/16-15:59:19.483629 7f03d4caa980                   Options.stats_dump_period_sec: 600
    2021/07/16-15:59:19.483631 7f03d4caa980                 Options.stats_persist_period_sec: 600
    2021/07/16-15:59:19.483633 7f03d4caa980                 Options.stats_history_buffer_size: 1048576
    2021/07/16-15:59:19.483635 7f03d4caa980                          Options.max_open_files: 65536
    2021/07/16-15:59:19.483637 7f03d4caa980                          Options.bytes_per_sync: 0
    2021/07/16-15:59:19.483639 7f03d4caa980                      Options.wal_bytes_per_sync: 0
    2021/07/16-15:59:19.483640 7f03d4caa980                   Options.strict_bytes_per_sync: 0
    2021/07/16-15:59:19.483642 7f03d4caa980       Options.compaction_readahead_size: 0
    2021/07/16-15:59:19.483644 7f03d4caa980 Compression algorithms supported:
    2021/07/16-15:59:19.483646 7f03d4caa980 	kZSTDNotFinalCompression supported: 1
    2021/07/16-15:59:19.485598 7f03d4caa980 	kZSTD supported: 1
    2021/07/16-15:59:19.485601 7f03d4caa980 	kXpressCompression supported: 0
    2021/07/16-15:59:19.485603 7f03d4caa980 	kLZ4HCCompression supported: 1
    2021/07/16-15:59:19.485606 7f03d4caa980 	kLZ4Compression supported: 1
    2021/07/16-15:59:19.485608 7f03d4caa980 	kBZip2Compression supported: 1
    2021/07/16-15:59:19.485610 7f03d4caa980 	kZlibCompression supported: 1
    2021/07/16-15:59:19.485612 7f03d4caa980 	kSnappyCompression supported: 1
    2021/07/16-15:59:19.485622 7f03d4caa980 Fast CRC32 supported: Supported on x86
    2021/07/16-15:59:19.486597 7f03d4caa980 [version_set.cc:4401] Recovering from manifest file: /data/storage/nebula/1/data/MANIFEST-000008
    2021/07/16-15:59:19.487208 7f03d4caa980 [column_family.cc:550] --------------- Options for column family [default]:
    2021/07/16-15:59:19.487218 7f03d4caa980               Options.comparator: leveldb.BytewiseComparator
    2021/07/16-15:59:19.487221 7f03d4caa980           Options.merge_operator: None
    2021/07/16-15:59:19.487223 7f03d4caa980        Options.compaction_filter: None
    2021/07/16-15:59:19.487225 7f03d4caa980        Options.compaction_filter_factory: StorageCompactionFilterFactory
    2021/07/16-15:59:19.487227 7f03d4caa980         Options.memtable_factory: SkipListFactory
    2021/07/16-15:59:19.487229 7f03d4caa980            Options.table_factory: BlockBasedTable
    2021/07/16-15:59:19.487296 7f03d4caa980            table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x7f03d38080b0)
      cache_index_and_filter_blocks: 0
      cache_index_and_filter_blocks_with_high_priority: 1
      pin_l0_filter_and_index_blocks_in_cache: 0
      pin_top_level_index_and_filter: 1
      index_type: 0
      data_block_index_type: 0
      index_shortening: 1
      data_block_hash_table_util_ratio: 0.750000
      hash_index_allow_collision: 1
      checksum: 1
      no_block_cache: 0
      block_cache: 0x7f03be433a90
      block_cache_name: LRUCache
      block_cache_options:
        capacity : 8388608
        num_shard_bits : 8
        strict_capacity_limit : 0
        memory_allocator : None
        high_pri_pool_ratio: 0.500
      block_cache_compressed: (nil)
      persistent_cache: (nil)
      block_size: 8192
      block_size_deviation: 10
      block_restart_interval: 16
      index_block_restart_interval: 1
      metadata_block_size: 4096
      partition_filters: 0
      use_delta_encoding: 1
      filter_policy: rocksdb.BuiltinBloomFilter
      whole_key_filtering: 1
      verify_compression: 0
      read_amp_bytes_per_bit: 0
      format_version: 2
      enable_index_compression: 1
      block_align: 0
    2021/07/16-15:59:19.487303 7f03d4caa980        Options.write_buffer_size: 134217728
    2021/07/16-15:59:19.487305 7f03d4caa980  Options.max_write_buffer_number: 10
    2021/07/16-15:59:19.487311 7f03d4caa980          Options.compression: LZ4
    2021/07/16-15:59:19.487313 7f03d4caa980                  Options.bottommost_compression: Disabled
    2021/07/16-15:59:19.487315 7f03d4caa980       Options.prefix_extractor: nullptr
    2021/07/16-15:59:19.487317 7f03d4caa980   Options.memtable_insert_with_hint_prefix_extractor: nullptr
    2021/07/16-15:59:19.487319 7f03d4caa980             Options.num_levels: 7
    2021/07/16-15:59:19.487321 7f03d4caa980        Options.min_write_buffer_number_to_merge: 4
    2021/07/16-15:59:19.487323 7f03d4caa980     Options.max_write_buffer_number_to_maintain: 0
    2021/07/16-15:59:19.487325 7f03d4caa980     Options.max_write_buffer_size_to_maintain: 0
    2021/07/16-15:59:19.487327 7f03d4caa980            Options.bottommost_compression_opts.window_bits: -14
    2021/07/16-15:59:19.487329 7f03d4caa980                  Options.bottommost_compression_opts.level: 32767
    2021/07/16-15:59:19.487331 7f03d4caa980               Options.bottommost_compression_opts.strategy: 0
    2021/07/16-15:59:19.487333 7f03d4caa980         Options.bottommost_compression_opts.max_dict_bytes: 0
    2021/07/16-15:59:19.487335 7f03d4caa980         Options.bottommost_compression_opts.zstd_max_train_bytes: 0
    2021/07/16-15:59:19.487337 7f03d4caa980                  Options.bottommost_compression_opts.enabled: false
    2021/07/16-15:59:19.487339 7f03d4caa980            Options.compression_opts.window_bits: -14
    2021/07/16-15:59:19.487340 7f03d4caa980                  Options.compression_opts.level: 32767
    2021/07/16-15:59:19.487342 7f03d4caa980               Options.compression_opts.strategy: 0
    2021/07/16-15:59:19.487344 7f03d4caa980         Options.compression_opts.max_dict_bytes: 0
    2021/07/16-15:59:19.487346 7f03d4caa980         Options.compression_opts.zstd_max_train_bytes: 0
    2021/07/16-15:59:19.487348 7f03d4caa980                  Options.compression_opts.enabled: false
    2021/07/16-15:59:19.487350 7f03d4caa980      Options.level0_file_num_compaction_trigger: 4
    2021/07/16-15:59:19.487366 7f03d4caa980          Options.level0_slowdown_writes_trigger: 20
    2021/07/16-15:59:19.487368 7f03d4caa980              Options.level0_stop_writes_trigger: 36
    2021/07/16-15:59:19.487370 7f03d4caa980                   Options.target_file_size_base: 134217728
    2021/07/16-15:59:19.487372 7f03d4caa980             Options.target_file_size_multiplier: 1
    2021/07/16-15:59:19.487374 7f03d4caa980                Options.max_bytes_for_level_base: 536870912
    2021/07/16-15:59:19.487376 7f03d4caa980 Options.level_compaction_dynamic_level_bytes: 0
    2021/07/16-15:59:19.487378 7f03d4caa980          Options.max_bytes_for_level_multiplier: 10.000000
    2021/07/16-15:59:19.487382 7f03d4caa980 Options.max_bytes_for_level_multiplier_addtl[0]: 1
    2021/07/16-15:59:19.487384 7f03d4caa980 Options.max_bytes_for_level_multiplier_addtl[1]: 1
    2021/07/16-15:59:19.487386 7f03d4caa980 Options.max_bytes_for_level_multiplier_addtl[2]: 1
    2021/07/16-15:59:19.487388 7f03d4caa980 Options.max_bytes_for_level_multiplier_addtl[3]: 1
    2021/07/16-15:59:19.487389 7f03d4caa980 Options.max_bytes_for_level_multiplier_addtl[4]: 1
    2021/07/16-15:59:19.487391 7f03d4caa980 Options.max_bytes_for_level_multiplier_addtl[5]: 1
    2021/07/16-15:59:19.487393 7f03d4caa980 Options.max_bytes_for_level_multiplier_addtl[6]: 1
    2021/07/16-15:59:19.487395 7f03d4caa980       Options.max_sequential_skip_in_iterations: 8
    2021/07/16-15:59:19.487397 7f03d4caa980                    Options.max_compaction_bytes: 3355443200
    2021/07/16-15:59:19.487399 7f03d4caa980                        Options.arena_block_size: 16777216
    2021/07/16-15:59:19.487401 7f03d4caa980   Options.soft_pending_compaction_bytes_limit: 68719476736
    2021/07/16-15:59:19.487403 7f03d4caa980   Options.hard_pending_compaction_bytes_limit: 274877906944
    2021/07/16-15:59:19.487405 7f03d4caa980       Options.rate_limit_delay_max_milliseconds: 100
    2021/07/16-15:59:19.487406 7f03d4caa980                Options.disable_auto_compactions: 0
    2021/07/16-15:59:19.487409 7f03d4caa980                        Options.compaction_style: kCompactionStyleLevel
    2021/07/16-15:59:19.487412 7f03d4caa980                          Options.compaction_pri: kMinOverlappingRatio
    2021/07/16-15:59:19.487414 7f03d4caa980 Options.compaction_options_universal.size_ratio: 1
    2021/07/16-15:59:19.487416 7f03d4caa980 Options.compaction_options_universal.min_merge_width: 2
    2021/07/16-15:59:19.487418 7f03d4caa980 Options.compaction_options_universal.max_merge_width: 4294967295
    2021/07/16-15:59:19.487420 7f03d4caa980 Options.compaction_options_universal.max_size_amplification_percent: 200
    2021/07/16-15:59:19.487422 7f03d4caa980 Options.compaction_options_universal.compression_size_percent: -1
    2021/07/16-15:59:19.487425 7f03d4caa980 Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize
    2021/07/16-15:59:19.487427 7f03d4caa980 Options.compaction_options_fifo.max_table_files_size: 1073741824
    2021/07/16-15:59:19.487429 7f03d4caa980 Options.compaction_options_fifo.allow_compaction: 0
    2021/07/16-15:59:19.487431 7f03d4caa980                   Options.table_properties_collectors: 
    2021/07/16-15:59:19.487433 7f03d4caa980                   Options.inplace_update_support: 0
    2021/07/16-15:59:19.487435 7f03d4caa980                 Options.inplace_update_num_locks: 10000
    2021/07/16-15:59:19.487437 7f03d4caa980               Options.memtable_prefix_bloom_size_ratio: 0.000000
    2021/07/16-15:59:19.487440 7f03d4caa980               Options.memtable_whole_key_filtering: 0
    2021/07/16-15:59:19.487441 7f03d4caa980   Options.memtable_huge_page_size: 0
    2021/07/16-15:59:19.487443 7f03d4caa980                           Options.bloom_locality: 0
    2021/07/16-15:59:19.487445 7f03d4caa980                    Options.max_successive_merges: 0
    2021/07/16-15:59:19.487447 7f03d4caa980                Options.optimize_filters_for_hits: 0
    2021/07/16-15:59:19.487449 7f03d4caa980                Options.paranoid_file_checks: 0
    2021/07/16-15:59:19.487451 7f03d4caa980                Options.force_consistency_checks: 0
    2021/07/16-15:59:19.487453 7f03d4caa980                Options.report_bg_io_stats: 0
    2021/07/16-15:59:19.487462 7f03d4caa980                               Options.ttl: 2592000
    2021/07/16-15:59:19.487465 7f03d4caa980          Options.periodic_compaction_seconds: 2592000
    2021/07/16-15:59:20.552839 7f03d4caa980 [version_set.cc:4544] Recovered from manifest file:/data/storage/nebula/1/data/MANIFEST-000008 succeeded,manifest_file_number is 8, next_file_number is 559463, last_sequence is 16270486869, log_number is 559460,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0
    2021/07/16-15:59:20.552860 7f03d4caa980 [version_set.cc:4560] Column family [default] (ID 0), log number is 559460
    2021/07/16-15:59:20.719806 7f03d4caa980 DB pointer 0x7f03d3860800
    2021/07/16-15:59:20.721101 7f0387fff700 [db_impl/db_impl.cc:849] ------- DUMPING STATS -------
    2021/07/16-15:59:20.721184 7f0387fff700 [db_impl/db_impl.cc:851] 
    ** DB Stats **
    Uptime(secs): 1.2 total, 1.2 interval
    Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s
    Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s
    Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
    Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s
    Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s
    Interval stall: 00:00:0.000 H:M:S, 0.0 percent

    ** Compaction Stats [default] **
    Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
    ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
      L0      2/0    2.33 KB   0.5      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0
      L5   8958/0    1.14 TB   0.2      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0
     Sum   8960/0    1.14 TB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0
     Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0

    ** Compaction Stats [default] **
    Priority    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
    -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    Uptime(secs): 1.2 total, 1.2 interval
    Flush(GB): cumulative 0.000, interval 0.000
    AddFile(GB): cumulative 0.000, interval 0.000
    AddFile(Total Files): cumulative 0, interval 0
    AddFile(L0 Files): cumulative 0, interval 0
    AddFile(Keys): cumulative 0, interval 0
    Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
    Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
    Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count

    ** File Read Latency Histogram By Level [default] **

    ** Compaction Stats [default] **
    Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
    ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
      L0      2/0    2.33 KB   0.5      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0
      L5   8958/0    1.14 TB   0.2      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0
     Sum   8960/0    1.14 TB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0
     Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0

    ** Compaction Stats [default] **
    Priority    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
    -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    Uptime(secs): 1.2 total, 0.0 interval
    Flush(GB): cumulative 0.000, interval 0.000
    AddFile(GB): cumulative 0.000, interval 0.000
    AddFile(Total Files): cumulative 0, interval 0
    AddFile(L0 Files): cumulative 0, interval 0
    AddFile(Keys): cumulative 0, interval 0
    Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
    Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
    Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count

    ** File Read Latency Histogram By Level [default] **
    2021/07/16-16:09:20.722256 7f0387fff700 [db_impl/db_impl.cc:849] ------- DUMPING STATS -------
    2021/07/16-16:09:20.722381 7f0387fff700 [db_impl/db_impl.cc:851] 
    ** DB Stats **
    Uptime(secs): 601.2 total, 600.0 interval
    Cumulative writes: 450 writes, 450 keys, 329 commit groups, 1.4 writes per commit group, ingest: 0.00 GB, 0.00 MB/s
    Cumulative WAL: 450 writes, 0 syncs, 450.00 writes per sync, written: 0.00 GB, 0.00 MB/s
    Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
    Interval writes: 450 writes, 450 keys, 329 commit groups, 1.4 writes per commit group, ingest: 0.02 MB, 0.00 MB/s
    Interval WAL: 450 writes, 0 syncs, 450.00 writes per sync, written: 0.00 MB, 0.00 MB/s
    Interval stall: 00:00:0.000 H:M:S, 0.0 percent

    ** Compaction Stats [default] **
    Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
    ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
      L0      2/0    2.33 KB   0.5      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0
      L5   8958/0    1.14 TB   0.2      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0
     Sum   8960/0    1.14 TB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0
     Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0

    ** Compaction Stats [default] **
    Priority    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
    -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    Uptime(secs): 601.2 total, 600.0 interval
    Flush(GB): cumulative 0.000, interval 0.000
    AddFile(GB): cumulative 0.000, interval 0.000
    AddFile(Total Files): cumulative 0, interval 0
    AddFile(L0 Files): cumulative 0, interval 0
    AddFile(Keys): cumulative 0, interval 0
    Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
    Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
    Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count

    ** File Read Latency Histogram By Level [default] **

    ** Compaction Stats [default] **
    Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
    ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------

已经按照你这个步骤启动好多次了, 都还是不行

吴老师, 这个我看了下, 除了max_open_files不是我配置的之外, 其他都是已经配置生效了, 在启动参数中了

  1. local_config为false,一切以meta为准,至于你起不来是update configs的时候少了什么符号 你对比下
  2. local_config为true时一切都以本地为准,show configs也没用

disable_auto_compact = true 再试试吧。
瞬间压力应该是flush memfile吧

1 个赞

false前提下, 我使用了下面这2句更新命令, 后来某些原因需要重启的时候就报错了, 报错大概日志是启动的时候会加载上次我在客户端update的配置(就下面这2句话)
UPDATE CONFIGS storage:rocksdb_db_options = {max_subcompactions: 32, max_background_jobs: 64,max_open_files: 100000000, max_background_compaction: 32, num_compaction_threads: 32, max_background_flushes: 32, rate_limit: 536870912}

UPDATE CONFIGS storage:rocksdb_column_family_options = {max_bytes_for_level_base: 536870912, max_write_buffer_number: 10, write_buffer_size: 134217728, min_write_buffer_number_to_merge: 4, level0_file_num_compaction_trigger: 4, target_file_size_base: 134217728, target_file_size_multiplier: 1}

true的时候, 我的配置如下, 少了max_background_compaction和num_compaction_threads这2个配置, 为啥不加呢? 因为加上任何一个都起不来,…还是加载配置出错, 对了, 有方法吧之前在客户端更新的配置缓存清除吗???
–rocksdb_column_family_options={“max_bytes_for_level_base”:“536870912”,“max_write_buffer_number”:“10”,“write_buffer_size”:“134217728”,“min_write_buffer_number_to_merge”:“4”,“level0_file_num_compaction_trigger”:“4”,“target_file_size_base”:“134217728”,“target_file_size_multiplier”:“1”}

--rocksdb_db_options={"max_subcompactions":"32","max_background_jobs":"64","max_open_files":"100000000","max_background_flushes":"32"}

disable_auto_compact = true这个能消除stall condition…吗? 我现在已经导入了50亿数据, 停的话还得重新开始, 有办法能让他在local_conifg=true的情况下不重启升生效吗???

我觉得就是周期性的flush而已,flush时候可能会略微慢一点。上层应用重试一下好了。
大批量写入的时候没有warning也不太可能的。

1 个赞

我应该知道为啥我在客户端修改了之后启动不了了, 在客户端update的时候不会校验参数的正确性, 随便杀参数都可以设置成功, 一旦设置了, 就可以用show config看到, 但是在重启的时候会加载之前在客户端配置的参数重新更新, 我加的max_background_compaction和num_compaction_threads这2个参数报错的原因是: 第一个少了…s
image

num_compaction_threads这我没在rocksdb的log中找到该参数, 估计没有这参数(个人理解)

那么现在问题来了, 怎么在不使用lcoal_conifg=1的情况下怎么清除客户端的参数设置缓存? 重新更新么???

1 个赞

不要启动storage 只启动graph和meta,把正确的update configs执行了再启动storage

@critical27 , 你好, 怎么更新到空{}呢? 默认是{}

浙ICP备20010487号