nebula2.0.1/2.6.1 导入sst数据丢失覆盖问题

  • nebula 版本:2.0.1 (现升级为2.6.1问题未解决)
  • 部署方式:分布式
  • 安装方式:RPM
  • 是否为线上版本:Y
  • 硬件信息
    • 磁盘 SSD
    • CPU、内存信息
      4台物理机 每台挂载6个ssd 内存512g
  • 数据量
    百亿点,千亿边
  • 问题的描述
    sst导入后旧数据被整体覆盖包括(点和关系)

storage配置

--local_config=true

--rate_limit=1000
########## basics ##########
# Whether to run as a daemon process
--daemonize=true
# The file to host the process id
--pid_file=pids/nebula-storaged.pid

########## logging ##########
# The directory to host logging files
--log_dir=logs
# Log level, 0, 1, 2, 3 for INFO, WARNING, ERROR, FATAL respectively
--minloglevel=0
# Verbose log level, 1, 2, 3, 4, the higher of the level, the more verbose of the logging
--v=0
# Maximum seconds to buffer the log messages
--logbufsecs=0
# Whether to redirect stdout and stderr to separate output files
--redirect_stdout=true
# Destination filename of stdout and stderr, which will also reside in log_dir.
--stdout_log_file=storaged-stdout.log
--stderr_log_file=storaged-stderr.log
# Copy log messages at or above this level to stderr in addition to logfiles. The numbers of severity levels INFO, WARNING, ERROR, and FATAL are 0, 1, 2, and 3, respectively.
--stderrthreshold=2

########## networking ##########
# Comma separated Meta server addresses
--meta_server_addrs=省略
# Local IP used to identify the nebula-storaged process.
# Change it to an address other than loopback if the service is distributed or
# will be accessed remotely.
--local_ip=省略
# Storage daemon listening port
--port=9779
# HTTP service ip
--ws_ip=0.0.0.0
# HTTP service port
--ws_http_port=19779
# HTTP2 service port
--ws_h2_port=19780
# heartbeat with meta service
--heartbeat_interval_secs=1



######### Raft #########
# Raft election timeout
--raft_heartbeat_interval_secs=60
# RPC timeout for raft client (ms)
--raft_rpc_timeout_ms=2000
## recycle Raft WAL
--wal_ttl=14400

########## Disk ##########
# Root data path. Split by comma. e.g. --data_path=/disk1/path1/,/disk2/path2/
# One path per Rocksdb instance.
--data_path=/data/sdb/storage,/data/sdc/storage,/data/sdd/storage,/data/sde/storage,/data/sdf/storage,/data/sdg/storage

# The default reserved bytes for one batch operation
--rocksdb_batch_size=8192
# The default block cache size used in BlockBasedTable.
# The unit is MB.
--rocksdb_block_cache=8
# The type of storage engine, `rocksdb', `memory', etc.
--engine_type=rocksdb

# Compression algorithm, options: no,snappy,lz4,lz4hc,zlib,bzip2,zstd
# For the sake of binary compatibility, the default value is snappy.
# Recommend to use:
#   * lz4 to gain more CPU performance, with the same compression ratio with snappy
#   * zstd to occupy less disk space
#   * lz4hc for the read-heavy write-light scenario
--rocksdb_compression=snappy

# Set different compressions for different levels
# For example, if --rocksdb_compression is snappy,
# "no:no:lz4:lz4::zstd" is identical to "no:no:lz4:lz4:snappy:zstd:snappy"
# In order to disable compression for level 0/1, set it to "no:no"
#--rocksdb_compression_per_level=

# Whether or not to enable rocksdb's statistics, disabled by default
--enable_rocksdb_statistics=false

# Statslevel used by rocksdb to collection statistics, optional values are
#   * kExceptHistogramOrTimers, disable timer stats, and skip histogram stats
#   * kExceptTimers, Skip timer stats
#   * kExceptDetailedTimers, Collect all stats except time inside mutex lock AND time spent on compression.
#   * kExceptTimeForMutex, Collect all stats except the counters requiring to get time inside the mutex lock.
#   * kAll, Collect all stats
--rocksdb_stats_level=kExceptHistogramOrTimers

# Whether or not to enable rocksdb's prefix bloom filter, disabled by default.
--enable_rocksdb_prefix_filtering=false
# Whether or not to enable the whole key filtering.
--enable_rocksdb_whole_key_filtering=true
# The prefix length for each key to use as the filter value.
# can be 12 bytes(PartitionId + VertexID), or 16 bytes(PartitionId + VertexID + TagID/EdgeType).
--rocksdb_filtering_prefix_ltrue=12

############## rocksdb Options ##############
# rocksdb DBOptions in json, each name and value of option is a string, given as "option_name":"option_value" separated by comma
--rocksdb_db_options={"max_background_compactions":"48","max_subcompactions":"48","max_background_flushes":"48","max_open_files":"-1","max_background_jobs":"49","max_file_opening_threads":"48","skip_stats_update_on_db_open":"false"}
# rocksdb ColumnFamilyOptions in json, each name and value of option is string, given as "option_name":"option_value" separated by comma
--rocksdb_column_family_options={"disable_auto_compactions":"true","write_buffer_size":"67108864","max_write_buffer_number":"16","max_bytes_for_level_base":"1073741824"}
# rocksdb BlockBasedTableOptions in json, each name and value of option is string, given as "option_name":"option_value" separated by comma
--rocksdb_block_based_table_options={"block_size":"8192"}

  • 问题的具体描述
    1.创建A空间定义 TAG t1、t2 定义 EGDE_TYPE e1,e2
    2.执行exchange任务生成sst 生成 t1/e1相关数据,执行download,执行ingest,校验数据,无任何问题
    3.执行major compact,过程失败
    4.执行exchange任务生成sst 生成 t2/e2相关数据,执行download,执行ingest
    5.执行major compact,成功
    6.校验数据发现部分t1点/e1关系数据丢失(t1、t2为相同的vertex_id),t1数据丢失的点现象表现为

    • 执行 match (v) where id(v)=‘xxxx’ return v; 无数据返回
    • 执行 match (v)-[e:e1]-(v2) where id(v)=‘xxxx’ return v; 无数据返回
    • 执行 match (v)-[e:e2]-(v2) where id(v)=‘xxxx’ return v; 有数据返回,但点上无任何tag
  • 相关的 meta / storage / graph info 日志信息
    storaged服务日志未见异常

你执行下submit job stats;show stats 看下每个tag和edgetype下的数据量是对的么。
然后你match 查不出来的数据试一下fetch看能不能查出来。

1.目前 submit job stats 任务在运行当中,使用fetch语句也查不出来
2.补充描述下,其中有部分点出现这个想象,具体数据量需要统计下
3.出现问题的点都呈现"空点的状态" 只有id,没有tag属性


4.因为两批数据导入过程中执行了major compact ,但是出错了, storaged服务因内存不足退出,不知道是不是这个原因导致丢数据

  1. 你两次生成sst时,exchange的配置文件中有分别更改tag 名和edgetype名吧。 以及生成sst时你的schema没有进行任务变化吧?
  2. 生成sst过程中是否有异常
  3. 你在将t1、e1数据ingest之后,有进行数据查看不,这时的数据是正常的吗

1.原有tag/edge_type未做任何改变,导入过程中有新增tag和edge_type
2.在生成/导入sst过程当中程序未报错
3.t1,e1导入后校验数据为正常状态

浙ICP备20010487号