nebula升级db_upgrader报错

问答 问题
1
  • nebula 版本：从2.6.2升级到3.1.0
  • 部署方式：分布式，三台机器，metad，storaged，graphd每台都启动
  • 安装方式：RPM
  • 是否为线上版本：N
  • 硬件信息
    • 磁盘 512G
    • CPU、内存信息： 4C16G
  • 问题的具体描述
    按照官方提供的升级步骤，升级nebula，从2.6.2到3.1.0，大致描述下步骤：
  1. 停止所有nebula服务
  2. 替换bin/* 文件
  3. 启动每台的metad，启动一台上的graphd，console连上去看下SHOW HOSTS meta和SHOW META LEADER，没问题则继续
  4. 每台机器执行 /usr/local/nebula/bin/db_upgrader
    –src_db_path=/data1/nebula/data/storage
    –dst_db_path=/data1/nebula/data-backup
    –upgrade_meta_server=10.xx.xxx.159:9559,10.xx.xxx.160:9559,10.xx.xxx.161:9559
    –upgrade_version=2:3
  5. 启动所有 Graph 和 Storage 服务
  6. 检查nebula> SHOW HOSTS;
    nebula> SHOW HOSTS storage;
    nebula> SHOW SPACES;
    nebula> USE <space_name>
    nebula> SHOW PARTS;
    nebula> SUBMIT JOB STATS;
    nebula> SHOW STATS;
  • 相关的 db_upgrader日志信息
    db_upgrader.ERROR:
Log file created at: 2022/06/28 06:03:38
Running on machine: Storage9007
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
E20220628 06:03:38.790912 28474 MetaClient.cpp:1366] Space 0 not found!
E20220628 06:03:38.791205 28474 DbUpgrader.cpp:76] Space id 0 no found
E20220628 06:03:38.791216 28474 DbUpgrader.cpp:59] Init /data1/nebula/data/storage space id 0 failed

db_upgrader.WARNING

Log file created at: 2022/06/28 06:03:38
Running on machine: Storage9007
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
E20220628 06:03:38.790912 28474 MetaClient.cpp:1366] Space 0 not found!
E20220628 06:03:38.791205 28474 DbUpgrader.cpp:76] Space id 0 no found
E20220628 06:03:38.791216 28474 DbUpgrader.cpp:59] Init /data1/nebula/data/storage space id 0 failed
W20220628 06:03:38.791224 28474 DbUpgrader.cpp:1180] Upgrade from path /data1/nebula/data/storage space id 0 to path /data1/nebula/data/data-backup init failed
W20220628 06:03:38.791231 28474 DbUpgrader.cpp:1182] Ignore upgrade /data1/nebula/data/storage space id 0

db_upgrader.INFO

Log file created at: 2022/06/28 06:03:37
Running on machine: Storage9007
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20220628 06:03:37.773161 28465 DbUpgraderTool.cpp:112] Prepare phase begin
I20220628 06:03:37.773564 28465 MetaClient.cpp:80] Create meta client to "10.65.101.159":9559
I20220628 06:03:37.773581 28465 MetaClient.cpp:81] root path: , data path size: 0
I20220628 06:03:38.787227 28465 MetaClient.cpp:3079] Load leader of "10.65.101.159":9779 in 0 space
I20220628 06:03:38.787305 28465 MetaClient.cpp:3079] Load leader of "10.65.101.160":9779 in 0 space
I20220628 06:03:38.787318 28465 MetaClient.cpp:3079] Load leader of "10.65.101.161":9779 in 0 space
I20220628 06:03:38.787333 28465 MetaClient.cpp:3085] Load leader ok
I20220628 06:03:38.790004 28465 MetaClient.cpp:148] Register time task for heartbeat!
I20220628 06:03:38.790349 28465 DbUpgraderTool.cpp:171] Prepare phase end
I20220628 06:03:38.790369 28465 DbUpgraderTool.cpp:174] Upgrade phase begin
I20220628 06:03:38.790704 28474 DbUpgraderTool.cpp:185] Upgrade from path /data1/nebula/data/storage to path /data1/nebula/data/data-backup begin
I20220628 06:03:38.790890 28474 DbUpgrader.cpp:1169] Upgrade from path /data1/nebula/data/storage to path /data1/nebula/data/data-backup in DbUpgrader run begin
E20220628 06:03:38.790912 28474 MetaClient.cpp:1366] Space 0 not found!
E20220628 06:03:38.791205 28474 DbUpgrader.cpp:76] Space id 0 no found
E20220628 06:03:38.791216 28474 DbUpgrader.cpp:59] Init /data1/nebula/data/storage space id 0 failed
W20220628 06:03:38.791224 28474 DbUpgrader.cpp:1180] Upgrade from path /data1/nebula/data/storage space id 0 to path /data1/nebula/data/data-backup init failed
W20220628 06:03:38.791231 28474 DbUpgrader.cpp:1182] Ignore upgrade /data1/nebula/data/storage space id 0
I20220628 06:03:38.791369 28474 RocksEngineConfig.cpp:366] Emplace rocksdb option max_subcompactions=10
I20220628 06:03:38.791379 28474 RocksEngineConfig.cpp:366] Emplace rocksdb option max_background_jobs=10
I20220628 06:03:38.791568 28474 RocksEngineConfig.cpp:366] Emplace rocksdb option hard_pending_compaction_bytes_limit=274877906944
I20220628 06:03:38.791579 28474 RocksEngineConfig.cpp:366] Emplace rocksdb option soft_pending_compaction_bytes_limit=137438953472
I20220628 06:03:38.791582 28474 RocksEngineConfig.cpp:366] Emplace rocksdb option level0_stop_writes_trigger=999999
I20220628 06:03:38.791586 28474 RocksEngineConfig.cpp:366] Emplace rocksdb option level0_slowdown_writes_trigger=999999
I20220628 06:03:38.791591 28474 RocksEngineConfig.cpp:366] Emplace rocksdb option max_bytes_for_level_base=268435456
I20220628 06:03:38.791595 28474 RocksEngineConfig.cpp:366] Emplace rocksdb option max_write_buffer_number=12
I20220628 06:03:38.791599 28474 RocksEngineConfig.cpp:366] Emplace rocksdb option write_buffer_size=134217728
I20220628 06:03:38.791602 28474 RocksEngineConfig.cpp:366] Emplace rocksdb option disable_auto_compactions=true
I20220628 06:03:38.975454 28474 RocksEngine.cpp:97] open rocksdb on /data1/nebula/data/storage/nebula/1/data
I20220628 06:03:38.975605 28474 RocksEngineConfig.cpp:366] Emplace rocksdb option max_subcompactions=10
I20220628 06:03:38.975620 28474 RocksEngineConfig.cpp:366] Emplace rocksdb option max_background_jobs=10
I20220628 06:03:38.975749 28474 RocksEngineConfig.cpp:366] Emplace rocksdb option hard_pending_compaction_bytes_limit=274877906944
I20220628 06:03:38.975759 28474 RocksEngineConfig.cpp:366] Emplace rocksdb option soft_pending_compaction_bytes_limit=137438953472
I20220628 06:03:38.975765 28474 RocksEngineConfig.cpp:366] Emplace rocksdb option level0_stop_writes_trigger=999999
I20220628 06:03:38.975775 28474 RocksEngineConfig.cpp:366] Emplace rocksdb option level0_slowdown_writes_trigger=999999
I20220628 06:03:38.975783 28474 RocksEngineConfig.cpp:366] Emplace rocksdb option max_bytes_for_level_base=268435456
I20220628 06:03:38.975791 28474 RocksEngineConfig.cpp:366] Emplace rocksdb option max_write_buffer_number=12
I20220628 06:03:38.975798 28474 RocksEngineConfig.cpp:366] Emplace rocksdb option write_buffer_size=134217728
I20220628 06:03:38.975807 28474 RocksEngineConfig.cpp:366] Emplace rocksdb option disable_auto_compactions=true
I20220628 06:03:38.980393 28474 RocksEngine.cpp:97] open rocksdb on /data1/nebula/data/data-backup/nebula/1/data
I20220628 06:03:38.980509 28474 DbUpgrader.cpp:94] Src data path: /data1/nebula/data/storage space id 1 has 34 parts
I20220628 06:03:38.980530 28474 DbUpgrader.cpp:138] Tag id 2 has 1 fields!
I20220628 06:03:38.980540 28474 DbUpgrader.cpp:178] Edgetype 3 has 0 fields!
I20220628 06:03:38.980558 28474 DbUpgrader.cpp:1196] Max concurrent spaces: 1
I20220628 06:03:38.980814 28516 DbUpgrader.cpp:1214] Upgrade from path /data1/nebula/data/storage space id 1 to path /data1/nebula/data/data-backup begin
I20220628 06:03:38.980834 28516 DbUpgrader.cpp:980] Start to handle data in space id 1
I20220628 06:03:38.980840 28516 DbUpgrader.cpp:983] Max concurrent parts: 10
I20220628 06:03:38.980845 28516 DbUpgrader.cpp:986] Start to handle vertex/edge/index of parts data in space id 1
I20220628 06:03:38.981081 28517 DbUpgrader.cpp:893] Start to handle vertex/edge/index data in space id 1 part id 1
I20220628 06:03:38.981328 28518 DbUpgrader.cpp:893] Start to handle vertex/edge/index data in space id 1 part id 4
I20220628 06:03:38.986409 28519 DbUpgrader.cpp:893] Start to handle vertex/edge/index data in space id 1 part id 7
I20220628 06:03:38.986835 28520 DbUpgrader.cpp:893] Start to handle vertex/edge/index data in space id 1 part id 10
I20220628 06:03:38.995560 28521 DbUpgrader.cpp:893] Start to handle vertex/edge/index data in space id 1 part id 13
I20220628 06:03:39.010919 28522 DbUpgrader.cpp:893] Start to handle vertex/edge/index data in space id 1 part id 16
I20220628 06:03:39.011953 28520 DbUpgrader.cpp:964] Handle vertex/edge/index data in space id 1 part id 10 succeed
I20220628 06:03:39.012537 28519 DbUpgrader.cpp:964] Handle vertex/edge/index data in space id 1 part id 7 succeed
I20220628 06:03:39.015501 28523 DbUpgrader.cpp:893] Start to handle vertex/edge/index data in space id 1 part id 19
I20220628 06:03:39.016440 28518 DbUpgrader.cpp:964] Handle vertex/edge/index data in space id 1 part id 4 succeed
I20220628 06:03:39.017381 28524 DbUpgrader.cpp:893] Start to handle vertex/edge/index data in space id 1 part id 22
I20220628 06:03:39.022332 28517 DbUpgrader.cpp:964] Handle vertex/edge/index data in space id 1 part id 1 succeed
I20220628 06:03:39.032713 28521 DbUpgrader.cpp:964] Handle vertex/edge/index data in space id 1 part id 13 succeed
I20220628 06:03:39.036319 28518 DbUpgrader.cpp:893] Start to handle vertex/edge/index data in space id 1 part id 25
I20220628 06:03:39.038450 28520 DbUpgrader.cpp:893] Start to handle vertex/edge/index data in space id 1 part id 28
I20220628 06:03:39.042603 28523 DbUpgrader.cpp:964] Handle vertex/edge/index data in space id 1 part id 19 succeed
I20220628 06:03:39.042637 28523 DbUpgrader.cpp:893] Start to handle vertex/edge/index data in space id 1 part id 31
I20220628 06:03:39.042779 28519 DbUpgrader.cpp:893] Start to handle vertex/edge/index data in space id 1 part id 34
I20220628 06:03:39.047322 28517 DbUpgrader.cpp:893] Start to handle vertex/edge/index data in space id 1 part id 37
中间没啥问题省略了些
I20220628 06:03:39.191460 28525 DbUpgrader.cpp:964] Handle vertex/edge/index data in space id 1 part id 91 succeed
I20220628 06:03:39.194231 28519 DbUpgrader.cpp:976] Handle vertex/edge/index of parts data in space id 1 finished
I20220628 06:03:39.194710 28518 DbUpgrader.cpp:976] Handle vertex/edge/index of parts data in space id 1 finished
I20220628 06:03:39.195211 28524 DbUpgrader.cpp:964] Handle vertex/edge/index data in space id 1 part id 100 succeed
I20220628 06:03:39.195230 28524 DbUpgrader.cpp:970] Handle last part: 100 vertex/edge/index data in space id 1 finished.
I20220628 06:03:39.196238 28523 DbUpgrader.cpp:976] Handle vertex/edge/index of parts data in space id 1 finished
I20220628 06:03:39.197458 28520 DbUpgrader.cpp:976] Handle vertex/edge/index of parts data in space id 1 finished
I20220628 06:03:39.201583 28525 DbUpgrader.cpp:976] Handle vertex/edge/index of parts data in space id 1 finished
I20220628 06:03:48.879096 28473 MetaClient.cpp:3079] Load leader of "10.65.101.159":9779 in 0 space
I20220628 06:03:48.879132 28473 MetaClient.cpp:3079] Load leader of "10.65.101.160":9779 in 0 space
I20220628 06:03:48.879138 28473 MetaClient.cpp:3079] Load leader of "10.65.101.161":9779 in 0 space
I20220628 06:03:48.879142 28473 MetaClient.cpp:3085] Load leader ok
I20220628 06:03:49.407317 28516 DbUpgrader.cpp:1084] Copy space id 1 wal file begin
I20220628 06:03:50.355752 28516 DbUpgrader.cpp:1231] Copy space id 1 wal file success
I20220628 06:03:50.355777 28516 DbUpgrader.cpp:1075] Path /data1/nebula/data/data-backup space id 1 compaction begin
I20220628 06:03:50.358397 28475 EventListener.h:21] Rocksdb start compaction column family: default because of ManualCompaction, status: OK, compacted 1 files into 0, base level is 0, output level is 1
I20220628 06:03:50.358744 28475 EventListener.h:35] Rocksdb compaction completed column family: default because of ManualCompaction, status: OK, compacted 1 files into 1, base level is 0, output level is 1
I20220628 06:03:50.358809 28516 DbUpgrader.cpp:1080] Path /data1/nebula/data/data-backup space id 1 compaction success!
I20220628 06:03:50.358817 28516 DbUpgrader.cpp:1241] Upgrade last space: 1 from /data1/nebula/data/storage to path /data1/nebula/data/data-backup end
I20220628 06:03:58.895022 28473 MetaClient.cpp:3079] Load leader of "10.65.101.159":9779 in 0 space
I20220628 06:03:58.895059 28473 MetaClient.cpp:3079] Load leader of "10.65.101.160":9779 in 0 space
I20220628 06:03:58.895066 28473 MetaClient.cpp:3079] Load leader of "10.65.101.161":9779 in 0 space
I20220628 06:03:58.895069 28473 MetaClient.cpp:3085] Load leader ok
I20220628 06:03:58.980964 28474 DbUpgrader.cpp:1206] Upgrade from path /data1/nebula/data/storage to path /data1/nebula/data/data-backup in DbUpgrader run end
I20220628 06:03:58.996440 28474 RocksEngine.h:196] Release rocksdb on /data1/nebula/data/data-backup/nebula/1
I20220628 06:03:58.996807 28474 RocksEngine.h:196] Release rocksdb on /data1/nebula/data/storage/nebula/1
I20220628 06:03:59.009667 28474 DbUpgraderTool.cpp:193] Upgrade from path /data1/nebula/data/storage to path /data1/nebula/data/data-backup end
I20220628 06:03:59.011302 28465 DbUpgraderTool.cpp:202] Upgrade phase end

其实就是会报Space 0 not found
但我跑了一些数据的case，发现好像数据没啥问题，服务也好像成功升级了
最后的检查里面，看起来都没问题，感觉报错了，但没完全报错

2

Space 0 not found，这个错误会有影响吗

3

这个space 0 没有影响。。。当时写数据升级工具的时候，storage 下space 0 这个目录还不存在。。。

1 个赞
4

image
image808×297 52.5 KB

storage/nebula/ 下面一开始就有0和1的，报错位置在这，找到0目录的时候去getSpaceNameByIdFromCache，但没找到0的name

StatusOr<std::string> MetaClient::getSpaceNameByIdFromCache(GraphSpaceID spaceId) {
  if (!ready_) {
    return Status::Error("Not ready!");
  }
  folly::rcu_reader guard;
  const auto& metadata = *metadata_.load();
  auto spaceIt = metadata.localCache_.find(spaceId);
  if (spaceIt == metadata.localCache_.end()) {
    LOG(ERROR) << "Space " << spaceId << " not found!";
    return Status::Error("Space %d not found", spaceId);
  }
  return spaceIt->second->spaceDesc_.get_space_name();
}
5

这个过程会从磁盘读取所有space id。0并不会存在磁盘上，所以localCache里不会有space 0相关数据。之所以无所谓是因为space0不含用户数据。

1 个赞
6

明白了，非常谢谢 :ok_hand:

7

此话题已在最后回复的 7 天后被自动关闭。不再允许新回复。