升配重启集群后删除出现-1005:Storage Error: part: 44, error: E_OUTDATED_TERM(-3071).

  • nebula 版本:3.2
  • 部署方式:分布式
  • 安装方式:源码编译
  • 是否上生产环境:Y
  • 硬件信息
    • 磁盘( 推荐使用 SSD)1T ssd
    • CPU、内存信息 32c64G
  • 问题的具体描述
    CPU 占用过高,升级了一下配置然后重启了集群
    另外重启的时候 机器的graphd和storaged会反复挂掉,要启动好几次才能成功

随后出现


再删除的时候一直出现

想知道要怎么才能恢复?

  • 相关的 meta / storage / graph info 日志信息(尽量使用文本形式方便检索)
    E20241031 11:55:05.401029 52135 StorageAccessExecutor.h:136] Storage Error: part: 27, error: E_OUTDATED_TERM(-3071).
    E20241031 11:55:05.401058 52135 QueryInstance.cpp:137] Storage Error: part: 27, error: E_OUTDATED_TERM(-3071).
    E20241031 11:55:05.526460 52131 StorageAccessExecutor.h:39] DeleteEdgesExecutor failed, error E_OUTDATED_TERM, part 44
    E20241031 11:55:05.526504 52131 StorageAccessExecutor.h:39] DeleteEdgesExecutor failed, error E_OUTDATED_TERM, part 45
    E20241031 11:55:05.526525 52131 StorageAccessExecutor.h:136] Storage Error: part: 44, error: E_OUTDATED_TERM(-3071).
    E20241031 11:55:05.526562 52129 QueryInstance.cpp:137] Storage Error: part: 44, error: E_OUTDATED_TERM(-3071).
    E20241031 11:55:05.537593 52134 StorageAccessExecutor.h:39] DeleteEdgesExecutor failed, error E_OUTDATED_TERM, part 19
    E20241031 11:55:05.537639 52134 StorageAccessExecutor.h:136] Storage Error: part: 19, error: E_OUTDATED_TERM(-3071).
    E20241031 11:55:05.537672 52135 QueryInstance.cpp:137] Storage Error: part: 19, error: E_OUTDATED_TERM(-3071).
    E20241031 11:55:05.572250 52118 StorageAccessExecutor.h:39] DeleteEdgesExecutor failed, error E_OUTDATED_TERM, part 17
    E20241031 11:55:05.572286 52118 StorageAccessExecutor.h:39] DeleteEdgesExecutor failed, error E_OUTDATED_TERM, part 15
    E20241031 11:55:05.572315 52118 StorageAccessExecutor.h:136] Storage Error: part: 17, error: E_OUTDATED_TERM(-3071).
    E20241031 11:55:05.572347 52138 QueryInstance.cpp:137] Storage Error: part: 17, error: E_OUTDATED_TERM(-3071).
    E20241031 11:55:05.699714 52138 StorageAccessExecutor.h:39] DeleteEdgesExecutor failed, error E_OUTDATED_TERM, part 44
    E20241031 11:55:05.699756 52138 StorageAccessExecutor.h:136] Storage Error: part: 44, error: E_OUTDATED_TERM(-3071).
    E20241031 11:55:05.699785 52134 QueryInstance.cpp:137] Storage Error: part: 44, error: E_OUTDATED_TERM(-3071).
    E20241031 11:55:05.708925 52138 StorageAccessExecutor.h:39] DeleteEdgesExecutor failed, error E_OUTDATED_TERM, part 12
    E20241031 11:55:05.708961 52138 StorageAccessExecutor.h:39] DeleteEdgesExecutor failed, error E_OUTDATED_TERM, part 35
    E20241031 11:55:05.708981 52138 StorageAccessExecutor.h:136] Storage Error: part: 12, error: E_OUTDATED_TERM(-3071).
    E20241031 11:55:05.709014 52138 QueryInstance.cpp:137] Storage Error: part: 12, error: E_OUTDATED_TERM(-3071).
    E20241031 11:55:05.972467 52136 StorageAccessExecutor.h:39] DeleteEdgesExecutor failed, error E_OUTDATED_TERM, part 53
    E20241031 11:55:05.972507 52136 StorageAccessExecutor.h:39] DeleteEdgesExecutor failed, error E_OUTDATED_TERM, part 6
    E20241031 11:55:05.972541 52136 StorageAccessExecutor.h:136] Storage Error: part: 53, error: E_OUTDATED_TERM(-3071).
    E20241031 11:55:05.972573 52129 QueryInstance.cpp:137] Storage Error: part: 53, error: E_OUTDATED_TERM(-3071).
    E20241031 11:55:06.122530 52125 StorageAccessExecutor.h:39] DeleteEdgesExecutor failed, error E_OUTDATED_TERM, part 38
    E20241031 11:55:06.122576 52125 StorageAccessExecutor.h:136] Storage Error: part: 38, error: E_OUTDATED_TERM(-3071).
    E20241031 11:55:06.122611 52131 QueryInstance.cpp:137] Storage Error: part: 38, error: E_OUTDATED_TERM(-3071).
    E20241031 11:55:06.175057 52138 StorageAccessExecutor.h:39] DeleteEdgesExecutor failed, error E_OUTDATED_TERM, part 13
    E20241031 11:55:06.175097 52138 StorageAccessExecutor.h:39] DeleteEdgesExecutor failed, error E_OUTDATED_TERM, part 43
    E20241031 11:55:06.175129 52138 StorageAccessExecutor.h:136] Storage Error: part: 13, error: E_OUTDATED_TERM(-3071).
    E20241031 11:55:06.175160 52136 QueryInstance.cpp:137] Storage Error: part: 13, error: E_OUTDATED_TERM(-3071).