升级版本后插入数据报错

  • nebula 版本:2.5.1
  • 部署方式:分布式
  • 是否为线上版本: N
  • 硬件信息
    • 磁盘: SSD
    • CPU、内存信息:4core 32G
  • 问题的具体描述
  • 相关的 meta / storage / graph info 日志信息(尽量使用文本形式方便检索)

之前使用rpm包方式安装的Nebula Graph集群,并且在里面插入了数据。后来为了研究Nebula Graph源码,手动在linux系统下编译了Nebula Graph的源码,然后用编译生成的二进制文件覆盖了原安装目录bin下的二进制文件,之后重启了集群,发现storage服务似乎就出问题了

1、show spaces; show tags;这些meta服务相关的能正常work

2、通过客户端向Nebula插入数据报错:

Storage Error: part: 1, error: E_RPC_FAILURE(-3).

3、storage异常日志:

E1019 11:35:51.663599 17363 RaftPart.cpp:1118] [Port: 9780, Space: 32240, Part: 10] Receive response about askForVote from "10.65.101.97":9780, error code is E_TERM_OUT_OF_DATE
E1019 11:36:02.515102 17361 RaftPart.cpp:1118] [Port: 9780, Space: 32240, Part: 13] Receive response about askForVote from "10.65.101.97":9780, error code is E_TERM_OUT_OF_DATE
E1019 11:36:06.818706 17364 RaftPart.cpp:1118] [Port: 9780, Space: 32471, Part: 9] Receive response about askForVote from "10.65.101.97":9780, error code is E_TERM_OUT_OF_DATE
E1019 11:36:21.671645 17363 RaftPart.cpp:1118] [Port: 9780, Space: 15, Part: 4] Receive response about askForVote from "10.65.101.97":9780, error code is E_TERM_OUT_OF_DATE
E1019 11:36:21.672749 17363 RaftPart.cpp:1118] [Port: 9780, Space: 32471, Part: 4] Receive response about askForVote from "10.65.101.97":9780, error code is E_TERM_OUT_OF_DATE
E1019 11:36:46.356030 17362 RaftPart.cpp:1118] [Port: 9780, Space: 32471, Part: 10] Receive response about askForVote from "10.65.101.97":9780, error code is E_TERM_OUT_OF_DATE
E1019 11:36:51.674474 17363 RaftPart.cpp:1118] [Port: 9780, Space: 40861, Part: 9] Receive response about askForVote from "10.65.101.97":9780, error code is E_TERM_OUT_OF_DATE
E1019 11:37:06.821878 17364 RaftPart.cpp:1118] [Port: 9780, Space: 32240, Part: 5] Receive response about askForVote from "10.65.101.97":9780, error code is E_TERM_OUT_OF_DATE
E1019 11:37:21.678627 17363 RaftPart.cpp:1118] [Port: 9780, Space: 196, Part: 12] Receive response about askForVote from "10.65.101.97":9780, error code is E_TERM_OUT_OF_DATE
E1019 11:37:21.679597 17363 RaftPart.cpp:1118] [Port: 9780, Space: 196, Part: 14] Receive response about askForVote from "10.65.101.97":9780, error code is E_TERM_OUT_OF_DATE
E1019 11:37:36.828266 17364 RaftPart.cpp:1118] [Port: 9780, Space: 32471, Part: 11] Receive response about askForVote from "10.65.101.97":9780, error code is E_TERM_OUT_OF_DATE

求助这种问题怎么解?

应该是Raft没有选出主。把10.65.101.97:9780这个机器的LOG发一下?

graph日志

Log file created at: 2021/10/18 03:49:09
Running on machine: nebula-graph8009
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E1018 03:49:09.454118 15890 MetaClient.cpp:131] Heartbeat failed, status:LeaderChanged: Leader changed!

meta日志:

Log file created at: 2021/10/15 13:02:24
Running on machine: nebula-graph8009
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E1015 13:02:24.233103 15882 ActiveHostsMan.cpp:256] Get last update time failed, error: E_LEADER_CHANGED
E1018 03:49:07.535555 15882 ActiveHostsMan.cpp:256] Get last update time failed, error: E_LEADER_CHANGED
E1018 03:49:08.864753 15882 ActiveHostsMan.cpp:256] Get last update time failed, error: E_LEADER_CHANGED
E1018 03:49:09.538585 15882 ActiveHostsMan.cpp:256] Get last update time failed, error: E_LEADER_CHANGED
E1018 03:49:10.871174 15882 ActiveHostsMan.cpp:256] Get last update time failed, error: E_LEADER_CHANGED
E1018 03:49:13.212797 15752 Host.cpp:375] [Port: 9560, Space: 0, Part: 0] [Host: 10.65.101.89:9560] Failed to append logs to the host (Err: E_UNKNOWN_PART)

storaged日志:

E1018 04:03:20.857259 15853 Host.cpp:375] [Port: 9780, Space: 32240, Part: 8] [Host: 10.65.101.89:9780] Failed to append logs to the host (Err: E_UNKNOWN_PART)
E1018 04:03:21.149430 15908 RaftPart.cpp:1118] [Port: 9780, Space: 32471, Part: 4] Receive response about askForVote from "10.65.101.89":9780, error code is E_UNKNOWN_
PART
E1018 04:03:21.720886 15908 RaftPart.cpp:1118] [Port: 9780, Space: 32240, Part: 6] Receive response about askForVote from "10.65.101.89":9780, error code is E_UNKNOWN_
PART
E1018 04:03:22.288300 15908 RaftPart.cpp:1118] [Port: 9780, Space: 40861, Part: 2] Receive response about askForVote from "10.65.101.89":9780, error code is E_TERM_OUT
_OF_DATE
E1018 04:03:22.798655 15908 RaftPart.cpp:1118] [Port: 9780, Space: 32240, Part: 14] Receive response about askForVote from "10.65.101.89":9780, error code is E_UNKNOWN
_PART
E1018 04:03:24.870258 15909 RaftPart.cpp:1118] [Port: 9780, Space: 32240, Part: 1] Receive response about askForVote from "10.65.101.89":9780, error code is E_UNKNOWN_
PART
E1018 04:03:25.381539 15909 RaftPart.cpp:1118] [Port: 9780, Space: 32393, Part: 4] Receive response about askForVote from "10.65.101.89":9780, error code is E_TERM_OUT
_OF_DATE
E1018 04:03:25.891803 15909 RaftPart.cpp:1118] [Port: 9780, Space: 40861, Part: 12] Receive response about askForVote from "10.65.101.89":9780, error code is E_TERM_OU
T_OF_DATE
E1018 04:03:27.041841 15910 RaftPart.cpp:1118] [Port: 9780, Space: 15, Part: 9] Receive response about askForVote from "10.65.101.89":9780, error code is E_UNKNOWN_PAR
T

编译安装的版本和rpm包安装的版本一致吗?

一样的,源码用v2.5.1的分支编译的

确认一下配置文件是否有变动。以及storage的data路径是否一致。

配置文件没有变更,data目录一致

问题机器上的storaged也变更了吗?可以尝试下创建一个新的space看看是不是同样的问题 感觉更像版本不一致 也许可以尝试一下storaged的数据格式升级操作
https://docs.nebula-graph.com.cn/2.5.1/4.deployment-and-installation/3.upgrade-nebula-graph/upgrade-nebula-graph-to-250/

此话题已在最后回复的 30 天后被自动关闭。不再允许新回复。