Nebula v2.0GA meta 服务启动失败

liyu04 · 2021 年4 月 7 日 13:13

提问参考模版：

nebula 版本：（v2.0GA
部署方式（分布式）：
硬件信息
- 磁盘（ SATA）
- CPU：Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
- 内存：125
- 内核：Linux 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
问题的具体描述
meta服务部署3个节点，其中一个节点启动一段时间后推出，再次启动失败。无端口冲突。
端口9560正常，9559不存在
相关的 meta / storage / graph info 日志信息

metad-stderr.log

*** Aborted at 1617798237 (unix time) try "date -d @1617798237" if you are using GNU date ***
PC: @     0x2ab1d564042d __GI_nanosleep
*** SIGTERM (@0x8e75) received by PID 31444 (TID 0x2ab1d484bd00) from PID 36469; stack trace: ***
    @          0x200d2a1 (unknown)
    @     0x2ab1d53760ff (unknown)
    @     0x2ab1d564042d __GI_nanosleep
    @     0x2ab1d56402c3 __sleep
    @          0x111504a initKV()
    @          0x10dd71b main
    @     0x2ab1d55a4b14 __libc_start_main
    @          0x1113a30 (unknown)

nebula-metad.ERROR

E0407 21:06:14.213541 36831 ThriftClientManager.inl:33] Invalid Channel: 0x2b02ce80e300 for host: "110.251.11.108":9560
E0407 21:06:15.214923 36812 RaftPart.cpp:1143] [Port: 9560, Space: 0, Part: 0] Receive response about askForVote from "10.251.11.110":9560, error code is -6
E0407 21:06:15.214949 36812 RaftPart.cpp:1143] [Port: 9560, Space: 0, Part: 0] Receive response about askForVote from "10.251.11.109":9560, error code is -6
E0407 21:06:16.146162 36832 ThriftClientManager.inl:33] Invalid Channel: 0x2b02cf60e300 for host: "110.251.11.108":9560
E0407 21:06:17.147369 36813 RaftPart.cpp:1143] [Port: 9560, Space: 0, Part: 0] Receive response about askForVote from "10.251.11.109":9560, error code is -6
E0407 21:06:17.147397 36813 RaftPart.cpp:1143] [Port: 9560, Space: 0, Part: 0] Receive response about askForVote from "10.251.11.110":9560, error code is -6
E0407 21:06:18.797339 36817 ThriftClientManager.inl:33] Invalid Channel: 0x2b02c240f300 for host: "110.251.11.108":9560
E0407 21:06:19.798599 36814 RaftPart.cpp:1143] [Port: 9560, Space: 0, Part: 0] Receive response about askForVote from "10.251.11.109":9560, error code is -6
E0407 21:06:19.798627 36814 RaftPart.cpp:1143] [Port: 9560, Space: 0, Part: 0] Receive response about askForVote from "10.251.11.110":9560, error code is -6
E0407 21:06:21.232398 36818 ThriftClientManager.inl:33] Invalid Channel: 0x2b02c320e300 for host: "110.251.11.108":9560
E0407 21:06:22.233525 36815 RaftPart.cpp:1143] [Port: 9560, Space: 0, Part: 0] Receive response about askForVote from "10.251.11.109":9560, error code is -6
E0407 21:06:22.233553 36815 RaftPart.cpp:1143] [Port: 9560, Space: 0, Part: 0] Receive response about askForVote from "10.251.11.110":9560, error code is -6
E0407 21:06:23.375968 36819 ThriftClientManager.inl:33] Invalid Channel: 0x2b02c400f300 for host: "110.251.11.108":9560
E0407 21:06:24.376824 36812 RaftPart.cpp:1143] [Port: 9560, Space: 0, Part: 0] Receive response about askForVote from "10.251.11.109":9560, error code is -6
E0407 21:06:24.376852 36812 RaftPart.cpp:1143] [Port: 9560, Space: 0, Part: 0] Receive response about askForVote from "10.251.11.110":9560, error code is -6
E0407 21:06:25.502238 36820 ThriftClientManager.inl:33] Invalid Channel: 0x2b02c4e0e300 for host: "110.251.11.108":9560
E0407 21:06:26.503350 36813 RaftPart.cpp:1143] [Port: 9560, Space: 0, Part: 0] Receive response about askForVote from "10.251.11.109":9560, error code is -6
E0407 21:06:26.503376 36813 RaftPart.cpp:1143] [Port: 9560, Space: 0, Part: 0] Receive response about askForVote from "10.251.11.110":9560, error code is -6

nebula-metad.WARNING

E0407 21:07:08.935899 36813 RaftPart.cpp:1143] [Port: 9560, Space: 0, Part: 0] Receive response about askForVote from "10.251.11.109":9560, error code is -6
E0407 21:07:08.935926 36813 RaftPart.cpp:1143] [Port: 9560, Space: 0, Part: 0] Receive response about askForVote from "10.251.11.110":9560, error code is -6
E0407 21:07:10.866134 36825 ThriftClientManager.inl:33] Invalid Channel: 0x2b02c940e300 for host: "110.251.11.108":9560
E0407 21:07:11.866343 36814 RaftPart.cpp:1143] [Port: 9560, Space: 0, Part: 0] Receive response about askForVote from "10.251.11.109":9560, error code is -6
E0407 21:07:11.866371 36814 RaftPart.cpp:1143] [Port: 9560, Space: 0, Part: 0] Receive response about askForVote from "10.251.11.110":9560, error code is -6
E0407 21:07:13.481236 36826 ThriftClientManager.inl:33] Invalid Channel: 0x2b02ca20e300 for host: "110.251.11.108":9560
E0407 21:07:14.481475 36815 RaftPart.cpp:1143] [Port: 9560, Space: 0, Part: 0] Receive response about askForVote from "10.251.11.109":9560, error code is -6
E0407 21:07:14.481523 36815 RaftPart.cpp:1143] [Port: 9560, Space: 0, Part: 0] Receive response about askForVote from "10.251.11.110":9560, error code is -6
E0407 21:07:16.379714 36827 ThriftClientManager.inl:33] Invalid Channel: 0x2b02cb00e300 for host: "110.251.11.108":9560
E0407 21:07:17.380057 36812 RaftPart.cpp:1143] [Port: 9560, Space: 0, Part: 0] Receive response about askForVote from "10.251.11.109":9560, error code is -6
E0407 21:07:17.380090 36812 RaftPart.cpp:1143] [Port: 9560, Space: 0, Part: 0] Receive response about askForVote from "10.251.11.110":9560, error code is -6
E0407 21:07:19.363390 36828 ThriftClientManager.inl:33] Invalid Channel: 0x2b02cbe0e300 for host: "110.251.11.108":9560
E0407 21:07:20.365741 36813 RaftPart.cpp:1143] [Port: 9560, Space: 0, Part: 0] Receive response about askForVote from "10.251.11.109":9560, error code is -6
E0407 21:07:20.365783 36813 RaftPart.cpp:1143] [Port: 9560, Space: 0, Part: 0] Receive response about askForVote from "10.251.11.110":9560, error code is -6

dingding · 2021 年4 月 8 日 01:11

把你启动不了的meta节点的info日志贴下，看stderr日志，是初始化rocksdb失败了，看是不是data目录访问不了，或者同个同时启动了多个metad服务。

liyu04 · 2021 年4 月 8 日 02:53

1.data目录可以正常访问
/home/aaa/data/meta/nebula/0/data:
total 5024
-rw-r–r-- 1 root root 943 Apr 7 20:16 000004.sst
-rw-r–r-- 1 root root 0 Apr 8 10:48 000012.log
-rw-r–r-- 1 root root 16 Apr 8 10:48 CURRENT
-rw-r–r-- 1 root root 37 Apr 7 20:15 IDENTITY
-rw-r–r-- 1 root root 0 Apr 7 20:15 LOCK
-rw-r–r-- 1 root root 21699 Apr 8 10:48 LOG
-rw-r–r-- 1 root root 21092 Apr 7 20:15 LOG.old.1617797800309682
-rw-r–r-- 1 root root 22776 Apr 7 20:16 LOG.old.1617798249083104
-rw-r–r-- 1 root root 436073 Apr 8 10:44 LOG.old.1617850088370619
-rw-r–r-- 1 root root 111 Apr 8 10:48 MANIFEST-000011
-rw-r–r-- 1 root root 5248 Apr 7 20:24 OPTIONS-000011
-rw-r–r-- 1 root root 5248 Apr 8 10:48 OPTIONS-000014
2.启动 nebula.service start metad ，9560端口可以启动，9559拒绝访问，应该不是同时启动两个metad

nebula-metad.INFO
I0408 10:48:36.051765 2282 RaftPart.cpp:1321] [Port: 9560, Space: 0, Part: 0] Schedule new task
I0408 10:48:36.429463 2228 MetaDaemon.cpp:112] Leader has not been elected, sleep 1s
I0408 10:48:37.429576 2228 MetaDaemon.cpp:112] Leader has not been elected, sleep 1s
I0408 10:48:37.662431 2283 RaftPart.cpp:1165] [Port: 9560, Space: 0, Part: 0] Start leader election…
I0408 10:48:37.662500 2283 RaftPart.cpp:1193] [Port: 9560, Space: 0, Part: 0] Sending out an election request (space = 0, part = 0, term = 13, lastLogId = 0, lastLogTerm = 0, candidateIP = 10.251.11.108, candidatePort = 9560)
I0408 10:48:37.662513 2283 RaftPart.cpp:1213] [Port: 9560, Space: 0, Part: 0] Sending AskForVoteRequest to [Port: 9560, Space: 0, Part: 0] [Host: 110.251.11.108:9560]
I0408 10:48:37.662529 2283 RaftPart.cpp:1213] [Port: 9560, Space: 0, Part: 0] Sending AskForVoteRequest to [Port: 9560, Space: 0, Part: 0] [Host: 10.251.11.109:9560]
I0408 10:48:37.662537 2283 RaftPart.cpp:1213] [Port: 9560, Space: 0, Part: 0] Sending AskForVoteRequest to [Port: 9560, Space: 0, Part: 0] [Host: 10.251.11.110:9560]
I0408 10:48:37.662546 2283 RaftPart.cpp:1232] [Port: 9560, Space: 0, Part: 0] AskForVoteRequest has been sent to all peers, waiting for responses
I0408 10:48:37.662562 2298 ThriftClientManager.inl:48] There is no existing client to “110.251.11.108”:9560, trying to create one
I0408 10:48:37.662608 2298 ThriftClientManager.inl:69] Connecting to “110.251.11.108”:9560 for 1 times
I0408 10:48:37.662859 2298 ThriftClientManager.inl:48] There is no existing client to “10.251.11.109”:9560, trying to create one
I0408 10:48:37.662878 2298 ThriftClientManager.inl:69] Connecting to “10.251.11.109”:9560 for 2 times
I0408 10:48:37.662994 2298 ThriftClientManager.inl:48] There is no existing client to “10.251.11.110”:9560, trying to create one
I0408 10:48:37.663013 2298 ThriftClientManager.inl:69] Connecting to “10.251.11.110”:9560 for 3 times
I0408 10:48:37.663548 2298 AsyncSocket.cpp:1931] this=0x2ac053e00710, AsyncSocket::handleRead() got 38 bytes
I0408 10:48:37.663825 2298 AsyncSocket.cpp:1931] this=0x2ac053e00a90, AsyncSocket::handleRead() got 38 bytes
I0408 10:48:38.429702 2228 MetaDaemon.cpp:112] Leader has not been elected, sleep 1s
I0408 10:48:38.664052 2298 CollectNSucceeded.inl:63] Set Value [completed=3, total=3, Result list size=2]
I0408 10:48:38.664149 2283 RaftPart.cpp:1239] [Port: 9560, Space: 0, Part: 0] Got AskForVote response back
E0408 10:48:38.664196 2283 RaftPart.cpp:1143] [Port: 9560, Space: 0, Part: 0] Receive response about askForVote from “10.251.11.109”:9560, error code is -6
E0408 10:48:38.664223 2283 RaftPart.cpp:1143] [Port: 9560, Space: 0, Part: 0] Receive response about askForVote from “10.251.11.110”:9560, error code is -6
I0408 10:48:38.664292 2283 RaftPart.cpp:1275] [Port: 9560, Space: 0, Part: 0] No one is elected, continue the election
I0408 10:48:38.664301 2283 RaftPart.cpp:1307] [Port: 9560, Space: 0, Part: 0] Wait for a while and continue the leader election
I0408 10:48:38.664309 2283 RaftPart.cpp:1321] [Port: 9560, Space: 0, Part: 0] Schedule new task
I0408 10:48:39.430748 2228 MetaDaemon.cpp:112] Leader has not been elected, sleep 1s
I0408 10:48:40.007707 2284 RaftPart.cpp:1165] [Port: 9560, Space: 0, Part: 0] Start leader election…
I0408 10:48:40.007753 2284 RaftPart.cpp:1193] [Port: 9560, Space: 0, Part: 0] Sending out an election request (space = 0, part = 0, term = 14, lastLogId = 0, lastLogTerm = 0, candidateIP = 10.251.11.108, candidatePort = 9560)
I0408 10:48:40.007767 2284 RaftPart.cpp:1213] [Port: 9560, Space: 0, Part: 0] Sending AskForVoteRequest to [Port: 9560, Space: 0, Part: 0] [Host: 110.251.11.108:9560]
I0408 10:48:40.007782 2284 RaftPart.cpp:1213] [Port: 9560, Space: 0, Part: 0] Sending AskForVoteRequest to [Port: 9560, Space: 0, Part: 0] [Host: 10.251.11.109:9560]
I0408 10:48:40.007791 2284 RaftPart.cpp:1213] [Port: 9560, Space: 0, Part: 0] Sending AskForVoteRequest to [Port: 9560, Space: 0, Part: 0] [Host: 10.251.11.110:9560]
I0408 10:48:40.007799 2284 RaftPart.cpp:1232] [Port: 9560, Space: 0, Part: 0] AskForVoteRequest has been sent to all peers, waiting for responses
I0408 10:48:40.007809 2299 ThriftClientManager.inl:48] There is no existing client to “110.251.11.108”:9560, trying to create one
I0408 10:48:40.007853 2299 ThriftClientManager.inl:69] Connecting to “110.251.11.108”:9560 for 1 times
I0408 10:48:40.008121 2299 ThriftClientManager.inl:48] There is no existing client to “10.251.11.109”:9560, trying to create one
I0408 10:48:40.008141 2299 ThriftClientManager.inl:69] Connecting to “10.251.11.109”:9560 for 2 times
I0408 10:48:40.008257 2299 ThriftClientManager.inl:48] There is no existing client to “10.251.11.110”:9560, trying to create one
I0408 10:48:40.008276 2299 ThriftClientManager.inl:69] Connecting to “10.251.11.110”:9560 for 3 times
I0408 10:48:40.008895 2299 AsyncSocket.cpp:1931] this=0x2ac054c00710, AsyncSocket::handleRead() got 38 bytes
I0408 10:48:40.009038 2299 AsyncSocket.cpp:1931] this=0x2ac054c00a90, AsyncSocket::handleRead() got 38 bytes
I0408 10:48:40.430891 2228 MetaDaemon.cpp:112] Leader has not been elected, sleep 1s

dingding · 2021 年4 月 8 日 03:05

你的metad leader没有选出来，所以不会去监听9559端口，你看下 10.251.11.110 的metad的日志，10.251.11.110 的meta一直连不上，是不是防火墙问题。

liyu04 · 2021 年4 月 8 日 03:13

网络没问题telnet都是通的
10.251.11.110 日志

nebula-metad.INFO
E0408 11:10:15.056470 33215 ThriftClientManager.inl:33] Invalid Channel: 0x2acc6c60e300 for host: “110.251.11.108”:9560
E0408 11:10:15.627365 33204 ThriftClientManager.inl:33] Invalid Channel: 0x2acc62c0f300 for host: “110.251.11.108”:9560
I0408 11:10:16.434839 33192 RaftPart.cpp:1361] [Port: 9560, Space: 0, Part: 0] Recieved a VOTING request: space = 0, partition = 0, candidateAddr = 10.251.11.108:9560, term = 594, lastLogId = 0, lastLogTerm = 0
I0408 11:10:16.434934 33192 RaftPart.cpp:1394] [Port: 9560, Space: 0, Part: 0] The partition currently is a Leader, lastLogId 641493, lastLogTerm 6, committedLogId 641493, term 6
I0408 11:10:16.434947 33192 RaftPart.cpp:1421] [Port: 9560, Space: 0, Part: 0] The partition’s last term to receive a log is 6, which is newer than the candidate’s log 0. So the candidate will be rejected
E0408 11:10:17.189090 33210 ThriftClientManager.inl:33] Invalid Channel: 0x2acc6800e900 for host: “110.251.11.108”:9560
I0408 11:10:18.085806 33192 RaftPart.cpp:1361] [Port: 9560, Space: 0, Part: 0] Recieved a VOTING request: space = 0, partition = 0, candidateAddr = 10.251.11.108:9560, term = 595, lastLogId = 0, lastLogTerm = 0
I0408 11:10:18.085887 33192 RaftPart.cpp:1394] [Port: 9560, Space: 0, Part: 0] The partition currently is a Leader, lastLogId 641497, lastLogTerm 6, committedLogId 641497, term 6
I0408 11:10:18.085901 33192 RaftPart.cpp:1421] [Port: 9560, Space: 0, Part: 0] The partition’s last term to receive a log is 6, which is newer than the candidate’s log 0. So the candidate will be rejected
E0408 11:10:18.225170 33214 ThriftClientManager.inl:33] Invalid Channel: 0x2acc6b80e600 for host: “110.251.11.108”:9560
E0408 11:10:18.874364 33215 ThriftClientManager.inl:33] Invalid Channel: 0x2acc6c60e300 for host: “110.251.11.108”:9560
E0408 11:10:19.534591 33205 ThriftClientManager.inl:33] Invalid Channel: 0x2acc63a0e300 for host: “110.251.11.108”:9560
E0408 11:10:20.070431 33212 ThriftClientManager.inl:33] Invalid Channel: 0x2acc69c0e600 for host: “110.251.11.108”:9560
E0408 11:10:20.625630 33205 ThriftClientManager.inl:33] Invalid Channel: 0x2acc63a0e300 for host: “110.251.11.108”:9560
I0408 11:10:20.901903 33192 RaftPart.cpp:1361] [Port: 9560, Space: 0, Part: 0] Recieved a VOTING request: space = 0, partition = 0, candidateAddr = 10.251.11.108:9560, term = 596, lastLogId = 0, lastLogTerm = 0
I0408 11:10:20.901952 33192 RaftPart.cpp:1394] [Port: 9560, Space: 0, Part: 0] The partition currently is a Leader, lastLogId 641525, lastLogTerm 6, committedLogId 641525, term 6
I0408 11:10:20.901962 33192 RaftPart.cpp:1421] [Port: 9560, Space: 0, Part: 0] The partition’s last term to receive a log is 6, which is newer than the candidate’s log 0. So the candidate will be rejected
E0408 11:10:21.153839 33215 ThriftClientManager.inl:33] Invalid Channel: 0x2acc6c60e300 for host: “110.251.11.108”:9560
E0408 11:10:21.728271 33207 ThriftClientManager.inl:33] Invalid Channel: 0x2acc6560e300 for host: “110.251.11.108”:9560
nebula-metad.WARNING
E0408 11:12:27.386147 33204 ThriftClientManager.inl:33] Invalid Channel: 0x2acc62c0f300 for host: “110.251.11.108”:9560
E0408 11:12:28.367835 33208 ThriftClientManager.inl:33] Invalid Channel: 0x2acc6640e300 for host: “110.251.11.108”:9560
E0408 11:12:29.035117 33209 ThriftClientManager.inl:33] Invalid Channel: 0x2acc6720e600 for host: “110.251.11.108”:9560
E0408 11:12:29.661476 33215 ThriftClientManager.inl:33] Invalid Channel: 0x2acc6c60e300 for host: “110.251.11.108”:9560
E0408 11:12:30.233407 33206 ThriftClientManager.inl:33] Invalid Channel: 0x2acc6480e300 for host: “110.251.11.108”:9560
E0408 11:12:30.744155 33213 ThriftClientManager.inl:33] Invalid Channel: 0x2acc6aa0e600 for host: “110.251.11.108”:9560
E0408 11:12:31.296614 33209 ThriftClientManager.inl:33] Invalid Channel: 0x2acc6720e600 for host: “110.251.11.108”:9560
nebula-metad.ERROR
E0408 11:12:41.875746 33203 ThriftClientManager.inl:33] Invalid Channel: 0x2acc61e0e300 for host: “110.251.11.108”:9560
E0408 11:12:42.504614 33211 ThriftClientManager.inl:33] Invalid Channel: 0x2acc68e0e600 for host: “110.251.11.108”:9560
E0408 11:12:43.128129 33203 ThriftClientManager.inl:33] Invalid Channel: 0x2acc61e0e300 for host: “110.251.11.108”:9560
E0408 11:12:43.681048 33210 ThriftClientManager.inl:33] Invalid Channel: 0x2acc6800e900 for host: “110.251.11.108”:9560
E0408 11:12:44.408169 33203 ThriftClientManager.inl:33] Invalid Channel: 0x2acc61e0e300 for host: “110.251.11.108”:9560

dingding · 2021 年4 月 8 日 03:17

你把三个meta的配置文件贴下，我咋看有四个meta地址。

critical27 · 2021 年4 月 8 日 03:18

大概率还是网络问题 netstat或者ss看下到底链接建立成功没有

liyu04 · 2021 年4 月 8 日 03:38

108

########## basics ##########
# Whether to run as a daemon process
--daemonize=true
# The file to host the process id
--pid_file=pids/nebula-metad.pid

########## logging ##########
# The directory to host logging files, which must already exists
--log_dir=logs
# Log level, 0, 1, 2, 3 for INFO, WARNING, ERROR, FATAL respectively
--minloglevel=0
# Verbose log level, 1, 2, 3, 4, the higher of the level, the more verbose of the logging
--v=4
# Maximum seconds to buffer the log messages
--logbufsecs=0
# Whether to redirect stdout and stderr to separate output files
--redirect_stdout=true
# Destination filename of stdout and stderr, which will also reside in log_dir.
--stdout_log_file=metad-stdout.log
--stderr_log_file=metad-stderr.log
# Copy log messages at or above this level to stderr in addition to logfiles. The numbers of severity levels INFO, WARNING, ERROR, and FATAL are 0, 1, 2, and 3, respectively.
--stderrthreshold=0

########## networking ##########
# Comma separated Meta Server addresses
--meta_server_addrs=110.251.11.108:9559,10.251.11.109:9559,10.251.11.110:9559
# Local IP used to identify the nebula-metad process.
# Change it to an address other than loopback if the service is distributed or
# will be accessed remotely.
--local_ip=10.251.11.108
# Meta daemon listening port
--port=9559
# HTTP service ip
--ws_ip=0.0.0.0
# HTTP service port
--ws_http_port=11000
# HTTP2 service port
--ws_h2_port=19560

########## storage ##########
# Root data path, here should be only single path for metad
--data_path=data/meta

########## Misc #########
# The default number of parts when a space is created
--default_parts_num=100
# The default replica factor when a space is created
--default_replica_factor=1

--heartbeat_interval_secs=10
############## rocksdb Options ##############
--rocksdb_wal_sync=true

109

########## basics ##########
# Whether to run as a daemon process
--daemonize=true
# The file to host the process id
--pid_file=pids/nebula-metad.pid

########## logging ##########
# The directory to host logging files, which must already exists
--log_dir=logs
# Log level, 0, 1, 2, 3 for INFO, WARNING, ERROR, FATAL respectively
--minloglevel=0
# Verbose log level, 1, 2, 3, 4, the higher of the level, the more verbose of the logging
--v=0
# Maximum seconds to buffer the log messages
--logbufsecs=0
# Whether to redirect stdout and stderr to separate output files
--redirect_stdout=true
# Destination filename of stdout and stderr, which will also reside in log_dir.
--stdout_log_file=metad-stdout.log
--stderr_log_file=metad-stderr.log
# Copy log messages at or above this level to stderr in addition to logfiles. The numbers of severity levels INFO, WARNING, ERROR, and FATAL are 0, 1, 2, and 3, respectively.
--stderrthreshold=2

########## networking ##########
# Comma separated Meta Server addresses
--meta_server_addrs=110.251.11.108:9559,10.251.11.109:9559,10.251.11.110:9559
# Local IP used to identify the nebula-metad process.
# Change it to an address other than loopback if the service is distributed or
# will be accessed remotely.
--local_ip=10.251.11.109
# Meta daemon listening port
--port=9559
# HTTP service ip
--ws_ip=0.0.0.0
# HTTP service port
--ws_http_port=11000
# HTTP2 service port
--ws_h2_port=19560

########## storage ##########
# Root data path, here should be only single path for metad
--data_path=data/meta

########## Misc #########
# The default number of parts when a space is created
--default_parts_num=100
# The default replica factor when a space is created
--default_replica_factor=1

--heartbeat_interval_secs=10
############## rocksdb Options ##############
--rocksdb_wal_sync=true

110

########## basics ##########
# Whether to run as a daemon process
--daemonize=true
# The file to host the process id
--pid_file=pids/nebula-metad.pid

########## logging ##########
# The directory to host logging files, which must already exists
--log_dir=logs
# Log level, 0, 1, 2, 3 for INFO, WARNING, ERROR, FATAL respectively
--minloglevel=0
# Verbose log level, 1, 2, 3, 4, the higher of the level, the more verbose of the logging
--v=0
# Maximum seconds to buffer the log messages
--logbufsecs=0
# Whether to redirect stdout and stderr to separate output files
--redirect_stdout=true
# Destination filename of stdout and stderr, which will also reside in log_dir.
--stdout_log_file=metad-stdout.log
--stderr_log_file=metad-stderr.log
# Copy log messages at or above this level to stderr in addition to logfiles. The numbers of severity levels INFO, WARNING, ERROR, and FATAL are 0, 1, 2, and 3, respectively.
--stderrthreshold=2

########## networking ##########
# Comma separated Meta Server addresses
--meta_server_addrs=110.251.11.108:9559,10.251.11.109:9559,10.251.11.110:9559
# Local IP used to identify the nebula-metad process.
# Change it to an address other than loopback if the service is distributed or
# will be accessed remotely.
--local_ip=10.251.11.110
# Meta daemon listening port
--port=9559
# HTTP service ip
--ws_ip=0.0.0.0
# HTTP service port
--ws_http_port=11000
# HTTP2 service port
--ws_h2_port=19560

########## storage ##########
# Root data path, here should be only single path for metad
--data_path=data/meta

########## Misc #########
# The default number of parts when a space is created
--default_parts_num=100
# The default replica factor when a space is created
--default_replica_factor=1

--heartbeat_interval_secs=10
############## rocksdb Options ##############
--rocksdb_wal_sync=true

netstat

[aaa@hdp200 nebula]$ netstat -anlp|grep 9559 |head
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 10.251.11.108:12816     10.251.11.110:9559      ESTABLISHED -                   
tcp        0      0 10.251.11.108:12764     10.251.11.110:9559      ESTABLISHED -                   
tcp        0      0 10.251.11.108:12589     10.251.11.110:9559      ESTABLISHED -                   
tcp        0      0 10.251.11.108:12777     10.251.11.110:9559      ESTABLISHED -                   
tcp        0      0 10.251.11.108:12845     10.251.11.110:9559      ESTABLISHED -                   
tcp        0      0 10.251.11.108:12554     10.251.11.110:9559      ESTABLISHED -                   
tcp        0      0 10.251.11.108:12621     10.251.11.110:9559      ESTABLISHED -                   
tcp        0      0 10.251.11.108:57962     10.251.11.109:9559      ESTABLISHED -                   
tcp        0      0 10.251.11.108:12861     10.251.11.110:9559      ESTABLISHED -                   
tcp        0      0 10.251.11.108:12755     10.251.11.110:9559      ESTABLISHED -                   
[aaa@hdp200 nebula]$ netstat -anlp|grep 9560 |head
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:9560            0.0.0.0:*               LISTEN      -                   
tcp        0      0 10.251.11.108:41306     10.251.11.110:9560      ESTABLISHED -                   
tcp        0      0 10.251.11.108:38469     10.251.11.109:9560      ESTABLISHED -                   
tcp        0      0 10.251.11.108:19560     10.251.11.175:9779      ESTABLISHED -                   
tcp        0      0 10.251.11.108:41253     10.251.11.110:9560      ESTABLISHED -                   
tcp        0      0 10.251.11.108:38511     10.251.11.109:9560      ESTABLISHED -                   
tcp        0      0 10.251.11.108:41279     10.251.11.110:9560      ESTABLISHED -                   
tcp        0      0 10.251.11.108:38462     10.251.11.109:9560      ESTABLISHED -                   
tcp        0      0 10.251.11.108:38476     10.251.11.109:9560      ESTABLISHED -                   
tcp        0      0 10.251.11.108:41257     10.251.11.110:9560      ESTABLISHED -

dingding · 2021 年4 月 8 日 03:51

I0408 11:10:16.434947 33192 RaftPart.cpp:1421] [Port: 9560, Space: 0, Part: 0] The partition’s last term to receive a log is 6, which is newer than the candidate’s log 0. So the candidate will be rejected

这里异常了，你是不是有删除data操作，然后重启过服务

liyu04 · 2021 年4 月 8 日 06:49

出现这个问题之后删除的，data目录下数据有重新生成

dingding · 2021 年4 月 8 日 06:53

出现这个问题应该是有的meta的data没有删除，导致下次启动的时候和其他meta的不一致了。你可以先把所有meta服务全部停掉，把所有data删除，确认删除干净，然后再重启meta服务，看还会吗？