Log file created at: 2022/02/19 21:06:21 Running on machine: nebula-server-prod-03 Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg I0219 21:06:21.330844 6303 MetaDaemon.cpp:262] localhost = "10.0.7.206":9559 I0219 21:06:21.334224 6303 NebulaStore.cpp:52] Start the raft service... I0219 21:06:21.334612 6303 NebulaSnapshotManager.cpp:25] Send snapshot is rate limited to 8388608 for each part by default I0219 21:06:21.334794 6303 RaftexService.cpp:62] Init thrift server for raft service, port: 9560 I0219 21:06:21.334897 6349 RaftexService.cpp:93] Starting the Raftex Service I0219 21:06:21.345938 6349 RaftexService.cpp:83] Starting the Raftex Service on 9560 I0219 21:06:21.345953 6349 RaftexService.cpp:103] Start the Raftex Service successfully I0219 21:06:21.346154 6303 NebulaStore.cpp:84] Scan the local path, and init the spaces_ I0219 21:06:21.346201 6303 NebulaStore.cpp:90] Scan path "data/meta/nebula/0" I0219 21:06:21.366240 6303 RocksEngine.cpp:128] open rocksdb on data/meta/nebula/0/data I0219 21:06:21.366252 6303 NebulaStore.cpp:114] Load space 0 from disk I0219 21:06:21.366277 6303 NebulaStore.cpp:142] Need to open 1 parts of space 0 I0219 21:06:21.513312 6344 FileBasedWal.cpp:66] [Port: 9560, Space: 0, Part: 0] lastLogId in wal is 1825911493, lastLogTerm is 455, path is data/meta/nebula/0/wal/0/0000000001825864258.wal I0219 21:06:21.513463 6344 RaftPart.cpp:278] [Port: 9560, Space: 0, Part: 0] There are 2 peer hosts, and total 3 copies. The quorum is 2, as learner 0, lastLogId 1825911493, lastLogTerm 455, committedLogId 1825911492, term 455 I0219 21:06:21.513473 6344 RaftPart.cpp:285] [Port: 9560, Space: 0, Part: 0] Add peer "10.0.7.204":9560 I0219 21:06:21.513489 6344 RaftPart.cpp:285] [Port: 9560, Space: 0, Part: 0] Add peer "10.0.7.205":9560 I0219 21:06:21.513615 6344 NebulaStore.cpp:146] Load part 0, 0 from disk I0219 21:06:21.513635 6303 NebulaStore.cpp:161] Load space 0 complete I0219 21:06:21.513664 6303 NebulaStore.cpp:170] Init data from partManager for "10.0.7.206":9559 I0219 21:06:21.513676 6303 NebulaStore.cpp:262] Data space 0 has existed! I0219 21:06:21.513684 6303 NebulaStore.cpp:302] [Space: 0, Part: 0] has existed! I0219 21:06:21.513705 6303 NebulaStore.cpp:77] Register handler... I0219 21:06:21.513708 6303 MetaDaemon.cpp:98] Waiting for the leader elected... I0219 21:06:21.513715 6303 MetaDaemon.cpp:110] Leader has not been elected, sleep 1s I0219 21:06:22.003826 6341 RaftPart.cpp:1241] [Port: 9560, Space: 0, Part: 0] Recieved a VOTING request: space = 0, partition = 0, candidateAddr = 10.0.7.204:9560, term = 467, lastLogId = 1825911478, lastLogTerm = 455 I0219 21:06:22.003861 6341 RaftPart.cpp:1268] [Port: 9560, Space: 0, Part: 0] The partition currently is a Follower, lastLogId 1825911493, lastLogTerm 455, committedLogId 1825911492, term 455 I0219 21:06:22.003868 6341 RaftPart.cpp:1302] [Port: 9560, Space: 0, Part: 0] The partition's last log id is 1825911493. The candidate's last log id 1825911478 is smaller, so it will be rejected, candidate is "10.0.7.204":9560 I0219 21:06:22.343511 6345 RaftPart.cpp:957] [Port: 9560, Space: 0, Part: 0] Start leader election, reason: lastMsgDur 967, term 455 I0219 21:06:22.343555 6345 RaftPart.cpp:1095] [Port: 9560, Space: 0, Part: 0] Sending out an election request (space = 0, part = 0, term = 456, lastLogId = 1825911493, lastLogTerm = 455, candidateIP = 10.0.7.206, candidatePort = 9560) I0219 21:06:22.344843 6345 RaftPart.cpp:1059] [Port: 9560, Space: 0, Part: 0] Partition is elected as the new leader for term 456 I0219 21:06:22.344854 6345 RaftPart.cpp:1137] [Port: 9560, Space: 0, Part: 0] The partition is elected as the leader I0219 21:06:22.345023 6352 Host.cpp:125] [Port: 9560, Space: 0, Part: 0] [Host: 10.0.7.204:9560] This is the first time to send the logs to this host, lastLogIdSent = 1825911493, lastLogTermSent = 455 I0219 21:06:22.345237 6352 Host.cpp:125] [Port: 9560, Space: 0, Part: 0] [Host: 10.0.7.205:9560] This is the first time to send the logs to this host, lastLogIdSent = 1825911493, lastLogTermSent = 455 I0219 21:06:22.499908 6341 SlowOpTracker.h:31] [Port: 9560, Space: 0, Part: 0] total time:155ms, Total send logs: 2 I0219 21:06:22.513850 6303 MetaDaemon.cpp:137] Get meta version is 2 I0219 21:06:22.611429 6341 MetaVersionMan.cpp:66] Write meta version 2 succeeds I0219 21:06:22.611464 6303 MetaDaemon.cpp:164] Nebula store init succeeded, clusterId 4083132708281413418 I0219 21:06:22.611487 6303 MetaDaemon.cpp:275] Start http service I0219 21:06:22.611649 6303 MetaDaemon.cpp:172] Starting Meta HTTP Service I0219 21:06:22.613037 6390 WebService.cpp:124] Web service started on HTTP[19559], HTTP2[19560] I0219 21:06:22.613107 6303 JobManager.cpp:56] JobManager initialized I0219 21:06:22.613123 6303 MetaDaemon.cpp:305] Check and init root user I0219 21:06:22.613139 6396 JobManager.cpp:77] JobManager::runJobBackground() enter I0219 21:06:22.613178 6303 RootUserMan.h:27] Root user exists I0219 21:06:22.613337 6303 MetaDaemon.cpp:331] The meta deamon start on "10.0.7.206":9559 E0219 21:06:32.347807 6365 Host.cpp:348] [Port: 9560, Space: 0, Part: 0] [Host: 10.0.7.205:9560] Failed to append logs to the host (Err: E_UNKNOWN_PART) I0219 21:06:33.227840 6341 RaftPart.cpp:1241] [Port: 9560, Space: 0, Part: 0] Recieved a VOTING request: space = 0, partition = 0, candidateAddr = 10.0.7.205:9560, term = 456, lastLogId = 1825911494, lastLogTerm = 455 I0219 21:06:33.227874 6341 RaftPart.cpp:1268] [Port: 9560, Space: 0, Part: 0] The partition currently is a Leader, lastLogId 1825911501, lastLogTerm 456, committedLogId 1825911501, term 456 I0219 21:06:33.227878 6341 RaftPart.cpp:1280] [Port: 9560, Space: 0, Part: 0] The partition currently is on term 456. The term proposed by the candidate is no greater, so it will be rejected I0220 09:38:57.672132 6418 HBProcessor.cpp:33] Receive heartbeat from "10.0.7.204":9779, role = STORAGE I0220 09:38:58.652906 6417 HBProcessor.cpp:33] Receive heartbeat from "10.0.7.204":9669, role = GRAPH I0220 09:39:12.100342 6416 HBProcessor.cpp:33] Receive heartbeat from "10.0.7.205":9669, role = GRAPH I0220 09:39:12.119222 6415 HBProcessor.cpp:33] Receive heartbeat from "10.0.7.205":9779, role = STORAGE I0220 09:39:20.508921 6414 HBProcessor.cpp:33] Receive heartbeat from "10.0.7.206":9669, role = GRAPH I0220 09:39:21.535485 6413 HBProcessor.cpp:33] Receive heartbeat from "10.0.7.206":9779, role = STORAGE