storaged服务宕机后，服务起了，但是status状态是OFFLINE

harrywang · 2022 年2 月 22 日 08:47

nebula 版本 2.5.0
现象描述：业务人员在使用过程中，通过studio连接不上，此时登陆服务器查看，grahpd和storaged服务已经宕机，本想先将服务启动后，再来排查问题，但是服务都已经启动了，通过 show hosts查看，Status 的状态都是 OFFLINE

截图如下：

但是：

已经尝试多次了

steam · 2022 年2 月 22 日 08:49

你们尝试过多次重启 storage 服务，一直都是 offline 是嘛？有 storage 的日志吗？贴一下

harrywang · 2022 年2 月 22 日 09:23

Log file created at: 2022/02/22 08:40:30
Running on machine: ip-10-17-254-247.cn-northwest-1.compute.internal
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0222 08:40:30.077006 22085 MetaClient.cpp:635] Send request to "10.17.254.247":9559, exceed retry limit
E0222 08:40:30.077471 22042 MetaClient.cpp:65] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0222 08:40:43.077920 22159 RaftPart.cpp:1118] [Port: 9780, Space: 340, Part: 13] Receive response about askForVote from "10.17.254.189":9780, error code is E_TERM_OUT_OF_DATE
E0222 08:40:43.077962 22159 RaftPart.cpp:1118] [Port: 9780, Space: 340, Part: 13] Receive response about askForVote from "10.17.254.132":9780, error code is E_TERM_OUT_OF_DATE
E0222 08:40:43.078943 22159 RaftPart.cpp:1118] [Port: 9780, Space: 340, Part: 11] Receive response about askForVote from "10.17.254.189":9780, error code is E_TERM_OUT_OF_DATE
E0222 08:40:43.078967 22159 RaftPart.cpp:1118] [Port: 9780, Space: 340, Part: 11] Receive response about askForVote from "10.17.254.132":9780, error code is E_TERM_OUT_OF_DATE
E0222 08:40:43.090245 22158 RaftPart.cpp:1118] [Port: 9780, Space: 340, Part: 4] Receive response about askForVote from "10.17.254.189":9780, error code is E_TERM_OUT_OF_DATE
E0222 08:40:43.090277 22158 RaftPart.cpp:1118] [Port: 9780, Space: 340, Part: 4] Receive response about askForVote from "10.17.254.132":9780, error code is E_TERM_OUT_OF_DATE
E0222 08:40:43.091658 22158 RaftPart.cpp:1118] [Port: 9780, Space: 340, Part: 12] Receive response about askForVote from "10.17.254.132":9780, error code is E_TERM_OUT_OF_DATE
E0222 08:40:43.091678 22158 RaftPart.cpp:1118] [Port: 9780, Space: 340, Part: 12] Receive response about askForVote from "10.17.254.189":9780, error code is E_TERM_OUT_OF_DATE
E0222 08:40:43.092697 22158 RaftPart.cpp:1118] [Port: 9780, Space: 340, Part: 16] Receive response about askForVote from "10.17.254.132":9780, error code is E_TERM_OUT_OF_DATE
E0222 08:40:43.092716 22158 RaftPart.cpp:1118] [Port: 9780, Space: 340, Part: 16] Receive response about askForVote from "10.17.254.189":9780, error code is E_TERM_OUT_OF_DATE
E0222 08:40:43.093966 22158 RaftPart.cpp:1118] [Port: 9780, Space: 340, Part: 11] Receive response about askForVote from "10.17.254.189":9780, error code is E_TERM_OUT_OF_DATE
E0222 08:40:43.093991 22158 RaftPart.cpp:1118] [Port: 9780, Space: 340, Part: 11] Receive response about askForVote from "10.17.254.132":9780, error code is E_TERM_OUT_OF_DATE

harrywang · 2022 年2 月 22 日 09:29

I0222 08:40:40.124135 22042 MetaClient.cpp:106] Register time task for heartbeat!
I0222 08:40:40.124148 22042 StorageServer.cpp:142] Init schema manager
I0222 08:40:40.124157 22042 StorageServer.cpp:145] Init index manager
I0222 08:40:40.124161 22042 StorageServer.cpp:148] Init kvstore
I0222 08:40:40.124173 22042 NebulaStore.cpp:49] Start the raft service...
I0222 08:40:40.124557 22042 RaftexService.cpp:65] Init thrift server for raft service, port: 9780
I0222 08:40:40.124783 22162 RaftexService.cpp:98] Starting the Raftex Service
I0222 08:40:40.139462 22162 RaftexService.cpp:86] Starting the Raftex Service on 9780
I0222 08:40:40.139503 22162 RaftexService.cpp:110] Start the Raftex Service successfully
I0222 08:40:40.139686 22042 NebulaStore.cpp:82] Scan the local path, and init the spaces_
I0222 08:40:40.139827 22042 NebulaStore.cpp:88] Scan path "data/storage/nebula/1"
I0222 08:40:40.140086 22042 RocksEngineConfig.cpp:308] Emplace rocksdb option max_bytes_for_level_base=268435456
I0222 08:40:40.140101 22042 RocksEngineConfig.cpp:308] Emplace rocksdb option max_write_buffer_number=4
I0222 08:40:40.140106 22042 RocksEngineConfig.cpp:308] Emplace rocksdb option write_buffer_size=67108864
I0222 08:40:40.140164 22042 RocksEngineConfig.cpp:308] Emplace rocksdb option block_size=8192
I0222 08:40:40.160823 22042 RocksEngine.cpp:132] open rocksdb on data/storage/nebula/1/data
I0222 08:40:40.160861 22042 NebulaStore.cpp:112] Load space 1 from disk
I0222 08:40:40.160889 22042 NebulaStore.cpp:143] Need to open 3 parts of space 1
I0222 08:40:41.004053 22157 FileBasedWal.cpp:67] [Port: 9780, Space: 1, Part: 2] lastLogId in wal is 321515, lastLogTerm is 30, path is data/storage/nebula/1/wal/2/0000000000000000001.wal
I0222 08:40:41.004185 22157 RaftPart.cpp:297] [Port: 9780, Space: 1, Part: 2] There are 0 peer hosts, and total 1 copies. The quorum is 1, as learner 0, lastLogId 321515, lastLogTerm 30, committedLogId 321515, term 30
I0222 08:40:41.004606 22157 NebulaStore.cpp:148] Load part 1, 2 from disk
I0222 08:40:41.045313 22159 FileBasedWal.cpp:67] [Port: 9780, Space: 1, Part: 8] lastLogId in wal is 321528, lastLogTerm is 30, path is data/storage/nebula/1/wal/8/0000000000000000001.wal
I0222 08:40:41.045400 22159 RaftPart.cpp:297] [Port: 9780, Space: 1, Part: 8] There are 0 peer hosts, and total 1 copies. The quorum is 1, as learner 0, lastLogId 321528, lastLogTerm 30, committedLogId 321528, term 30
I0222 08:40:41.045572 22159 NebulaStore.cpp:148] Load part 1, 8 from disk
I0222 08:40:41.049929 22158 FileBasedWal.cpp:67] [Port: 9780, Space: 1, Part: 5] lastLogId in wal is 321529, lastLogTerm is 30, path is data/storage/nebula/1/wal/5/0000000000000000001.wal
I0222 08:40:41.049929 22158 FileBasedWal.cpp:67] [Port: 9780, Space: 1, Part: 5] lastLogId in wal is 321529, lastLogTerm is 30, path is data/storage/nebula/1/wal/5/0000000000000000001.wal
I0222 08:40:41.049984 22158 RaftPart.cpp:297] [Port: 9780, Space: 1, Part: 5] There are 0 peer hosts, and total 1 copies. The quorum is 1, as learner 0, lastLogId 321529, lastLogTerm 30, committedLogId 321529, term 30
I0222 08:40:41.050132 22158 NebulaStore.cpp:148] Load part 1, 5 from disk
I0222 08:40:41.050176 22158 RaftPart.cpp:1018] [Port: 9780, Space: 1, Part: 5] Start leader election, reason: lastMsgDur 879, term 30
I0222 08:40:41.050194 22158 RaftPart.cpp:1168] [Port: 9780, Space: 1, Part: 5] Sending out an election request (space = 1, part = 5, term = 31, lastLogId = 321529, lastLogTerm = 30, candidateIP = 10.17.254.247, candidatePort = 9780)
I0222 08:40:41.050191 22042 NebulaStore.cpp:163] Load space 1 complete
I0222 08:40:41.050278 22042 NebulaStore.cpp:88] Scan path "data/storage/nebula/6"
I0222 08:40:41.050290 22042 NebulaStore.cpp:88] Scan path "data/storage/nebula/16"
I0222 08:40:41.050206 22158 RaftPart.cpp:1127] [Port: 9780, Space: 1, Part: 5] Partition is elected as the new leader for term 31
I0222 08:40:41.050309 22158 RaftPart.cpp:1223] [Port: 9780, Space: 1, Part: 5] The partition is elected as the leader
I0222 08:40:41.050297 22042 NebulaStore.cpp:88] Scan path "data/storage/nebula/26"
I0222 08:40:41.050324 22042 NebulaStore.cpp:88] Scan path "data/storage/nebula/36"
I0222 08:40:41.050330 22042 NebulaStore.cpp:88] Scan path "data/storage/nebula/46"
I0222 08:40:41.050335 22042 NebulaStore.cpp:88] Scan path "data/storage/nebula/56"
I0222 08:40:41.050341 22042 NebulaStore.cpp:88] Scan path "data/storage/nebula/66"
I0222 08:40:41.050345 22042 NebulaStore.cpp:88] Scan path "data/storage/nebula/76"
I0222 08:40:41.050350 22042 NebulaStore.cpp:88] Scan path "data/storage/nebula/100"
I0222 08:40:41.050355 22042 NebulaStore.cpp:88] Scan path "data/storage/nebula/124"
I0222 08:40:41.050360 22042 NebulaStore.cpp:88] Scan path "data/storage/nebula/148"
I0222 08:40:41.050364 22042 NebulaStore.cpp:88] Scan path "data/storage/nebula/172"
I0222 08:40:41.050369 22042 NebulaStore.cpp:88] Scan path "data/storage/nebula/196"
I0222 08:40:41.050374 22042 NebulaStore.cpp:88] Scan path "data/storage/nebula/220"
I0222 08:40:41.050379 22042 NebulaStore.cpp:88] Scan path "data/storage/nebula/244"
I0222 08:40:41.050382 22042 NebulaStore.cpp:88] Scan path "data/storage/nebula/268"
I0222 08:40:41.050388 22042 NebulaStore.cpp:88] Scan path "data/storage/nebula/292"
I0222 08:40:41.050395 22042 NebulaStore.cpp:88] Scan path "data/storage/nebula/316"
I0222 08:40:41.050504 22042 RocksEngineConfig.cpp:308] Emplace rocksdb option max_bytes_for_level_base=268435456
I0222 08:40:41.050516 22042 RocksEngineConfig.cpp:308] Emplace rocksdb option max_write_buffer_number=4
I0222 08:40:41.050527 22042 RocksEngineConfig.cpp:308] Emplace rocksdb option write_buffer_size=67108864
I0222 08:40:41.050582 22042 RocksEngineConfig.cpp:308] Emplace rocksdb option block_size=8192
I0222 08:40:41.078554 22042 RocksEngine.cpp:132] open rocksdb on data/storage/nebula/340/data
I0222 08:40:41.078574 22042 NebulaStore.cpp:112] Load space 340 from disk
I0222 08:40:41.078617 22042 NebulaStore.cpp:143] Need to open 24 parts of space 340
I0222 08:40:41.079131 22158 FileBasedWal.cpp:67] [Port: 9780, Space: 340, Part: 2] lastLogId in wal is 418283, lastLogTerm is 179, path is data/storage/nebula/340/wal/2/0000000000000418105.wal
I0222 08:40:41.079154 22160 FileBasedWal.cpp:67] [Port: 9780, Space: 340, Part: 4] lastLogId in wal is 418228, lastLogTerm is 183, path is data/storage/nebula/340/wal/4/0000000000000418051.wal
I0222 08:40:41.079190 22158 RaftPart.cpp:297] [Port: 9780, Space: 340, Part: 2] There are 2 peer hosts, and total 3 copies. The quorum is 2, as learner 0, lastLogId 418283, lastLogTerm 179, committedLogId 418282, term 179
I0222 08:40:41.079226 22159 FileBasedWal.cpp:67] [Port: 9780, Space: 340, Part: 3] lastLogId in wal is 418254, lastLogTerm is 193, path is data/storage/nebula/340/wal/3/0000000000000418077.wal
I0222 08:40:41.079193 22160 RaftPart.cpp:297] [Port: 9780, Space: 340, Part: 4] There are 2 peer hosts, and total 3 copies. The quorum is 2, as learner 0, lastLogId 418228, lastLogTerm 183, committedLogId 418227, term 183
I0222 08:40:41.079282 22159 RaftPart.cpp:297] [Port: 9780, Space: 340, Part: 3] There are 2 peer hosts, and total 3 copies. The quorum is 2, as learner 0, lastLogId 418254, lastLogTerm 193, committedLogId 418253, term 193
I0222 08:40:41.079299 22160 RaftPart.cpp:310] [Port: 9780, Space: 340, Part: 4] Add peer "10.17.254.189":9780
I0222 08:40:41.079224 22158 RaftPart.cpp:310] [Port: 9780, Space: 340, Part: 2] Add peer "10.17.254.132":9780
I0222 08:40:41.079317 22160 RaftPart.cpp:310] [Port: 9780, Space: 340, Part: 4] Add peer "10.17.254.132":9780
I0222 08:40:41.079362 22158 RaftPart.cpp:310] [Port: 9780, Space: 340, Part: 2] Add peer "10.17.254.189":9780
I0222 08:40:41.079232 22157 FileBasedWal.cpp:67] [Port: 9780, Space: 340, Part: 1] lastLogId in wal is 418206, lastLogTerm is 188, path is data/storage/nebula/340/wal/1/0000000000000418029.wal
I0222 08:40:41.079308 22159 RaftPart.cpp:310] [Port: 9780, Space: 340, Part: 3] Add peer "10.17.254.132":9780
I0222 08:40:41.079429 22157 RaftPart.cpp:297] [Port: 9780, Space: 340, Part: 1] There are 2 peer hosts, and total 3 copies. The quorum is 2, as learner 0, lastLogId 418206, lastLogTerm 188, committedLogId 418205, term 188
I0222 08:40:41.079933 22160 FileBasedWal.cpp:67] [Port: 9780, Space: 340, Part: 8] lastLogId in wal is 418253, lastLogTerm is 195, path is data/storage/nebula/340/wal/8/0000000000000418075.wal
I0222 08:40:41.079963 22160 RaftPart.cpp:297] [Port: 9780, Space: 340, Part: 8] There are 2 peer hosts, and total 3 copies. The quorum is 2, as learner 0, lastLogId 418253, lastLogTerm 195, committedLogId 418252, term 195
I0222 08:40:41.079972 22160 RaftPart.cpp:310] [Port: 9780, Space: 340, Part: 8] Add peer "10.17.254.132":9780
I0222 08:40:41.079982 22160 RaftPart.cpp:310] [Port: 9780, Space: 340, Part: 8] Add peer "10.17.254.189":9780
I0222 08:40:41.080020 22160 NebulaStore.cpp:148] Load part 340, 8 from disk
I0222 08:40:41.080026 22158 FileBasedWal.cpp:67] [Port: 9780, Space: 340, Part: 6] lastLogId in wal is 418221, lastLogTerm is 114, path is data/storage/nebula/340/wal/6/0000000000000418044.wal
I0222 08:40:41.080058 22158 RaftPart.cpp:297] [Port: 9780, Space: 340, Part: 6] There are 2 peer hosts, and total 3 copies. The quorum is 2, as learner 0, lastLogId 418221, lastLogTerm 114, committedLogId 418220, term 114
I0222 08:40:41.080065 22158 RaftPart.cpp:310] [Port: 9780, Space: 340, Part: 6] Add peer "10.17.254.132":9780
I0222 08:40:41.080073 22158 RaftPart.cpp:310] [Port: 9780, Space: 340, Part: 6] Add peer "10.17.254.189":9780
I0222 08:40:41.080106 22158 NebulaStore.cpp:148] Load part 340, 6 from disk
I0222 08:40:41.080114 22159 FileBasedWal.cpp:67] [Port: 9780, Space: 340, Part: 7] lastLogId in wal is 418252, lastLogTerm is 141, path is data/storage/nebula/340/wal/7/0000000000000418075.wal
I0222 08:40:41.080106 22157 FileBasedWal.cpp:67] [Port: 9780, Space: 340, Part: 5] lastLogId in wal is 418241, lastLogTerm is 141, path is data/storage/nebula/340/wal/5/0000000000000418064.wal
I0222 08:40:41.080147 22159 RaftPart.cpp:297] [Port: 9780, Space: 340, Part: 7] There are 2 peer hosts, and total 3 copies. The quorum is 2, as learner 0, lastLogId 418252, lastLogTerm 141, committedLogId 418251, term 141
I0222 08:40:41.080161 22157 RaftPart.cpp:297] [Port: 9780, Space: 340, Part: 5] There are 2 peer hosts, and total 3 copies. The quorum is 2, as learner 0, lastLogId 418241, lastLogTerm 141, committedLogId 418240, term 141
I0222 08:40:41.080190 22157 RaftPart.cpp:310] [Port: 9780, Space: 340, Part: 5] Add peer "10.17.254.132":9780
I0222 08:40:41.080164 22159 RaftPart.cpp:310] [Port: 9780, Space: 340, Part: 7] Add peer "10.17.254.189":9780
I0222 08:40:41.080256 22159 RaftPart.cpp:310] [Port: 9780, Space: 340, Part: 7] Add peer "10.17.254.132":9780
I0222 08:40:41.080219 22157 RaftPart.cpp:310] [Port: 9780, Space: 340, Part: 5] Add peer "10.17.254.189":9780
I0222 08:40:41.080361 22159 NebulaStore.cpp:148] Load part 340, 7 from disk
I0222 08:40:41.080397 22157 NebulaStore.cpp:148] Load part 340, 5 from disk
I0222 08:40:41.080569 22158 FileBasedWal.cpp:67] [Port: 9780, Space: 340, Part: 10] lastLogId in wal is 418230, lastLogTerm is 142, path is data/storage/nebula/340/wal/10/0000000000000418053.wal
I0222 08:40:41.080579 22160 FileBasedWal.cpp:67] [Port: 9780, Space: 340, Part: 12] lastLogId in wal is 418243, lastLogTerm is 117, path is data/storage/nebula/340/wal/12/0000000000000418064.wal
I0222 08:40:41.080600 22158 RaftPart.cpp:297] [Port: 9780, Space: 340, Part: 10] There are 2 peer hosts, and total 3 copies. The quorum is 2, as learner 0, lastLogId 418230, lastLogTerm 142, committedLogId 418229, term 142
I0222 08:40:41.080607 22158 RaftPart.cpp:310] [Port: 9780, Space: 340, Part: 10] Add peer "10.17.254.189":9780
I0222 08:40:41.080610 22160 RaftPart.cpp:297] [Port: 9780, Space: 340, Part: 12] There are 2 peer hosts, and total 3 copies. The quorum is 2, as learner 0, lastLogId 418243, lastLogTerm 117, committedLogId 418242, term 117
I0222 08:40:41.080615 22158 RaftPart.cpp:310] [Port: 9780, Space: 340, Part: 10] Add peer "10.17.254.132":9780
I0222 08:40:41.080618 22160 RaftPart.cpp:310] [Port: 9780, Space: 340, Part: 12] Add peer "10.17.254.132":9780
I0222 08:40:41.080626 22160 RaftPart.cpp:310] [Port: 9780, Space: 340, Part: 12] Add peer "10.17.254.189":9780
I0222 08:40:41.080665 22158 NebulaStore.cpp:148] Load part 340, 10 from disk
I0222 08:40:41.080703 22160 NebulaStore.cpp:148] Load part 340, 12 from disk
I0222 08:40:41.080917 22159 FileBasedWal.cpp:67] [Port: 9780, Space: 340, Part: 11] lastLogId in wal is 418239, lastLogTerm is 157, path is data/storage/nebula/340/wal/11/0000000000000418061.wal
I0222 08:40:41.080931 22157 FileBasedWal.cpp:67] [Port: 9780, Space: 340, Part: 9] lastLogId in wal is 418222, lastLogTerm is 179, path is data/storage/nebula/340/wal/9/0000000000000418043.wal
I0222 08:40:41.080948 22159 RaftPart.cpp:297] [Port: 9780, Space: 340, Part: 11] There are 2 peer hosts, and total 3 copies. The quorum is 2, as learner 0, lastLogId 418239, lastLogTerm 157, committedLogId 418238, term 157
I0222 08:40:41.080983 22159 RaftPart.cpp:310] [Port: 9780, Space: 340, Part: 11] Add peer "10.17.254.132":9780
I0222 08:40:41.080950 22157 RaftPart.cpp:297] [Port: 9780, Space: 340, Part: 9] There are 2 peer hosts, and total 3 copies. The quorum is 2, as learner 0, lastLogId 418222, lastLogTerm 179, committedLogId 418221, term 179
I0222 08:40:41.081168 22157 RaftPart.cpp:310] [Port: 9780, Space: 340, Part: 9] Add peer "10.17.254.132":9780
I0222 08:40:41.081178 22158 FileBasedWal.cpp:67] [Port: 9780, Space: 340, Part: 14] lastLogId in wal is 418204, lastLogTerm is 117, path is data/storage/nebula/340/wal/14/0000000000000418025.wal
I0222 08:40:41.081030 22159 RaftPart.cpp:310] [Port: 9780, Space: 340, Part: 11] Add peer "10.17.254.189":9780
I0222 08:40:41.081269 22158 RaftPart.cpp:297] [Port: 9780, Space: 340, Part: 14] There are 2 peer hosts, and total 3 copies. The quorum is 2, as learner 0, lastLogId 418204, lastLogTerm 117, committedLogId 418203, term 117
I0222 08:40:41.081277 22158 RaftPart.cpp:310] [Port: 9780, Space: 340, Part: 14] Add peer "10.17.254.132":9780
I0222 08:40:41.081975 22160 FileBasedWal.cpp:67] [Port: 9780, Space: 340, Part: 20] lastLogId in wal is 418281, lastLogTerm is 146, path is data/storage/nebula/340/wal/20/0000000000000418103.wal
I0222 08:40:41.081991 22157 FileBasedWal.cpp:67] [Port: 9780, Space: 340, Part: 13] lastLogId in wal is 418246, lastLogTerm is 120, path is data/storage/nebula/340/wal/13/0000000000000418068.wal
I0222 08:40:41.082016 22157 RaftPart.cpp:297] [Port: 9780, Space: 340, Part: 13] There are 2 peer hosts, and total 3 copies. The quorum is 2, as learner 0, lastLogId 418246, lastLogTerm 120, committedLogId 418245, term 120
I0222 08:40:41.082048 22157 RaftPart.cpp:310] [Port: 9780, Space: 340, Part: 13] Add peer "10.17.254.189":9780
I0222 08:40:41.082017 22160 RaftPart.cpp:297] [Port: 9780, Space: 340, Part: 20] There are 2 peer hosts, and total 3 copies. The quorum is 2, as learner 0, lastLogId 418281, lastLogTerm 146, committedLogId 418280, term 146
I0222 08:40:41.082068 22158 FileBasedWal.cpp:67] [Port: 9780, Space: 340, Part: 18] lastLogId in wal is 418248, lastLogTerm is 151, path is data/storage/nebula/340/wal/18/0000000000000418070.wal
I0222 08:40:41.082087 22157 RaftPart.cpp:310] [Port: 9780, Space: 340, Part: 13] Add peer "10.17.254.132":9780
I0222 08:40:41.082070 22160 RaftPart.cpp:310] [Port: 9780, Space: 340, Part: 20] Add peer "10.17.254.132":9780
I0222 08:40:41.081979 22159 RaftPart.cpp:297] [Port: 9780, Space: 340, Part: 15] There are 2 peer hosts, and total 3 copies. The quorum is 2, as learner 0, lastLogId 418273, lastLogTerm 160, committedLogId 418272, term 160
I0222 08:40:41.082129 22159 RaftPart.cpp:310] [Port: 9780, Space: 340, Part: 15] Add peer "10.17.254.132":9780
I0222 08:40:41.082140 22159 RaftPart.cpp:310] [Port: 9780, Space: 340, Part: 15] Add peer "10.17.254.189":9780
I0222 08:40:41.082147 22160 RaftPart.cpp:310] [Port: 9780, Space: 340, Part: 20] Add peer "10.17.254.189":9780
I0222 08:40:41.082108 22158 RaftPart.cpp:297] [Port: 9780, Space: 340, Part: 18] There are 2 peer hosts, and total 3 copies. The quorum is 2, as learner 0, lastLogId 418248, lastLogTerm 151, committedLogId 418247, term 151
I0222 08:40:41.082149 22157 NebulaStore.cpp:148] Load part 340, 13 from disk
I0222 08:40:41.082195 22159 NebulaStore.cpp:148] Load part 340, 15 from disk
I0222 08:40:41.082271 22160 NebulaStore.cpp:148] Load part 340, 20 from disk
I0222 08:40:41.082186 22158 RaftPart.cpp:310] [Port: 9780, Space: 340, Part: 18] Add peer "10.17.254.132":9780
I0222 08:40:41.082350 22158 RaftPart.cpp:310] [Port: 9780, Space: 340, Part: 18] Add peer "10.17.254.189":9780
I0222 08:40:41.082536 22158 NebulaStore.cpp:148] Load part 340, 18 from disk
I0222 08:40:41.082710 22159 FileBasedWal.cpp:67] [Port: 9780, Space: 340, Part: 19] lastLogId in wal is 418252, lastLogTerm is 151, path is data/storage/nebula/340/wal/19/0000000000000418075.wal
I0222 08:40:41.082720 22157 FileBasedWal.cpp:67] [Port: 9780, Space: 340, Part: 17] lastLogId in wal is 418250, lastLogTerm is 152, path is data/storage/nebula/340/wal/17/0000000000000418071.wal
I0222 08:40:41.354291 22159 RaftPart.cpp:297] [Port: 9780, Space: 376, Part: 8] There are 0 peer hosts, and total 1 copies. The quorum is 1, as learner 0, lastLogId 131520, lastLogTerm 23, committedLogId 131520, term 23
I0222 08:40:41.354346 22159 NebulaStore.cpp:148] Load part 376, 8 from disk
I0222 08:40:41.400444 22160 FileBasedWal.cpp:67] [Port: 9780, Space: 376, Part: 11] lastLogId in wal is 131520, lastLogTerm is 23, path is data/storage/nebula/376/wal/11/0000000000000000001.wal
I0222 08:40:41.400532 22160 RaftPart.cpp:297] [Port: 9780, Space: 376, Part: 11] There are 0 peer hosts, and total 1 copies. The quorum is 1, as learner 0, lastLogId 131520, lastLogTerm 23, committedLogId 131520, term 23
I0222 08:40:41.400584 22160 NebulaStore.cpp:148] Load part 376, 11 from disk
I0222 08:40:41.431392 22157 FileBasedWal.cpp:67] [Port: 9780, Space: 376, Part: 2] lastLogId in wal is 131521, lastLogTerm is 23, path is data/storage/nebula/376/wal/2/0000000000000000001.wal
I0222 08:40:41.431566 22157 RaftPart.cpp:297] [Port: 9780, Space: 376, Part: 2] There are 0 peer hosts, and total 1 copies. The quorum is 1, as learner 0, lastLogId 131521, lastLogTerm 23, committedLogId 131521, term 23
I0222 08:40:41.431622 22157 NebulaStore.cpp:148] Load part 376, 2 from disk
I0222 08:40:41.594666 22158 FileBasedWal.cpp:67] [Port: 9780, Space: 376, Part: 17] lastLogId in wal is 131521, lastLogTerm is 23, path is data/storage/nebula/376/wal/17/0000000000000000001.wal
I0222 08:40:41.594743 22158 RaftPart.cpp:297] [Port: 9780, Space: 376, Part: 17] There are 0 peer hosts, and total 1 copies. The quorum is 1, as learner 0, lastLogId 131521, lastLogTerm 23, committedLogId 131521, term 23
I0222 08:40:41.594797 22158 NebulaStore.cpp:148] Load part 376, 17 from disk
I0222 08:40:41.595165 22159 FileBasedWal.cpp:67] [Port: 9780, Space: 376, Part: 20] lastLogId in wal is 131519, lastLogTerm is 23, path is data/storage/nebula/376/wal/20/0000000000000000001.wal
I0222 08:40:41.595242 22159 RaftPart.cpp:297] [Port: 9780, Space: 376, Part: 20] There are 0 peer hosts, and total 1 copies. The quorum is 1, as learner 0, lastLogId 131519, lastLogTerm 23, committedLogId 131519, term 23
I0222 08:40:41.595293 22159 NebulaStore.cpp:148] Load part 376, 20 from disk
I0222 08:40:41.707675 22160 FileBasedWal.cpp:67] [Port: 9780, Space: 376, Part: 23] lastLogId in wal is 131520, lastLogTerm is 23, path is data/storage/nebula/376/wal/23/0000000000000000001.wal
I0222 08:40:41.707777 22160 RaftPart.cpp:297] [Port: 9780, Space: 376, Part: 23] There are 0 peer hosts, and total 1 copies. The quorum is 1, as learner 0, lastLogId 131520, lastLogTerm 23, committedLogId 131520, term 23
I0222 08:40:41.707828 22160 NebulaStore.cpp:148] Load part 376, 23 from disk
I0222 08:40:41.751071 22157 FileBasedWal.cpp:67] [Port: 9780, Space: 376, Part: 14] lastLogId in wal is 131522, lastLogTerm is 23, path is data/storage/nebula/376/wal/14/0000000000000000001.wal
I0222 08:40:44.114183 22292 AdminTaskManager.cpp:89] waiting for incoming task
I0222 08:40:44.114872 22294 StorageServer.cpp:222] The admin service start on "10.17.254.247":9778
I0222 08:40:44.114946 22295 StorageServer.cpp:244] The internal storage service start(same with admin) on "10.17.254.247":9777
I0222 08:40:44.122349 22293 StorageServer.cpp:196] The storage service start on "10.17.254.247":9779
I0222 08:40:44.210500 22160 RaftPart.cpp:1018] [Port: 9780, Space: 448, Part: 62] Start leader election, reason: lastMsgDur 129, term 14
I0222 08:40:44.210547 22160 RaftPart.cpp:1168] [Port: 9780, Space: 448, Part: 62] Sending out an election request (space = 448, part = 62, term = 15, lastLogId = 532, lastLogTerm = 14, candidateIP = 10.17.254.247, candidatePort = 9780)
I0222 08:40:44.210553 22160 RaftPart.cpp:1127] [Port: 9780, Space: 448, Part: 62] Partition is elected as the new leader for term 15
I0222 08:40:44.210559 22160 RaftPart.cpp:1223] [Port: 9780, Space: 448, Part: 62] The partition is elected as the leader
I0222 08:40:44.234727 22160 RaftPart.cpp:1018] [Port: 9780, Space: 389, Part: 14] Start leader election, reason: lastMsgDur 408, term 23
I0222 08:40:44.234760 22160 RaftPart.cpp:1168] [Port: 9780, Space: 389, Part: 14] Sending out an election request (space = 389, part = 14, term = 24, lastLogId = 124600, lastLogTerm = 23, candidateIP = 10.17.254.247, candidatePort = 9780)
I0222 08:40:44.234766 22160 RaftPart.cpp:1127] [Port: 9780, Space: 389, Part: 14] Partition is elected as the new leader for term 24
I0222 08:40:44.234771 22160 RaftPart.cpp:1223] [Port: 9780, Space: 389, Part: 14] The partition is elected as the leader
I0222 08:40:44.236729 22158 RaftPart.cpp:1018] [Port: 9780, Space: 376, Part: 86] Start leader election, reason: lastMsgDur 1165, term 23
I0222 08:40:44.236775 22158 RaftPart.cpp:1168] [Port: 9780, Space: 376, Part: 86] Sending out an election request (space = 376, part = 86, term = 24, lastLogId = 131519, lastLogTerm = 23, candidateIP = 10.17.254.247, candidatePort = 9780)
I0222 08:40:44.236781 22158 RaftPart.cpp:1127] [Port: 9780, Space: 376, Part: 86] Partition is elected as the new leader for term 24
I0222 08:40:44.236786 22158 RaftPart.cpp:1223] [Port: 9780, Space: 376, Part: 86] The partition is elected as the leader
I0222 08:40:44.243957 22160 RaftPart.cpp:1018] [Port: 9780, Space: 448, Part: 86] Start leader election, reason: lastMsgDur 160, term 14
I0222 08:40:44.243994 22160 RaftPart.cpp:1168] [Port: 9780, Space: 448, Part: 86] Sending out an election request (space = 448, part = 86, term = 15, lastLogId = 532, lastLogTerm = 14, candidateIP = 10.17.254.247, candidatePort = 9780)
I0222 08:40:44.244000 22160 RaftPart.cpp:1127] [Port: 9780, Space: 448, Part: 86] Partition is elected as the new leader for term 15
I0222 08:40:44.244005 22160 RaftPart.cpp:1223] [Port: 9780, Space: 448, Part: 86] The partition is elected as the leader
I0222 08:40:44.247967 22158 RaftPart.cpp:1018] [Port: 9780, Space: 448, Part: 92] Start leader election, reason: lastMsgDur 167, term 14
I0222 08:40:44.247983 22158 RaftPart.cpp:1168] [Port: 9780, Space: 448, Part: 92] Sending out an election request (space = 448, part = 92, term = 15, lastLogId = 532, lastLogTerm = 14, candidateIP = 10.17.254.247, candidatePort = 9780)
I0222 08:40:44.247989 22158 RaftPart.cpp:1127] [Port: 9780, Space: 448, Part: 92] Partition is elected as the new leader for term 15
I0222 08:40:44.247993 22158 RaftPart.cpp:1223] [Port: 9780, Space: 448, Part: 92] The partition is elected as the leader
I0222 08:40:44.251179 22159 RaftPart.cpp:1018] [Port: 9780, Space: 448, Part: 71] Start leader election, reason: lastMsgDur 170, term 14

critical27 · 2022 年2 月 24 日 04:17

查询正常吗？如果不正常，麻烦把所有storage节点的INFO日志都上传上来。我记不太清2.5版本的遗留问题了，有一种可能是在compaction，所以看起来离线了，可以看rocksdb的日志

system · 2022 年3 月 26 日 04:18

此话题已在最后回复的 30 天后被自动关闭。不再允许新回复。