nebula2.6.2 rebulid 索引失败，同时也会使部分storage 服务节点挂掉

zhengshuai1030 · 2022 年6 月 21 日 07:55

提问参考模版：

nebula 版本：2.6.2
部署方式：分布式
安装方式：源码编译
是否为线上版本：Y /
硬件信息
- 磁盘500Gssd
- CPU 32c、32G内存信息
问题的具体描述
nebula2.6.2 rebulid 索引失败，同时也会使部分storage 服务节点挂掉

I0621 14:35:50.543932 213058 RebuildIndexTask.cpp:101] Building index successful, space=16, part=47
I0621 14:35:50.544512 213058 RebuildIndexTask.cpp:104] Processing operation logs, space=16, part=47
I0621 14:35:50.559070 209473 SlowOpTracker.h:31] [Port: 9780, Space: 16, Part: 12] total time:184ms, Total send logs: 2
I0621 14:35:50.590597 209462 SlowOpTracker.h:31] [Port: 9780, Space: 16, Part: 6] total time:127ms, Total send logs: 2
I0621 14:35:50.620716 209473 SlowOpTracker.h:31] [Port: 9780, Space: 16, Part: 30] total time:137ms, Total send logs: 2
I0621 14:35:50.669212 213056 RebuildIndexTask.cpp:101] Building index successful, space=16, part=30
I0621 14:35:50.669250 213056 RebuildIndexTask.cpp:104] Processing operation logs, space=16, part=30
I0621 14:35:50.694604 209462 SlowOpTracker.h:31] [Port: 9780, Space: 16, Part: 17] total time:149ms, Total send logs: 2
I0621 14:35:50.718991 209462 SlowOpTracker.h:31] [Port: 9780, Space: 16, Part: 42] total time:160ms, Total send logs: 2
I0621 14:35:50.759819 209468 SlowOpTracker.h:31] [Port: 9780, Space: 16, Part: 24] total time:174ms, Total send logs: 2
I0621 14:35:50.774507 209473 SlowOpTracker.h:31] [Port: 9780, Space: 16, Part: 5] total time:132ms, Total send logs: 2
I0621 14:35:50.792866 209474 SlowOpTracker.h:31] [Port: 9780, Space: 16, Part: 12] total time:108ms, Total send logs: 2
I0621 14:35:50.795786 209462 SlowOpTracker.h:31] [Port: 9780, Space: 16, Part: 42] total time:77ms, Total commit: 1
I0621 14:35:50.809310 209462 SlowOpTracker.h:31] [Port: 9780, Space: 16, Part: 6] total time:76ms, Total send logs: 2
I0621 14:35:50.818338 209468 SlowOpTracker.h:31] [Port: 9780, Space: 16, Part: 24] total time:59ms, Total commit: 1
I0621 14:35:50.839812 209473 SlowOpTracker.h:31] [Port: 9780, Space: 16, Part: 5] total time:65ms, Total commit: 1
I0621 14:35:50.855896 209467 SlowOpTracker.h:31] [Port: 9780, Space: 16, Part: 47] total time:74ms, Total send logs: 2
I0621 14:35:50.861594 209474 SlowOpTracker.h:31] [Port: 9780, Space: 16, Part: 12] total time:69ms, Total commit: 1
I0621 14:35:50.877748 213053 RebuildIndexTask.cpp:101] Building index successful, space=16, part=12
I0621 14:35:50.877784 213053 RebuildIndexTask.cpp:104] Processing operation logs, space=16, part=12
I0621 14:35:50.878032 209462 SlowOpTracker.h:31] [Port: 9780, Space: 16, Part: 6] total time:68ms, Total commit: 1
I0621 14:35:50.894052 213052 RebuildIndexTask.cpp:101] Building index successful, space=16, part=6
I0621 14:35:50.894091 213052 RebuildIndexTask.cpp:104] Processing operation logs, space=16, part=6
I0621 14:35:50.919631 209467 SlowOpTracker.h:31] [Port: 9780, Space: 16, Part: 47] total time:64ms, Total commit: 1
I0621 14:35:50.943254 209462 SlowOpTracker.h:31] [Port: 9780, Space: 16, Part: 17] total time:52ms, Total commit: 1
I0621 14:35:50.988693 209473 SlowOpTracker.h:31] [Port: 9780, Space: 16, Part: 30] total time:207ms, Total send logs: 2
I0621 14:35:51.000066 213057 RebuildIndexTask.cpp:101] Building index successful, space=16, part=42
I0621 14:35:51.000097 213057 RebuildIndexTask.cpp:104] Processing operation logs, space=16, part=42
I0621 14:35:51.028744 209462 SlowOpTracker.h:31] [Port: 9780, Space: 16, Part: 24] total time:54ms, Total commit: 1
I0621 14:35:51.032083 209462 SlowOpTracker.h:31] [Port: 9780, Space: 16, Part: 5] total time:60ms, Total send logs: 2
I0621 14:35:51.032712 213058 RebuildIndexTask.cpp:113] RebuildIndexTask Finished, space=16, part=47
I0621 14:35:51.032914 213058 AdminTaskManager.cpp:319] subtask of task(19, 4) finished, unfinished task 7
I0621 14:35:51.044389 213058 AdminTaskManager.cpp:327] task(19, 4) runSubTask() exit
I0621 14:35:51.045456 213055 RebuildIndexTask.cpp:101] Building index successful, space=16, part=24
I0621 14:35:51.045486 213055 RebuildIndexTask.cpp:104] Processing operation logs, space=16, part=24
I0621 14:35:51.115078 209462 SlowOpTracker.h:31] [Port: 9780, Space: 16, Part: 12] total time:187ms, Total send logs: 2
I0621 14:35:51.119371 209462 SlowOpTracker.h:31] [Port: 9780, Space: 16, Part: 17] total time:63ms, Total send logs: 2
I0621 14:35:51.123402 213052 RebuildIndexTask.cpp:113] RebuildIndexTask Finished, space=16, part=6
I0621 14:35:51.123438 213052 AdminTaskManager.cpp:319] subtask of task(19, 4) finished, unfinished task 6
I0621 14:35:51.133909 213052 AdminTaskManager.cpp:327] task(19, 4) runSubTask() exit
I0621 14:35:51.150528 213054 RebuildIndexTask.cpp:101] Building index successful, space=16, part=17
I0621 14:35:51.150561 213054 RebuildIndexTask.cpp:104] Processing operation logs, space=16, part=17
I0621 14:35:51.164101 209467 SlowOpTracker.h:31] [Port: 9780, Space: 16, Part: 24] total time:69ms, Total send logs: 2
I0621 14:35:51.185819 213057 RebuildIndexTask.cpp:113] RebuildIndexTask Finished, space=16, part=42
I0621 14:35:51.185851 213057 AdminTaskManager.cpp:319] subtask of task(19, 4) finished, unfinished task 5
I0621 14:35:51.196148 213057 AdminTaskManager.cpp:327] task(19, 4) runSubTask() exit
I0621 14:35:51.214577 209467 SlowOpTracker.h:31] [Port: 9780, Space: 16, Part: 30] total time:135ms, Total send logs: 2
I0621 14:35:51.218935 213056 RebuildIndexTask.cpp:113] RebuildIndexTask Finished, space=16, part=30
I0621 14:35:51.218959 213056 AdminTaskManager.cpp:319] subtask of task(19, 4) finished, unfinished task 4
I0621 14:35:51.229563 213056 AdminTaskManager.cpp:327] task(19, 4) runSubTask() exit
I0621 14:35:51.295143 213055 RebuildIndexTask.cpp:113] RebuildIndexTask Finished, space=16, part=24
I0621 14:35:51.295176 213055 AdminTaskManager.cpp:319] subtask of task(19, 4) finished, unfinished task 3
I0621 14:35:51.305573 213055 AdminTaskManager.cpp:327] task(19, 4) runSubTask() exit
I0621 14:35:51.352042 209467 SlowOpTracker.h:31] [Port: 9780, Space: 16, Part: 12] total time:176ms, Total send logs: 2
I0621 14:35:51.356070 213053 RebuildIndexTask.cpp:113] RebuildIndexTask Finished, space=16, part=12
I0621 14:35:51.356096 213053 AdminTaskManager.cpp:319] subtask of task(19, 4) finished, unfinished task 2
I0621 14:35:51.366653 213053 AdminTaskManager.cpp:327] task(19, 4) runSubTask() exit
I0621 14:35:51.377038 213054 RebuildIndexTask.cpp:113] RebuildIndexTask Finished, space=16, part=17
I0621 14:35:51.377063 213054 AdminTaskManager.cpp:319] subtask of task(19, 4) finished, unfinished task 1
I0621 14:35:51.387492 213054 AdminTaskManager.cpp:327] task(19, 4) runSubTask() exit
I0621 14:35:51.444142 213051 RebuildIndexTask.cpp:101] Building index successful, space=16, part=5
I0621 14:35:51.444175 213051 RebuildIndexTask.cpp:104] Processing operation logs, space=16, part=5
I0621 14:35:51.603047 213051 RebuildIndexTask.cpp:113] RebuildIndexTask Finished, space=16, part=5
I0621 14:35:51.603081 213051 AdminTaskManager.cpp:319] subtask of task(19, 4) finished, unfinished task 0
I0621 14:35:51.603250 213051 AdminTask.h:79] task(19, 4) finished, rc=[SUCCEEDED]
I0621 14:35:51.605798 210409 AdminTaskManager.cpp:82] reportTaskFinish(), job=19, task=4, rc=SUCCEEDED
I0621 14:35:51.616662 210409 AdminTaskManager.cpp:119] reportTaskFinish(), job=19, task=4, rc=SUCCEEDED
I0621 14:35:51.651948 209473 RaftPart.cpp:1241] [Port: 9780, Space: 1, Part: 23] Recieved a VOTING request: space = 1, partition = 23, candidateAddr = 10.128.67.56:9780, term = 10, lastLogId = 7813, lastLogTerm = 9
I0621 14:35:51.651978 209473 RaftPart.cpp:1268] [Port: 9780, Space: 1, Part: 23] The partition currently is a Follower, lastLogId 7813, lastLogTerm 9, committedLogId 7812, term 9
I0621 14:35:51.651989 209473 RaftPart.cpp:1329] [Port: 9780, Space: 1, Part: 23] The partition will vote for the candidate "10.128.67.56":9780
I0621 14:35:51.652479 209556 RaftPart.cpp:1613] [Port: 9780, Space: 1, Part: 23] The current role is Follower. Will follow the new leader 10.128.67.56:9780 [Term: 10]
I0621 14:35:51.652561 209562 Part.cpp:209] [Port: 9780, Space: 1, Part: 23] Find the new leader "10.128.67.56":9780
I0621 14:35:52.521404 209462 RaftPart.cpp:1241] [Port: 9780, Space: 16, Part: 23] Recieved a VOTING request: space = 16, partition = 23, candidateAddr = 10.128.67.56:9780, term = 8, lastLogId = 1376, lastLogTerm = 7
I0621 14:35:52.521440 209462 RaftPart.cpp:1268] [Port: 9780, Space: 16, Part: 23] The partition currently is a Follower, lastLogId 1376, lastLogTerm 7, committedLogId 1375, term 7
I0621 14:35:52.521450 209462 RaftPart.cpp:1329] [Port: 9780, Space: 16, Part: 23] The partition will vote for the candidate "10.128.67.56":9780
I0621 14:35:52.521836 209548 RaftPart.cpp:1613] [Port: 9780, Space: 16, Part: 23] The current role is Follower. Will follow the new leader 10.128.67.56:9780 [Term: 8]
I0621 14:35:52.521919 209561 Part.cpp:209] [Port: 9780, Space: 16, Part: 23] Find the new leader "10.128.67.56":9780
I0621 14:35:53.429816 209468 SlowOpTracker.h:31] [Port: 9780, Space: 16, Part: 47] total time:87ms, Total send logs: 2
I0621 14:35:53.429823 209474 SlowOpTracker.h:31] [Port: 9780, Space: 16, Part: 5] total time:87ms, Total send logs: 2
I0621 14:35:55.655488 209474 RaftPart.cpp:1241] [Port: 9780, Space: 10, Part: 23] Recieved a VOTING request: space = 10, partition = 23, candidateAddr = 10.128.67.56:9780, term = 9, lastLogId = 2694, lastLogTerm = 8
I0621 14:35:55.655522 209474 RaftPart.cpp:1268] [Port: 9780, Space: 10, Part: 23] The partition currently is a Follower, lastLogId 2694, lastLogTerm 8, committedLogId 2693, term 8
I0621 14:35:55.655531 209474 RaftPart.cpp:1329] [Port: 9780, Space: 10, Part: 23] The partition will vote for the candidate "10.128.67.56":9780
I0621 14:35:55.655851 209543 RaftPart.cpp:1613] [Port: 9780, Space: 10, Part: 23] The current role is Follower. Will follow the new leader 10.128.67.56:9780 [Term: 9]
I0621 14:35:55.655918 209560 Part.cpp:209] [Port: 9780, Space: 10, Part: 23] Find the new leader "10.128.67.56":9780
I0621 14:35:56.253960 209462 RaftPart.cpp:1241] [Port: 9780, Space: 10, Part: 47] Recieved a VOTING request: space = 10, partition = 47, candidateAddr = 10.128.67.56:9780, term = 7, lastLogId = 2699, lastLogTerm = 6
I0621 14:35:56.253994 209462 RaftPart.cpp:1268] [Port: 9780, Space: 10, Part: 47] The partition currently is a Follower, lastLogId 2699, lastLogTerm 6, committedLogId 2698, term 6
I0621 14:35:56.254006 209462 RaftPart.cpp:1329] [Port: 9780, Space: 10, Part: 47] The partition will vote for the candidate "10.128.67.56":9780
I0621 14:35:56.254447 209549 RaftPart.cpp:1613] [Port: 9780, Space: 10, Part: 47] The current role is Follower. Will follow the new leader 10.128.67.56:9780 [Term: 7]
I0621 14:35:56.254516 209561 Part.cpp:209] [Port: 9780, Space: 10, Part: 47] Find the new leader "10.128.67.56":9780
I0621 14:35:56.926414 209462 RaftPart.cpp:1241] [Port: 9780, Space: 10, Part: 5] Recieved a VOTING request: space = 10, partition = 5, candidateAddr = 10.128.67.56:9780, term = 8, lastLogId = 2822, lastLogTerm = 7
I0621 14:35:56.926451 209462 RaftPart.cpp:1268] [Port: 9780, Space: 10, Part: 5] The partition currently is a Follower, lastLogId 2822, lastLogTerm 7, committedLogId 2821, term 7
I0621 14:35:56.926463 209462 RaftPart.cpp:1329] [Port: 9780, Space: 10, Part: 5] The partition will vote for the candidate "10.128.67.56":9780
I0621 14:35:56.926848 209480 RaftPart.cpp:1613] [Port: 9780, Space: 10, Part: 5] The current role is Follower. Will follow the new leader 10.128.67.56:9780 [Term: 8]
I0621 14:35:56.926940 209562 Part.cpp:209] [Port: 9780, Space: 10, Part: 5] Find the new leader "10.128.67.56":9780
I0621 14:35:57.016826 209558 MetaClient.cpp:3014] Load leader of "10.128.67.56":9779 in 5 space
I0621 14:35:57.016894 209558 MetaClient.cpp:3014] Load leader of "10.130.196.46":9779 in 5 space
I0621 14:35:57.016937 209558 MetaClient.cpp:3014] Load leader of "10.130.196.47":9779 in 5 space
I0621 14:35:57.016973 209558 MetaClient.cpp:3014] Load leader of "10.129.207.193":9779 in 5 space
I0621 14:35:57.017004 209558 MetaClient.cpp:3014] Load leader of "10.129.207.194":9779 in 5 space
I0621 14:35:57.017030 209558 MetaClient.cpp:3014] Load leader of "10.129.207.195":9779 in 5 space
I0621 14:35:57.017038 209558 MetaClient.cpp:3020] Load leader ok
I0621 14:35:58.750746 209467 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:58.773281 209468 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:58.795662 209467 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:58.819770 209473 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:58.840596 209473 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:58.863293 209467 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:58.886844 209473 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:58.907294 209467 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:58.928766 209473 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:58.947981 209468 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:58.970209 209467 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:58.995322 209468 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.113083 209467 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.134294 209467 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.166854 209467 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.192138 209473 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.212406 209467 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.235790 209467 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.258039 209473 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.279484 209467 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.301730 209468 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.323335 209468 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.344214 209467 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.371186 209473 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.394651 209467 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.424351 209473 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.450767 209467 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.480841 209468 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.506043 209462 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.529296 209467 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.551656 209468 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.575181 209468 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.596292 209473 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.618263 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.641559 209473 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.662075 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.683830 209473 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.706593 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.728981 209473 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.752532 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.781608 209473 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.803089 209473 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.823774 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.852023 209473 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.872402 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.894341 209467 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.913357 209467 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.933373 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.955303 209468 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.976428 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:35:59.995046 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.013823 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.033180 209473 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.052004 209468 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.070726 209468 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.091935 209468 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.112533 209468 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.131542 209473 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.151239 209468 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.173591 209468 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.195986 209473 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.214107 209473 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.234768 209473 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.252986 209473 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.271709 209468 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.292634 209473 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.314736 209468 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.335058 209473 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.356040 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.377058 209473 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.400096 209468 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.424083 209473 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.446590 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.467679 209467 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.488638 209467 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.511415 209473 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.530361 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.556769 209468 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.577240 209467 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.596477 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.616187 209467 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.636878 209467 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.659809 209467 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.680505 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.700632 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.720088 209467 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.741537 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.762370 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.782665 209467 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.803648 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.824740 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.844413 209467 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.865914 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.886667 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.908830 209468 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.930528 209467 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.953186 209468 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.973531 209468 RaftPart.cpp:1241] [Port: 9780, Space: 1, Part: 47] Recieved a VOTING request: space = 1, partition = 47, candidateAddr = 10.128.67.56:9780, term = 6, lastLogId = 7804, lastLogTerm = 5
I0621 14:36:00.973549 209468 RaftPart.cpp:1268] [Port: 9780, Space: 1, Part: 47] The partition currently is a Follower, lastLogId 7804, lastLogTerm 5, committedLogId 7803, term 5
I0621 14:36:00.973558 209468 RaftPart.cpp:1329] [Port: 9780, Space: 1, Part: 47] The partition will vote for the candidate "10.128.67.56":9780
I0621 14:36:00.973984 209556 RaftPart.cpp:1613] [Port: 9780, Space: 1, Part: 47] The current role is Follower. Will follow the new leader 10.128.67.56:9780 [Term: 6]
I0621 14:36:00.974071 209468 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:00.974068 209559 Part.cpp:209] [Port: 9780, Space: 1, Part: 47] Find the new leader "10.128.67.56":9780
I0621 14:36:00.995203 209462 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.017858 209468 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.040779 209468 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.061579 209462 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.083220 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.103986 209462 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.124976 209462 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.149461 209468 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.172142 209462 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.195155 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.223366 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.244762 209462 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.267014 209462 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.290163 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.312734 209462 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.334205 209468 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.355294 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.377626 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.399441 209462 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.422288 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.443372 209462 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.467530 209464 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.489058 209468 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.513319 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.535061 209464 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.557608 209462 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.579139 209464 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.601327 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.623456 209462 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.644798 209462 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.667112 209462 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.688565 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.713559 209464 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.737299 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.759570 209462 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.785656 209462 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.808544 209464 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.829730 209462 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.850279 209462 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.871666 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.893577 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.915238 209464 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.937281 209462 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.957862 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:01.982198 209462 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:02.004869 209462 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:02.026682 209462 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:02.047466 209462 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:02.069974 209464 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:02.089921 209462 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:02.109092 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:02.145213 209462 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:02.166513 209464 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:02.190075 209464 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:02.216920 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:02.234519 209464 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:02.254544 209462 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:02.278642 209474 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:02.300756 209462 RaftPart.cpp:1578] [Port: 9780, Space: 2, Part: 23] The current role is Leader. The local term is 16. The remote term is not newer
I0621 14:36:05.656262 209474 RaftPart.cpp:1241] [Port: 9780, Space: 10, Part: 29] Recieved a VOTING request: space = 10, partition = 29, candidateAddr = 10.128.67.56:9780, term = 244, lastLogId = 2696, lastLogTerm = 243
I0621 14:36:05.656298 209474 RaftPart.cpp:1268] [Port: 9780, Space: 10, Part: 29] The partition currently is a Follower, lastLogId 2696, lastLogTerm 243, committedLogId 2695, term 243
I0621 14:36:05.656311 209474 RaftPart.cpp:1329] [Port: 9780, Space: 10, Part: 29] The partition will vote for the candidate "10.128.67.56":9780
I0621 14:36:05.656790 209553 RaftPart.cpp:1613] [Port: 9780, Space: 10, Part: 29] The current role is Follower. Will follow the new leader 10.128.67.56:9780 [Term: 244]
I0621 14:36:05.656875 209560 Part.cpp:209] [Port: 9780, Space: 10, Part: 29] Find the new leader "10.128.67.56":9780
I0621 14:36:07.145681 209558 MetaClient.cpp:3014] Load leader of "10.128.67.56":9779 in 5 space
I0621 14:36:07.145748 209558 MetaClient.cpp:3014] Load leader of "10.130.196.46":9779 in 5 space
I0621 14:36:07.145782 209558 MetaClient.cpp:3014] Load leader of "10.130.196.47":9779 in 5 space
I0621 14:36:07.145817 209558 MetaClient.cpp:3014] Load leader of "10.129.207.193":9779 in 5 space
I0621 14:36:07.145845 209558 MetaClient.cpp:3014] Load leader of "10.129.207.194":9779 in 5 space
I0621 14:36:07.145862 209558 MetaClient.cpp:3014] Load leader of "10.129.207.195":9779 in 5 space
I0621 14:36:07.145869 209558 MetaClient.cpp:3020] Load leader ok
I0621 14:36:07.799854 209562 RaftPart.cpp:957] [Port: 9780, Space: 10, Part: 41] Start leader election, reason: lastMsgDur 30001, term 246
I0621 14:36:07.799906 209562 RaftPart.cpp:1095] [Port: 9780, Space: 10, Part: 41] Sending out an election request (space = 10, part = 41, term = 247, lastLogId = 2696, lastLogTerm = 246, candidateIP = 10.130.196.46, candidatePort = 9780)
I0621 14:36:07.800345 209562 RaftPart.cpp:1059] [Port: 9780, Space: 10, Part: 41] Partition is elected as the new leader for term 247
I0621 14:36:07.800362 209562 RaftPart.cpp:1137] [Port: 9780, Space: 10, Part: 41] The partition is elected as the leader
I0621 14:36:07.800753 209545 Host.cpp:125] [Port: 9780, Space: 10, Part: 41] [Host: 10.129.207.195:9780] This is the first time to send the logs to this host, lastLogIdSent = 2696, lastLogTermSent = 246
I0621 14:36:07.800806 209545 Host.cpp:125] [Port: 9780, Space: 10, Part: 41] [Host: 10.128.67.56:9780] This is the first time to send the logs to this host, lastLogIdSent = 2696, lastLogTermSent = 246
I0621 14:36:07.801255 209562 TransactionManager.cpp:172] leader get do scanPrimes space=10, part=41, term=247
I0621 14:36:07.801272 209562 TransactionManager.cpp:178] scanPrimes(), spaceId=10, partId=41
I0621 14:36:08.173882 209562 TransactionManager.cpp:227] insert space=10, part=41, into white list suc=1
I0621 14:36:10.975080 209474 RaftPart.cpp:1241] [Port: 9780, Space: 1, Part: 41] Recieved a VOTING request: space = 1, partition = 41, candidateAddr = 10.128.67.56:9780, term = 18, lastLogId = 7823, lastLogTerm = 17
I0621 14:36:10.975109 209474 RaftPart.cpp:1268] [Port: 9780, Space: 1, Part: 41] The partition currently is a Follower, lastLogId 7823, lastLogTerm 17, committedLogId 7822, term 17
I0621 14:36:10.975118 209474 RaftPart.cpp:1329] [Port: 9780, Space: 1, Part: 41] The partition will vote for the candidate "10.128.67.56":9780
I0621 14:36:10.975409 209553 RaftPart.cpp:1613] [Port: 9780, Space: 1, Part: 41] The current role is Follower. Will follow the new leader 10.128.67.56:9780 [Term: 18]
I0621 14:36:10.975474 209559 Part.cpp:209] [Port: 9780, Space: 1, Part: 41] Find the new leader "10.128.67.56":9780
I0621 14:36:11.860471 209561 RaftPart.cpp:957] [Port: 9780, Space: 5, Part: 5] Start leader election, reason: lastMsgDur 35593, term 13
I0621 14:36:11.860522 209561 RaftPart.cpp:1095] [Port: 9780, Space: 5, Part: 5] Sending out an election request (space = 5, part = 5, term = 14, lastLogId = 8230, lastLogTerm = 13, candidateIP = 10.130.196.46, candidatePort = 9780)
I0621 14:36:11.861205 209561 RaftPart.cpp:1059] [Port: 9780, Space: 5, Part: 5] Partition is elected as the new leader for term 14
I0621 14:36:11.861213 209561 RaftPart.cpp:1137] [Port: 9780, Space: 5, Part: 5] The partition is elected as the leader
I0621 14:36:11.861600 209543 Host.cpp:125] [Port: 9780, Space: 5, Part: 5] [Host: 10.129.207.195:9780] This is the first time to send the logs to this host, lastLogIdSent = 8230, lastLogTermSent = 13
I0621 14:36:11.861645 209543 Host.cpp:125] [Port: 9780, Space: 5, Part: 5] [Host: 10.128.67.56:9780] This is the first time to send the logs to this host, lastLogIdSent = 8230, lastLogTermSent = 13
I0621 14:36:11.862160 209559 TransactionManager.cpp:172] leader get do scanPrimes space=5, part=5, term=14
I0621 14:36:11.862192 209559 TransactionManager.cpp:178] scanPrimes(), spaceId=5, partId=5
I0621 14:36:12.230818 209560 RaftPart.cpp:957] [Port: 9780, Space: 5, Part: 47] Start leader election, reason: lastMsgDur 31733, term 4
I0621 14:36:12.230854 209560 RaftPart.cpp:1095] [Port: 9780, Space: 5, Part: 47] Sending out an election request (space = 5, part = 47, term = 5, lastLogId = 8207, lastLogTerm = 4, candidateIP = 10.130.196.46, candidatePort = 9780)
I0621 14:36:12.231348 209560 RaftPart.cpp:1059] [Port: 9780, Space: 5, Part: 47] Partition is elected as the new leader for term 5
I0621 14:36:12.231356 209560 RaftPart.cpp:1137] [Port: 9780, Space: 5, Part: 47] The partition is elected as the leader
I0621 14:36:12.231685 209549 Host.cpp:125] [Port: 9780, Space: 5, Part: 47] [Host: 10.129.207.195:9780] This is the first time to send the logs to this host, lastLogIdSent = 8207, lastLogTermSent = 4
I0621 14:36:12.231773 209549 Host.cpp:125] [Port: 9780, Space: 5, Part: 47] [Host: 10.128.67.56:9780] This is the first time to send the logs to this host, lastLogIdSent = 8207, lastLogTermSent = 4
I0621 14:36:12.232231 209559 TransactionManager.cpp:227] insert space=5, part=5, into white list suc=0
I0621 14:36:12.232295 209560 TransactionManager.cpp:172] leader get do scanPrimes space=5, part=47, term=5
I0621 14:36:12.232306 209560 TransactionManager.cpp:178] scanPrimes(), spaceId=5, partId=47
I0621 14:36:12.412972 209560 TransactionManager.cpp:227] insert space=5, part=47, into white list suc=1
I0621 14:36:12.434989 209561 RaftPart.cpp:957] [Port: 9780, Space: 5, Part: 35] Start leader election, reason: lastMsgDur 31938, term 22
I0621 14:36:12.435053 209561 RaftPart.cpp:1095] [Port: 9780, Space: 5, Part: 35] Sending out an election request (space = 5, part = 35, term = 23, lastLogId = 8230, lastLogTerm = 22, candidateIP = 10.130.196.46, candidatePort = 9780)
I0621 14:36:12.435490 209561 RaftPart.cpp:1059] [Port: 9780, Space: 5, Part: 35] Partition is elected as the new leader for term 23
I0621 14:36:12.435516 209561 RaftPart.cpp:1137] [Port: 9780, Space: 5, Part: 35] The partition is elected as the leader
I0621 14:36:12.435828 209556 Host.cpp:125] [Port: 9780, Space: 5, Part: 35] [Host: 10.129.207.195:9780] This is the first time to send the logs to this host, lastLogIdSent = 8230, lastLogTermSent = 22
I0621 14:36:12.435886 209556 Host.cpp:125] [Port: 9780, Space: 5, Part: 35] [Host: 10.128.67.56:9780] This is the first time to send the logs to this host, lastLogIdSent = 8230, lastLogTermSent = 22
I0621 14:36:12.436372 209559 TransactionManager.cpp:172] leader get do scanPrimes space=5, part=35, term=23
I0621 14:36:12.436403 209559 TransactionManager.cpp:178] scanPrimes(), spaceId=5, partId=35
I0621 14:36:12.582638 209559 TransactionManager.cpp:227] insert space=5, part=35, into white list suc=1
I0621 14:36:13.544412 209474 SlowOpTracker.h:31] [Port: 9780, Space: 16, Part: 5] total time:199ms, Total send logs: 2
I0621 14:36:17.217455 209558 MetaClient.cpp:3014] Load leader of "10.128.67.56":9779 in 5 space
I0621 14:36:17.217530 209558 MetaClient.cpp:3014] Load leader of "10.130.196.46":9779 in 5 space
I0621 14:36:17.217557 209558 MetaClient.cpp:3014] Load leader of "10.130.196.47":9779 in 5 space
I0621 14:36:17.217592 209558 MetaClient.cpp:3014] Load leader of "10.129.207.193":9779 in 5 space
I0621 14:36:17.217618 209558 MetaClient.cpp:3014] Load leader of "10.129.207.194":9779 in 5 space
I0621 14:36:17.217625 209558 MetaClient.cpp:3014] Load leader of "10.129.207.195":9779 in 0 space
I0621 14:36:17.217631 209558 MetaClient.cpp:3020] Load leader ok
I0621 14:36:23.277294 209467 RaftPart.cpp:1241] [Port: 9780, Space: 16, Part: 35] Recieved a VOTING request: space = 16, partition = 35, candidateAddr = 10.128.67.56:9780, term = 8, lastLogId = 1125, lastLogTerm = 7
I0621 14:36:23.277330 209467 RaftPart.cpp:1268] [Port: 9780, Space: 16, Part: 35] The partition currently is a Follower, lastLogId 1125, lastLogTerm 7, committedLogId 1124, term 7
I0621 14:36:23.277341 209467 RaftPart.cpp:1329] [Port: 9780, Space: 16, Part: 35] The partition will vote for the candidate "10.128.67.56":9780
I0621 14:36:23.277765 209545 RaftPart.cpp:1613] [Port: 9780, Space: 16, Part: 35] The current role is Follower. Will follow the new leader 10.128.67.56:9780 [Term: 8]
I0621 14:36:23.277855 209560 Part.cpp:209] [Port: 9780, Space: 16, Part: 35] Find the new leader "10.128.67.56":9780
I0621 14:36:23.621798 209467 SlowOpTracker.h:31] [Port: 9780, Space: 16, Part: 47] total time:274ms, Total send logs: 2
I0621 14:36:26.124953 209466 RaftPart.cpp:1241] [Port: 9780, Space: 2, Part: 47] Recieved a VOTING request: space = 2, partition = 47, candidateAddr = 10.128.67.56:9780, term = 25, lastLogId = 7791, lastLogTerm = 24
I0621 14:36:26.124989 209466 RaftPart.cpp:1268] [Port: 9780, Space: 2, Part: 47] The partition currently is a Follower, lastLogId 7791, lastLogTerm 24, committedLogId 7790, term 24
I0621 14:36:26.125002 209466 RaftPart.cpp:1329] [Port: 9780, Space: 2, Part: 47] The partition will vote for the candidate "10.128.67.56":9780
I0621 14:36:26.125389 209549 RaftPart.cpp:1613] [Port: 9780, Space: 2, Part: 47] The current role is Follower. Will follow the new leader 10.128.67.56:9780 [Term: 25]
I0621 14:36:26.125504 209559 Part.cpp:209] [Port: 9780, Space: 2, Part: 47] Find the new leader "10.128.67.56":9780
I0621 14:36:27.291397 209558 MetaClient.cpp:3014] Load leader of "10.128.67.56":9779 in 5 space
I0621 14:36:27.291463 209558 MetaClient.cpp:3014] Load leader of "10.130.196.46":9779 in 5 space
I0621 14:36:27.291496 209558 MetaClient.cpp:3014] Load leader of "10.130.196.47":9779 in 5 space
I0621 14:36:27.291528 209558 MetaClient.cpp:3014] Load leader of "10.129.207.193":9779 in 5 space
I0621 14:36:27.291565 209558 MetaClient.cpp:3014] Load leader of "10.129.207.194":9779 in 5 space
I0621 14:36:27.291572 209558 MetaClient.cpp:3014] Load leader of "10.129.207.195":9779 in 0 space
I0621 14:36:27.291579 209558 MetaClient.cpp:3020] Load leader ok
I0621 14:36:31.771555 209474 RaftPart.cpp:1241] [Port: 9780, Space: 2, Part: 29] Recieved a VOTING request: space = 2, partition = 29, candidateAddr = 10.129.207.195:9780, term = 19, lastLogId = 7790, lastLogTerm = 18
I0621 14:36:31.771591 209474 RaftPart.cpp:1268] [Port: 9780, Space: 2, Part: 29] The partition currently is a Follower, lastLogId 7796, lastLogTerm 18, committedLogId 7795, term 18
I0621 14:36:31.771600 209474 RaftPart.cpp:1302] [Port: 9780, Space: 2, Part: 29] The partition's last log id is 7796. The candidate's last log id 7790 is smaller, so it will be rejected, candidate is "10.129.207.195":9780
I0621 14:36:37.383539 209558 MetaClient.cpp:3014] Load leader of "10.128.67.56":9779 in 5 space
I0621 14:36:37.383605 209558 MetaClient.cpp:3014] Load leader of "10.130.196.46":9779 in 5 space
I0621 14:36:37.383635 209558 MetaClient.cpp:3014] Load leader of "10.130.196.47":9779 in 5 space
I0621 14:36:37.383671 209558 MetaClient.cpp:3014] Load leader of "10.129.207.193":9779 in 5 space
I0621 14:36:37.383697 209558 MetaClient.cpp:3014] Load leader of "10.129.207.194":9779 in 5 space
I0621 14:36:37.383703 209558 MetaClient.cpp:3014] Load leader of "10.129.207.195":9779 in 0 space
I0621 14:36:37.383709 209558 MetaClient.cpp:3020] Load leader ok
I0621 14:37:01.865082 209560 RaftPart.cpp:957] [Port: 9780, Space: 1, Part: 35] Start leader election, reason: lastMsgDur 30102, term 11
I0621 14:37:01.865216 209560 RaftPart.cpp:1095] [Port: 9780, Space: 1, Part: 35] Sending out an election request (space = 1, part = 35, term = 12, lastLogId = 7818, lastLogTerm = 11, candidateIP = 10.130.196.46, candidatePort = 9780)
I0621 14:37:01.865901 209560 RaftPart.cpp:1059] [Port: 9780, Space: 1, Part: 35] Partition is elected as the new leader for term 12
I0621 14:37:01.865916 209560 RaftPart.cpp:1137] [Port: 9780, Space: 1, Part: 35] The partition is elected as the leader
I0621 14:37:01.866138 209555 Host.cpp:125] [Port: 9780, Space: 1, Part: 35] [Host: 10.129.207.195:9780] This is the first time to send the logs to this host, lastLogIdSent = 7818, lastLogTermSent = 11
I0621 14:37:01.866242 209555 Host.cpp:125] [Port: 9780, Space: 1, Part: 35] [Host: 10.128.67.56:9780] This is the first time to send the logs to this host, lastLogIdSent = 7818, lastLogTermSent = 11
I0621 14:37:01.866748 209562 TransactionManager.cpp:172] leader get do scanPrimes space=1, part=35, term=12
I0621 14:37:01.866940 209562 TransactionManager.cpp:178] scanPrimes(), spaceId=1, partId=35
I0621 14:37:01.867125 209562 TransactionManager.cpp:227] insert space=1, part=35, into white list suc=1
I0621 14:37:04.651602 209462 RaftPart.cpp:1241] [Port: 9780, Space: 5, Part: 23] Recieved a VOTING request: space = 5, partition = 23, candidateAddr = 10.129.207.195:9780, term = 8, lastLogId = 8201, lastLogTerm = 7
I0621 14:37:04.651649 209462 RaftPart.cpp:1268] [Port: 9780, Space: 5, Part: 23] The partition currently is a Follower, lastLogId 8212, lastLogTerm 7, committedLogId 8211, term 7
I0621 14:37:04.651657 209462 RaftPart.cpp:1302] [Port: 9780, Space: 5, Part: 23] The partition's last log id is 8212. The candidate's last log id 8201 is smaller, so it will be rejected, candidate is "10.129.207.195":9780
I0621 14:37:07.498966 209558 MetaClient.cpp:3014] Load leader of "10.128.67.56":9779 in 5 space
I0621 14:37:07.499071 209558 MetaClient.cpp:3014] Load leader of "10.130.196.46":9779 in 5 space
I0621 14:37:07.499127 209558 MetaClient.cpp:3014] Load leader of "10.130.196.47":9779 in 5 space
I0621 14:37:07.499181 209558 MetaClient.cpp:3014] Load leader of "10.129.207.193":9779 in 5 space
I0621 14:37:07.499229 209558 MetaClient.cpp:3014] Load leader of "10.129.207.194":9779 in 5 space
I0621 14:37:07.499244 209558 MetaClient.cpp:3014] Load leader of "10.129.207.195":9779 in 0 space
I0621 14:37:07.499267 209558 MetaClient.cpp:3020] Load leader ok

spw · 2022 年6 月 21 日 08:51

感觉这些日志没有关于挂掉的有用信息？

zhengshuai1030 · 2022 年6 月 21 日 08:54

是的我看日志也没有为什么挂掉日志打出。是不是这个日志没有记录还是级别不够？？这个是nebula-storaged.INFO 日志

spw · 2022 年6 月 21 日 10:23

有没有 storaged 挂掉的出错栈

zhengshuai1030 · 2022 年6 月 21 日 10:32

这个是corefiles
corefiles.zip (5.0 MB)

zhengshuai1030 · 2022 年6 月 22 日 02:59

这边发现内存有点高，就优化了下面两个参数后测试了几遍感觉稳定了一些，具体什么原因还不明确，大神你那边麻烦再帮分析一下
–rocksdb_block_based_table_options={“block_size”:“16384”,“cache_index_and_filter_blocks”:“true”}
–rocksdb_db_options={“max_subcompactions”:“4”,“max_background_jobs”:“4”,“max_open_files”:“3000”}

sworduo · 2022 年6 月 22 日 03:14

之前是不是oom了，dmesg看看

spw · 2022 年6 月 22 日 03:46

好的，我来看下；能把优化前后的参数都贴一下吗？

zhengshuai1030 · 2022 年6 月 22 日 03:48

优化前就是2.6.2 默认的
–rocksdb_block_based_table_options={“block_size”:“8192”,“cache_index_and_filter_blocks”:“true”}
–rocksdb_db_options={“max_subcompactions”:“4”,“max_background_jobs”:“4”}

spw · 2022 年6 月 22 日 04:45

core file 我这打不开，毕竟环境和参数不一样。你能通过 gdb 打开，看下错误栈然后贴上来吗？

zhengshuai1030 · 2022 年6 月 22 日 06:17

gdb 我不熟，我先找c++ 的同事帮看看

zhengshuai1030 · 2022 年6 月 22 日 06:58

这是错误用栈吗

zhengshuai1030 · 2022 年6 月 22 日 08:12

这边c++ 同事说这个没有日志打出，要重新打包debug 模式啥的才有日志？？。我这边该怎么处理啊，这个问题出现频率很高了，rebulaid index 就蹦了storaged没法使用了，大神帮忙看看

spw · 2022 年6 月 22 日 08:37

是因为 OOM 么？

zhengshuai1030 · 2022 年6 月 22 日 08:37

现在内存一直高位不下降，是什么原因。现在也没rebulid index 和导入数据

zhengshuai1030 · 2022 年6 月 22 日 08:38

可能吧，看内存监控有内存有到95%

zhengshuai1030 · 2022 年6 月 22 日 08:44

好的我看看,下面是现在storaged 配置，也帮分析看看

########## basics ##########
# Whether to run as a daemon process
--daemonize=true
# The file to host the process id
--pid_file=pids/nebula-storaged.pid
# Whether to use the configuration obtained from the configuration file
#--local_config=true

########## logging ##########
# The directory to host logging files
--log_dir=/home/service/var/logs/nebula-graph-2.6.2/storaged
# Log level, 0, 1, 2, 3 for INFO, WARNING, ERROR, FATAL respectively
--minloglevel=0
# Verbose log level, 1, 2, 3, 4, the higher of the level, the more verbose of the logging
--v=0
# Maximum seconds to buffer the log messages
--logbufsecs=0
# Whether to redirect stdout and stderr to separate output files
--redirect_stdout=true
# Destination filename of stdout and stderr, which will also reside in log_dir.
--stdout_log_file=storaged-stdout.log
--stderr_log_file=storaged-stderr.log
# Copy log messages at or above this level to stderr in addition to logfiles. The numbers of severity levels INFO, WARNING, ERROR, and FATAL are 0, 1, 2, and 3, respectively.
--stderrthreshold=2

########## networking ##########
# Comma separated Meta server addresses
--meta_server_addrs=10.128.67.56:9559,10.129.207.193:9559,10.129.207.194:9559
# Local IP used to identify the nebula-storaged process.
# Change it to an address other than loopback if the service is distributed or
# will be accessed remotely.
--local_ip=10.128.67.56
# Storage daemon listening port
--port=9779
# HTTP service ip
--ws_ip=10.128.67.56
# HTTP service port
--ws_http_port=19779
# HTTP2 service port
--ws_h2_port=19780
# heartbeat with meta service
--heartbeat_interval_secs=10

######### Raft #########
# Raft election timeout
--raft_heartbeat_interval_secs=30
# RPC timeout for raft client (ms)
--raft_rpc_timeout_ms=500
## recycle Raft WAL
--wal_ttl=1800

########## Disk ##########
# Root data path. split by comma. e.g. --data_path=/disk1/path1/,/disk2/path2/
# One path per Rocksdb instance.
--data_path=/home/service/var/data/nebula-graph-2.6.2/storaged

# Minimum reserved bytes of each data path
--minimum_reserved_bytes=268435456

# The default reserved bytes for one batch operation
--rocksdb_batch_size=8192
# The default block cache size used in BlockBasedTable. (MB)
# recommend: 1/3 of all memory
--rocksdb_block_cache=8192

# Compression algorithm, options: no,snappy,lz4,lz4hc,zlib,bzip2,zstd
# For the sake of binary compatibility, the default value is snappy.
# Recommend to use:
#   * lz4 to gain more CPU performance, with the same compression ratio with snappy
#   * zstd to occupy less disk space
#   * lz4hc for the read-heavy write-light scenario
--rocksdb_compression=lz4

# Set different compressions for different levels
# For example, if --rocksdb_compression is snappy,
# "no:no:lz4:lz4::zstd" is identical to "no:no:lz4:lz4:snappy:zstd:snappy"
# In order to disable compression for level 0/1, set it to "no:no"
--rocksdb_compression_per_level=

############## rocksdb Options ##############
# rocksdb DBOptions in json, each name and value of option is a string, given as "option_name":"option_value" separated by comma
--rocksdb_db_options={"max_subcompactions":"4","max_background_jobs":"4","max_open_files":"3000"}
# rocksdb ColumnFamilyOptions in json, each name and value of option is string, given as "option_name":"option_value" separated by comma
--rocksdb_column_family_options={"disable_auto_compactions":"false","write_buffer_size":"67108864","max_write_buffer_number":"4","max_bytes_for_level_base":"268435456"}
# rocksdb BlockBasedTableOptions in json, each name and value of option is string, given as "option_name":"option_value" separated by comma
--rocksdb_block_based_table_options={"block_size":"16384","cache_index_and_filter_blocks":"true"}

# Whether or not to enable rocksdb's statistics, disabled by default
--enable_rocksdb_statistics=false

# Statslevel used by rocksdb to collection statistics, optional values are
#   * kExceptHistogramOrTimers, disable timer stats, and skip histogram stats
#   * kExceptTimers, Skip timer stats
#   * kExceptDetailedTimers, Collect all stats except time inside mutex lock AND time spent on compression.
#   * kExceptTimeForMutex, Collect all stats except the counters requiring to get time inside the mutex lock.
#   * kAll, Collect all stats
--rocksdb_stats_level=kExceptHistogramOrTimers

# Whether or not to enable rocksdb's prefix bloom filter, enabled by default.
--enable_rocksdb_prefix_filtering=true
# Whether or not to enable rocksdb's whole key bloom filter, disabled by default.
--enable_rocksdb_whole_key_filtering=false

############### misc ####################
--snapshot_part_rate_limit=8388608
--snapshot_batch_size=1048576
--rebuild_index_part_rate_limit=4194304
--rebuild_index_batch_size=1048576
--cache_index_and_filter_blocks=true
--pin_l0_filter_and_index_blocks_in_cache=true
--clean_wal_interval_secs=60

spw · 2022 年6 月 22 日 08:48

也可以搜搜论坛中 OOM 帖子，看能不能有些帮助；初步怀疑是 rocksdb cache 以及 bloom filter 所占内存太多。

zhengshuai1030 · 2022 年6 月 22 日 08:52

帖子我看了现在就是优化后的,导入测试了几次还行，下午有测试几次内存就还是变高了，这边在调整一下看看吧

spw · 2022 年6 月 23 日 05:39

可以使用内存分析工具定位下占内存的地方。