- nebula 版本:2.6.1
- 部署方式:分布式
重启集群后数据一致性好像有问题啊,某个space插入数据报错:
INSERT EDGE `TypeAttributeRelation` (`id`) VALUES "Ff1v4mKFieXuvt5IvKPPAZT3cefgL7AH"->"BnIEY0hX_sys_typeName":("sys_typeName"),"Ff1v4mKFieXuvt5IvKPPAZT3cefgL7AH"->"BnIEY0hX_description":("description"),"Ff1v4mKFieXuvt5IvKPPAZT3cefgL7AH"->"BnIEY0hX_discarded":("discarded"),"Ff1v4mKFieXuvt5IvKPPAZT3cefgL7AH"->"BnIEY0hX_discardedTime":("discardedTime"),"Ff1v4mKFieXuvt5IvKPPAZT3cefgL7AH"->"BnIEY0hX_sys_createdTime":("sys_createdTime"),"Ff1v4mKFieXuvt5IvKPPAZT3cefgL7AH"->"BnIEY0hX_number":("number"),"Ff1v4mKFieXuvt5IvKPPAZT3cefgL7AH"->"BnIEY0hX_sys_updatedTime":("sys_updatedTime"),"Ff1v4mKFieXuvt5IvKPPAZT3cefgL7AH"->"BnIEY0hX_sys_updatedBy":("sys_updatedBy"),"Ff1v4mKFieXuvt5IvKPPAZT3cefgL7AH"->"BnIEY0hX_organizationId":("organizationId"),"Ff1v4mKFieXuvt5IvKPPAZT3cefgL7AH"->"BnIEY0hX_sys_createdBy":("sys_createdBy"),"Ff1v4mKFieXuvt5IvKPPAZT3cefgL7AH"->"BnIEY0hX_name":("name");
Storage Error: part: 15, error: E_OUTDATED_TERM(-3071).
space信息:
show hosts信息:
storaged日志信息:
I1228 14:53:38.945047 6439 FileBasedWal.cpp:356] [Port: 9780, Space: 20260, Part: 12] Rollback to log 333382
I1228 14:53:38.945150 6439 RaftPart.cpp:1482] [Port: 9780, Space: 20260, Part: 12] Rollback succeeded! lastLogId is 333382, logLogTerm is 3, committedLogId is 333382, term is 179
I1228 14:53:39.077759 6439 RaftPart.cpp:1505] [Port: 9780, Space: 19024, Part: 3] Stale log! Local lastLogId 698082, lastLogTerm 63, lastLogIdSent 698081, lastLogTermSent 63
I1228 14:53:39.095302 6414 FileBasedWal.cpp:356] [Port: 9780, Space: 19024, Part: 8] Rollback to log 694823
I1228 14:53:39.095402 6414 RaftPart.cpp:1482] [Port: 9780, Space: 19024, Part: 8] Rollback succeeded! lastLogId is 694823, logLogTerm is 60, committedLogId is 694823, term is 62
I1228 14:53:39.523882 6437 FileBasedWal.cpp:356] [Port: 9780, Space: 19024, Part: 14] Rollback to log 694389
I1228 14:53:39.523989 6437 RaftPart.cpp:1482] [Port: 9780, Space: 19024, Part: 14] Rollback succeeded! lastLogId is 694389, logLogTerm is 44, committedLogId is 694389, term is 53
I1228 14:53:39.854019 6416 RaftPart.cpp:1505] [Port: 9780, Space: 19024, Part: 2] Stale log! Local lastLogId 698265, lastLogTerm 50, lastLogIdSent 698264, lastLogTermSent 50
I1228 14:53:40.503317 6411 SlowOpTracker.h:31] [Port: 9780, Space: 20280, Part: 3] total time:694ms, Total send logs: 2
I1228 14:53:41.045962 6424 RaftPart.cpp:1473] [Port: 9780, Space: 20260, Part: 12] The local last log term is 3, which is different from the leader's prevLogTerm 7, the prevLogId is 336640. So need to rollback to last committedLogId_ 333382
I1228 14:53:41.288285 6437 RaftPart.cpp:1473] [Port: 9780, Space: 19024, Part: 8] The local last log term is 60, which is different from the leader's prevLogTerm 61,the prevLogId is 696710. So need to rollback to last committedLogId_ 694823
I1228 14:53:41.837641 6429 RaftPart.cpp:1473] [Port: 9780, Space: 19024, Part: 14] The local last log term is 44, which is different from the leader's prevLogTerm 52, the prevLogId is 697912. So need to rollback to last committedLogId_ 694389
I1228 14:53:42.120430 6424 FileBasedWal.cpp:356] [Port: 9780, Space: 20260, Part: 12] Rollback to log 333382
I1228 14:53:42.120548 6424 RaftPart.cpp:1482] [Port: 9780, Space: 20260, Part: 12] Rollback succeeded! lastLogId is 333382, logLogTerm is 3, committedLogId is 333382, term is 179
I1228 14:53:42.561540 6437 FileBasedWal.cpp:356] [Port: 9780, Space: 19024, Part: 8] Rollback to log 694823
I1228 14:53:42.561650 6437 RaftPart.cpp:1482] [Port: 9780, Space: 19024, Part: 8] Rollback succeeded! lastLogId is 694823, logLogTerm is 60, committedLogId is 694823, term is 62
I1228 14:53:42.945832 6429 FileBasedWal.cpp:356] [Port: 9780, Space: 19024, Part: 14] Rollback to log 694389
I1228 14:53:42.945941 6429 RaftPart.cpp:1482] [Port: 9780, Space: 19024, Part: 14] Rollback succeeded! lastLogId is 694389, logLogTerm is 44, committedLogId is 694389, term is 53
I1228 14:53:43.328434 6415 SlowOpTracker.h:31] [Port: 9780, Space: 20280, Part: 3] total time:186ms, Total send logs: 2
I1228 14:53:44.217707 6424 RaftPart.cpp:1505] [Port: 9780, Space: 20538, Part: 12] Stale log! Local lastLogId 170726, lastLogTerm 10, lastLogIdSent 170725, lastLogTermSent 10
I1228 14:53:44.379016 6424 RaftPart.cpp:1473] [Port: 9780, Space: 20260, Part: 12] The local last log term is 3, which is different from the leader's prevLogTerm 7, the prevLogId is 336641. So need to rollback to last committedLogId_ 333382
I1228 14:53:44.621739 6412 RaftPart.cpp:1473] [Port: 9780, Space: 19024, Part: 8] The local last log term is 60, which is different from the leader's prevLogTerm 61,the prevLogId is 696711. So need to rollback to last committedLogId_ 694823
I1228 14:53:45.110424 6435 RaftPart.cpp:1505] [Port: 9780, Space: 20538, Part: 1] Stale log! Local lastLogId 178455, lastLogTerm 12, lastLogIdSent 178454, lastLogTermSent 12
I1228 14:53:45.113150 6435 RaftPart.cpp:1473] [Port: 9780, Space: 19024, Part: 14] The local last log term is 44, which is different from the leader's prevLogTerm 52, the prevLogId is 697913. So need to rollback to last committedLogId_ 694389
I1228 14:53:45.123811 6419 RaftPart.cpp:1505] [Port: 9780, Space: 20260, Part: 14] Stale log! Local lastLogId 337007, lastLogTerm 17, lastLogIdSent 337006, lastLogTermSent 17
I1228 14:53:45.195915 6419 RaftPart.cpp:1505] [Port: 9780, Space: 20538, Part: 3] Stale log! Local lastLogId 177015, lastLogTerm 23, lastLogIdSent 177014, lastLogTermSent 23
I1228 14:53:45.196208 6440 SlowOpTracker.h:31] [Port: 9780, Space: 19024, Part: 6] total time:638ms, Total send logs: 2
I1228 14:53:45.268522 6440 RaftPart.cpp:1505] [Port: 9780, Space: 19024, Part: 5] Stale log! Local lastLogId 696696, lastLogTerm 100, lastLogIdSent 696695, lastLogTermSent 100
I1228 14:53:45.382441 6424 FileBasedWal.cpp:356] [Port: 9780, Space: 20260, Part: 12] Rollback to log 333382
I1228 14:53:45.382540 6424 RaftPart.cpp:1482] [Port: 9780, Space: 20260, Part: 12] Rollback succeeded! lastLogId is 333382, logLogTerm is 3, committedLogId is 333382, term is 179
I1228 14:53:45.704020 6412 FileBasedWal.cpp:356] [Port: 9780, Space: 19024, Part: 8] Rollback to log 694823
I1228 14:53:45.704124 6412 RaftPart.cpp:1482] [Port: 9780, Space: 19024, Part: 8] Rollback succeeded! lastLogId is 694823, logLogTerm is 60, committedLogId is 694823, term is 62
I1228 14:53:46.188400 6435 FileBasedWal.cpp:356] [Port: 9780, Space: 19024, Part: 14] Rollback to log 694389
I1228 14:53:46.188508 6435 RaftPart.cpp:1482] [Port: 9780, Space: 19024, Part: 14] Rollback succeeded! lastLogId is 694389, logLogTerm is 44, committedLogId is 694389, term is 53
I1228 14:53:47.061414 6428 SlowOpTracker.h:31] [Port: 9780, Space: 20280, Part: 3] total time:587ms, Total send logs: 2
I1228 14:53:47.625368 6423 RaftPart.cpp:1505] [Port: 9780, Space: 19024, Part: 13] Stale log! Local lastLogId 698078, lastLogTerm 94, lastLogIdSent 698077, lastLogTermSent 94
I1228 14:53:47.711839 6412 RaftPart.cpp:1473] [Port: 9780, Space: 20260, Part: 12] The local last log term is 3, which is different from the leader's prevLogTerm 7, the prevLogId is 336642. So need to rollback to last committedLogId_ 333382
I1228 14:53:47.954623 6421 RaftPart.cpp:1473] [Port: 9780, Space: 19024, Part: 8] The local last log term is 60, which is different from the leader's prevLogTerm 61,the prevLogId is 696712. So need to rollback to last committedLogId_ 694823
I1228 14:53:48.447055 6438 RaftPart.cpp:1473] [Port: 9780, Space: 19024, Part: 14] The local last log term is 44, which is different from the leader's prevLogTerm 52, the prevLogId is 697914. So need to rollback to last committedLogId_ 694389
I1228 14:53:48.779824 6412 FileBasedWal.cpp:356] [Port: 9780, Space: 20260, Part: 12] Rollback to log 333382
I1228 14:53:48.779927 6412 RaftPart.cpp:1482] [Port: 9780, Space: 20260, Part: 12] Rollback succeeded! lastLogId is 333382, logLogTerm is 3, committedLogId is 333382, term is 179
I1228 14:53:49.019721 6421 FileBasedWal.cpp:356] [Port: 9780, Space: 19024, Part: 8] Rollback to log 694823
I1228 14:53:49.019825 6421 RaftPart.cpp:1482] [Port: 9780, Space: 19024, Part: 8] Rollback succeeded! lastLogId is 694823, logLogTerm is 60, committedLogId is 694823, term is 62
I1228 14:53:49.461599 6438 FileBasedWal.cpp:356] [Port: 9780, Space: 19024, Part: 14] Rollback to log 694389
I1228 14:53:49.461704 6438 RaftPart.cpp:1482] [Port: 9780, Space: 19024, Part: 14] Rollback succeeded! lastLogId is 694389, logLogTerm is 44, committedLogId is 694389, term is 53
I1228 14:53:49.792132 6423 RaftPart.cpp:1505] [Port: 9780, Space: 20934, Part: 1] Stale log! Local lastLogId 5883, lastLogTerm 156, lastLogIdSent 5882, lastLogTermSent 156
I1228 14:53:49.842617 6417 RaftPart.cpp:1505] [Port: 9780, Space: 20538, Part: 13] Stale log! Local lastLogId 171094, lastLogTerm 13, lastLogIdSent 171093, lastLogTermSent 13
I1228 14:53:49.863600 6417 SlowOpTracker.h:31] [Port: 9780, Space: 20280, Part: 3] total time:56ms, Total send logs: 2
I1228 14:53:51.045637 6428 RaftPart.cpp:1473] [Port: 9780, Space: 20260, Part: 12] The local last log term is 3, which is different from the leader's prevLogTerm 7, the prevLogId is 336643. So need to rollback to last committedLogId_ 333382
I1228 14:53:51.289005 6412 RaftPart.cpp:1473] [Port: 9780, Space: 19024, Part: 8] The local last log term is 60, which is different from the leader's prevLogTerm 61,the prevLogId is 696713. So need to rollback to last committedLogId_ 694823
I1228 14:53:51.570561 6435 RaftPart.cpp:1505] [Port: 9780, Space: 20934, Part: 12] Stale log! Local lastLogId 7203, lastLogTerm 37, lastLogIdSent 7202, lastLogTermSent 37
I1228 14:53:51.780447 6435 RaftPart.cpp:1473] [Port: 9780, Space: 19024, Part: 14] The local last log term is 44, which is different from the leader's prevLogTerm 52, the prevLogId is 697915. So need to rollback to last committedLogId_ 694389
I1228 14:53:52.118641 6428 FileBasedWal.cpp:356] [Port: 9780, Space: 20260, Part: 12] Rollback to log 333382
I1228 14:53:52.118750 6428 RaftPart.cpp:1482] [Port: 9780, Space: 20260, Part: 12] Rollback succeeded! lastLogId is 333382, logLogTerm is 3, committedLogId is 333382, term is 179
I1228 14:53:52.119367 6428 SlowOpTracker.h:31] [Port: 9780, Space: 19024, Part: 6] total time:894ms, Total send logs: 2
I1228 14:53:52.370895 6412 FileBasedWal.cpp:356] [Port: 9780, Space: 19024, Part: 8] Rollback to log 694823
I1228 14:53:52.371003 6412 RaftPart.cpp:1482] [Port: 9780, Space: 19024, Part: 8] Rollback succeeded! lastLogId is 694823, logLogTerm is 60, committedLogId is 694823, term is 62
I1228 14:53:52.852320 6435 FileBasedWal.cpp:356] [Port: 9780, Space: 19024, Part: 14] Rollback to log 694389
I1228 14:53:52.852418 6435 RaftPart.cpp:1482] [Port: 9780, Space: 19024, Part: 14] Rollback succeeded! lastLogId is 694389, logLogTerm is 44, committedLogId is 694389, term is 53
I1228 14:53:53.968729 6435 SlowOpTracker.h:31] [Port: 9780, Space: 19514, Part: 6] total time:791ms, Total send logs: 2
I1228 14:53:53.969296 6435 SlowOpTracker.h:31] [Port: 9780, Space: 20280, Part: 3] total time:829ms, Total send logs: 2
I1228 14:53:54.623088 6417 RaftPart.cpp:1473] [Port: 9780, Space: 19024, Part: 8] The local last log term is 60, which is different from the leader's prevLogTerm 61,the prevLogId is 696714. So need to rollback to last committedLogId_ 694823
I1228 14:53:54.821745 6418 RaftPart.cpp:1473] [Port: 9780, Space: 20260, Part: 12] The local last log term is 3, which is different from the leader's prevLogTerm 7, the prevLogId is 336644. So need to rollback to last committedLogId_ 333382
I1228 14:53:54.889267 6435 RaftPart.cpp:1505] [Port: 9780, Space: 20538, Part: 14] Stale log! Local lastLogId 178723, lastLogTerm 12, lastLogIdSent 178722, lastLogTermSent 12
I1228 14:53:55.736474 6417 FileBasedWal.cpp:356] [Port: 9780, Space: 19024, Part: 8] Rollback to log 694823
I1228 14:53:55.736596 6417 RaftPart.cpp:1482] [Port: 9780, Space: 19024, Part: 8] Rollback succeeded! lastLogId is 694823, logLogTerm is 60, committedLogId is 694823, term is 62
I1228 14:53:55.744369 6417 RaftPart.cpp:1505] [Port: 9780, Space: 19024, Part: 3] Stale log! Local lastLogId 698087, lastLogTerm 63, lastLogIdSent 698086, lastLogTermSent 63
I1228 14:53:55.744586 6414 RaftPart.cpp:1473] [Port: 9780, Space: 19024, Part: 14] The local last log term is 44, which is different from the leader's prevLogTerm 52, the prevLogId is 697916. So need to rollback to last committedLogId_ 694389
I1228 14:53:55.841154 6485 MetaClient.cpp:3006] Load leader of "10.65.101.89":9779 in 9 space
I1228 14:53:55.841286 6485 MetaClient.cpp:3006] Load leader of "10.65.101.97":9779 in 11 space
I1228 14:53:55.841361 6485 MetaClient.cpp:3006] Load leader of "10.65.101.98":9779 in 10 space
I1228 14:53:55.841423 6485 MetaClient.cpp:3006] Load leader of "10.65.101.118":9779 in 5 space
I1228 14:53:55.841430 6485 MetaClient.cpp:3012] Load leader ok
I1228 14:53:56.027770 6418 FileBasedWal.cpp:356] [Port: 9780, Space: 20260, Part: 12] Rollback to log 333382
I1228 14:53:56.027870 6418 RaftPart.cpp:1482] [Port: 9780, Space: 20260, Part: 12] Rollback succeeded! lastLogId is 333382, logLogTerm is 3, committedLogId is 333382, term is 179
I1228 14:53:56.604343 6412 RaftPart.cpp:1505] [Port: 9780, Space: 20934, Part: 7] Stale log! Local lastLogId 6022, lastLogTerm 61, lastLogIdSent 6021, lastLogTermSent 61
I1228 14:53:56.754393 6412 RaftPart.cpp:1505] [Port: 9780, Space: 20280, Part: 5] Stale log! Local lastLogId 331788, lastLogTerm 22, lastLogIdSent 331787, lastLogTermSent 22
I1228 14:53:56.755139 6412 SlowOpTracker.h:31] [Port: 9780, Space: 20280, Part: 3] total time:281ms, Total send logs: 2
I1228 14:53:57.026911 6414 FileBasedWal.cpp:356] [Port: 9780, Space: 19024, Part: 14] Rollback to log 694389
I1228 14:53:57.027014 6414 RaftPart.cpp:1482] [Port: 9780, Space: 19024, Part: 14] Rollback succeeded! lastLogId is 694389, logLogTerm is 44, committedLogId is 694389, term is 53
I1228 14:53:57.711652 6414 RaftPart.cpp:1473] [Port: 9780, Space: 20260, Part: 12] The local last log term is 3, which is different from the leader's prevLogTerm 7, the prevLogId is 336645. So need to rollback to last committedLogId_ 333382
I1228 14:53:57.955528 6422 RaftPart.cpp:1473] [Port: 9780, Space: 19024, Part: 8] The local last log term is 60, which is different from the leader's prevLogTerm 61,the prevLogId is 696715. So need to rollback to last committedLogId_ 694823
I1228 14:53:58.236528 6417 RaftPart.cpp:1505] [Port: 9780, Space: 20934, Part: 15] Stale log! Local lastLogId 7109, lastLogTerm 18, lastLogIdSent 7108, lastLogTermSent 18
I1228 14:53:58.520157 6412 RaftPart.cpp:1505] [Port: 9780, Space: 20260, Part: 2] Stale log! Local lastLogId 336997, lastLogTerm 22, lastLogIdSent 336996, lastLogTermSent 22
I1228 14:53:58.693619 6417 RaftPart.cpp:1473] [Port: 9780, Space: 19024, Part: 14] The local last log term is 44, which is different from the leader's prevLogTerm 52, the prevLogId is 697917. So need to rollback to last committedLogId_ 694389
I1228 14:53:58.747620 6414 FileBasedWal.cpp:356] [Port: 9780, Space: 20260, Part: 12] Rollback to log 333382
I1228 14:53:58.747731 6414 RaftPart.cpp:1482] [Port: 9780, Space: 20260, Part: 12] Rollback succeeded! lastLogId is 333382, logLogTerm is 3, committedLogId is 333382, term is 179
I1228 14:53:59.044570 6422 FileBasedWal.cpp:356] [Port: 9780, Space: 19024, Part: 8] Rollback to log 694823
I1228 14:53:59.044692 6422 RaftPart.cpp:1482] [Port: 9780, Space: 19024, Part: 8] Rollback succeeded! lastLogId is 694823, logLogTerm is 60, committedLogId is 694823, term is 62
I1228 14:53:59.838361 6417 FileBasedWal.cpp:356] [Port: 9780, Space: 19024, Part: 14] Rollback to log 694389
I1228 14:53:59.838464 6417 RaftPart.cpp:1482] [Port: 9780, Space: 19024, Part: 14] Rollback succeeded! lastLogId is 694389, logLogTerm is 44, committedLogId is 694389, term is 53
I1228 14:54:00.578605 6417 SlowOpTracker.h:31] [Port: 9780, Space: 20280, Part: 3] total time:772ms, Total send logs: 2
I1228 14:54:01.045114 6417 RaftPart.cpp:1473] [Port: 9780, Space: 20260, Part: 12] The local last log term is 3, which is different from the leader's prevLogTerm 7, the prevLogId is 336646. So need to rollback to last committedLogId_ 333382
I1228 14:54:01.262164 6434 RaftPart.cpp:1505] [Port: 9780, Space: 19024, Part: 12] Stale log! Local lastLogId 697745, lastLogTerm 62, lastLogIdSent 697744, lastLogTermSent 62
I1228 14:54:01.288755 6425 RaftPart.cpp:1473] [Port: 9780, Space: 19024, Part: 8] The local last log term is 60, which is different from the leader's prevLogTerm 61,the prevLogId is 696716. So need to rollback to last committedLogId_ 694823
I1228 14:54:01.779491 6410 RaftPart.cpp:1473] [Port: 9780, Space: 19024, Part: 14] The local last log term is 44, which is different from the leader's prevLogTerm 52, the prevLogId is 697918. So need to rollback to last committedLogId_ 694389
I1228 14:54:02.186241 6417 FileBasedWal.cpp:356] [Port: 9780, Space: 20260, Part: 12] Rollback to log 333382
I1228 14:54:02.186343 6417 RaftPart.cpp:1482] [Port: 9780, Space: 20260, Part: 12] Rollback succeeded! lastLogId is 333382, logLogTerm is 3, committedLogId is 333382, term is 179
I1228 14:54:02.523494 6425 FileBasedWal.cpp:356] [Port: 9780, Space: 19024, Part: 8] Rollback to log 694823
I1228 14:54:02.523620 6425 RaftPart.cpp:1482] [Port: 9780, Space: 19024, Part: 8] Rollback succeeded! lastLogId is 694823, logLogTerm is 60, committedLogId is 694823, term is 62