2.6.1 flink消费kafka数据导入nebula出现locked问题

问答 问题
1
  • nebula 版本：2.6.1
  • 部署方式：分布式
  • 安装方式： RPM
  • 是否为线上版本：Y / N
  • 硬件信息
    • 磁盘（ 推荐使用 SSD）
    • CPU、内存信息
  • 问题的具体描述
    使用flink消费kafka中的日志时，storage日志中出现vertex locked，edge locked 问题
  • 部分日志如下
E0222 18:35:15.013926 30466 AddEdgesProcessor.cpp:172] edge locked : src 6bc1b2f359e4d80f715e5945e7f91734+768++, type -80, rank 0, dst 6bc1b2f359e4d80f715e5945e7f91734+768+768+1644593337
qE0222 18:35:25.226378 30454 AddVerticesProcessor.cpp:164] The vertex locked : tag 70, vid cfbf6280625d34662f23fc780d3298db+968+10268+1645495407
E0222 18:35:25.226411 30472 AddVerticesProcessor.cpp:164] The vertex locked : tag 70, vid b31b370281533d3a8284a9d4f3a4b4db+888+27964+1645522192
E0222 18:35:25.226871 30454 AddVerticesProcessor.cpp:164] The vertex locked : tag 70, vid dbefa577dcffe37d041c69faf2e9004b+848+12364+1645423336
E0222 18:35:25.254930 30473 AddEdgesProcessor.cpp:172] edge locked : src 4b760ec0ab679e00247fea2fbf4e51aa+772+5616+1644399911, type 79, rank 0, dst 4b760ec0ab679e00247fea2fbf4e51aa+772+6548+1645526114
E0222 18:35:25.255162 30460 AddEdgesProcessor.cpp:172] edge locked : src b31b370281533d3a8284a9d4f3a4b4db+888+29624+1645522188, type 79, rank 0, dst b31b370281533d3a8284a9d4f3a4b4db+888++
E0222 18:35:25.255209 30460 AddEdgesProcessor.cpp:172] edge locked : src 2222a5fd6410c38bf30a7b5e00d3c877+13204+42976+1645526112, type 79, rank 0, dst 2222a5fd6410c38bf30a7b5e00d3c877+13204+58960+1645526112
E0222 18:35:25.255856 30460 AddEdgesProcessor.cpp:172] edge locked : src d69198d02dec538e8cc991bcb9a6bddb+988+29476+1645526114, type 79, rank 0, dst d69198d02dec538e8cc991bcb9a6bddb+988+29784+1645526114
E0222 18:35:25.255916 30460 AddEdgesProcessor.cpp:172] edge locked : src dbefa577dcffe37d041c69faf2e9004b+848+12364+1645423336, type 79, rank 0, dst dbefa577dcffe37d041c69faf2e9004b+848++
E0222 18:35:25.256270 30473 AddEdgesProcessor.cpp:172] edge locked : src 7e51f579630cc17d4ff35a3ea24b4fe7++16292+1645514237, type 79, rank 0, dst 7e51f579630cc17d4ff35a3ea24b4fe7+++
E0222 18:35:25.283780 30473 AddEdgesProcessor.cpp:172] edge locked : src faf98ff19e35b66b612d4d3086cc5ff7+12896+12896+1645516691, type 80, rank 0, dst faf98ff19e35b66b612d4d3086cc5ff7+12896++
E0222 18:35:25.283815 30473 AddEdgesProcessor.cpp:172] edge locked : src 28c38f0b9fc30246d25fd26cef30d968+892+892+1645523965, type 80, rank 0, dst 28c38f0b9fc30246d25fd26cef30d968+892++
E0222 18:35:25.284233 30473 AddEdgesProcessor.cpp:172] edge locked : src 4bf016123fdc667aba4face678e85031+824+824+1645495821, type 80, rank 0, dst 4bf016123fdc667aba4face678e85031+824++
E0222 18:35:25.284649 30472 AddEdgesProcessor.cpp:172] edge locked : src d69198d02dec538e8cc991bcb9a6bddb+988+29784+1645526114, type -80, rank 0, dst d69198d02dec538e8cc991bcb9a6bddb+988+988+1644889456
E0222 18:35:25.285028 30472 AddEdgesProcessor.cpp:172] edge locked : src b03ab82f009f2da182c650a70f12c428+804+804+1644977405, type 80, rank 0, dst b03ab82f009f2da182c650a70f12c428+804+19904+1645013128
E0222 18:35:25.285050 30472 AddEdgesProcessor.cpp:172] edge locked : src ed184e3425d986890d348a715609fe8a+3736+3736+1644808425, type 80, rank 0, dst ed184e3425d986890d348a715609fe8a+3736++
E0222 18:35:25.285663 30473 AddEdgesProcessor.cpp:172] edge locked : src feade63997527f30745fbfa4e95cb4ff+820+41788+1645526114, type -80, rank 0, dst feade63997527f30745fbfa4e95cb4ff+820+820+1644681243
  • 问题:请问这是什么原因导致的？有什么影响？应该怎么处理？谢谢
2

:thinking: 你是不是同时在写同一个点和同一条边啊

3

1.日志是打点日志，存在重复日志，点和边可能会重复
2.并发度为2，不知道和并发度有没有关系

4

flink消费kafka日志数据后是按批写入nebula，会有同时写入点和边的情况。
1.这种点和边被锁的情况会导致数据丢失吗？还会有其他影响吗？
2.这种情况怎么解决？想到的方式：（1）按批写入调整为按条写入，flinkconnector支持吗？（2）每个批次的数据去重后再导入

5

此话题已在最后回复的 30 天后被自动关闭。不再允许新回复。