part x raft buffer is full

  • nebula 版本:3.4.1
  • 部署方式: 分布式
  • 安装方式:RPM
  • 是否上生产环境:N
  • 硬件信息
    • 磁盘 :机械硬盘
    • CPU 8c
    • 内存信息 64g
  • 问题的具体描述
    现象:
    使用ngbatis插入数据时,出现raft buffer is full异常。
    客户端有10个节点,nebula节点有11个,创建的space分片数为10个。
    根据论坛相关buffer is full问题帖子参考,修改过–rocksdb_db_options={“max_subcompactions”:“4”,“max_background_jobs”:“4”} 和max_write_buff_number=8,均无法解决。具体报错日志及cpu使用截图见下.
Caused by: org.nebula.contrib.ngbatis.exception.QueryException: execute failed: ResultSet error: Storage Error: Part 4 raft buffer is full. Please retry later.
	at org.nebula.contrib.ngbatis.proxy,MapperProxy.executeBySessionPool(MapperProxy.Java:297) 
	at org.nebula.contrib.ngbatis.proxy. MapperProxy.invoke(MapperProxy. java:134) 
	at org.nebula.contrib.ngbatis.proxy.MapperProxy.invoke (MapperProxy.java:170)
	at org.nebula.contrib.ngbatis.proxy.MapperProxy.pageSupport (MapperProxy. java:186) 
	at org.nebula.contrib.ngbatis.proxy.MapperProxy.invoke(MapperProxy.java:90)
	at com.pat.abc.graph.dao.CommonGraphDao$Proxy.insertEdgeIfNotExistBatch (Unknown Source)
	at com.pat.abc.graph.service.CalloutGraphServiceImp1.lambda$nul1$6(CallOutGraphServiceImpl. java:287)
	at com.pat.abc.component.asyncmanager.threadhelper.CommonAsyncThreadPoolHelper.submit(CommonAsyncThreadPoolHelper.java:70)
Caused by: org.nebula.contrib.ngbatis.exception,QueryException; ResultSet error: Storage Error: Part 4 raft buffer is full. Please retry later.
	org.nebula. contrib.ngbatis.proxy.MapperProxy. executeBySessionPool (MapperProxy.Java:293)
	... 85 common frames omitted

dashboard的cpu概览如下

尝试修改下; 调整 raft_heartbeat_interval_secs 参数

1 个赞

并发调小一些

调整到10台机器每台机器2个线程,每个线程批量插入5条,还是有问题

这又是什么参数,看了很多文章和回复,没有涉及这个参数的…