内存又又又又溢出，我的集群到底何去何从！

问答 问题
, ,
1
  • nebula 版本：3.8.0
  • 部署方式：分布式 5节点 3meta5storage5graph
  • 安装方式：源码编译
  • 是否上生产环境：Y
  • 硬件信息
    • 磁盘 HHD
    • CPU、内存信息 40c128g
  • 问题的具体描述：
  • 凌晨一点和五点告警监控出现内存溢出报错，查看日志如下，问了业务侧没有定时任务再跑，希望各位老师可以帮助找到引发原因并提供解决方案。
  • 相关的 meta / storage / graph info 日志信息
    一点半时storage日志：
    F20250701 01:23:41.403143 404676 AsyncProcessor.h:1017] Check failed: queue.chainLength() <= static_cast<size_t>(std::numeric_limits::max()) (25568053224 vs. 2147483647)
五点半时storage日志：
F20250701 05:46:00.441386 3692204 AsyncProcessor.h:1017] Check failed: queue.chainLength() <= static_cast<size_t>(std::numeric_limits<int>::max()) (25576041482 vs. 2147483647) 
Log file created at: 2025/07/01 05:46:32
Running on machine: Nebulagraph-02
Running duration (h:mm:ss): 0:00:03
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
E20250701 05:46:32.367923 3888118 MetaClient.cpp:772] Send request to "172.29.5.95":9559, exceed retry limit
E20250701 05:46:32.452920 3888118 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20250701 05:46:32.452971 3888072 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
2

graphd /metad的日志 呢

3

这两个没有相关的错误日志，metad根本没有日志打印，graphd的日志都是UDF的：
E20250701 05:32:45.095602 705989 FunctionUdfManager.cpp:41] UDF Folder doesn’t Exist!
E20250701 05:37:45.095722 705989 FunctionUdfManager.cpp:41] UDF Folder doesn’t Exist!
E20250701 05:42:45.095856 705989 FunctionUdfManager.cpp:41] UDF Folder doesn’t Exist!
E20250701 05:47:45.095973 705989 FunctionUdfManager.cpp:41] UDF Folder doesn’t Exist!
E20250701 05:52:45.096094 705989 FunctionUdfManager.cpp:41] UDF Folder doesn’t Exist!
E20250701 05:57:45.096462 705989 FunctionUdfManager.cpp:41] UDF Folder doesn’t Exist!

4

先确认下，你机器的时间和实际时间是一致的么？注意可能有时区问题

5

啊？我服务器时间就是现在的时间，三个组件配置文件都是–timezone_name=UTC+08:00