在Arm环境下通过编译源码方式启动nebula，nebula-metad服务内存占用高问题

shiwxyz · 2022 年4 月 26 日 06:08

nebula 版本：v2.6.2
部署方式：单机
安装方式：源码编译
硬件信息
- 磁盘（推荐使用 SSD）：HDD
- CPU、内存信息： CPU 128核内存512G
- 机器信息：Linux ds1 4.14.0-115.el7a.0.1.aarch64 #1 SMP Sun Nov 25 20:54:21 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux
问题的具体描述：

通过编译源码包，启动nebula-metad服务后，内存占用率奇高，参考论坛遇到同样问题的帖子（https://discuss.nebula-graph.com.cn/t/topic/3836/39），尝试了重新编译，添加ENABLE_ASAN参数，重启，依然无法降低内存占用

    cmake -DCMAKE_INSTALL_PREFIX=/home/bigdata/nebula -DENABLE_TESTING=OFF --ENABLE_ASAN=ON -DCMAKE_BUILD_TYPE=Release ..

nebula-meta-info

    Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
    I0426 13:36:44.626706 20328 MetaDaemon.cpp:262] localhost = "ds1":9559
    I0426 13:36:47.132624 20328 NebulaStore.cpp:52] Start the raft service...
    I0426 13:36:47.448185 20328 NebulaSnapshotManager.cpp:25] Send snapshot is rate limited to 8388608 for each part by default
    I0426 13:36:47.481490 20328 RaftexService.cpp:62] Init thrift server for raft service, port: 9560
    I0426 13:36:47.627097 20412 RaftexService.cpp:93] Starting the Raftex Service
    I0426 13:36:48.937203 20412 RaftexService.cpp:83] Starting the Raftex Service on 9560
    I0426 13:36:48.937261 20412 RaftexService.cpp:103] Start the Raftex Service successfully
    I0426 13:36:48.937597 20328 NebulaStore.cpp:84] Scan the local path, and init the spaces_
    I0426 13:36:48.937798 20328 NebulaStore.cpp:90] Scan path "/data/nebula/metad/data/nebula/0"
    I0426 13:36:49.206513 20328 RocksEngine.cpp:128] open rocksdb on /data/nebula/metad/data/nebula/0/data
    I0426 13:36:49.206548 20328 NebulaStore.cpp:114] Load space 0 from disk
    I0426 13:36:49.206614 20328 NebulaStore.cpp:142] Need to open 1 parts of space 0
    I0426 13:36:49.212071 20407 FileBasedWal.cpp:66] [Port: 9560, Space: 0, Part: 0] lastLogId in wal is 1596, lastLogTerm is 3, path is /data/nebula/metad/data/nebula/0/wal/0/0000000000000000001.wal
    I0426 13:36:49.212250 20407 RaftPart.cpp:278] [Port: 9560, Space: 0, Part: 0] There are 0 peer hosts, and total 1 copies. The quorum is 1, as learner 0, lastLogId 1596, lastLogTerm 3, committedLogId 1596, term 3
    I0426 13:36:49.212491 20407 NebulaStore.cpp:146] Load part 0, 0 from disk
    I0426 13:36:49.212527 20328 NebulaStore.cpp:161] Load space 0 complete
    I0426 13:36:49.212553 20328 NebulaStore.cpp:170] Init data from partManager for "ds1":9559
    I0426 13:36:49.212569 20328 NebulaStore.cpp:262] Data space 0 has existed!
    I0426 13:36:49.212579 20328 NebulaStore.cpp:302] [Space: 0, Part: 0] has existed!
    I0426 13:36:49.212611 20328 NebulaStore.cpp:77] Register handler...
    I0426 13:36:49.212620 20328 MetaDaemon.cpp:98] Waiting for the leader elected...
    I0426 13:36:49.212631 20328 MetaDaemon.cpp:110] Leader has not been elected, sleep 1s
    I0426 13:36:49.855301 20408 RaftPart.cpp:957] [Port: 9560, Space: 0, Part: 0] Start leader election, reason: lastMsgDur 648, term 3
    I0426 13:36:49.855407 20408 RaftPart.cpp:1095] [Port: 9560, Space: 0, Part: 0] Sending out an election request (space = 0, part = 0, term = 4, lastLogId = 1596, lastLogTerm = 3, candidateIP = ds1, candidatePort = 9560)
    I0426 13:36:49.855435 20408 RaftPart.cpp:1059] [Port: 9560, Space: 0, Part: 0] Partition is elected as the new leader for term 4
    I0426 13:36:49.855451 20408 RaftPart.cpp:1137] [Port: 9560, Space: 0, Part: 0] The partition is elected as the leader
    I0426 13:36:50.212819 20328 MetaDaemon.cpp:137] Get meta version is 2
    I0426 13:36:50.213088 20403 MetaVersionMan.cpp:66] Write meta version 2 succeeds
    I0426 13:36:50.213135 20328 MetaDaemon.cpp:164] Nebula store init succeeded, clusterId 6816554851720541959
    I0426 13:36:50.213162 20328 MetaDaemon.cpp:275] Start http service
    I0426 13:36:50.271596 20328 MetaDaemon.cpp:172] Starting Meta HTTP Service
    I0426 13:36:50.786607 20460 WebService.cpp:124] Web service started on HTTP[19559], HTTP2[19560]
    I0426 13:36:50.786952 20328 JobManager.cpp:56] JobManager initialized
    I0426 13:36:50.787016 20328 MetaDaemon.cpp:305] Check and init root user
    I0426 13:36:50.787137 20328 RootUserMan.h:27] Root user exists
    I0426 13:36:50.787387 20328 MetaDaemon.cpp:331] The meta deamon start on "ds1":9559
    I0426 13:36:50.860303 20467 JobManager.cpp:77] JobManager::runJobBackground() enter

同文件

nebula-meta服务启动内存占用高问题.md (4.7 KB)

steam · 2022 年4 月 26 日 06:09

来，图片连接引用的是你的本地路径，cv 贴一下图吧。

shiwxyz · 2022 年4 月 26 日 06:18

liwenhui · 2022 年4 月 26 日 06:50

[quote=“shiwxyz, post:1, topic:8545”]

Web service started on HTTP[19559], HTTP2[19560]
    I0426 13:36:50.786952 20328 JobManager.cpp:56] JobManager initialized
    I0426 13:36:50.787016 20328 MetaDaemon.cpp:305] Check and init root user
    I0426 13:36:50.787137 20328 RootUserMan.h:27] Root user exists
    I0426 13:36:50.787387 20328 MetaDaemon.cpp:331] The meta deamon start on "ds1":9559
    I0426 13:36:50.860303 20467 JobManager.cpp:77] JobManager::runJobBackground() enter

[/quote]日志就这些吗，meta配置里加个–v=3，再启动看看有什么信息

shiwxyz · 2022 年4 月 26 日 06:58

这个配置加上去报错了，是不是–v=3?

shiwxyz · 2022 年4 月 26 日 06:59

我还有一张通过pref监控的图片

liwenhui · 2022 年4 月 26 日 07:03

是 --v

shiwxyz · 2022 年4 月 26 日 07:05

ok，我重新设置了，我把日志贴出来，这只是一部分的

I0426 15:03:38.115151 49795 RaftPart.cpp:802] [Port: 9560, Space: 0, Part: 0] Received enough response
I0426 15:03:38.115168 49795 RaftPart.cpp:842] [Port: 9560, Space: 0, Part: 0] 0 hosts have accepted the logs
I0426 15:03:38.115182 49795 AtomicLogBuffer.h:323] current node firstLogId = 4938, the seeking logId = 4968
I0426 15:03:38.115200 49795 Part.cpp:224] [Port: 9560, Space: 0, Part: 0] Skip the heartbeat!
I0426 15:03:38.115272 49795 AtomicLogBuffer.h:343] Release ref, readers = 1
I0426 15:03:38.115288 49795 RaftPart.cpp:885] [Port: 9560, Space: 0, Part: 0] Leader succeeded in committing the logs 4968 to 4968
I0426 15:03:38.115304 49795 RaftPart.cpp:906] [Port: 9560, Space: 0, Part: 0] logs size 0
I0426 15:03:38.115317 49795 RaftPart.cpp:933] [Port: 9560, Space: 0, Part: 0] No more log to be replicated
I0426 15:03:38.746673 49802 DiskManager.cpp:123] Refresh filesystem info of "/data/nebula/metad/data"
I0426 15:03:39.782763 49799 RaftPart.cpp:1196] [Port: 9560, Space: 0, Part: 0] Need to send heartbeat
I0426 15:03:39.782881 49799 RaftPart.cpp:1768] [Port: 9560, Space: 0, Part: 0] Send heartbeat
I0426 15:03:39.782913 49799 RaftPart.cpp:1821] [Port: 9560, Space: 0, Part: 0] Heartbeat is accepted by quorum
I0426 15:03:39.782936 49799 RaftPart.cpp:1206] [Port: 9560, Space: 0, Part: 0] Schedule new task
I0426 15:03:39.782913 49795 RaftPart.cpp:608] [Port: 9560, Space: 0, Part: 0] Checking whether buffer overflow
I0426 15:03:39.783006 49795 RaftPart.cpp:620] [Port: 9560, Space: 0, Part: 0] Appending logs to the buffer
I0426 15:03:39.783031 49795 RaftPart.cpp:640] [Port: 9560, Space: 0, Part: 0] Preparing to send AppendLog request
I0426 15:03:39.783047 49795 RaftPart.cpp:672] [Port: 9560, Space: 0, Part: 0] Calling appendLogsInternal()
I0426 15:03:39.783062 49795 RaftPart.cpp:697] [Port: 9560, Space: 0, Part: 0] Ready to append logs from id 4969 (Current term is 0)
I0426 15:03:39.783082 49795 Part.cpp:397] [Port: 9560, Space: 0, Part: 0] logId 4969, termId 6, clusterId 0
I0426 15:03:39.783125 49795 RaftPart.cpp:726] [Port: 9560, Space: 0, Part: 0] Succeeded writing logs [4969, 4969] to WAL
I0426 15:03:39.783145 49795 RaftPart.cpp:765] [Port: 9560, Space: 0, Part: 0] About to replicate logs to all peer hosts
I0426 15:03:39.783193 49795 RaftPart.cpp:802] [Port: 9560, Space: 0, Part: 0] Received enough response
I0426 15:03:39.783210 49795 RaftPart.cpp:842] [Port: 9560, Space: 0, Part: 0] 0 hosts have accepted the logs
I0426 15:03:39.783226 49795 AtomicLogBuffer.h:323] current node firstLogId = 4938, the seeking logId = 4969
I0426 15:03:39.783241 49795 Part.cpp:224] [Port: 9560, Space: 0, Part: 0] Skip the heartbeat!
I0426 15:03:39.783327 49795 AtomicLogBuffer.h:343] Release ref, readers = 1
I0426 15:03:39.783344 49795 RaftPart.cpp:885] [Port: 9560, Space: 0, Part: 0] Leader succeeded in committing the logs 4969 to 4969
I0426 15:03:39.783360 49795 RaftPart.cpp:906] [Port: 9560, Space: 0, Part: 0] logs size 0
I0426 15:03:39.783372 49795 RaftPart.cpp:933] [Port: 9560, Space: 0, Part: 0] No more log to be replicated
I0426 15:03:41.456777 49800 RaftPart.cpp:1196] [Port: 9560, Space: 0, Part: 0] Need to send heartbeat
I0426 15:03:41.456887 49800 RaftPart.cpp:1768] [Port: 9560, Space: 0, Part: 0] Send heartbeat
I0426 15:03:41.456921 49800 RaftPart.cpp:1821] [Port: 9560, Space: 0, Part: 0] Heartbeat is accepted by quorum
I0426 15:03:41.456913 49795 RaftPart.cpp:608] [Port: 9560, Space: 0, Part: 0] Checking whether buffer overflow
I0426 15:03:41.456954 49800 RaftPart.cpp:1206] [Port: 9560, Space: 0, Part: 0] Schedule new task
I0426 15:03:41.457000 49795 RaftPart.cpp:620] [Port: 9560, Space: 0, Part: 0] Appending logs to the buffer
I0426 15:03:41.457052 49795 RaftPart.cpp:640] [Port: 9560, Space: 0, Part: 0] Preparing to send AppendLog request
I0426 15:03:41.457069 49795 RaftPart.cpp:672] [Port: 9560, Space: 0, Part: 0] Calling appendLogsInternal()
I0426 15:03:41.457084 49795 RaftPart.cpp:697] [Port: 9560, Space: 0, Part: 0] Ready to append logs from id 4970 (Current term is 0)
I0426 15:03:41.457104 49795 Part.cpp:397] [Port: 9560, Space: 0, Part: 0] logId 4970, termId 6, clusterId 0
I0426 15:03:41.457145 49795 RaftPart.cpp:726] [Port: 9560, Space: 0, Part: 0] Succeeded writing logs [4970, 4970] to WAL
I0426 15:03:41.457165 49795 RaftPart.cpp:765] [Port: 9560, Space: 0, Part: 0] About to replicate logs to all peer hosts
I0426 15:03:41.457209 49795 RaftPart.cpp:802] [Port: 9560, Space: 0, Part: 0] Received enough response
I0426 15:03:41.457226 49795 RaftPart.cpp:842] [Port: 9560, Space: 0, Part: 0] 0 hosts have accepted the logs
I0426 15:03:41.457242 49795 AtomicLogBuffer.h:323] current node firstLogId = 4938, the seeking logId = 4970
I0426 15:03:41.457257 49795 Part.cpp:224] [Port: 9560, Space: 0, Part: 0] Skip the heartbeat!
I0426 15:03:41.457322 49795 AtomicLogBuffer.h:343] Release ref, readers = 1
I0426 15:03:41.457340 49795 RaftPart.cpp:885] [Port: 9560, Space: 0, Part: 0] Leader succeeded in committing the logs 4970 to 4970
I0426 15:03:41.457355 49795 RaftPart.cpp:906] [Port: 9560, Space: 0, Part: 0] logs size 0
I0426 15:03:41.457370 49795 RaftPart.cpp:933] [Port: 9560, Space: 0, Part: 0] No more log to be replicated
I0426 15:03:43.124781 49801 RaftPart.cpp:1196] [Port: 9560, Space: 0, Part: 0] Need to send heartbeat
I0426 15:03:43.124892 49801 RaftPart.cpp:1768] [Port: 9560, Space: 0, Part: 0] Send heartbeat
I0426 15:03:43.124927 49801 RaftPart.cpp:1821] [Port: 9560, Space: 0, Part: 0] Heartbeat is accepted by quorum
I0426 15:03:43.124948 49801 RaftPart.cpp:1206] [Port: 9560, Space: 0, Part: 0] Schedule new task
I0426 15:03:43.124927 49795 RaftPart.cpp:608] [Port: 9560, Space: 0, Part: 0] Checking whether buffer overflow
I0426 15:03:43.125022 49795 RaftPart.cpp:620] [Port: 9560, Space: 0, Part: 0] Appending logs to the buffer
I0426 15:03:43.125047 49795 RaftPart.cpp:640] [Port: 9560, Space: 0, Part: 0] Preparing to send AppendLog request
I0426 15:03:43.125063 49795 RaftPart.cpp:672] [Port: 9560, Space: 0, Part: 0] Calling appendLogsInternal()
I0426 15:03:43.125079 49795 RaftPart.cpp:697] [Port: 9560, Space: 0, Part: 0] Ready to append logs from id 4971 (Current term is 0)
I0426 15:03:43.125098 49795 Part.cpp:397] [Port: 9560, Space: 0, Part: 0] logId 4971, termId 6, clusterId 0
I0426 15:03:43.125140 49795 RaftPart.cpp:726] [Port: 9560, Space: 0, Part: 0] Succeeded writing logs [4971, 4971] to WAL
I0426 15:03:43.125160 49795 RaftPart.cpp:765] [Port: 9560, Space: 0, Part: 0] About to replicate logs to all peer hosts
I0426 15:03:43.125209 49795 RaftPart.cpp:802] [Port: 9560, Space: 0, Part: 0] Received enough response
I0426 15:03:43.125226 49795 RaftPart.cpp:842] [Port: 9560, Space: 0, Part: 0] 0 hosts have accepted the logs
I0426 15:03:43.125254 49795 AtomicLogBuffer.h:323] current node firstLogId = 4938, the seeking logId = 4971
I0426 15:03:43.125272 49795 Part.cpp:224] [Port: 9560, Space: 0, Part: 0] Skip the heartbeat!
I0426 15:03:43.125340 49795 AtomicLogBuffer.h:343] Release ref, readers = 1
I0426 15:03:43.125358 49795 RaftPart.cpp:885] [Port: 9560, Space: 0, Part: 0] Leader succeeded in committing the logs 4971 to 4971
I0426 15:03:43.125375 49795 RaftPart.cpp:906] [Port: 9560, Space: 0, Part: 0] logs size 0
I0426 15:03:43.125388 49795 RaftPart.cpp:933] [Port: 9560, Space: 0, Part: 0] No more log to be replicated
I0426 15:03:44.792739 49798 RaftPart.cpp:1196] [Port: 9560, Space: 0, Part: 0] Need to send heartbeat
I0426 15:03:44.792865 49798 RaftPart.cpp:1768] [Port: 9560, Space: 0, Part: 0] Send heartbeat
I0426 15:03:44.792891 49795 RaftPart.cpp:608] [Port: 9560, Space: 0, Part: 0] Checking whether buffer overflow
I0426 15:03:44.793141 49798 RaftPart.cpp:1821] [Port: 9560, Space: 0, Part: 0] Heartbeat is accepted by quorum
I0426 15:03:44.793159 49795 RaftPart.cpp:620] [Port: 9560, Space: 0, Part: 0] Appending logs to the buffer
I0426 15:03:44.793206 49798 RaftPart.cpp:1206] [Port: 9560, Space: 0, Part: 0] Schedule new task
I0426 15:03:44.793234 49795 RaftPart.cpp:640] [Port: 9560, Space: 0, Part: 0] Preparing to send AppendLog request
I0426 15:03:44.793277 49795 RaftPart.cpp:672] [Port: 9560, Space: 0, Part: 0] Calling appendLogsInternal()
I0426 15:03:44.793292 49795 RaftPart.cpp:697] [Port: 9560, Space: 0, Part: 0] Ready to append logs from id 4972 (Current term is 0)
I0426 15:03:44.793311 49795 Part.cpp:397] [Port: 9560, Space: 0, Part: 0] logId 4972, termId 6, clusterId 0
I0426 15:03:44.793350 49795 RaftPart.cpp:726] [Port: 9560, Space: 0, Part: 0] Succeeded writing logs [4972, 4972] to WAL
I0426 15:03:44.793370 49795 RaftPart.cpp:765] [Port: 9560, Space: 0, Part: 0] About to replicate logs to all peer hosts
I0426 15:03:44.793411 49795 RaftPart.cpp:802] [Port: 9560, Space: 0, Part: 0] Received enough response
I0426 15:03:44.793427 49795 RaftPart.cpp:842] [Port: 9560, Space: 0, Part: 0] 0 hosts have accepted the logs
I0426 15:03:44.793442 49795 AtomicLogBuffer.h:323] current node firstLogId = 4938, the seeking logId = 4972
I0426 15:03:44.793457 49795 Part.cpp:224] [Port: 9560, Space: 0, Part: 0] Skip the heartbeat!
I0426 15:03:44.793517 49795 AtomicLogBuffer.h:343] Release ref, readers = 1
I0426 15:03:44.793534 49795 RaftPart.cpp:885] [Port: 9560, Space: 0, Part: 0] Leader succeeded in committing the logs 4972 to 4972
I0426 15:03:44.793548 49795 RaftPart.cpp:906] [Port: 9560, Space: 0, Part: 0] logs size 0
I0426 15:03:44.793561 49795 RaftPart.cpp:933] [Port: 9560, Space: 0, Part: 0] No more log to be replicated
I0426 15:03:46.467089 49799 RaftPart.cpp:1196] [Port: 9560, Space: 0, Part: 0] Need to send heartbeat
I0426 15:03:46.467226 49799 RaftPart.cpp:1768] [Port: 9560, Space: 0, Part: 0] Send heartbeat
I0426 15:03:46.467258 49799 RaftPart.cpp:1821] [Port: 9560, Space: 0, Part: 0] Heartbeat is accepted by quorum
I0426 15:03:46.467278 49799 RaftPart.cpp:1206] [Port: 9560, Space: 0, Part: 0] Schedule new task
I0426 15:03:46.467262 49795 RaftPart.cpp:608] [Port: 9560, Space: 0, Part: 0] Checking whether buffer overflow
I0426 15:03:46.467360 49795 RaftPart.cpp:620] [Port: 9560, Space: 0, Part: 0] Appending logs to the buffer
I0426 15:03:46.467388 49795 RaftPart.cpp:640] [Port: 9560, Space: 0, Part: 0] Preparing to send AppendLog request
I0426 15:03:46.467404 49795 RaftPart.cpp:672] [Port: 9560, Space: 0, Part: 0] Calling appendLogsInternal()
I0426 15:03:46.467420 49795 RaftPart.cpp:697] [Port: 9560, Space: 0, Part: 0] Ready to append logs from id 4973 (Current term is 0)
I0426 15:03:46.467447 49795 Part.cpp:397] [Port: 9560, Space: 0, Part: 0] logId 4973, termId 6, clusterId 0
I0426 15:03:46.467494 49795 RaftPart.cpp:726] [Port: 9560, Space: 0, Part: 0] Succeeded writing logs [4973, 4973] to WAL
I0426 15:03:46.467517 49795 RaftPart.cpp:765] [Port: 9560, Space: 0, Part: 0] About to replicate logs to all peer hosts
I0426 15:03:46.467567 49795 RaftPart.cpp:802] [Port: 9560, Space: 0, Part: 0] Received enough response
I0426 15:03:46.467600 49795 RaftPart.cpp:842] [Port: 9560, Space: 0, Part: 0] 0 hosts have accepted the logs
I0426 15:03:46.467617 49795 AtomicLogBuffer.h:323] current node firstLogId = 4938, the seeking logId = 4973
I0426 15:03:46.467633 49795 Part.cpp:224] [Port: 9560, Space: 0, Part: 0] Skip the heartbeat!
I0426 15:03:46.467705 49795 AtomicLogBuffer.h:343] Release ref, readers = 1
I0426 15:03:46.467725 49795 RaftPart.cpp:885] [Port: 9560, Space: 0, Part: 0] Leader succeeded in committing the logs 4973 to 4973
I0426 15:03:46.467742 49795 RaftPart.cpp:906] [Port: 9560, Space: 0, Part: 0] logs size 0
I0426 15:03:46.467761 49795 RaftPart.cpp:933] [Port: 9560, Space: 0, Part: 0] No more log to be replicated
I0426 15:03:48.135061 49800 RaftPart.cpp:1196] [Port: 9560, Space: 0, Part: 0] Need to send heartbeat
I0426 15:03:48.135201 49800 RaftPart.cpp:1768] [Port: 9560, Space: 0, Part: 0] Send heartbeat
I0426 15:03:48.135236 49795 RaftPart.cpp:608] [Port: 9560, Space: 0, Part: 0] Checking whether buffer overflow
I0426 15:03:48.135241 49800 RaftPart.cpp:1821] [Port: 9560, Space: 0, Part: 0] Heartbeat is accepted by quorum
I0426 15:03:48.135320 49795 RaftPart.cpp:620] [Port: 9560, Space: 0, Part: 0] Appending logs to the buffer
I0426 15:03:48.135351 49800 RaftPart.cpp:1206] [Port: 9560, Space: 0, Part: 0] Schedule new task
I0426 15:03:48.135390 49795 RaftPart.cpp:640] [Port: 9560, Space: 0, Part: 0] Preparing to send AppendLog request
I0426 15:03:48.135438 49795 RaftPart.cpp:672] [Port: 9560, Space: 0, Part: 0] Calling appendLogsInternal()
I0426 15:03:48.135458 49795 RaftPart.cpp:697] [Port: 9560, Space: 0, Part: 0] Ready to append logs from id 4974 (Current term is 0)
I0426 15:03:48.135483 49795 Part.cpp:397] [Port: 9560, Space: 0, Part: 0] logId 4974, termId 6, clusterId 0
I0426 15:03:48.135530 49795 RaftPart.cpp:726] [Port: 9560, Space: 0, Part: 0] Succeeded writing logs [4974, 4974] to WAL
I0426 15:03:48.135555 49795 RaftPart.cpp:765] [Port: 9560, Space: 0, Part: 0] About to replicate logs to all peer hosts
I0426 15:03:48.135602 49795 RaftPart.cpp:802] [Port: 9560, Space: 0, Part: 0] Received enough response
I0426 15:03:48.135623 49795 RaftPart.cpp:842] [Port: 9560, Space: 0, Part: 0] 0 hosts have accepted the logs
I0426 15:03:48.135639 49795 AtomicLogBuffer.h:323] current node firstLogId = 4938, the seeking logId = 4974
I0426 15:03:48.135658 49795 Part.cpp:224] [Port: 9560, Space: 0, Part: 0] Skip the heartbeat!
I0426 15:03:48.135730 49795 AtomicLogBuffer.h:343] Release ref, readers = 1
I0426 15:03:48.135752 49795 RaftPart.cpp:885] [Port: 9560, Space: 0, Part: 0] Leader succeeded in committing the logs 4974 to 4974
I0426 15:03:48.135773 49795 RaftPart.cpp:906] [Port: 9560, Space: 0, Part: 0] logs size 0
I0426 15:03:48.135788 49795 RaftPart.cpp:933] [Port: 9560, Space: 0, Part: 0] No more log to be replicated
I0426 15:03:48.751941 49802 DiskManager.cpp:123] Refresh filesystem info of "/data/nebula/metad/data"
I0426 15:03:49.803160 49801 RaftPart.cpp:1196] [Port: 9560, Space: 0, Part: 0] Need to send heartbeat
I0426 15:03:49.803280 49801 RaftPart.cpp:1768] [Port: 9560, Space: 0, Part: 0] Send heartbeat
I0426 15:03:49.803308 49795 RaftPart.cpp:608] [Port: 9560, Space: 0, Part: 0] Checking whether buffer overflow
I0426 15:03:49.803531 49801 RaftPart.cpp:1821] [Port: 9560, Space: 0, Part: 0] Heartbeat is accepted by quorum
I0426 15:03:49.803617 49795 RaftPart.cpp:620] [Port: 9560, Space: 0, Part: 0] Appending logs to the buffer
I0426 15:03:49.803656 49801 RaftPart.cpp:1206] [Port: 9560, Space: 0, Part: 0] Schedule new task
I0426 15:03:49.803689 49795 RaftPart.cpp:640] [Port: 9560, Space: 0, Part: 0] Preparing to send AppendLog request
I0426 15:03:49.803741 49795 RaftPart.cpp:672] [Port: 9560, Space: 0, Part: 0] Calling appendLogsInternal()
I0426 15:03:49.803762 49795 RaftPart.cpp:697] [Port: 9560, Space: 0, Part: 0] Ready to append logs from id 4975 (Current term is 0)
I0426 15:03:49.803786 49795 Part.cpp:397] [Port: 9560, Space: 0, Part: 0] logId 4975, termId 6, clusterId 0
I0426 15:03:49.803839 49795 RaftPart.cpp:726] [Port: 9560, Space: 0, Part: 0] Succeeded writing logs [4975, 4975] to WAL
I0426 15:03:49.803862 49795 RaftPart.cpp:765] [Port: 9560, Space: 0, Part: 0] About to replicate logs to all peer hosts
I0426 15:03:49.803915 49795 RaftPart.cpp:802] [Port: 9560, Space: 0, Part: 0] Received enough response
I0426 15:03:49.803930 49795 RaftPart.cpp:842] [Port: 9560, Space: 0, Part: 0] 0 hosts have accepted the logs
I0426 15:03:49.803946 49795 AtomicLogBuffer.h:323] current node firstLogId = 4938, the seeking logId = 4975
I0426 15:03:49.803962 49795 Part.cpp:224] [Port: 9560, Space: 0, Part: 0] Skip the heartbeat!
I0426 15:03:49.804028 49795 AtomicLogBuffer.h:343] Release ref, readers = 1
I0426 15:03:49.804049 49795 RaftPart.cpp:885] [Port: 9560, Space: 0, Part: 0] Leader succeeded in committing the logs 4975 to 4975
I0426 15:03:49.804065 49795 RaftPart.cpp:906] [Port: 9560, Space: 0, Part: 0] logs size 0
I0426 15:03:49.804081 49795 RaftPart.cpp:933] [Port: 9560, Space: 0, Part: 0] No more log to be replicated
I0426 15:03:51.471468 49798 RaftPart.cpp:1196] [Port: 9560, Space: 0, Part: 0] Need to send heartbeat
I0426 15:03:51.471590 49798 RaftPart.cpp:1768] [Port: 9560, Space: 0, Part: 0] Send heartbeat
I0426 15:03:51.471619 49795 RaftPart.cpp:608] [Port: 9560, Space: 0, Part: 0] Checking whether buffer overflow
I0426 15:03:51.471624 49798 RaftPart.cpp:1821] [Port: 9560, Space: 0, Part: 0] Heartbeat is accepted by quorum
I0426 15:03:51.471698 49795 RaftPart.cpp:620] [Port: 9560, Space: 0, Part: 0] Appending logs to the buffer
I0426 15:03:51.471730 49798 RaftPart.cpp:1206] [Port: 9560, Space: 0, Part: 0] Schedule new task
I0426 15:03:51.471765 49795 RaftPart.cpp:640] [Port: 9560, Space: 0, Part: 0] Preparing to send AppendLog request
I0426 15:03:51.471810 49795 RaftPart.cpp:672] [Port: 9560, Space: 0, Part: 0] Calling appendLogsInternal()
I0426 15:03:51.471832 49795 RaftPart.cpp:697] [Port: 9560, Space: 0, Part: 0] Ready to append logs from id 4976 (Current term is 0)
I0426 15:03:51.471853 49795 Part.cpp:397] [Port: 9560, Space: 0, Part: 0] logId 4976, termId 6, clusterId 0
I0426 15:03:51.471897 49795 RaftPart.cpp:726] [Port: 9560, Space: 0, Part: 0] Succeeded writing logs [4976, 4976] to WAL
I0426 15:03:51.471920 49795 RaftPart.cpp:765] [Port: 9560, Space: 0, Part: 0] About to replicate logs to all peer hosts
I0426 15:03:51.471971 49795 RaftPart.cpp:802] [Port: 9560, Space: 0, Part: 0] Received enough response
I0426 15:03:51.471992 49795 RaftPart.cpp:842] [Port: 9560, Space: 0, Part: 0] 0 hosts have accepted the logs
I0426 15:03:51.472007 49795 AtomicLogBuffer.h:323] current node firstLogId = 4938, the seeking logId = 4976
I0426 15:03:51.472024 49795 Part.cpp:224] [Port: 9560, Space: 0, Part: 0] Skip the heartbeat!
I0426 15:03:51.472090 49795 AtomicLogBuffer.h:343] Release ref, readers = 1
I0426 15:03:51.472110 49795 RaftPart.cpp:885] [Port: 9560, Space: 0, Part: 0] Leader succeeded in committing the logs 4976 to 4976
I0426 15:03:51.472126 49795 RaftPart.cpp:906] [Port: 9560, Space: 0, Part: 0] logs size 0
I0426 15:03:51.472143 49795 RaftPart.cpp:933] [Port: 9560, Space: 0, Part: 0] No more log to be replicated
I0426 15:03:53.144793 49799 RaftPart.cpp:1196] [Port: 9560, Space: 0, Part: 0] Need to send heartbeat
I0426 15:03:53.144874 49799 RaftPart.cpp:1768] [Port: 9560, Space: 0, Part: 0] Send heartbeat
I0426 15:03:53.144896 49799 RaftPart.cpp:1821] [Port: 9560, Space: 0, Part: 0] Heartbeat is accepted by quorum
I0426 15:03:53.144909 49799 RaftPart.cpp:1206] [Port: 9560, Space: 0, Part: 0] Schedule new task
I0426 15:03:53.144906 49795 RaftPart.cpp:608] [Port: 9560, Space: 0, Part: 0] Checking whether buffer overflow
I0426 15:03:53.145022 49795 RaftPart.cpp:620] [Port: 9560, Space: 0, Part: 0] Appending logs to the buffer
I0426 15:03:53.145043 49795 RaftPart.cpp:640] [Port: 9560, Space: 0, Part: 0] Preparing to send AppendLog request
I0426 15:03:53.145056 49795 RaftPart.cpp:672] [Port: 9560, Space: 0, Part: 0] Calling appendLogsInternal()
I0426 15:03:53.145069 49795 RaftPart.cpp:697] [Port: 9560, Space: 0, Part: 0] Ready to append logs from id 4977 (Current term is 0)
I0426 15:03:53.145095 49795 Part.cpp:397] [Port: 9560, Space: 0, Part: 0] logId 4977, termId 6, clusterId 0
I0426 15:03:53.145131 49795 RaftPart.cpp:726] [Port: 9560, Space: 0, Part: 0] Succeeded writing logs [4977, 4977] to WAL
I0426 15:03:53.145148 49795 RaftPart.cpp:765] [Port: 9560, Space: 0, Part: 0] About to replicate logs to all peer hosts
I0426 15:03:53.145184 49795 RaftPart.cpp:802] [Port: 9560, Space: 0, Part: 0] Received enough response
I0426 15:03:53.145197 49795 RaftPart.cpp:842] [Port: 9560, Space: 0, Part: 0] 0 hosts have accepted the logs
I0426 15:03:53.145208 49795 AtomicLogBuffer.h:323] current node firstLogId = 4938, the seeking logId = 4977
I0426 15:03:53.145222 49795 Part.cpp:224] [Port: 9560, Space: 0, Part: 0] Skip the heartbeat!
I0426 15:03:53.145282 49795 AtomicLogBuffer.h:343] Release ref, readers = 1
I0426 15:03:53.145294 49795 RaftPart.cpp:885] [Port: 9560, Space: 0, Part: 0] Leader succeeded in committing the logs 4977 to 4977
I0426 15:03:53.145308 49795 RaftPart.cpp:906] [Port: 9560, Space: 0, Part: 0] logs size 0
I0426 15:03:53.145316 49795 RaftPart.cpp:933] [Port: 9560, Space: 0, Part: 0] No more log to be replicated
I0426 15:03:54.812745 49800 RaftPart.cpp:1196] [Port: 9560, Space: 0, Part: 0] Need to send heartbeat
I0426 15:03:54.812912 49800 RaftPart.cpp:1768] [Port: 9560, Space: 0, Part: 0] Send heartbeat
I0426 15:03:54.812947 49795 RaftPart.cpp:608] [Port: 9560, Space: 0, Part: 0] Checking whether buffer overflow
I0426 15:03:54.812952 49800 RaftPart.cpp:1821] [Port: 9560, Space: 0, Part: 0] Heartbeat is accepted by quorum
I0426 15:03:54.813036 49795 RaftPart.cpp:620] [Port: 9560, Space: 0, Part: 0] Appending logs to the buffer
I0426 15:03:54.813068 49800 RaftPart.cpp:1206] [Port: 9560, Space: 0, Part: 0] Schedule new task
I0426 15:03:54.813105 49795 RaftPart.cpp:640] [Port: 9560, Space: 0, Part: 0] Preparing to send AppendLog request
I0426 15:03:54.813155 49795 RaftPart.cpp:672] [Port: 9560, Space: 0, Part: 0] Calling appendLogsInternal()
I0426 15:03:54.813180 49795 RaftPart.cpp:697] [Port: 9560, Space: 0, Part: 0] Ready to append logs from id 4978 (Current term is 0)
I0426 15:03:54.813207 49795 Part.cpp:397] [Port: 9560, Space: 0, Part: 0] logId 4978, termId 6, clusterId 0
I0426 15:03:54.813262 49795 RaftPart.cpp:726] [Port: 9560, Space: 0, Part: 0] Succeeded writing logs [4978, 4978] to WAL
I0426 15:03:54.813288 49795 RaftPart.cpp:765] [Port: 9560, Space: 0, Part: 0] About to replicate logs to all peer hosts
I0426 15:03:54.813344 49795 RaftPart.cpp:802] [Port: 9560, Space: 0, Part: 0] Received enough response
I0426 15:03:54.813364 49795 RaftPart.cpp:842] [Port: 9560, Space: 0, Part: 0] 0 hosts have accepted the logs
I0426 15:03:54.813381 49795 AtomicLogBuffer.h:323] current node firstLogId = 4938, the seeking logId = 4978
I0426 15:03:54.813400 49795 Part.cpp:224] [Port: 9560, Space: 0, Part: 0] Skip the heartbeat!
I0426 15:03:54.813484 49795 AtomicLogBuffer.h:343] Release ref, readers = 1
I0426 15:03:54.813508 49795 RaftPart.cpp:885] [Port: 9560, Space: 0, Part: 0] Leader succeeded in committing the logs 4978 to 4978
I0426 15:03:54.813529 49795 RaftPart.cpp:906] [Port: 9560, Space: 0, Part: 0] logs size 0
I0426 15:03:54.813547 49795 RaftPart.cpp:933] [Port: 9560, Space: 0, Part: 0] No more log to be replicated
I0426 15:03:56.486971 49801 RaftPart.cpp:1196] [Port: 9560, Space: 0, Part: 0] Need to send heartbeat
I0426 15:03:56.487090 49801 RaftPart.cpp:1768] [Port: 9560, Space: 0, Part: 0] Send heartbeat
I0426 15:03:56.487128 49801 RaftPart.cpp:1821] [Port: 9560, Space: 0, Part: 0] Heartbeat is accepted by quorum
I0426 15:03:56.487119 49795 RaftPart.cpp:608] [Port: 9560, Space: 0, Part: 0] Checking whether buffer overflow
I0426 15:03:56.487233 49801 RaftPart.cpp:1206] [Port: 9560, Space: 0, Part: 0] Schedule new task
I0426 15:03:56.487286 49795 RaftPart.cpp:620] [Port: 9560, Space: 0, Part: 0] Appending logs to the buffer
I0426 15:03:56.487346 49795 RaftPart.cpp:640] [Port: 9560, Space: 0, Part: 0] Preparing to send AppendLog request
I0426 15:03:56.487380 49795 RaftPart.cpp:672] [Port: 9560, Space: 0, Part: 0] Calling appendLogsInternal()
I0426 15:03:56.487396 49795 RaftPart.cpp:697] [Port: 9560, Space: 0, Part: 0] Ready to append logs from id 4979 (Current term is 0)
I0426 15:03:56.487419 49795 Part.cpp:397] [Port: 9560, Space: 0, Part: 0] logId 4979, termId 6, clusterId 0
I0426 15:03:56.487463 49795 RaftPart.cpp:726] [Port: 9560, Space: 0, Part: 0] Succeeded writing logs [4979, 4979] to WAL
I0426 15:03:56.487485 49795 RaftPart.cpp:765] [Port: 9560, Space: 0, Part: 0] About to replicate logs to all peer hosts
I0426 15:03:56.487533 49795 RaftPart.cpp:802] [Port: 9560, Space: 0, Part: 0] Received enough response
I0426 15:03:56.487552 49795 RaftPart.cpp:842] [Port: 9560, Space: 0, Part: 0] 0 hosts have accepted the logs
I0426 15:03:56.487568 49795 AtomicLogBuffer.h:323] current node firstLogId = 4938, the seeking logId = 4979
I0426 15:03:56.487586 49795 Part.cpp:224] [Port: 9560, Space: 0, Part: 0] Skip the heartbeat!
I0426 15:03:56.487655 49795 AtomicLogBuffer.h:343] Release ref, readers = 1
I0426 15:03:56.487676 49795 RaftPart.cpp:885] [Port: 9560, Space: 0, Part: 0] Leader succeeded in committing the logs 4979 to 4979
I0426 15:03:56.487694 49795 RaftPart.cpp:906] [Port: 9560, Space: 0, Part: 0] logs size 0
I0426 15:03:56.487711 49795 RaftPart.cpp:933] [Port: 9560, Space: 0, Part: 0] No more log to be replicated

HarrisChu · 2022 年4 月 26 日 08:01

看一下大页关了没

cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/defrag
cat /proc/meminfo | grep Huge

shiwxyz · 2022 年4 月 26 日 08:05

shiwxyz · 2022 年4 月 26 日 08:25

运行一段时间后，内存占用降了下来，但是好奇为什么启动时需要占用内存那么大？如果机器的内存低于占用内存，是不是服务会起不来？

wenhaocs · 2022 年5 月 5 日 03:16

听上去像rocksdb的启动。你们meta文件夹有多大？另外，如果机器内存占用过大，不会起不来，只是会很慢，而且如果graphd跟metad在同一台机器，可能导致graphd停止接受查询。

shiwxyz · 2022 年5 月 5 日 06:13

你的意思是说启动meta服务时，即启动rocksdb会占用比较大的内存吗？我不太理解你说的meta的文件夹有多大？是挂载目录的磁盘大小吗？这个有10T。

wenhaocs · 2022 年5 月 5 日 18:51

在启动meta时候，会加载schema等信息，这个过程中rocksdb会将数据读到内存里，需要很多prefix + get操作。这也是为什么你perf结果里显示rocksdb bytewise comparator overhead比较高。除此之外，还会将一些essential blocks比如fitler/index blocks加载到内存中，这些都会占用内存。你可以看一下你meta目录有多大。也就是你nebula-metad.conf里配置的用来存放meta data的目录。可以用du -h 查看。

另外你perf显示结果里开销最大的是上下文切换。能否再看一下你们磁盘容量够不够？可以尝试换台机器，看看问题还在不在？