nebula3.0 内存占用

lopn · 2022 年9 月 23 日 08:38

5434 root      20   0 2763560   1.4g   3620 S   2.3  9.1 123:13.45 nebula-storaged

9.23 1.4G

liwenhui · 2022 年9 月 23 日 10:12

block_cache设的多少，可能跟这个有关

lopn · 2022 年9 月 26 日 03:58

@Lisa

 5434 root      20   0 3713832   2.1g   3900 S  14.4 13.7 179:51.52 nebula-storaged

9.26 2.1g

lopn · 2022 年9 月 26 日 04:01

@Lisa @liwenhui

# The default reserved bytes for one batch operation
--rocksdb_batch_size=4096
# The default block cache size used in BlockBasedTable.
# The unit is MB.
--rocksdb_block_cache=4

是说这个吗？默认值

lopn · 2022 年9 月 26 日 04:02

@Lisa @liwenhui

############## rocksdb Options ##############
--rocksdb_db_options={"max_open_files":"50000"}
--rocksdb_block_based_table_options={"block_size":"32768","cache_index_and_filter_blocks":"true"}


# rocksdb DBOptions in json, each name and value of option is a string, given as "option_name":"option_value" separated by comma
#--rocksdb_db_options={}
# rocksdb ColumnFamilyOptions in json, each name and value of option is string, given as "option_name":"option_value" separated by comma
--rocksdb_column_family_options={"write_buffer_size":"67108864","max_write_buffer_number":"4","max_bytes_for_level_base":"268435456"}
# rocksdb BlockBasedTableOptions in json, each name and value of option is string, given as "option_name":"option_value" separated by comma
#--rocksdb_block_based_table_options={"block_size":"8192"}

liwenhui · 2022 年9 月 27 日 07:00

看看这个有没有帮助记一次 nebula-storaged 内存占用高解决的过程 - #4，来自 liuqian1990

lopn · 2022 年9 月 28 日 03:39

@liwenhui @Lisa
没看我发的配置吗？就是看上面的链接改过的，没有用

lopn · 2022 年9 月 28 日 03:43

@Lisa @liwenhui

5434 root      20   0 4082472   2.5g   4448 S   0.7 16.3 204:38.37 nebula-storaged

9.23 1.4G
9.26 2.1g
9.28 2.5g

xjc · 2022 年9 月 28 日 04:20

数据量多大？每天是否有数据写入？一共几个space，space的分片多少？
麻烦这些信息发一下。

lopn · 2022 年9 月 29 日 09:44

@xjc 每天都有数据写入 100W 的点插入用了 INSERT VERTEX 和 INSERT VERTEX IF NOT EXISTS

一共5个space

lopn · 2022 年9 月 29 日 10:11

(root@nebula) [(none)]> SHOW HOSTS GRAPH;
+-------------+------+----------+---------+--------------+---------+
| Host        | Port | Status   | Role    | Git Info Sha | Version |
+-------------+------+----------+---------+--------------+---------+
| "127.0.0.1" | 9669 | "ONLINE" | "GRAPH" | "02b2091"    | "3.0.0" |
+-------------+------+----------+---------+--------------+---------+
Got 1 rows (time spent 555/1273 us)

Thu, 29 Sep 2022 18:09:30 CST

(root@nebula) [(none)]> SHOW HOSTS GRAPH;
+-------------+------+----------+---------+--------------+---------+
| Host        | Port | Status   | Role    | Git Info Sha | Version |
+-------------+------+----------+---------+--------------+---------+
| "127.0.0.1" | 9669 | "ONLINE" | "GRAPH" | "02b2091"    | "3.0.0" |
+-------------+------+----------+---------+--------------+---------+
Got 1 rows (time spent 509/1062 us)

Thu, 29 Sep 2022 18:09:36 CST

(root@nebula) [(none)]> SHOW HOSTS STORAGE;
+-------------+------+-----------+-----------+--------------+---------+
| Host        | Port | Status    | Role      | Git Info Sha | Version |
+-------------+------+-----------+-----------+--------------+---------+
| "127.0.0.1" | 9779 | "OFFLINE" | "STORAGE" | ""           |         |
+-------------+------+-----------+-----------+--------------+---------+
Got 1 rows (time spent 522/1169 us)

Thu, 29 Sep 2022 18:09:38 CST

(root@nebula) [(none)]> SHOW HOSTS META;
+-------------+------+----------+--------+--------------+---------+
| Host        | Port | Status   | Role   | Git Info Sha | Version |
+-------------+------+----------+--------+--------------+---------+
| "127.0.0.1" | 9559 | "ONLINE" | "META" | "02b2091"    | "3.0.0" |
+-------------+------+----------+--------+--------------+---------+
Got 1 rows (time spent 467/985 us)

Thu, 29 Sep 2022 18:09:47 CST

(root@nebula) [(none)]> show spaces;
+---------------------+
| Name                |
+---------------------+
| "space_2OD5XBFqPCX" |
| "space_8G7d2TDqHwc" |
| "space_ASAASdbMg"   |
| "space_PbUHvAP3khw" |
| "space_rNDIFizDj4U" |
+---------------------+
Got 5 rows (time spent 455/1007 us)

xjc · 2022 年9 月 29 日 10:11

space的分片数呢？这里可能是因为写数据时候的wal log的缓存。

lopn · 2022 年9 月 29 日 10:12

还有一个问题，storage 显示offline ，但是服务正常，这个怎么回事

lopn · 2022 年9 月 29 日 10:20

看了下，分片没有设，用了默认值 100

lopn · 2022 年9 月 29 日 10:24

storage 部分日志

6589 I20220929 11:26:41.838034  5449 SlowOpTracker.h:31] [Port: 9780, Space: 4, Part: 25] total time:933ms, Write WAL, total 2
6590 I20220929 11:26:42.826150  5449 SlowOpTracker.h:31] [Port: 9780, Space: 6, Part: 35] total time:822ms, Write WAL, total 2
6591 I20220929 11:27:35.846838  5471 SlowOpTracker.h:31] [Port: 9780, Space: 5, Part: 43] total time:143ms, Write WAL, total 2
6592 I20220929 11:27:37.837945  5471 SlowOpTracker.h:31] [Port: 9780, Space: 6, Part: 87] total time:612ms, Write WAL, total 2
6593 I20220929 11:27:38.836521  5471 SlowOpTracker.h:31] [Port: 9780, Space: 6, Part: 3] total time:957ms, Write WAL, total 2
6594 I20220929 11:27:39.841322  5471 SlowOpTracker.h:31] [Port: 9780, Space: 4, Part: 52] total time:938ms, Write WAL, total 2
6595 I20220929 11:27:40.840849  5471 SlowOpTracker.h:31] [Port: 9780, Space: 4, Part: 38] total time:339ms, Write WAL, total 2
6596 I20220929 11:27:46.871877  5449 SlowOpTracker.h:31] [Port: 9780, Space: 3, Part: 60] total time:402ms, Write WAL, total 2
6597 I20220929 11:27:47.856581  5446 SlowOpTracker.h:31] [Port: 9780, Space: 6, Part: 74] total time:921ms, Write WAL, total 2
6598 I20220929 11:30:04.936359  5449 SlowOpTracker.h:31] [Port: 9780, Space: 3, Part: 74] total time:708ms, Write WAL, total 2
6599 I20220929 11:31:02.955296  5448 SlowOpTracker.h:31] [Port: 9780, Space: 4, Part: 63] total time:742ms, Write WAL, total 2
6600 I20220929 11:34:05.115072  5456 SlowOpTracker.h:31] [Port: 9780, Space: 4, Part: 79] total time:330ms, Write WAL, total 2
6601 I20220929 11:35:06.154806  5449 SlowOpTracker.h:31] [Port: 9780, Space: 4, Part: 26] total time:436ms, Write WAL, total 2
6602 I20220929 11:36:06.172777  5456 SlowOpTracker.h:31] [Port: 9780, Space: 4, Part: 91] total time:544ms, Write WAL, total 2
6603 I20220929 11:40:09.324720  5449 SlowOpTracker.h:31] [Port: 9780, Space: 3, Part: 71] total time:208ms, Write WAL, total 2
6604 I20220929 11:40:10.321213  5456 SlowOpTracker.h:31] [Port: 9780, Space: 3, Part: 83] total time:353ms, Write WAL, total 2
6605 I20220929 11:40:12.323447  5448 SlowOpTracker.h:31] [Port: 9780, Space: 3, Part: 17] total time:763ms, Write WAL, total 2
6606 I20220929 11:44:11.474040  5446 SlowOpTracker.h:31] [Port: 9780, Space: 5, Part: 42] total time:832ms, Write WAL, total 2
6607 I20220929 12:55:28.459122  5505 EventListener.h:18] Rocksdb start compaction column family: default because of LevelL0FilesNum, status: OK, compacted 5 files into 0, ba     se level is 0, output level is 1
6608 I20220929 12:55:28.460000  5505 CompactionFilter.h:54] Do full/manual compaction!
6609 I20220929 12:55:29.591056  5505 EventListener.h:28] Rocksdb compaction completed column family: default because of LevelL0FilesNum, status: OK, compacted 5 files into 1     , base level is 0, output level is 1
6610 I20220929 18:15:37.203197  5471 AdminTask.cpp:19] createAdminTask (7, 0)
6611 I20220929 18:15:37.238875  5471 AdminTaskManager.cpp:153] enqueue task(7, 0), con req=2147483647
6612 I20220929 18:15:37.240670  7081 AdminTaskManager.cpp:253] dequeue task(7, 0)
6613 I20220929 18:15:37.240902  7081 AdminTaskManager.cpp:298] run task(7, 0), 100 subtasks in 10 thread
6614 I20220929 18:15:37.241179 17419 StatsTask.cpp:102] Start stats task
6615 I20220929 18:15:37.241336 17420 StatsTask.cpp:102] Start stats task
6616 I20220929 18:15:37.241464 17421 StatsTask.cpp:102] Start stats task
6617 I20220929 18:15:37.241601 17422 StatsTask.cpp:102] Start stats task
6618 I20220929 18:15:37.241712 17423 StatsTask.cpp:102] Start stats task
6619 I20220929 18:15:37.241858 17424 StatsTask.cpp:102] Start stats task
6620 I20220929 18:15:37.241998 17425 StatsTask.cpp:102] Start stats task
6621 I20220929 18:15:37.242154 17426 StatsTask.cpp:102] Start stats task
6622 I20220929 18:15:37.242300 17427 StatsTask.cpp:102] Start stats task
6623 I20220929 18:15:37.242435 17428 StatsTask.cpp:102] Start stats task
6624 I20220929 18:15:37.242455  7081 AdminTaskManager.cpp:241] waiting for incoming task
6625 I20220929 18:15:37.409646 17426 StatsTask.cpp:279] Stats task finished
6626 I20220929 18:15:37.409679 17426 AdminTaskManager.cpp:334] subtask of task(7, 0) finished, unfinished task 99
6627 I20220929 18:15:37.409699 17426 StatsTask.cpp:102] Start stats task
6628 I20220929 18:15:37.410228 17427 StatsTask.cpp:279] Stats task finished
6629 I20220929 18:15:37.410252 17427 AdminTaskManager.cpp:334] subtask of task(7, 0) finished, unfinished task 98
6630 I20220929 18:15:37.410269 17427 StatsTask.cpp:102] Start stats task
6631 I20220929 18:15:37.411021 17428 StatsTask.cpp:279] Stats task finished
6632 I20220929 18:15:37.411046 17428 AdminTaskManager.cpp:334] subtask of task(7, 0) finished, unfinished task 97
6633 I20220929 18:15:37.411062 17428 StatsTask.cpp:102] Start stats task
6634 I20220929 18:15:37.415784 17426 StatsTask.cpp:279] Stats task finished
6635 I20220929 18:15:37.415804 17426 AdminTaskManager.cpp:334] subtask of task(7, 0) finished, unfinished task 96
6636 I20220929 18:15:37.415822 17426 StatsTask.cpp:102] Start stats task

xjc · 2022 年9 月 29 日 14:43

默认的配置下如果5个space都有写入，光这部分内存最大可能会达到5x100x16M = 8G。你单个节点配置也不太高的情况下，把分片数量都降到20看下。
目前分片数量不能改，只好重新创建图空间来导入了。

lopn · 2022 年9 月 30 日 04:11

16M 指是的什么呢，每个分片 wal log的缓存的大小?

lopn · 2022 年9 月 30 日 04:12

还有内存持续增长，不释放吗？

xjc · 2022 年10 月 5 日 14:16

是缓存的最大值，每个分片默认不会超过16M

system · 2022 年11 月 4 日 14:16

此话题已在最后回复的 30 天后被自动关闭。不再允许新回复。