GET SUBGRAPH 2 steps from VID 性能较低,如何优化

  • nebula 版本:V2.0.1
  • 部署方式: 分布式 三节点,三副本,30 partition, 三个meta,三个graphd, 三个storaged
  • 是否为线上版本: N (测试中)
  • 硬件信息
    • 磁盘:SSD
    • CPU、内存信息 : 16核,128G

问题: 当前Space 中只有三种关系,Rel_A/ Rel_B/ Rel_C 都是需要的,所以语句直接用了GET SUBGRAPH 2 steps from VID
压力测试时发现

  1. 小数据集的查询结果 该语句性能比Neo4j 慢了1/3
  2. 较大数据集的查询Neo4j timeout, Nebula 可以正常返回
    请问目前对于,性能慢了1/3 有优化方案吗?

压测结果如下:

1、 可以把 neo4j的 语句贴一下
2、 可以 profile 一下 看看哪里慢了

3,现在storage 取数据时 用的是单线程, 已经有个PR 在取数据时候修改为多线程, 会加快一些速度

2 个赞
  1. 因为当前小数据集结果没有有5000, limit 应该可以忽略
     MATCH (c:Entity) WHERE c.keyNo = "kkkkkkk"
     CALL apoc.path.subgraphAll(c, {maxLevel: 1, relationshipFilter:'Rel_A|Rel_B|Rel_C', limit: 5000})
     YIELD nodes, relationships
     WITH nodes, relationships
     UNWIND relationships as r
     WITH nodes, r
     RETURN nodes, COLLECT(DISTINCT r) AS rel
  1. 随意找一个profile如下: 主要耗时在GetNeighbors 上
Execution Plan

-----+--------------+--------------+-------------------------------------------------------------------+----------------------------------
| id | name         | dependencies | profiling data                                                    | operator info                   |
-----+--------------+--------------+-------------------------------------------------------------------+----------------------------------
|  5 | DataCollect  | 4            | ver: 0, rows: 3, execTime: 4876us, totalTime: 4879us              | outputVar: [                    |
|    |              |              |                                                                   |   {                             |
|    |              |              |                                                                   |     "colNames": [               |
|    |              |              |                                                                   |       "_vertices",              |
|    |              |              |                                                                   |       "_edges"                  |
|    |              |              |                                                                   |     ],                          |
|    |              |              |                                                                   |     "name": "__DataCollect_5",  |
|    |              |              |                                                                   |     "type": "DATASET"           |
|    |              |              |                                                                   |   }                             |
|    |              |              |                                                                   | ]                               |
|    |              |              |                                                                   | inputVar: [                     |
|    |              |              |                                                                   |   {                             |
|    |              |              |                                                                   |     "colNames": [],             |
|    |              |              |                                                                   |     "name": "__GetNeighbors_2", |
|    |              |              |                                                                   |     "type": "DATASET"           |
|    |              |              |                                                                   |   },                            |
|    |              |              |                                                                   |   {                             |
|    |              |              |                                                                   |     "colNames": [],             |
|    |              |              |                                                                   |     "name": "__VAR_2",          |
|    |              |              |                                                                   |     "type": "DATASET"           |
|    |              |              |                                                                   |   }                             |
|    |              |              |                                                                   | ]                               |
|    |              |              |                                                                   | kind: SUBGRAPH                  |
-----+--------------+--------------+-------------------------------------------------------------------+----------------------------------
|  4 | Loop         | 0            | ver: 0, rows: 1, execTime: 6us, totalTime: 6us                    | outputVar: [                    |
|    |              |              | ver: 1, rows: 1, execTime: 3us, totalTime: 4us                    |   {                             |
|    |              |              | ver: 2, rows: 1, execTime: 5us, totalTime: 5us                    |     "colNames": [],             |
|    |              |              | ver: 3, rows: 1, execTime: 5us, totalTime: 6us                    |     "name": "__Loop_4",         |
|    |              |              |                                                                   |     "type": "DATASET"           |
|    |              |              |                                                                   |   }                             |
|    |              |              |                                                                   | ]                               |
|    |              |              |                                                                   | inputVar:                       |
|    |              |              |                                                                   | condition: (++($__VAR_0)<=3)    |
|    |              |              |                                                                   | loopBody: 3                     |
-----+--------------+--------------+-------------------------------------------------------------------+----------------------------------
|  3 | Subgraph     | 2            | ver: 0, rows: 21, execTime: 40us, totalTime: 41us                 | branch: true, nodeId: 4         |
|    |              |              | ver: 1, rows: 332, execTime: 247us, totalTime: 248us              |                                 |
|    |              |              | ver: 2, rows: 0, execTime: 1526us, totalTime: 1527us              | outputVar: [                    |
|    |              |              |                                                                   |   {                             |
|    |              |              |                                                                   |     "colNames": [               |
|    |              |              |                                                                   |       "_vid"                    |
|    |              |              |                                                                   |     ],                          |
|    |              |              |                                                                   |     "name": "__VAR_1",          |
|    |              |              |                                                                   |     "type": "DATASET"           |
|    |              |              |                                                                   |   }                             |
|    |              |              |                                                                   | ]                               |
|    |              |              |                                                                   | inputVar: __GetNeighbors_2      |
-----+--------------+--------------+-------------------------------------------------------------------+----------------------------------
|  2 | GetNeighbors | 1            | {                                                                 | outputVar: [                    |
|    |              |              | ver: 0, rows: 0, execTime: 56us, totalTime: 3272us                |   {                             |
|    |              |              | "10.0.7.250":9779 exec/total/vertices: 2806(us)/3136(us)/1,       |     "colNames": [],             |
|    |              |              | total_rpc_time: 3198(us)                                          |     "name": "__GetNeighbors_2", |
|    |              |              | }                                                                 |     "type": "DATASET"           |
|    |              |              | {                                                                 |   }                             |
|    |              |              | ver: 1, rows: 0, execTime: 117us, totalTime: 17810us              | ]                               |
|    |              |              | "10.0.7.251":9779 exec/total/vertices: 17004(us)/17602(us)/8,     | inputVar: __VAR_1               |
|    |              |              | "10.0.7.252":9779 exec/total/vertices: 14084(us)/14866(us)/7,     | space: 68                       |
|    |              |              | total_rpc_time: 17675(us)                                         | dedup: false                    |
|    |              |              | "10.0.7.250":9779 exec/total/vertices: 10634(us)/11234(us)/6,     | limit: -1                       |
|    |              |              | }                                                                 | filter:                         |
|    |              |              | {                                                                 | orderBy: []                     |
|    |              |              | ver: 2, rows: 0, execTime: 157us, totalTime: 229993us             | src: $__VAR_1._vid              |
|    |              |              | "10.0.7.251":9779 exec/total/vertices: 207173(us)/212023(us)/99,  | edgeTypes: []                   |
|    |              |              | "10.0.7.250":9779 exec/total/vertices: 225212(us)/229644(us)/123, | edgeDirection: OUT_EDGE         |
|    |              |              | "10.0.7.252":9779 exec/total/vertices: 207480(us)/212668(us)/110, | vertexProps: [                  |
|    |              |              | total_rpc_time: 229806(us)                                        |   {                             |
|    |              |              | }                                                                 |     "props": [                  |
|    |              |              |                                                                   |       "keyno",                  |
|    |              |              |                                                                   |       "name",                   |
|    |              |              |                                                                   |       "shortstatus",            |
|    |              |              |                                                                   |       "econkind",               |
|    |              |              |                                                                   |       "registcapi",             |
|    |              |              |                                                                   |       "hasimage",               |
|    |              |              |                                                                   |       "groupid",                |
|    |              |              |                                                                   |       "ismain",                 |
|    |              |              |                                                                   |       "isinvestor",             |
|    |              |              |                                                                   |       "create_time"             |
|    |              |              |                                                                   |     ],                          |
|    |              |              |                                                                   |     "tagId": 69                 |
|    |              |              |                                                                   |   },                            |
|    |              |              |                                                                   |   {                             |
|    |              |              |                                                                   |     "props": [                  |
|    |              |              |                                                                   |       "keyno",                  |
|    |              |              |                                                                   |       "name",                   |
|    |              |              |                                                                   |       "job",                    |
|    |              |              |                                                                   |       "hasimage",               |
|    |              |              |                                                                   |       "create_time"             |
|    |              |              |                                                                   |     ],                          |
|    |              |              |                                                                   |     "tagId": 70                 |
|    |              |              |                                                                   |   }                             |
|    |              |              |                                                                   | ]                               |
|    |              |              |                                                                   | edgeProps: [                    |
|    |              |              |                                                                   |   {                             |
|    |              |              |                                                                   |     "props": [                  |
|    |              |              |                                                                   |       "_src",                   |
|    |              |              |                                                                   |       "_type",                  |
|    |              |              |                                                                   |       "_rank",                  |
|    |              |              |                                                                   |       "_dst",                   |
|    |              |              |                                                                   |       "type",                   |
|    |              |              |                                                                   |       "startid",                |
|    |              |              |                                                                   |       "endid",                  |
|    |              |              |                                                                   |       "role",                   |
|    |              |              |                                                                   |       "stockpercent",           |
|    |              |              |                                                                   |       "shouldcapi",             |
|    |              |              |                                                                   |       "iscal",                  |
|    |              |              |                                                                   |       "create_time"             |
|    |              |              |                                                                   |     ],                          |
|    |              |              |                                                                   |     "type": "71"                |
|    |              |              |                                                                   |   },                            |
|    |              |              |                                                                   |   {                             |
|    |              |              |                                                                   |     "props": [                  |
|    |              |              |                                                                   |       "_src",                   |
|    |              |              |                                                                   |       "_type",                  |
|    |              |              |                                                                   |       "_rank",                  |
|    |              |              |                                                                   |       "_dst",                   |
|    |              |              |                                                                   |       "type",                   |
|    |              |              |                                                                   |       "startid",                |
|    |              |              |                                                                   |       "endid",                  |
|    |              |              |                                                                   |       "role",                   |
|    |              |              |                                                                   |       "stockpercent",           |
|    |              |              |                                                                   |       "shouldcapi",             |
|    |              |              |                                                                   |       "iscal",                  |
|    |              |              |                                                                   |       "create_time"             |
|    |              |              |                                                                   |     ],                          |
|    |              |              |                                                                   |     "type": "-71"               |
|    |              |              |                                                                   |   },                            |
|    |              |              |                                                                   |   {                             |
|    |              |              |                                                                   |     "props": [                  |
|    |              |              |                                                                   |       "_src",                   |
|    |              |              |                                                                   |       "_type",                  |
|    |              |              |                                                                   |       "_rank",                  |
|    |              |              |                                                                   |       "_dst",                   |
|    |              |              |                                                                   |       "type",                   |
|    |              |              |                                                                   |       "startid",                |
|    |              |              |                                                                   |       "endid",                  |
|    |              |              |                                                                   |       "role",                   |
|    |              |              |                                                                   |       "stockpercent",           |
|    |              |              |                                                                   |       "shouldcapi",             |
|    |              |              |                                                                   |       "create_time"             |
|    |              |              |                                                                   |     ],                          |
|    |              |              |                                                                   |     "type": "72"                |
|    |              |              |                                                                   |   },                            |
|    |              |              |                                                                   |   {                             |
|    |              |              |                                                                   |     "props": [                  |
|    |              |              |                                                                   |       "_src",                   |
|    |              |              |                                                                   |       "_type",                  |
|    |              |              |                                                                   |       "_rank",                  |
|    |              |              |                                                                   |       "_dst",                   |
|    |              |              |                                                                   |       "type",                   |
|    |              |              |                                                                   |       "startid",                |
|    |              |              |                                                                   |       "endid",                  |
|    |              |              |                                                                   |       "role",                   |
|    |              |              |                                                                   |       "stockpercent",           |
|    |              |              |                                                                   |       "shouldcapi",             |
|    |              |              |                                                                   |       "create_time"             |
|    |              |              |                                                                   |     ],                          |
|    |              |              |                                                                   |     "type": "-72"               |
|    |              |              |                                                                   |   },                            |
|    |              |              |                                                                   |   {                             |
|    |              |              |                                                                   |     "props": [                  |
|    |              |              |                                                                   |       "_src",                   |
|    |              |              |                                                                   |       "_type",                  |
|    |              |              |                                                                   |       "_rank",                  |
|    |              |              |                                                                   |       "_dst",                   |
|    |              |              |                                                                   |       "type",                   |
|    |              |              |                                                                   |       "startid",                |
|    |              |              |                                                                   |       "endid",                  |
|    |              |              |                                                                   |       "role",                   |
|    |              |              |                                                                   |       "stockpercent",           |
|    |              |              |                                                                   |       "shouldcapi",             |
|    |              |              |                                                                   |       "create_time"             |
|    |              |              |                                                                   |     ],                          |
|    |              |              |                                                                   |     "type": "73"                |
|    |              |              |                                                                   |   },                            |
|    |              |              |                                                                   |   {                             |
|    |              |              |                                                                   |     "props": [                  |
|    |              |              |                                                                   |       "_src",                   |
|    |              |              |                                                                   |       "_type",                  |
|    |              |              |                                                                   |       "_rank",                  |
|    |              |              |                                                                   |       "_dst",                   |
|    |              |              |                                                                   |       "type",                   |
|    |              |              |                                                                   |       "startid",                |
|    |              |              |                                                                   |       "endid",                  |
|    |              |              |                                                                   |       "role",                   |
|    |              |              |                                                                   |       "stockpercent",           |
|    |              |              |                                                                   |       "shouldcapi",             |
|    |              |              |                                                                   |       "create_time"             |
|    |              |              |                                                                   |     ],                          |
|    |              |              |                                                                   |     "type": "-73"               |
|    |              |              |                                                                   |   }                             |
|    |              |              |                                                                   | ]                               |
|    |              |              |                                                                   | statProps:                      |
|    |              |              |                                                                   | exprs:                          |
|    |              |              |                                                                   | random: false                   |
-----+--------------+--------------+-------------------------------------------------------------------+----------------------------------
|  1 | Start        |              | ver: 0, rows: 0, execTime: 0us, totalTime: 7us                    | outputVar: [                    |
|    |              |              | ver: 1, rows: 0, execTime: 0us, totalTime: 6us                    |   {                             |
|    |              |              | ver: 2, rows: 0, execTime: 0us, totalTime: 13us                   |     "colNames": [],             |
|    |              |              |                                                                   |     "name": "__Start_1",        |
|    |              |              |                                                                   |     "type": "DATASET"           |
|    |              |              |                                                                   |   }                             |
|    |              |              |                                                                   | ]                               |
-----+--------------+--------------+-------------------------------------------------------------------+----------------------------------
|  0 | Start        |              | ver: 0, rows: 0, execTime: 0us, totalTime: 13us                   | outputVar: [                    |
|    |              |              |                                                                   |   {                             |
|    |              |              |                                                                   |     "colNames": [],             |
|    |              |              |                                                                   |     "name": "__Start_0",        |
|    |              |              |                                                                   |     "type": "DATASET"           |
|    |              |              |                                                                   |   }                             |
|    |              |              |                                                                   | ]                               |
-----+--------------+--------------+-------------------------------------------------------------------+----------------------------------
  1. 多线程PR 大概多久可以release? 是在下个小版本v2.0.2 发布吗?
1 个赞

感觉小数据集没什么可比性啊, 并且neo4j和nebula的存储方式不同,
neo4j中的点和边是分别存储在一起的,nebula中的边和起始点存储在一起的, 小数据集下neo4j是有可能占优的
你可以使用大一点的数据集测试一下
多线程的PR 内部还在测试,应该在下一个版本会出

2 个赞

谢谢,明白,之前预期以为会比Neo4j 好,测试下来发现没提升,想上线就得想怎么优化的问题。
现在对于这个问题,只能等下个版本的更新吗? 有没有性能提升的方案,比如某个参数的修改?

补充: 突然感觉是不是小数据集 理解不一致。
我说的小数据集是指: GET SUBGRAPH 2 steps from VID 这个返回结果的大小,有些子图比较大的情况,nebula 占优势,但是子图比较小的情况下,测试如上图,性能差距比较大。
整体数据集的大小两者是一致的,节点2亿, 边3亿

neo4j 这个match 有个where字段, 应该会过滤一些数据
@critical27 请问我们storage 多线程的PR,在下个小版本合入吗

neo4j 的 where 语法应该是为了确定起始点,根据属性过滤。
Nebula 中get subgraph 直接在From 中指定了VID 的起始点,应该和where 关系不大。
另外,想知道storage 里面多线程的查询,在测试中,对子图这种语句有多大性能提升有评估过吗?

内部测试在3亿点,4亿条边,3台机器3副本的情况下,拓展5步,性能大概提高了30%左右

2 个赞

哦哦,那提升挺多的。大概什么时候可以release呀?

PR在这里 Concurrently go/lookup/fetch by critical27 · Pull Request #503 · vesoft-inc/nebula-storage · GitHub 应该是要在下一个大版本,大概在下一个季度了

该主题在最后一个回复创建后7天后自动关闭。不再允许新的回复。