Match 语句在 多 Tag 查询下如何使用索引

  • nebula 版本:3.1.0
  • 部署方式:单机
  • 安装方式:源码编译

数据库为 2k 个节点,31个 tag。在所有 Tag 上建立的对应的 tag index。

执行单 Tag 查询语句,profile match (v:label_9) return v.label_global.position limit 10,执行计划使用执行节点 IndexScan,但执行语句 profile match (v:label_6:label_808) return v.label_global.position limit 80,即同时对多Tag 进行查询时,执行计划中的执行节点变为了 ScanVertices。根据我对代码的阅读和执行节点的理解,ScanVertices 计划是没有对索引进行利用的。不知道我的理解对吗?如果希望利用索引的话应该如何查询?

以下为执行的语句和返回结果:

执行单 Tag 查询语句,profile match (v:label_9) return v.label_global.position limit 10
执行结果如下

+-------------------------+
| v.label_global.position |
+-------------------------+
| 14                      |
| 11                      |
| 1                       |
| 17                      |
| 20                      |
| 11                      |
| 1                       |
| 7                       |
| 4                       |
| 5                       |
+-------------------------+
Got 10 rows (time spent 16064/16911 us)
-----+----------------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------
| id | name           | dependencies | profiling data                                                                                                                                                           | operator info                            |
-----+----------------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------
| 11 | Project        | 9            | ver: 0, rows: 10, execTime: 36us, totalTime: 38us                                                                                                                        | outputVar: {                             |
|    |                |              |                                                                                                                                                                          |   "colNames": [                          |
|    |                |              |                                                                                                                                                                          |     "v.label_global.position"            |
|    |                |              |                                                                                                                                                                          |   ],                                     |
|    |                |              |                                                                                                                                                                          |   "type": "DATASET",                     |
|    |                |              |                                                                                                                                                                          |   "name": "__DataCollect_7"              |
|    |                |              |                                                                                                                                                                          | }                                        |
|    |                |              |                                                                                                                                                                          | inputVar: __Limit_9                      |
|    |                |              |                                                                                                                                                                          | columns: [                               |
|    |                |              |                                                                                                                                                                          |   "-.v.label_global.position"            |
|    |                |              |                                                                                                                                                                          | ]                                        |
-----+----------------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------
|  9 | Limit          | 3            | ver: 0, rows: 10, execTime: 41us, totalTime: 42us                                                                                                                        | outputVar: {                             |
|    |                |              |                                                                                                                                                                          |   "colNames": [                          |
|    |                |              |                                                                                                                                                                          |     "v"                                  |
|    |                |              |                                                                                                                                                                          |   ],                                     |
|    |                |              |                                                                                                                                                                          |   "type": "DATASET",                     |
|    |                |              |                                                                                                                                                                          |   "name": "__Limit_9"                    |
|    |                |              |                                                                                                                                                                          | }                                        |
|    |                |              |                                                                                                                                                                          | inputVar: __AppendVertices_3             |
|    |                |              |                                                                                                                                                                          | offset: 0                                |
|    |                |              |                                                                                                                                                                          | count: 10                                |
-----+----------------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------
|  3 | AppendVertices | 1            | {                                                                                                                                                                        | outputVar: {                             |
|    |                |              | ver: 0, rows: 106, execTime: 679us, totalTime: 13793us                                                                                                                   |   "colNames": [                          |
|    |                |              | "127.0.0.1":9779 exec/total: 12089(us)/13227(us)                                                                                                                         |     "v"                                  |
|    |                |              | total_rpc: 13395(us)                                                                                                                                                     |   ],                                     |
|    |                |              | }                                                                                                                                                                        |   "type": "DATASET",                     |
|    |                |              |                                                                                                                                                                          |   "name": "__AppendVertices_3"           |
|    |                |              |                                                                                                                                                                          | }                                        |
|    |                |              |                                                                                                                                                                          | inputVar: __IndexScan_1                  |
|    |                |              |                                                                                                                                                                          | space: 5644                              |
|    |                |              |                                                                                                                                                                          | dedup: true                              |
|    |                |              |                                                                                                                                                                          | limit: -1                                |
|    |                |              |                                                                                                                                                                          | filter:                                  |
|    |                |              |                                                                                                                                                                          | orderBy: []                              |
|    |                |              |                                                                                                                                                                          | src: $_vid                               |
|    |                |              |                                                                                                                                                                          | props: [                                 |
|    |                |              |                                                                                                                                                                          |   {                                      |
|    |                |              |                                                                                                                                                                          |     "props": [                           |
|    |                |              |                                                                                                                                                                          |       "position"                         |
|    |                |              |                                                                                                                                                                          |     ],                                   |
|    |                |              |                                                                                                                                                                          |     "tagId": 5666                        |
|    |                |              |                                                                                                                                                                          |   },                                     |
|    |                |              |                                                                                                                                                                          |   {                                      |
|    |                |              |                                                                                                                                                                          |     "props": [                           |
|    |                |              |                                                                                                                                                                          |       "_tag"                             |
|    |                |              |                                                                                                                                                                          |     ],                                   |
|    |                |              |                                                                                                                                                                          |     "tagId": 5664                        |
|    |                |              |                                                                                                                                                                          |   }                                      |
|    |                |              |                                                                                                                                                                          | ]                                        |
|    |                |              |                                                                                                                                                                          | exprs:                                   |
|    |                |              |                                                                                                                                                                          | vertex_filter: label_9._tag IS NOT EMPTY |
|    |                |              |                                                                                                                                                                          | if_track_previous_path: false            |
-----+----------------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------
|  1 | IndexScan      | 2            | {                                                                                                                                                                        | outputVar: {                             |
|    |                |              | ver: 0, rows: 106, execTime: 0us, totalTime: 1240us                                                                                                                      |   "colNames": [                          |
|    |                |              | storage_detail: {IndexLimitNode(limit=9223372036854775807):240(us),IndexProjectionNode(projectColumn=[_vid]):236(us),IndexVertexScanNode(IndexID=5705, Path=()):204(us)} |     "_vid"                               |
|    |                |              | "127.0.0.1":9779 exec/total: 335(us)/1081(us)                                                                                                                            |   ],                                     |
|    |                |              | }                                                                                                                                                                        |   "type": "DATASET",                     |
|    |                |              |                                                                                                                                                                          |   "name": "__IndexScan_1"                |
|    |                |              |                                                                                                                                                                          | }                                        |
|    |                |              |                                                                                                                                                                          | inputVar:                                |
|    |                |              |                                                                                                                                                                          | space: 5644                              |
|    |                |              |                                                                                                                                                                          | dedup: false                             |
|    |                |              |                                                                                                                                                                          | limit: 9223372036854775807               |
|    |                |              |                                                                                                                                                                          | filter:                                  |
|    |                |              |                                                                                                                                                                          | orderBy: []                              |
|    |                |              |                                                                                                                                                                          | schemaId: 5664                           |
|    |                |              |                                                                                                                                                                          | isEdge: false                            |
|    |                |              |                                                                                                                                                                          | returnCols: [                            |
|    |                |              |                                                                                                                                                                          |   "_vid"                                 |
|    |                |              |                                                                                                                                                                          | ]                                        |
|    |                |              |                                                                                                                                                                          | indexCtx: [                              |
|    |                |              |                                                                                                                                                                          |   {                                      |
|    |                |              |                                                                                                                                                                          |     "columnHints": [],                   |
|    |                |              |                                                                                                                                                                          |     "filter": "",                        |
|    |                |              |                                                                                                                                                                          |     "index_id": 5705                     |
|    |                |              |                                                                                                                                                                          |   }                                      |
|    |                |              |                                                                                                                                                                          | ]                                        |
-----+----------------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------
|  2 | Start          |              | ver: 0, rows: 0, execTime: 0us, totalTime: 29us                                                                                                                          | outputVar: {                             |
|    |                |              |                                                                                                                                                                          |   "colNames": [],                        |
|    |                |              |                                                                                                                                                                          |   "type": "DATASET",                     |
|    |                |              |                                                                                                                                                                          |   "name": "__Start_2"                    |
|    |                |              |                                                                                                                                                                          | }                                        |
-----+----------------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------

可以看到,结果中使用了 IndexScan。

但执行语句 profile match (v:label_6:label_808) return v.label_global.position limit 80,即同时对两种 Tag 进行查询时,执行计划中的执行节点变为了 ScanVertices。

+-------------------------+
| v.label_global.position |
+-------------------------+
| 10                      |
| 15                      |
| 11                      |
| 9                       |
| 1                       |
| 6                       |
| 16                      |
+-------------------------+
Got 7 rows (time spent 8562/9283 us)
-----+----------------+--------------+-----------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------
| id | name           | dependencies | profiling data                                      | operator info                                                                                                                         |
-----+----------------+--------------+-----------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------
| 15 | Project        | 16           | ver: 0, rows: 7, execTime: 14us, totalTime: 15us    | outputVar: {                                                                                                                          |
|    |                |              |                                                     |   "colNames": [                                                                                                                       |
|    |                |              |                                                     |     "v.label_global.position"                                                                                                         |
|    |                |              |                                                     |   ],                                                                                                                                  |
|    |                |              |                                                     |   "type": "DATASET",                                                                                                                  |
|    |                |              |                                                     |   "name": "__DataCollect_8"                                                                                                           |
|    |                |              |                                                     | }                                                                                                                                     |
|    |                |              |                                                     | inputVar: __Limit_13                                                                                                                  |
|    |                |              |                                                     | columns: [                                                                                                                            |
|    |                |              |                                                     |   "-.v.label_global.position"                                                                                                         |
|    |                |              |                                                     | ]                                                                                                                                     |
-----+----------------+--------------+-----------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------
| 16 | Limit          | 17           | ver: 0, rows: 7, execTime: 4us, totalTime: 6us      | outputVar: {                                                                                                                          |
|    |                |              |                                                     |   "colNames": [                                                                                                                       |
|    |                |              |                                                     |     "v"                                                                                                                               |
|    |                |              |                                                     |   ],                                                                                                                                  |
|    |                |              |                                                     |   "type": "DATASET",                                                                                                                  |
|    |                |              |                                                     |   "name": "__Limit_13"                                                                                                                |
|    |                |              |                                                     | }                                                                                                                                     |
|    |                |              |                                                     | inputVar: __AppendVertices_17                                                                                                         |
|    |                |              |                                                     | offset: 0                                                                                                                             |
|    |                |              |                                                     | count: 80                                                                                                                             |
-----+----------------+--------------+-----------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------
| 17 | AppendVertices | 18           | {                                                   | outputVar: {                                                                                                                          |
|    |                |              | ver: 0, rows: 7, execTime: 185us, totalTime: 1203us |   "colNames": [                                                                                                                       |
|    |                |              | "127.0.0.1":9779 exec/total: 534(us)/1013(us)       |     "v"                                                                                                                               |
|    |                |              | total_rpc: 1106(us)                                 |   ],                                                                                                                                  |
|    |                |              | }                                                   |   "type": "DATASET",                                                                                                                  |
|    |                |              |                                                     |   "name": "__AppendVertices_17"                                                                                                       |
|    |                |              |                                                     | }                                                                                                                                     |
|    |                |              |                                                     | inputVar: __ScanVertices_18                                                                                                           |
|    |                |              |                                                     | space: 5644                                                                                                                           |
|    |                |              |                                                     | dedup: true                                                                                                                           |
|    |                |              |                                                     | limit: -1                                                                                                                             |
|    |                |              |                                                     | filter:                                                                                                                               |
|    |                |              |                                                     | orderBy: []                                                                                                                           |
|    |                |              |                                                     | src: $-._vid                                                                                                                          |
|    |                |              |                                                     | props: [                                                                                                                              |
|    |                |              |                                                     |   {                                                                                                                                   |
|    |                |              |                                                     |     "props": [                                                                                                                        |
|    |                |              |                                                     |       "position"                                                                                                                      |
|    |                |              |                                                     |     ],                                                                                                                                |
|    |                |              |                                                     |     "tagId": 5666                                                                                                                     |
|    |                |              |                                                     |   }                                                                                                                                   |
|    |                |              |                                                     | ]                                                                                                                                     |
|    |                |              |                                                     | exprs:                                                                                                                                |
|    |                |              |                                                     | vertex_filter:                                                                                                                        |
|    |                |              |                                                     | if_track_previous_path: false                                                                                                         |
-----+----------------+--------------+-----------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------
| 18 | ScanVertices   | 3            | {                                                   | outputVar: {                                                                                                                          |
|    |                |              | ver: 0, rows: 7, execTime: 85us, totalTime: 6424us  |   "colNames": [                                                                                                                       |
|    |                |              | "127.0.0.1":9779 exec/total: 5551(us)/6303(us)      |     "_vid",                                                                                                                           |
|    |                |              | total_rpc: 6377(us)                                 |     "label_6._tag",                                                                                                                   |
|    |                |              | }                                                   |     "label_808._tag"                                                                                                                  |
|    |                |              |                                                     |   ],                                                                                                                                  |
|    |                |              |                                                     |   "type": "DATASET",                                                                                                                  |
|    |                |              |                                                     |   "name": "__ScanVertices_18"                                                                                                         |
|    |                |              |                                                     | }                                                                                                                                     |
|    |                |              |                                                     | inputVar:                                                                                                                             |
|    |                |              |                                                     | space: 5644                                                                                                                           |
|    |                |              |                                                     | dedup: false                                                                                                                          |
|    |                |              |                                                     | limit: 80                                                                                                                             |
|    |                |              |                                                     | filter: ((label_6._tag IS NOT EMPTY AND label_808._tag IS NOT EMPTY) AND (label_6._tag IS NOT EMPTY AND label_808._tag IS NOT EMPTY)) |
|    |                |              |                                                     | orderBy: []                                                                                                                           |
|    |                |              |                                                     | props: [                                                                                                                              |
|    |                |              |                                                     |   {                                                                                                                                   |
|    |                |              |                                                     |     "props": [                                                                                                                        |
|    |                |              |                                                     |       "_tag"                                                                                                                          |
|    |                |              |                                                     |     ],                                                                                                                                |
|    |                |              |                                                     |     "tagId": 5660                                                                                                                     |
|    |                |              |                                                     |   },                                                                                                                                  |
|    |                |              |                                                     |   {                                                                                                                                   |
|    |                |              |                                                     |     "props": [                                                                                                                        |
|    |                |              |                                                     |       "_tag"                                                                                                                          |
|    |                |              |                                                     |     ],                                                                                                                                |
|    |                |              |                                                     |     "tagId": 5675                                                                                                                     |
|    |                |              |                                                     |   }                                                                                                                                   |
|    |                |              |                                                     | ]                                                                                                                                     |
|    |                |              |                                                     | exprs:                                                                                                                                |
-----+----------------+--------------+-----------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------
|  3 | Start          |              | ver: 0, rows: 0, execTime: 0us, totalTime: 20us     | outputVar: {                                                                                                                          |
|    |                |              |                                                     |   "colNames": [],                                                                                                                     |
|    |                |              |                                                     |   "type": "DATASET",                                                                                                                  |
|    |                |              |                                                     |   "name": "__Start_3"                                                                                                                 |
|    |                |              |                                                     | }                                                                                                                                     |
-----+----------------+--------------+-----------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------
1 个赞

抛砖,不知道我的理解对不对
索引的 key 设计是索引 id 在前,(n:tag_foo:tag_bar) 语义是 tag_foo && tag_bar,如果利用索引的话是扫两个索引全量然后做 && 操作,filter没法下推到存储,但是用 scanVerteices 的话这个条件可以下推 filter: ((label_6._tag IS NOT EMPTY AND label_808._tag IS NOT EMPTY) AND (label_6._tag IS NOT EMPTY AND label_808._tag IS NOT EMPTY))

更新:理解的不对哈

Vertex KeyValue

Tag index KeyValue

-----+----------------+--------------+-----------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------
| 18 | ScanVertices   | 3            | {                                                   | outputVar: {                                                                                                                          |
|    |                |              | ver: 0, rows: 7, execTime: 85us, totalTime: 6424us  |   "colNames": [                                                                                                                       |
|    |                |              | "127.0.0.1":9779 exec/total: 5551(us)/6303(us)      |     "_vid",                                                                                                                           |
|    |                |              | total_rpc: 6377(us)                                 |     "label_6._tag",                                                                                                                   |
|    |                |              | }                                                   |     "label_808._tag"                                                                                                                  |
|    |                |              |                                                     |   ],                                                                                                                                  |
|    |                |              |                                                     |   "type": "DATASET",                                                                                                                  |
|    |                |              |                                                     |   "name": "__ScanVertices_18"                                                                                                         |
|    |                |              |                                                     | }                                                                                                                                     |
|    |                |              |                                                     | inputVar:                                                                                                                             |
|    |                |              |                                                     | space: 5644                                                                                                                           |
|    |                |              |                                                     | dedup: false                                                                                                                          |
|    |                |              |                                                     | limit: 80                                                                                                                             |
|    |                |              |                                                     | **filter: ((label_6._tag IS NOT EMPTY AND label_808._tag IS NOT EMPTY) AND (label_6._tag IS NOT EMPTY AND label_808._tag IS NOT EMPTY))** |
|    |                |              |                                                     | orderBy: []                                                                                                                           |
|    |                |              |                                                     | props: [                                                                                                                              |
|    |                |              |                                                     |   {                                                                                                                                   |
|    |                |              |                                                     |     "props": [                                                                                                                        |
|    |                |              |                                                     |       "_tag"                                                                                                                          |
|    |                |              |                                                     |     ],                                                                                                                                |
|    |                |              |                                                     |     "tagId": 5660                                                                                                                     |
|    |                |              |                                                     |   },                                                                                                                                  |
|    |                |              |                                                     |   {                                                                                                                                   |
|    |                |              |                                                     |     "props": [                                                                                                                        |
|    |                |              |                                                     |       "_tag"                                                                                                                          |
|    |                |              |                                                     |     ],                                                                                                                                |
|    |                |              |                                                     |     "tagId": 5675                                                                                                                     |
|    |                |              |                                                     |   }                                                                                                                                   |
|    |                |              |                                                     | ]                                                                                                                                     |
|    |                |              |                                                     | exprs:                                                                                                                                |
-----+----------------+--------------+-----------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------

不对吧。

1 个赞

你理解是对的。
应该算是个 bug,暂时先改写下语句吧:

MATCH (v:label_6) WHERE "label_808" IN labels(v)
RETURN v.label_global.position
LIMIT 80
1 个赞

嗯嗯,我划掉了哈 o(╥﹏╥)o,这块是优化可以利用索引去做的对么?
啊,看到了

嗯,倒不是优化不优化的事。类似的语句执行计划应该在 ScanVertices 的 pr 加入之后被非预期地修改了,语句行为也发生了变化。

1 个赞

奥奥,有 issue 了么?

还没,你提一下叭

1 个赞

在这里哈 MATCH pattern for multiple TAG end up with ScanVertices even when all involved indexes exist · Issue #4775 · vesoft-inc/nebula · GitHub

是不是 StartVidFinder 在决定执行节点时,由于 index 的 match 存在一定问题,导致 indexScan 认为无索引可用,导致未能使用 indexScan?

嗯。

要不要来 PR fix StartVidFinder? :slight_smile:

多索引具体怎么执行没思路呀(之前看文档以为是支持的),两次 IndexScan 然后求交吗

@Shylock-Hg 现在计划成单独查询然后 AND 不行么?

16be9e9084c2b3878dd653e12eb1e79
对源码读了一下,可能目前 nebula 的设计就是这样的吧。
p.s. 如果想参与到优化中的话有什么讨论群吗?或者 remote 的实习岗位?自己阅读代码 + PR 感觉还是很吃力的

确实是这样设计的,这里指的是多 tag 的索引搜索是不支持的。但是单 tag 的 多索引是支持的,比如 MATCH (v:player) where v.age>20 or v.name=="Tim Duncan" return v,执行计划上会在 IndexScan 中生成多个 column hint,在存储侧会分别去扫 age 和 name 的索引之后将 vid 都 union 起来传给计算层。

Remote 实习我帮你问问哈!咱们现在还没建立这样的微信群,不过可以在 github discussion 里发一些 RFC 的讨论,如果你想要做一些优化上的贡献可以找找一些 open 的 issue,如果觉得没有有成就感想要做的部分,也可以开一个 issue/ discussion 问下,我帮也帮你问问大家心里有没有可以分的 task。