find path 查询导致graph 崩溃退出

提问参考模版:

  • nebula 版本:2.6.0
  • 部署方式:分布式
  • 安装方式:RPM
  • 是否为线上版本: N
  • 硬件信息
    • 磁盘( 推荐使用 SSD)HDD
    • CPU、内存信息
  • 问题的具体描述
  • 相关的 meta / storage / graph info 日志信息(尽量使用文本形式方便检索)

执行find path ,查询的数据较多,在等待一会之后graph 崩溃退出了。

(root@nebula) [product_29]> explain FIND NOLOOP PATH WITH PROP FROM '100' TO '1500' OVER * BIDIRECT UPTO 10 STEPS
Execution succeeded (time spent 642/1177 us)

Execution Plan (optimize time 109 us)

-----+-----------------+--------------+----------------+----------------------------------------
| id | name            | dependencies | profiling data | operator info                         |
-----+-----------------+--------------+----------------+----------------------------------------
| 19 | DataCollect     | 15,18        |                | outputVar: [                          |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "colNames": [                     |
|    |                 |              |                |       "path"                          |
|    |                 |              |                |     ],                                |
|    |                 |              |                |     "type": "DATASET",                |
|    |                 |              |                |     "name": "__DataCollect_19"        |
|    |                 |              |                |   }                                   |
|    |                 |              |                | ]                                     |
|    |                 |              |                | inputVar: [                           |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "colNames": [],                   |
|    |                 |              |                |     "type": "DATASET",                |
|    |                 |              |                |     "name": "__GetVertices_15"        |
|    |                 |              |                |   },                                  |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "colNames": [],                   |
|    |                 |              |                |     "type": "DATASET",                |
|    |                 |              |                |     "name": "__GetEdges_18"           |
|    |                 |              |                |   },                                  |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "colNames": [                     |
|    |                 |              |                |       "path"                          |
|    |                 |              |                |     ],                                |
|    |                 |              |                |     "type": "DATASET",                |
|    |                 |              |                |     "name": "__DataCollect_11"        |
|    |                 |              |                |   }                                   |
|    |                 |              |                | ]                                     |
|    |                 |              |                | kind: PathProp                        |
-----+-----------------+--------------+----------------+----------------------------------------
| 15 | GetVertices     | 14           |                | outputVar: [                          |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "colNames": [],                   |
|    |                 |              |                |     "type": "DATASET",                |
|    |                 |              |                |     "name": "__GetVertices_15"        |
|    |                 |              |                |   }                                   |
|    |                 |              |                | ]                                     |
|    |                 |              |                | inputVar: __Unwind_14                 |
|    |                 |              |                | space: 38                             |
|    |                 |              |                | dedup: true                           |
|    |                 |              |                | limit: 9223372036854775807            |
|    |                 |              |                | filter:                               |
|    |                 |              |                | orderBy: []                           |
|    |                 |              |                | src: id(COLUMN[1])                    |
|    |                 |              |                | props: [                              |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "props": [                        |
|    |                 |              |                |       "namelist",                     |
|    |                 |              |                |       "_tag"                          |
|    |                 |              |                |     ],                                |
|    |                 |              |                |     "tagId": 40                       |
|    |                 |              |                |   },                                  |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "props": [                        |
|    |                 |              |                |       "_tag"                          |
|    |                 |              |                |     ],                                |
|    |                 |              |                |     "tagId": 39                       |
|    |                 |              |                |   },                                  |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "props": [                        |
|    |                 |              |                |       "_tag"                          |
|    |                 |              |                |     ],                                |
|    |                 |              |                |     "tagId": 41                       |
|    |                 |              |                |   }                                   |
|    |                 |              |                | ]                                     |
|    |                 |              |                | exprs:                                |
-----+-----------------+--------------+----------------+----------------------------------------
| 14 | Unwind          | 13           |                | outputVar: [                          |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "colNames": [                     |
|    |                 |              |                |       "nodes"                         |
|    |                 |              |                |     ],                                |
|    |                 |              |                |     "type": "DATASET",                |
|    |                 |              |                |     "name": "__Unwind_14"             |
|    |                 |              |                |   }                                   |
|    |                 |              |                | ]                                     |
|    |                 |              |                | inputVar: __Project_13                |
|    |                 |              |                | alias:                                |
|    |                 |              |                | unwindExpr: COLUMN[0]                 |
-----+-----------------+--------------+----------------+----------------------------------------
| 13 | Project         | 12           |                | outputVar: [                          |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "colNames": [                     |
|    |                 |              |                |       "nodes"                         |
|    |                 |              |                |     ],                                |
|    |                 |              |                |     "type": "DATASET",                |
|    |                 |              |                |     "name": "__Project_13"            |
|    |                 |              |                |   }                                   |
|    |                 |              |                | ]                                     |
|    |                 |              |                | inputVar: __DataCollect_11            |
|    |                 |              |                | columns: [                            |
|    |                 |              |                |   "nodes(COLUMN[0]) AS nodes"         |
|    |                 |              |                | ]                                     |
-----+-----------------+--------------+----------------+----------------------------------------
| 12 | PassThrough     | 11           |                | outputVar: [                          |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "colNames": [],                   |
|    |                 |              |                |     "type": "DATASET",                |
|    |                 |              |                |     "name": "__PassThrough_12"        |
|    |                 |              |                |   }                                   |
|    |                 |              |                | ]                                     |
|    |                 |              |                | inputVar: __DataCollect_11            |
-----+-----------------+--------------+----------------+----------------------------------------
| 11 | DataCollect     | 10           |                | outputVar: [                          |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "colNames": [                     |
|    |                 |              |                |       "path"                          |
|    |                 |              |                |     ],                                |
|    |                 |              |                |     "type": "DATASET",                |
|    |                 |              |                |     "name": "__DataCollect_11"        |
|    |                 |              |                |   }                                   |
|    |                 |              |                | ]                                     |
|    |                 |              |                | inputVar: [                           |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "colNames": [                     |
|    |                 |              |                |       "_path"                         |
|    |                 |              |                |     ],                                |
|    |                 |              |                |     "type": "DATASET",                |
|    |                 |              |                |     "name": "__ConjunctPath_7"        |
|    |                 |              |                |   }                                   |
|    |                 |              |                | ]                                     |
|    |                 |              |                | kind: ALL PATHS                       |
-----+-----------------+--------------+----------------+----------------------------------------
| 10 | Loop            | 9            |                | outputVar: [                          |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "colNames": [],                   |
|    |                 |              |                |     "type": "DATASET",                |
|    |                 |              |                |     "name": "__Loop_10"               |
|    |                 |              |                |   }                                   |
|    |                 |              |                | ]                                     |
|    |                 |              |                | inputVar: __VAR_1                     |
|    |                 |              |                | condition: (++($__VAR_2)<=5)          |
|    |                 |              |                | loopBody: 7                           |
-----+-----------------+--------------+----------------+----------------------------------------
|  7 | ConjunctPath    | 4,6          |                | branch: true, nodeId: 10              |
|    |                 |              |                |                                       |
|    |                 |              |                | outputVar: [                          |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "colNames": [                     |
|    |                 |              |                |       "_path"                         |
|    |                 |              |                |     ],                                |
|    |                 |              |                |     "type": "DATASET",                |
|    |                 |              |                |     "name": "__ConjunctPath_7"        |
|    |                 |              |                |   }                                   |
|    |                 |              |                | ]                                     |
|    |                 |              |                | inputVar: {                           |
|    |                 |              |                |   "rightVar": "__VAR_1",              |
|    |                 |              |                |   "leftVar": "__VAR_0"                |
|    |                 |              |                | }                                     |
|    |                 |              |                | kind: AllPath                         |
|    |                 |              |                | conditionalVar:                       |
|    |                 |              |                | noloop: true                          |
-----+-----------------+--------------+----------------+----------------------------------------
|  4 | ProduceAllPaths | 3            |                | outputVar: [                          |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "colNames": [                     |
|    |                 |              |                |       "_vid",                         |
|    |                 |              |                |       "_path"                         |
|    |                 |              |                |     ],                                |
|    |                 |              |                |     "type": "DATASET",                |
|    |                 |              |                |     "name": "__VAR_0"                 |
|    |                 |              |                |   }                                   |
|    |                 |              |                | ]                                     |
|    |                 |              |                | noloop : false                        |
-----+-----------------+--------------+----------------+----------------------------------------
|  3 | GetNeighbors    | 2            |                | outputVar: [                          |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "colNames": [],                   |
|    |                 |              |                |     "type": "DATASET",                |
|    |                 |              |                |     "name": "__GetNeighbors_3"        |
|    |                 |              |                |   }                                   |
|    |                 |              |                | ]                                     |
|    |                 |              |                | inputVar: __VAR_0                     |
|    |                 |              |                | space: 38                             |
|    |                 |              |                | dedup: true                           |
|    |                 |              |                | limit: -1                             |
|    |                 |              |                | filter:                               |
|    |                 |              |                | orderBy: []                           |
|    |                 |              |                | src: COLUMN[0]                        |
|    |                 |              |                | edgeTypes: []                         |
|    |                 |              |                | edgeDirection: OUT_EDGE               |
|    |                 |              |                | vertexProps:                          |
|    |                 |              |                | edgeProps: [                          |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "props": [                        |
|    |                 |              |                |       "_dst",                         |
|    |                 |              |                |       "_type",                        |
|    |                 |              |                |       "_rank"                         |
|    |                 |              |                |     ],                                |
|    |                 |              |                |     "type": "-42"                     |
|    |                 |              |                |   },                                  |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "props": [                        |
|    |                 |              |                |       "_dst",                         |
|    |                 |              |                |       "_type",                        |
|    |                 |              |                |       "_rank"                         |
|    |                 |              |                |     ],                                |
|    |                 |              |                |     "type": "42"                      |
|    |                 |              |                |   }                                   |
|    |                 |              |                | ]                                     |
|    |                 |              |                | statProps:                            |
|    |                 |              |                | exprs:                                |
|    |                 |              |                | random: false                         |
-----+-----------------+--------------+----------------+----------------------------------------
|  2 | PassThrough     | 1            |                | outputVar: [                          |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "colNames": [],                   |
|    |                 |              |                |     "type": "DATASET",                |
|    |                 |              |                |     "name": "__PassThrough_2"         |
|    |                 |              |                |   }                                   |
|    |                 |              |                | ]                                     |
|    |                 |              |                | inputVar: __Start_1                   |
-----+-----------------+--------------+----------------+----------------------------------------
|  1 | Start           |              |                | outputVar: [                          |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "colNames": [],                   |
|    |                 |              |                |     "type": "DATASET",                |
|    |                 |              |                |     "name": "__Start_1"               |
|    |                 |              |                |   }                                   |
|    |                 |              |                | ]                                     |
-----+-----------------+--------------+----------------+----------------------------------------
|  6 | ProduceAllPaths | 5            |                | outputVar: [                          |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "colNames": [                     |
|    |                 |              |                |       "_vid",                         |
|    |                 |              |                |       "_path"                         |
|    |                 |              |                |     ],                                |
|    |                 |              |                |     "type": "DATASET",                |
|    |                 |              |                |     "name": "__VAR_1"                 |
|    |                 |              |                |   }                                   |
|    |                 |              |                | ]                                     |
|    |                 |              |                | noloop : false                        |
-----+-----------------+--------------+----------------+----------------------------------------
|  5 | GetNeighbors    | 2            |                | outputVar: [                          |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "colNames": [],                   |
|    |                 |              |                |     "type": "DATASET",                |
|    |                 |              |                |     "name": "__GetNeighbors_5"        |
|    |                 |              |                |   }                                   |
|    |                 |              |                | ]                                     |
|    |                 |              |                | inputVar: __VAR_1                     |
|    |                 |              |                | space: 38                             |
|    |                 |              |                | dedup: true                           |
|    |                 |              |                | limit: -1                             |
|    |                 |              |                | filter:                               |
|    |                 |              |                | orderBy: []                           |
|    |                 |              |                | src: COLUMN[0]                        |
|    |                 |              |                | edgeTypes: []                         |
|    |                 |              |                | edgeDirection: OUT_EDGE               |
|    |                 |              |                | vertexProps:                          |
|    |                 |              |                | edgeProps: [                          |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "props": [                        |
|    |                 |              |                |       "_dst",                         |
|    |                 |              |                |       "_type",                        |
|    |                 |              |                |       "_rank"                         |
|    |                 |              |                |     ],                                |
|    |                 |              |                |     "type": "42"                      |
|    |                 |              |                |   },                                  |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "props": [                        |
|    |                 |              |                |       "_dst",                         |
|    |                 |              |                |       "_type",                        |
|    |                 |              |                |       "_rank"                         |
|    |                 |              |                |     ],                                |
|    |                 |              |                |     "type": "-42"                     |
|    |                 |              |                |   }                                   |
|    |                 |              |                | ]                                     |
|    |                 |              |                | statProps:                            |
|    |                 |              |                | exprs:                                |
|    |                 |              |                | random: false                         |
-----+-----------------+--------------+----------------+----------------------------------------
|  9 | Project         | 8            |                | outputVar: [                          |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "colNames": [                     |
|    |                 |              |                |       "_vid",                         |
|    |                 |              |                |       "_path"                         |
|    |                 |              |                |     ],                                |
|    |                 |              |                |     "type": "DATASET",                |
|    |                 |              |                |     "name": "__VAR_1"                 |
|    |                 |              |                |   }                                   |
|    |                 |              |                | ]                                     |
|    |                 |              |                | inputVar: __VAR_1                     |
|    |                 |              |                | columns: [                            |
|    |                 |              |                |   "COLUMN[0] AS _vid",                |
|    |                 |              |                |   "[PathBuild[COLUMN[0]]] AS _path"   |
|    |                 |              |                | ]                                     |
-----+-----------------+--------------+----------------+----------------------------------------
|  8 | Project         | 0            |                | outputVar: [                          |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "colNames": [                     |
|    |                 |              |                |       "_vid",                         |
|    |                 |              |                |       "_path"                         |
|    |                 |              |                |     ],                                |
|    |                 |              |                |     "type": "DATASET",                |
|    |                 |              |                |     "name": "__VAR_0"                 |
|    |                 |              |                |   }                                   |
|    |                 |              |                | ]                                     |
|    |                 |              |                | inputVar: __VAR_0                     |
|    |                 |              |                | columns: [                            |
|    |                 |              |                |   "COLUMN[0] AS _vid",                |
|    |                 |              |                |   "[PathBuild[COLUMN[0]]] AS _path"   |
|    |                 |              |                | ]                                     |
-----+-----------------+--------------+----------------+----------------------------------------
|  0 | Start           |              |                | outputVar: [                          |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "colNames": [],                   |
|    |                 |              |                |     "type": "DATASET",                |
|    |                 |              |                |     "name": "__Start_0"               |
|    |                 |              |                |   }                                   |
|    |                 |              |                | ]                                     |
-----+-----------------+--------------+----------------+----------------------------------------
| 18 | GetEdges        | 17           |                | outputVar: [                          |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "colNames": [],                   |
|    |                 |              |                |     "type": "DATASET",                |
|    |                 |              |                |     "name": "__GetEdges_18"           |
|    |                 |              |                |   }                                   |
|    |                 |              |                | ]                                     |
|    |                 |              |                | inputVar: __Unwind_17                 |
|    |                 |              |                | space: 38                             |
|    |                 |              |                | dedup: true                           |
|    |                 |              |                | limit: 9223372036854775807            |
|    |                 |              |                | filter:                               |
|    |                 |              |                | orderBy: []                           |
|    |                 |              |                | src: src(COLUMN[1])                   |
|    |                 |              |                | type: typeid(COLUMN[1])               |
|    |                 |              |                | ranking: rank(COLUMN[1])              |
|    |                 |              |                | dst: dst(COLUMN[1])                   |
|    |                 |              |                | props: [                              |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "props": [                        |
|    |                 |              |                |       "_src",                         |
|    |                 |              |                |       "_type",                        |
|    |                 |              |                |       "_rank",                        |
|    |                 |              |                |       "_dst",                         |
|    |                 |              |                |       "eventtime"                     |
|    |                 |              |                |     ],                                |
|    |                 |              |                |     "type": "42"                      |
|    |                 |              |                |   }                                   |
|    |                 |              |                | ]                                     |
|    |                 |              |                | exprs:                                |
-----+-----------------+--------------+----------------+----------------------------------------
| 17 | Unwind          | 16           |                | outputVar: [                          |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "colNames": [                     |
|    |                 |              |                |       "edges"                         |
|    |                 |              |                |     ],                                |
|    |                 |              |                |     "type": "DATASET",                |
|    |                 |              |                |     "name": "__Unwind_17"             |
|    |                 |              |                |   }                                   |
|    |                 |              |                | ]                                     |
|    |                 |              |                | inputVar: __Project_16                |
|    |                 |              |                | alias:                                |
|    |                 |              |                | unwindExpr: COLUMN[0]                 |
-----+-----------------+--------------+----------------+----------------------------------------
| 16 | Project         | 12           |                | outputVar: [                          |
|    |                 |              |                |   {                                   |
|    |                 |              |                |     "colNames": [                     |
|    |                 |              |                |       "edges"                         |
|    |                 |              |                |     ],                                |
|    |                 |              |                |     "type": "DATASET",                |
|    |                 |              |                |     "name": "__Project_16"            |
|    |                 |              |                |   }                                   |
|    |                 |              |                | ]                                     |
|    |                 |              |                | inputVar: __DataCollect_11            |
|    |                 |              |                | columns: [                            |
|    |                 |              |                |   "relationships(COLUMN[0]) AS edges" |
|    |                 |              |                | ]                                     |
-----+-----------------+--------------+----------------+----------------------------------------

又测试几次,似乎跟数据有关系,也有能查询出来的,但是查询对应的数据就会挂掉。

(root@nebula) [product_29]> FIND SHORTEST PATH WITH PROP FROM '100' TO '1500' OVER * BIDIRECT UPTO 10 STEPS
[ERROR (-1005)]: N5folly13BrokenPromiseE: Broken promise for type name `nebula::Status`

Mon, 22 Nov 2021 18:20:24 CST

(root@nebula) [product_29]> FIND SHORTEST PATH WITH PROP FROM '100' TO '1500' OVER * BIDIRECT UPTO 9 STEPS
+------+
| path |
+------+
+------+
Empty set (time spent 1086468/1086383 us)

数据如下

修改一个节点值则能查询出来,这个是9步可以到达的。

机器具体是什么配置?你观察一下,应该是OOM了。

感觉跟我提的这个issue 比较类似,我之前也是这个情况,find path 算法性能大点的时候 不太好
https://github.com/vesoft-inc/nebula/issues/3221

测试了一下,查询一共需要40个G的内存,刚好有一台机器空闲内存不足40G,所以直接挂掉了,内存足的可以查询。但是不是有那个内存百分位的配置吗?这好像没有生效。

是的,你的内存百分位设置的是多少呢?

设置的0.9,但是一执行内存都用完了,就崩溃退出了

内存水位是在每个算子执行完之后检测的,所以这个可能是在算子执行的时候发生了OOM,可以把这台机器的阈值调低一点。

OOM是不是应该做一下处理,让查询失败就行了,graph直接退出影响挺大的。

Broken promise这个报错稳定复现吗? 那时候内存足够吗?

是稳定复现的,查询的时候内存会不断增加,直到超过机器的内存graphd就挂掉了。

但是Broken promise这个报错的时候, graphd还没挂掉的吧?

应该是的,当时好像没太注意到这个报错,只是关心那个崩溃的问题了。

刚才我又测试了一下,就是提交一个find path 把内存耗尽就报错了

内存耗尽了然后console显示broken promise报错, 但这个时候graphd服务还没挂吧? 因为我看到报错后你还正常执行了一条语句

是的,我刚才用2.6.1版本测试的,报错后没有挂掉。挂掉应该不是这个错误引起的吧。

挂掉应该就是oom导致的, 现在oom检测不是及时的, 可能在某次oom检测之后发生了oom, 这个时候就做不到优雅退出了.

1 个赞

我现在是想复现Broken promise这个报错, 本地没复现出来

需要有关联度较多的数据,然后去执行6步甚至10步的路径查找,应该可以复现的。我这里多线程去提交查询,导致内存直接把机器剩余内存用完,就出现这个错误了。
我使用的数据2步关联2w,3步关联80w。

1 个赞