rebuild index的问题

版本号:V2.0.0

问题:rebuild index对于新的prop失败。

issue的复现步骤:
创建一个tag:player

(root@nebula) [nba]> desc tag player
+--------+----------+-------+---------+
| Field  | Type     | Null  | Default |
+--------+----------+-------+---------+
| "name" | "string" | "YES" |         |
+--------+----------+-------+---------+
| "age"  | "int64"  | "YES" |         |

插入一些数据:

(root@nebula) [nba]> lookup on player yield player.name, player.age
+---------------------+---------------------+------------+
| VertexID            | player.name         | player.age |
+---------------------+---------------------+------------+
| "Aron Baynes"       | "Aron Baynes"       | 32         |
+---------------------+---------------------+------------+
| "Ben Simmons"       | "Ben Simmons"       | 22         |
+---------------------+---------------------+------------+
| "Blake Griffin"     | "Blake Griffin"     | 30         |
+---------------------+---------------------+------------+

执行alter tag增加一个属性:

(root@nebula) [nba]> ALTER TAG player ADD (new_name string);
Execution succeeded (time spent 5773/6190 us)
(root@nebula) [nba]> lookup on player yield player.name, player.age, player.new_name
+---------------------+---------------------+------------+-----------------+
| VertexID            | player.name         | player.age | player.new_name |
+---------------------+---------------------+------------+-----------------+
| "Aron Baynes"       | "Aron Baynes"       | 32         | __NULL__        |
+---------------------+---------------------+------------+-----------------+
| "Ben Simmons"       | "Ben Simmons"       | 22         | __NULL__        |
+---------------------+---------------------+------------+-----------------+
| "Blake Griffin"     | "Blake Griffin"     | 30         | __NULL__        |

创建一个新的组合索引:player_new_name

(root@nebula) [nba]> create tag index player_new_name on player(name(10), new_name(10))
Execution succeeded (time spent 7067/7389 us)

Thu, 04 Nov 2021 20:43:32 CST

重新rebuild这个新的index:player_new_name

(root@nebula) [nba]> rebuild tag index player_new_name
+------------+
| New Job Id |
+------------+
| 9168       |
+------------+
(root@nebula) [nba]> show job 9168
+----------------+---------------------+------------+-------------------------+-------------------------+
| Job Id(TaskId) | Command(Dest)       | Status     | Start Time              | Stop Time               |
+----------------+---------------------+------------+-------------------------+-------------------------+
| 9168           | "REBUILD_TAG_INDEX" | "FINISHED" | 2021-11-04T12:43:58.000 | 2021-11-04T12:43:58.000 |
+----------------+---------------------+------------+-------------------------+-------------------------+
| 0              | "xxx.xxx.xxx.xxx"     | "FINISHED" | 2021-11-04T12:43:58.000 | 2021-11-04T12:45:50.000 |
+----------------+---------------------+------------+-------------------------+-------------------------+
| 1              | "xxx.xxx.xxx.xxx"    | "FINISHED" | 2021-11-04T12:43:58.000 | 2021-11-04T12:45:50.000 |
+----------------+---------------------+------------+-------------------------+-------------------------+
| 2              | "xxx.xxx.xxx.xxx"   | "FINISHED" | 2021-11-04T12:43:58.000 | 2021-11-04T12:45:50.000 |
+----------------+---------------------+------------+-------------------------+-------------------------+

但其实看nebula-storaged的日志会发现很多错误,实际对于已经插入的数据,rebuild index是失败的。

E1104 20:43:58.509814 18879 IndexKeyUtils.cpp:112] prop error by : new_name. status : Unknown prop
W1104 20:43:58.509853 18879 RebuildTagIndexTask.cpp:128] Collect index value failed
E1104 20:43:58.509876 18879 IndexKeyUtils.cpp:112] prop error by : new_name. status : Unknown prop
W1104 20:43:58.509896 18879 RebuildTagIndexTask.cpp:128] Collect index value failed
E1104 20:43:58.509945 18879 IndexKeyUtils.cpp:112] prop error by : new_name. status : Unknown prop
W1104 20:43:58.509963 18879 RebuildTagIndexTask.cpp:128] Collect index value failed
E1104 20:43:58.509982 18879 IndexKeyUtils.cpp:112] prop error by : new_name. status : Unknown prop
W1104 20:43:58.510000 18879 RebuildTagIndexTask.cpp:128] Collect index value failed
E1104 20:43:58.510020 18879 IndexKeyUtils.cpp:112] prop error by : new_name. status : Unknown prop
W1104 20:43:58.510036 18879 RebuildTagIndexTask.cpp:128] Collect index value failed

这个问题的原因是,老的数据里面的val使用的是老的schema_ver=0,schema_ver=0的schema里面还没有这个新的prop(new_name),在IndexKeyUtils::collectIndexValues中会找不到player_new_name这个index需要的属性(new_name),所以才会报错。

IndexKeyUtils::collectIndexValues(RowReader* reader,
                                  const std::vector<nebula::meta::cpp2::ColumnDef>& cols) {
    if (reader == nullptr) {
        return Status::Error("Invalid row reader");
    }
    std::vector<Value> values;
    for (const auto& col : cols) {
        auto v = reader->getValueByName(col.get_name());
        auto isNullable = col.__isset.nullable && *(col.get_nullable());
        auto ret = checkValue(v, isNullable);
        if (!ret.ok()) {
            LOG(ERROR) << "prop error by : " << col.get_name()
                       << ". status : " << ret;
            return ret;
        }
        values.emplace_back(std::move(v));
    }
    return encodeValues(std::move(values), cols);
}

问题就是对于新的prop创建的index,在rebuild包含这个prop的index时,老的val里面找不到这个prop,导致rebuild index失败。从用户角度来说这种情况应该按照null处理,而不是按照prop error处理。

- -.郭老师你都是老用户了,下次提问可以给个版本号吗

好的。

v2.0.0 ?

是的,我也看了最新的代码,这个问题是没有修复的。

我印象你在增加字段的时候指定nullable应该就不会报错了?

默认就是nullable的。现在的问题是schema_ver=0的那个schema找不到新的prop

已经记录,感谢反馈
https://github.com/vesoft-inc/nebula/issues/3274

2 个赞

如果是带nullable的应该有问题,我回头找个人看吧

@sworduo 可以尝试修复一下这个问题。