storaged崩溃，有core文件

111 · 2021 年10 月 21 日 03:00

nebula 版本：2.5.1
部署方式:分布式
硬件信息
SSD 400G
CPU 20核、内存 30g
问题的具体描述
部署了30台机器的集群，使用过程中发现有3台的storaged崩溃，在nebula目录下有core文件,gdb bin/nebula-storaged core.389_executor-pri3-2，然后bt full显示信息如下:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/nebula/bin/nebula-storaged --flagfile /usr/local/nebula/etc/nebula-s'.
Program terminated with signal 6, Aborted.
#0  0x00007ff1ffdea1f7 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install nebula-graph-2.5.1-1.x86_64
(gdb) bt full
#0  0x00007ff1ffdea1f7 in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007ff1ffdeb8e8 in abort () from /lib64/libc.so.6
No symbol table info available.
#2  0x0000000002081dad in google::LogMessage::Fail() ()
No symbol table info available.
#3  0x0000000002086b8e in google::LogMessage::SendToLog() ()
No symbol table info available.
#4  0x0000000002081a7e in google::LogMessage::Flush() ()
No symbol table info available.
#5  0x00000000020822d9 in google::LogMessageFatal::~LogMessageFatal() ()
No symbol table info available.
#6  0x00000000010d0272 in nebula::RowWriterV2::checkUnsetFields() ()
No symbol table info available.
#7  0x00000000010d082d in nebula::RowWriterV2::finish() ()
No symbol table info available.
#8  0x0000000000ecec42 in nebula::storage::BaseProcessor<nebula::storage::cpp2::ExecResponse>::encodeRowVal(nebula::meta::NebulaSchemaProvider const*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<nebula::Value, std::allocator<nebula::Value> > const&, nebula::WriteResult&) [clone .constprop.0] ()
No symbol table info available.
#9  0x0000000000ecf728 in nebula::storage::AddVerticesProcessor::doProcess(nebula::storage::cpp2::AddVerticesRequest const&) ()
No symbol table info available.
#10 0x0000000000ed4bbb in nebula::storage::AddVerticesProcessor::process(nebula::storage::cpp2::AddVerticesRequest const&) ()
No symbol table info available.
#11 0x0000000000ec0741 in nebula::storage::GraphStorageServiceHandler::future_addVertices(nebula::storage::cpp2::AddVerticesRequest const&) ()
No symbol table info available.
#12 0x00000000013711e3 in nebula::storage::cpp2::GraphStorageServiceSvIf::async_tm_addVertices(std::unique_ptr<apache::thrift::HandlerCallback<nebula::storage::cpp2::ExecResponse>, std::default_delete<apache::thrift::HandlerCallback<nebula::storage::cpp2::ExecResponse> > >, nebula::storage::cpp2::AddVerticesRequest const&) ()
No symbol table info available.
#13 0x0000000001385bfa in void nebula::storage::cpp2::GraphStorageServiceAsyncProcessor::process_addVertices<apache::thrift::CompactProtocolReader, apache::thrift::CompactProtocolWriter>(std::unique_ptr<apache::thrift::ResponseChannelRequest, apache::thrift::RequestsRegistry::Deleter>, apache::thrift::SerializedRequest&&, apache::thrift::Cpp2RequestContext*, folly::EventBase*, apache::thrift::concurrency::ThreadManager*) ()
No symbol table info available.
#14 0x000000000138ba39 in void folly::detail::function::FunctionTraits<void (std::unique_ptr<apache::thrift::ResponseChannelRequest, apache::thrift::RequestsRegistry::Deleter>)>::callBig<std::shared_ptr<apache::thrift::EventTask> apache::thrift::GeneratedAsyncProcessor::makeEventTaskForRequest<nebula::storage::cpp2::GraphStorageServiceAsyncProcessor>(std::unique_ptr<apache::thrift::ResponseChannelRequest, apache::thrift::RequestsRegistry::Deleter>, apache::thrift::SerializedRequest&&, apache::thrift::Cpp2RequestContext*, folly::EventBase*, apache::thrift::concurrency::ThreadManager*, apache::thrift::RpcKind, void (nebula::storage::cpp2::GraphStorageServiceAsyncProcessor::*)(std::unique_ptr<apache::thrift::ResponseChannelRequest, apache::thrift::RequestsRegistry::Deleter>, apache::thrift::SerializedRequest&&, apache::thrift::Cpp2RequestContext*, folly::EventBase*, apache::thrift::concurrency::ThreadManager*), nebula::storage::cpp2::GraphStorageServiceAsyncProcessor*, apache::thrift::Tile*)::{lambda(std::unique_ptr<apache::thrift::ResponseChannelRequest, apache::thrift::RequestsRegistry::Deleter>)#1}>(std::unique_ptr<apache::thrift::ResponseChannelRequest, apache::thrift::RequestsRegistry::Deleter>&&, folly::detail::function::Data&) ()
No symbol table info available.
#15 0x0000000001c330e2 in apache::thrift::EventTask::run() ()
No symbol table info available.
#16 0x0000000001c38627 in virtual thunk to apache::thrift::concurrency::FunctionRunner::run() ()
No symbol table info available.
#17 0x0000000001d6a3f3 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
No symbol table info available.
#18 0x0000000001d6e04d in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
No symbol table info available.
#19 0x00007ff200180e25 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#20 0x00007ff1ffead35d in clone () from /lib64/libc.so.6
No symbol table info available.

critical27 · 2021 年10 月 21 日 04:38

写了啥类型的数据，schema是啥，日志里有一个FATAL日志，也贴一下

111 · 2021 年10 月 21 日 05:10

Log file created at: 2021/10/21 11:15:58
Running on machine: 9-146-152-196
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
F1021 11:15:58.618381 3679769 RowWriterV2.cpp:858] Unsupported default value type: EMPTY, default value: EMPTY, default value expr: EMPTY

111 · 2021 年10 月 21 日 05:13

critical27 · 2021 年10 月 21 日 06:03

另外方便的话可以发下插入的query是啥样的暂时还没看出来为啥会挂

插入语句如果所有字段都指定是不会出现现在这个问题

111 · 2021 年10 月 21 日 06:22

一直在持续写入，不知道具体是什么语句导致的。
插入userid语句没有指定所有字段。创建tag userid的语句是
CREATE TAG userid(attr string, flush_num string DEFAULT ‘0’)
使用中有如下几种语句：
INSERT VERTEX IF NOT EXISTS userid () VALUES ‘abcd’:()
UPDATE VERTEX ON userid SET attr = ‘qwer’
UPDATE VERTEX ON userid SET flush_num = ‘0’

critical27 · 2021 年10 月 21 日 11:06

我这按
CREATE TAG userid(attr string, flush_num string DEFAULT ‘0’)
CREATE TAG newsid(attr string, t timestamp)
和你贴的query语句没复现出来

挂的那个地方很奇怪说是某个字段的默认值是EMPTY

111 · 2021 年10 月 22 日 03:33

无法复现，这个log说有字段empty的问题，我就先把userid的attr改成 string DEFAULT ‘‘了。然后INSERT 语句也填上字段值’’，目前暂时没继续出错

system · 2021 年11 月 21 日 03:33

此话题已在最后回复的 30 天后被自动关闭。不再允许新回复。