Rocksdb compaction的时候会发生死锁

问答 问题
1

提问参考模版：

nebula 版本：v2.0 ga
部署方式（分布式 / 单机 / Docker / DBaaS）： 分布式
是否为线上版本：Y
硬件信息

update configs开启自动compaction：

update configs STORAGE:rocksdb_column_family_options={disable_auto_compactions: false, max_bytes_for_level_base: 268435456, max_write_buffer_number: 4, write_buffer_size: 67108864}

发现rocksdb死锁，所有的write rocksdb的操作会hung住：

Thread 30 (Thread 0x7f5d15dfd700 (LWP 2340)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000000003eff10d in rocksdb::port::CondVar::Wait() ()
#2  0x0000000003e1744d in rocksdb::WriteThread::JoinBatchGroup(rocksdb::WriteThread::Writer*) ()
#3  0x0000000003d34e24 in rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsign
ed long, bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*) ()
#4  0x0000000003d368e2 in rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*) ()
#5  0x0000000002f1a7cf in nebula::kvstore::RocksEngine::commitBatchWrite (this=0x7f5d20a6c600, batch=..., disableWAL=true, sync=false)
    at /home/guojun/source/bigo/dfs_nebula-graph/modules/storage/src/kvstore/RocksEngine.cpp:141
#6  0x0000000002eff76d in nebula::kvstore::Part::commitLogs (this=0x7f5d01403810, iter=...)
    at /home/guojun/source/bigo/dfs_nebula-graph/modules/storage/src/kvstore/Part.cpp:324
#7  0x0000000002f89050 in nebula::raftex::RaftPart::processAppendLogRequest (this=0x7f5d01403810, req=..., resp=...)
    at /home/guojun/source/bigo/dfs_nebula-graph/modules/storage/src/kvstore/raftex/RaftPart.cpp:1702
#8  0x0000000002fbc638 in nebula::raftex::RaftexService::appendLog (this=0x7f5d20a2d110, resp=..., req=...)
    at /home/guojun/source/bigo/dfs_nebula-graph/modules/storage/src/kvstore/raftex/RaftexService.cpp:217
#9  0x0000000003259850 in nebula::raftex::cpp2::RaftexServiceSvIf::<lambda(nebula::raftex::cpp2::AppendLogResponse&)>::operator()(nebula::raftex::cpp2::
AppendLogResponse &) const (__closure=0x7f5d15dfa8c0, _return=...)
    at /home/guojun/source/bigo/dfs_nebula-graph/build/modules/common/src/common/interface/gen-cpp2/RaftexService.cpp:36
#10 0x0000000003261179 in apache::thrift::detail::si::returning<nebula::raftex::cpp2::RaftexServiceSvIf::future_appendLog(const nebula::raftex::cpp2::Ap
pendLogRequest&)::<lambda(nebula::raftex::cpp2::AppendLogResponse&)> >(<unknown type in /usr/local/nebula/bin/nebula-storaged, CU 0x4195c14, DIE 0x4270d
a4>) (f=<unknown type in /usr/local/nebula/bin/nebula-storaged, CU 0x4195c14, DIE 0x4270da4>)
    at /opt/vesoft/third-party/include/thrift/lib/cpp2/GeneratedCodeHelper.h:1159
#11 0x000000000325ba31 in apache::thrift::detail::si::<lambda()>::operator()(void) const (this=0x7f5d15dfa888)
    at /opt/vesoft/third-party/include/thrift/lib/cpp2/GeneratedCodeHelper.h:1167
#12 0x0000000003266a99 in folly::<lambda()>::operator()(void) (this=0x7f5d15dfa838) at /opt/vesoft/third-party/include/folly/futures/Future-inl.h:1342
#13 0x000000000326aa93 in folly::makeTryWith<folly::makeFutureWith(F&&) [with F = apache::thrift::detail::si::future_returning(F&&) [with F = nebula::ra
ftex::cpp2::RaftexServiceSvIf::future_appendLog(const nebula::raftex::cpp2::AppendLogRequest&)::<lambda(nebula::raftex::cpp2::AppendLogResponse&)>]::<la
mbda()>]::<lambda()> >(<unknown type in /usr/local/nebula/bin/nebula-storaged, CU 0x4195c14, DIE 0x4263513>) (
2

看了下rocksdb的代码，如果compaction一直不完成的话，确实会一直等待。这个有什么解决办法吗？

// Don't wait if there's a background error, even if its a soft error. We
// might wait here indefinitely as the background compaction may never
// finish successfully, resulting in the stall condition lasting
// indefinitely
while (error_handler_.GetBGError().ok() && write_controller_.IsStopped()) {
  if (write_options.no_slowdown) {
    return Status::Incomplete("Write stall");
  }
  delayed = true;

  // Notify write_thread_ about the stall so it can setup a barrier and
  // fail any pending writers with no_slowdown
  write_thread_.BeginWriteStall();
  TEST_SYNC_POINT("DBImpl::DelayWrite:Wait");
  bg_cv_.Wait();
  write_thread_.EndWriteStall();
}
3

不想卡住的话 参考https://github.com/facebook/rocksdb/wiki/Write-Stalls 改下相关参数

1 个赞
4

修改参数确实可以解决：
update configs STORAGE:rocksdb_column_family_options={disable_auto_compactions: false, max_bytes_for_level_base: 268435456, max_write_buffer_number: 4, write_buffer_size: 67108864, level0_file_num_compaction_trigger: 2000, soft_pending_compaction_bytes_limit: 5596752720043, hard_pending_compaction_bytes_limit : 6696752720043}

5

该话题在最后一个回复创建后30天后自动关闭。不再允许新的回复。