metad服务高可用问题

问答 问题
1

我在之前用3.0.0版本做metad服务的高可用测试的时候遇到，一台metad宕机不恢复的话，会出现过几分钟以后整个metad集群不可用的问题，可以见这一个帖子：

当时怀疑可能是3.0.0版本有raft的问题，以及可能宕机那一台不启动2台运行出错很容易继续报错导致集群不可用。
所以我后续就升级到了3.1.0版本，但是测试还是以前的现象，宕机几分钟后不可以用。
然后我又在这个基础上把metad服务5台部署，但是这个时候宕机一台metad不启动，剩下四台metad运行，还是会出现重新选举出leader以后几分钟metad集群不可用的问题，如果把宕机的那台恢复起来，那么集群又会恢复正常，不可用期间也没有什么报错，metad的leader还是打印正常接收心跳的日志。
感觉这个metad的高可用是不是存在什么问题，请大神帮忙分析下。

2 个赞
2

麻烦把Meta的LOG和config发一下

3

这是leader宕机后，被选为leader的meta日志

I20220725 16:21:59.537278 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:21:59.591295 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:00.524657 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:00.540706 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:00.545104 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:01.105242 35899 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220725 16:22:01.521961 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:01.541182 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:01.559562 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:02.481869 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:02.497274 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:02.497910 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:02.499619 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:02.501617 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:02.520973 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:02.536698 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:02.543053 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:02.564285 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:02.576593 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:02.584111 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:02.584229 35900 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:03.212482 35900 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220725 16:22:03.510136 35900 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:03.557230 35900 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:03.566668 35900 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:03.587493 35900 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:03.593374 35900 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:04.499981 35900 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:04.506630 35900 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:04.558667 35900 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:04.568188 35900 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:04.594892 35900 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:05.264606 35900 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220725 16:22:05.264652 35900 HBProcessor.cpp:40] Machine "172.17.141.118":9779 is not registed
I20220725 16:22:05.501238 35900 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:05.501308 35900 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:22:05.501684 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:54:56.611974 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:54:56.664724 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:54:56.669981 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:54:56.707473 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:54:56.724134 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:54:56.743085 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:54:56.777791 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:54:57.627602 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:54:57.649260 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:54:57.687135 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:54:57.688445 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:54:57.695662 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:54:57.752501 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:54:58.610741 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:54:58.611057 35899 SessionManagerProcessor.cpp:165] Session id `1658739548614253' not found
I20220725 16:54:58.611232 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:54:58.696204 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:54:58.710554 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:54:59.419565 35899 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220725 16:54:59.647428 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:54:59.647557 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:54:59.647630 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:54:59.668568 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:54:59.709892 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:54:59.716383 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:54:59.727981 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:54:59.781507 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:54:59.781884 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:54:59.787315 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:54:59.787716 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:54:59.794054 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:55:00.313145 35899 SessionManagerProcessor.cpp:56] Session id '1658739255384860' not found
I20220725 16:55:00.421262 35899 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220725 16:55:00.645769 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:55:00.678966 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:55:00.681766 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:55:00.719975 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:55:00.745143 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:55:01.637696 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:55:01.646111 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:55:01.657822 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:55:01.682483 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:55:01.722852 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:55:01.758858 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:55:01.784348 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:55:01.832651 35899 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220725 16:55:10.291703 35871 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9669, role = GRAPH
I20220725 16:55:10.393400 35864 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220725 16:55:10.432087 35877 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220725 16:55:12.262527 35877 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220725 16:55:12.376930 35864 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
I20220725 16:55:14.859941 35893 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220725 16:55:20.303342 35877 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9669, role = GRAPH
I20220725 16:55:20.405483 35871 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220725 16:55:20.443569 35893 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220725 16:55:22.268245 35871 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220725 16:55:22.389254 35886 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
I20220725 16:55:24.872553 35864 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220725 16:55:27.308903 35893 SessionManagerProcessor.cpp:56] Session id '1658739328472242' not found
I20220725 16:55:30.315764 35884 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9669, role = GRAPH
I20220725 16:55:30.417281 35877 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220725 16:55:30.445289 35877 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220725 16:55:32.275889 35884 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220725 16:55:32.401593 35884 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
I20220725 16:55:34.885471 35880 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220725 16:55:40.327484 35880 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9669, role = GRAPH
I20220725 16:55:40.422817 35884 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220725 16:55:40.457468 35880 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220725 16:55:42.284366 35871 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220725 16:55:42.413880 35871 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
4

meta的配置

########## basics ##########
# Whether to run as a daemon process
--daemonize=true
# The file to host the process id
--pid_file=pids/nebula-metad.pid

########## logging ##########
# The directory to host logging files
--log_dir=logs
# Log level, 0, 1, 2, 3 for INFO, WARNING, ERROR, FATAL respectively
--minloglevel=0
# Verbose log level, 1, 2, 3, 4, the higher of the level, the more verbose of the logging
--v=0
# Maximum seconds to buffer the log messages
--logbufsecs=0
# Whether to redirect stdout and stderr to separate output files
--redirect_stdout=true
# Destination filename of stdout and stderr, which will also reside in log_dir.
--stdout_log_file=metad-stdout.log
--stderr_log_file=metad-stderr.log
# Copy log messages at or above this level to stderr in addition to logfiles. The numbers of severity levels INFO, WARNING, ERROR, and FATAL are 0, 1, 2, and 3, respectively.
--stderrthreshold=2
# wether logging files' name contain time stamp, If Using logrotate to rotate logging files, than should set it to true.
--timestamp_in_logfile_name=true

########## networking ##########
# Comma separated Meta Server addresses
--meta_server_addrs=172.17.141.116:9559,172.17.141.117:9559,172.17.141.118:9559,10.200.90.67:9559,10.200.90.114:9559
# Local IP used to identify the nebula-metad process.
# Change it to an address other than loopback if the service is distributed or
# will be accessed remotely.
--local_ip=172.17.141.116
# Meta daemon listening port
--port=9559
# HTTP service ip
--ws_ip=0.0.0.0
# HTTP service port
--ws_http_port=19559
# Port to listen on Storage with HTTP protocol, it corresponds to ws_http_port in storage's configuration file
--ws_storage_http_port=19779

########## storage ##########
# Root data path, here should be only single path for metad
--data_path=data/meta

########## Misc #########
# The default number of parts when a space is created
--default_parts_num=100
# The default replica factor when a space is created
--default_replica_factor=1

--heartbeat_interval_secs=10
--agent_heartbeat_interval_secs=60
5

确定吗？这个E_LEADER_CHANGED明显是在说它不是Leader

6

我刚才又重试了一次，五台metad组成的集群，可以看到18:42分我killI掉原来的主metad后，当时成功率波动了一下，日志也可以看出来当时leader change了，但是18:46分突然错误率就百分之100了，日志看不出异常

image
image1373×549 56.8 KB

几台metad的日志是这样的,新的主应该是metad4，几台机器时钟没有校准，但是metad4和图中时间是一致的。

metad1：

Log file created at: 2022/07/28 18:28:47
Running on machine: localhost.localdomain
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20220728 18:28:47.447583 19170 MetaDaemon.cpp:135] localhost = "172.17.141.117":9559
I20220728 18:28:47.496351 19170 NebulaStore.cpp:51] Start the raft service...
I20220728 18:28:47.499578 19170 NebulaSnapshotManager.cpp:25] Send snapshot is rate limited to 10485760 for each part by default
I20220728 18:28:47.560793 19170 RaftexService.cpp:46] Start raft service on 9560
I20220728 18:28:47.561079 19170 NebulaStore.cpp:85] Scan the local path, and init the spaces_
I20220728 18:28:47.561190 19170 NebulaStore.cpp:92] Scan path "/home/nebula/install/nebula-graph-3.1.0.el7.x86_64/data/meta/nebula/0"
I20220728 18:28:47.561220 19170 NebulaStore.cpp:271] Init data from partManager for "172.17.141.117":9559
I20220728 18:28:47.561269 19170 NebulaStore.cpp:387] Create data space 0
I20220728 18:28:47.591284 19170 RocksEngine.cpp:97] open rocksdb on /home/nebula/install/nebula-graph-3.1.0.el7.x86_64/data/meta/nebula/0/data
I20220728 18:28:47.780952 19170 NebulaStore.cpp:459] Space 0, part 0 has been added, asLearner 0
I20220728 18:28:47.780995 19170 NebulaStore.cpp:78] Register handler...
I20220728 18:28:47.781004 19170 MetaDaemonInit.cpp:101] Waiting for the leader elected...
I20220728 18:28:47.781014 19170 MetaDaemonInit.cpp:113] Leader has not been elected, sleep 1s
I20220728 18:28:48.781400 19170 MetaDaemonInit.cpp:140] Get meta version is 3
I20220728 18:28:48.781437 19170 MetaDaemonInit.cpp:157] Nebula store init succeeded, clusterId 8219658889359283833
I20220728 18:28:48.781452 19170 MetaDaemon.cpp:148] Start http service
I20220728 18:28:48.782518 19170 MetaDaemonInit.cpp:162] Starting Meta HTTP Service
I20220728 18:28:48.795019 19392 WebService.cpp:124] Web service started on HTTP[19559]
I20220728 18:28:48.795164 19170 JobManager.cpp:69] Not leader, skip reading remaining jobs
I20220728 18:28:48.796056 19170 JobManager.cpp:56] JobManager initialized
I20220728 18:28:48.796113 19398 JobManager.cpp:119] JobManager::scheduleThread enter
I20220728 18:28:48.802747 19170 MetaDaemon.cpp:213] The meta daemon start on "172.17.141.117":9559
I20220728 18:28:55.747956 19444 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220728 18:28:55.748066 19444 HBProcessor.cpp:40] Machine "172.17.141.116":9779 is not registed
I20220728 18:32:13.795495 19444 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:13.835883 19444 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:14.810793 19444 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:14.813948 19444 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:15.746635 19444 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:15.788278 19444 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:15.822481 19444 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:15.860555 19444 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:16.767855 19444 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:16.787717 19444 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:16.811937 19444 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:16.812008 19444 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:16.813032 19443 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:16.818012 19443 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:16.836871 19443 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:16.836988 19443 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:16.841684 19443 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:17.764895 19443 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:17.766693 19443 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:17.769488 19443 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:17.790755 19443 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:17.809702 19443 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:17.830999 19443 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:18.765919 19443 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:18.769357 19443 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:18.792172 19443 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:18.812530 19444 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:18.812593 19443 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:18.829912 19443 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:18.855652 19443 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:18.857617 19443 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:32:19.668825 19443 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220728 18:32:19.762288 19443 SessionManagerProcessor.cpp:165] Session id `1659006807367960' not found

meta2：

Log file created at: 2022/07/28 18:39:48
Running on machine: vm-10-200-90-114
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20220728 18:39:48.550541 20048 MetaDaemon.cpp:135] localhost = "10.200.90.114":9559
I20220728 18:39:48.551421 20049 PosixThreadFactory.cpp:192] setpriority failed (are you root?) with error 13: Permission denied
I20220728 18:39:48.552479 20050 PosixThreadFactory.cpp:192] setpriority failed (are you root?) with error 13: Permission denied
I20220728 18:39:48.554711 20053 PosixThreadFactory.cpp:192] setpriority failed (are you root?) with error 13: Permission denied
I20220728 18:39:48.555203 20054 PosixThreadFactory.cpp:192] setpriority failed (are you root?) with error 13: Permission denied
I20220728 18:39:48.555729 20059 PosixThreadFactory.cpp:192] setpriority failed (are you root?) with error 13: Permission denied
I20220728 18:39:48.556727 20060 PosixThreadFactory.cpp:192] setpriority failed (are you root?) with error 13: Permission denied
I20220728 18:39:48.564836 20048 NebulaStore.cpp:51] Start the raft service...
I20220728 18:39:48.565176 20048 NebulaSnapshotManager.cpp:25] Send snapshot is rate limited to 10485760 for each part by default
I20220728 18:39:48.565901 20048 ThriftServer.cpp:366] libevent 2.1.12-stable method epoll
I20220728 18:39:48.568706 20048 RaftexService.cpp:46] Start raft service on 9560
I20220728 18:39:48.568771 20048 NebulaStore.cpp:85] Scan the local path, and init the spaces_
I20220728 18:39:48.568804 20048 NebulaStore.cpp:92] Scan path "/home/uc_appas/install/nebula-graph-3.1.0.el7.x86_64/data/meta/nebula/0"
I20220728 18:39:48.568811 20048 NebulaStore.cpp:271] Init data from partManager for "10.200.90.114":9559
I20220728 18:39:48.568825 20048 NebulaStore.cpp:387] Create data space 0
I20220728 18:39:48.584308 20122 EventListener.h:21] Rocksdb start compaction column family: default because of LevelL0FilesNum, status: OK, compacted 5 files into 0, base level is 0, output level is 1
I20220728 18:39:48.584489 20048 RocksEngine.cpp:97] open rocksdb on /home/uc_appas/install/nebula-graph-3.1.0.el7.x86_64/data/meta/nebula/0/data
I20220728 18:39:48.617400 20122 EventListener.h:35] Rocksdb compaction completed column family: default because of LevelL0FilesNum, status: OK, compacted 5 files into 1, base level is 0, output level is 1
I20220728 18:39:48.671799 20048 RaftPart.cpp:422] [Port: 9560, Space: 0, Part: 0] There are 4 peer hosts, and total 5 copies. The quorum is 3, as learner 0, lastLogId 3012027, lastLogTerm 7, committedLogId 3012026, committedLogTerm 7, term 7
I20220728 18:39:48.671814 20048 RaftPart.cpp:429] [Port: 9560, Space: 0, Part: 0] Add peer "172.17.141.116":9560
I20220728 18:39:48.671823 20048 RaftPart.cpp:429] [Port: 9560, Space: 0, Part: 0] Add peer "172.17.141.117":9560
I20220728 18:39:48.671826 20048 RaftPart.cpp:429] [Port: 9560, Space: 0, Part: 0] Add peer "172.17.141.118":9560
I20220728 18:39:48.671830 20048 RaftPart.cpp:429] [Port: 9560, Space: 0, Part: 0] Add peer "10.200.90.67":9560
I20220728 18:39:48.672196 20048 NebulaStore.cpp:459] Space 0, part 0 has been added, asLearner 0
I20220728 18:39:48.672217 20048 NebulaStore.cpp:78] Register handler...
I20220728 18:39:48.672222 20048 MetaDaemonInit.cpp:101] Waiting for the leader elected...
I20220728 18:39:48.672230 20048 MetaDaemonInit.cpp:113] Leader has not been elected, sleep 1s
I20220728 18:39:49.044883 20095 RaftPart.cpp:1115] [Port: 9560, Space: 0, Part: 0] Start leader election, reason: lastMsgDur 450, term 7
I20220728 18:39:49.044919 20095 RaftPart.cpp:1261] [Port: 9560, Space: 0, Part: 0] Start leader election...
I20220728 18:39:49.044927 20095 RaftPart.cpp:1289] [Port: 9560, Space: 0, Part: 0] Sending out an election request (space = 0, part = 0, term = 8, lastLogId = 3012027, lastLogTerm = 7, candidateIP = 10.200.90.114, candidatePort = 9560), isPreVote = 1
I20220728 18:39:49.672341 20048 MetaDaemonInit.cpp:113] Leader has not been elected, sleep 1s
I20220728 18:39:50.597570 20091 RaftPart.cpp:1828] [Port: 9560, Space: 0, Part: 0] The current role is Follower. Will follow the new leader "172.17.141.118":9560 on term 8
I20220728 18:39:50.672570 20048 MetaDaemonInit.cpp:140] Get meta version is 3
I20220728 18:39:50.672590 20048 MetaDaemonInit.cpp:157] Nebula store init succeeded, clusterId 8219658889359283833
I20220728 18:39:50.672597 20048 MetaDaemon.cpp:148] Start http service
I20220728 18:39:50.672765 20048 MetaDaemonInit.cpp:162] Starting Meta HTTP Service
I20220728 18:39:50.673875 20144 WebService.cpp:124] Web service started on HTTP[19559]
I20220728 18:39:50.673933 20048 JobManager.cpp:69] Not leader, skip reading remaining jobs
I20220728 18:39:50.673986 20048 JobManager.cpp:56] JobManager initialized
I20220728 18:39:50.673995 20150 JobManager.cpp:119] JobManager::scheduleThread enter
I20220728 18:39:50.677088 20048 StatsManager.cpp:107] Registered stats num_heartbeats
I20220728 18:39:50.677178 20048 StatsManager.cpp:159] Registered histogram heartbeat_latency_us [bucketSize: 1000, min value: 0, max value: 20000]
I20220728 18:39:50.677184 20048 HBProcessor.h:26] Succeeded in initializing the HBCounters
I20220728 18:39:50.677193 20048 StatsManager.cpp:107] Registered stats num_agent_heartbeats
I20220728 18:39:50.677256 20048 StatsManager.cpp:159] Registered histogram agent_heartbeat_latency_us [bucketSize: 1000, min value: 0, max value: 20000]
I20220728 18:39:50.677260 20048 AgentHBProcessor.h:26] Succeeded in initializing the AgentHBCounters
I20220728 18:39:50.677263 20048 MetaDaemon.cpp:213] The meta daemon start on "10.200.90.114":9559
I20220728 18:39:50.677340 20048 ThriftServer.cpp:366] libevent 2.1.12-stable method epoll
I20220728 18:39:50.677515 20151 PosixThreadFactory.cpp:192] setpriority failed (are you root?) with error 13: Permission denied
I20220728 18:39:50.677557 20152 PosixThreadFactory.cpp:192] setpriority failed (are you root?) with error 13: Permission denied
I20220728 18:39:50.677600 20153 PosixThreadFactory.cpp:192] setpriority failed (are you root?) with error 13: Permission denied
I20220728 18:39:50.677632 20154 PosixThreadFactory.cpp:192] setpriority failed (are you root?) with error 13: Permission denied
I20220728 18:39:50.677685 20155 PosixThreadFactory.cpp:192] setpriority failed (are you root?) with error 13: Permission denied
I20220728 18:39:50.677752 20156 PosixThreadFactory.cpp:192] setpriority failed (are you root?) with error 13: Permission denied
I20220728 18:42:57.577656 20159 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:42:57.609035 20159 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:42:57.609592 20159 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:42:57.635215 20159 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:42:58.624420 20159 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:42:58.624473 20160 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:42:58.672060 20160 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:42:58.684394 20160 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:42:59.569967 20160 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:42:59.570699 20160 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:42:59.586678 20160 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:42:59.593523 20160 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:42:59.603679 20160 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:42:59.623036 20160 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:42:59.632143 20160 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:42:59.665585 20160 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:00.576879 20160 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:00.589663 20160 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:00.593113 20160 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:00.594604 20160 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:00.610172 20160 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:00.626823 20160 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:00.671814 20160 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:00.676103 20160 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:00.798130 20160 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:00.863040 20160 SessionManagerProcessor.cpp:56] Session id '1659001583242263' not found
I20220728 18:43:01.589637 20160 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:01.634562 20160 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:01.636073 20160 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:01.642019 20160 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:01.677575 20160 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:01.804684 20160 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:02.634482 20160 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:02.636919 20160 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:02.643862 20160 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:03.149328 20091 RaftPart.cpp:1433] [Port: 9560, Space: 0, Part: 0] Received a VOTING request: space = 0, partition = 0, candidateAddr = 10.200.90.67:9560, term = 9, lastLogId = 3047379, lastLogTerm = 8, isPreVote = 1
I20220728 18:43:03.149360 20091 RaftPart.cpp:1462] [Port: 9560, Space: 0, Part: 0] The partition currently is a Follower, lastLogId 3047379, lastLogTerm 8, committedLogId 3047378, term 8
I20220728 18:43:03.149374 20091 RaftPart.cpp:1541] [Port: 9560, Space: 0, Part: 0] The partition will vote for the candidate "10.200.90.67":9560, isPreVote = 1
I20220728 18:43:03.150432 20091 RaftPart.cpp:1433] [Port: 9560, Space: 0, Part: 0] Received a VOTING request: space = 0, partition = 0, candidateAddr = 10.200.90.67:9560, term = 9, lastLogId = 3047379, lastLogTerm = 8, isPreVote = 0
I20220728 18:43:03.150446 20091 RaftPart.cpp:1462] [Port: 9560, Space: 0, Part: 0] The partition currently is a Follower, lastLogId 3047379, lastLogTerm 8, committedLogId 3047378, term 8
I20220728 18:43:03.150451 20091 RaftPart.cpp:1541] [Port: 9560, Space: 0, Part: 0] The partition will vote for the candidate "10.200.90.67":9560, isPreVote = 0
I20220728 18:43:03.152009 20109 RaftPart.cpp:1828] [Port: 9560, Space: 0, Part: 0] The current role is Follower. Will follow the new leader "10.200.90.67":9560 on term 9
I20220728 18:43:07.032359 20160 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
I20220728 18:43:07.032402 20160 HBProcessor.cpp:40] Machine "172.17.141.117":9779 is not registed

meta3:

Log file created at: 2022/07/28 18:28:02
Running on machine: localhost.localdomain
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20220728 18:28:02.042500 26595 MetaDaemon.cpp:135] localhost = "172.17.141.116":9559
I20220728 18:28:02.054723 26595 NebulaStore.cpp:51] Start the raft service...
I20220728 18:28:02.055593 26595 NebulaSnapshotManager.cpp:25] Send snapshot is rate limited to 10485760 for each part by default
I20220728 18:28:02.097920 26595 RaftexService.cpp:46] Start raft service on 9560
I20220728 18:28:02.098073 26595 NebulaStore.cpp:85] Scan the local path, and init the spaces_
I20220728 18:28:02.098134 26595 NebulaStore.cpp:92] Scan path "/home/nebula/install/nebula-graph-3.1.0.el7.x86_64/data/meta/nebula/0"
I20220728 18:28:02.098153 26595 NebulaStore.cpp:271] Init data from partManager for "172.17.141.116":9559
I20220728 18:28:02.098183 26595 NebulaStore.cpp:387] Create data space 0
I20220728 18:28:02.118677 26595 RocksEngine.cpp:97] open rocksdb on /home/nebula/install/nebula-graph-3.1.0.el7.x86_64/data/meta/nebula/0/data
I20220728 18:28:02.323769 26595 NebulaStore.cpp:459] Space 0, part 0 has been added, asLearner 0
I20220728 18:28:02.323807 26595 NebulaStore.cpp:78] Register handler...
I20220728 18:28:02.323814 26595 MetaDaemonInit.cpp:101] Waiting for the leader elected...
I20220728 18:28:02.323823 26595 MetaDaemonInit.cpp:113] Leader has not been elected, sleep 1s
I20220728 18:28:03.323947 26595 MetaDaemonInit.cpp:113] Leader has not been elected, sleep 1s
I20220728 18:28:04.324111 26595 MetaDaemonInit.cpp:113] Leader has not been elected, sleep 1s
I20220728 18:28:05.324254 26595 MetaDaemonInit.cpp:113] Leader has not been elected, sleep 1s
I20220728 18:28:06.324404 26595 MetaDaemonInit.cpp:113] Leader has not been elected, sleep 1s
I20220728 18:28:07.324558 26595 MetaDaemonInit.cpp:113] Leader has not been elected, sleep 1s
I20220728 18:28:08.324918 26595 MetaDaemonInit.cpp:140] Get meta version is 3
I20220728 18:28:08.324954 26595 MetaDaemonInit.cpp:157] Nebula store init succeeded, clusterId 8219658889359283833
I20220728 18:28:08.324968 26595 MetaDaemon.cpp:148] Start http service
I20220728 18:28:08.325317 26595 MetaDaemonInit.cpp:162] Starting Meta HTTP Service
I20220728 18:28:08.328691 26827 WebService.cpp:124] Web service started on HTTP[19559]
I20220728 18:28:08.328830 26595 JobManager.cpp:69] Not leader, skip reading remaining jobs
I20220728 18:28:08.328959 26595 JobManager.cpp:56] JobManager initialized
I20220728 18:28:08.328977 26833 JobManager.cpp:119] JobManager::scheduleThread enter
I20220728 18:28:08.336094 26595 MetaDaemon.cpp:213] The meta daemon start on "172.17.141.116":9559
I20220728 18:28:15.139086 26879 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220728 18:28:15.139191 26879 HBProcessor.cpp:87] Update host "172.17.141.116":9669 dir info, root path: /home/nebula/install/nebula-graph-3.1.0.el7.x86_64, data path size: 0
I20220728 18:28:17.659394 26879 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220728 18:28:17.659456 26879 HBProcessor.cpp:87] Update host "172.17.141.118":9669 dir info, root path: /home/nebula/install/nebula-graph-3.1.0.el7.x86_64, data path size: 0
I20220728 18:31:33.216642 26879 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:31:33.245743 26879 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:31:33.260334 26879 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:31:33.283233 26879 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:31:33.288092 26879 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:31:34.184671 26879 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:31:34.185675 26879 GetSpaceProcessor.cpp:18] Get space Failed, SpaceName test error: E_LEADER_CHANGED
I20220728 18:31:34.185802 26879 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:31:34.219655 26879 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:31:34.284715 26879 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:31:34.285147 26879 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:31:35.242933 26879 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:31:35.253029 26879 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:31:36.187497 26879 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:31:36.188000 26879 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:31:36.222322 26879 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:31:36.231119 26879 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:31:36.240599 26879 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:31:36.248783 26879 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:31:36.248900 26879 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:31:36.253616 26879 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:31:36.305997 26879 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:31:37.216768 26879 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:31:37.283432 26879 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:31:37.480149 26879 SessionManagerProcessor.cpp:56] Session id '1659001583242263' not found
I20220728 18:31:38.122874 26879 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220728 18:31:38.122951 26879 HBProcessor.cpp:40] Machine "172.17.141.116":9779 is not registed
I20220728 18:31:38.210417 26879 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:31:38.211685 26879 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:31:38.252080 26879 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:31:38.284956 26879 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:31:39.065415 26879 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220728 18:31:39.065459 26879 HBProcessor.cpp:40] Machine "172.17.141.118":9779 is not registed
I20220728 18:31:40.124140 26879 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220728 18:31:40.124190 26879 HBProcessor.cpp:40] Machine "172.17.141.116":9779 is not registed

metad4:

Log file created at: 2022/07/28 18:39:41
Running on machine: vm-10-200-90-67
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20220728 18:39:41.199898  2106 MetaDaemon.cpp:135] localhost = "10.200.90.67":9559
I20220728 18:39:41.237147  2106 NebulaStore.cpp:51] Start the raft service...
I20220728 18:39:41.237545  2106 NebulaSnapshotManager.cpp:25] Send snapshot is rate limited to 10485760 for each part by default
I20220728 18:39:41.241037  2106 RaftexService.cpp:46] Start raft service on 9560
I20220728 18:39:41.241103  2106 NebulaStore.cpp:85] Scan the local path, and init the spaces_
I20220728 18:39:41.241138  2106 NebulaStore.cpp:92] Scan path "/home/cap_tidb/install/nebula-graph-3.1.0.el7.x86_64/data/meta/nebula/0"
I20220728 18:39:41.241145  2106 NebulaStore.cpp:271] Init data from partManager for "10.200.90.67":9559
I20220728 18:39:41.241159  2106 NebulaStore.cpp:387] Create data space 0
I20220728 18:39:41.257648  2180 EventListener.h:21] Rocksdb start compaction column family: default because of LevelL0FilesNum, status: OK, compacted 5 files into 0, base level is 0, output level is 1
I20220728 18:39:41.257845  2106 RocksEngine.cpp:97] open rocksdb on /home/cap_tidb/install/nebula-graph-3.1.0.el7.x86_64/data/meta/nebula/0/data
I20220728 18:39:41.305992  2180 EventListener.h:35] Rocksdb compaction completed column family: default because of LevelL0FilesNum, status: OK, compacted 5 files into 1, base level is 0, output level is 1
I20220728 18:39:41.348297  2106 NebulaStore.cpp:459] Space 0, part 0 has been added, asLearner 0
I20220728 18:39:41.348330  2106 NebulaStore.cpp:78] Register handler...
I20220728 18:39:41.348335  2106 MetaDaemonInit.cpp:101] Waiting for the leader elected...
I20220728 18:39:41.348341  2106 MetaDaemonInit.cpp:113] Leader has not been elected, sleep 1s
I20220728 18:39:42.348529  2106 MetaDaemonInit.cpp:140] Get meta version is 3
I20220728 18:39:42.348554  2106 MetaDaemonInit.cpp:157] Nebula store init succeeded, clusterId 8219658889359283833
I20220728 18:39:42.348562  2106 MetaDaemon.cpp:148] Start http service
I20220728 18:39:42.348733  2106 MetaDaemonInit.cpp:162] Starting Meta HTTP Service
I20220728 18:39:42.349809  2202 WebService.cpp:124] Web service started on HTTP[19559]
I20220728 18:39:42.349861  2106 JobManager.cpp:69] Not leader, skip reading remaining jobs
I20220728 18:39:42.349906  2106 JobManager.cpp:56] JobManager initialized
I20220728 18:39:42.349913  2208 JobManager.cpp:119] JobManager::scheduleThread enter
I20220728 18:39:42.353091  2106 MetaDaemon.cpp:213] The meta daemon start on "10.200.90.67":9559
I20220728 18:39:44.625006  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
I20220728 18:39:44.625061  2218 HBProcessor.cpp:40] Machine "172.17.141.117":9779 is not registed
I20220728 18:42:57.758019  2218 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:42:57.815436  2218 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:42:57.819687  2218 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:42:57.855662  2218 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:42:58.769405  2218 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:42:58.776320  2218 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:42:58.797281  2218 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:42:58.810030  2218 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:42:58.819370  2218 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:42:58.833786  2218 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:42:59.849952  2218 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:42:59.856837  2218 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:00.769449  2218 SessionManagerProcessor.cpp:165] Session id `1659006807367960' not found
I20220728 18:43:00.774585  2218 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:00.817797  2218 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:00.819005  2218 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:00.827730  2218 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:00.828171  2218 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:00.850142  2218 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:00.856781  2218 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:00.863032  2218 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:01.825320  2218 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:01.864058  2218 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:01.864862  2218 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:02.803141  2218 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:02.832173  2218 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:02.857163  2218 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:02.865293  2218 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:02.866415  2218 SessionManagerProcessor.cpp:16] User does not exist, errorCode: E_LEADER_CHANGED
I20220728 18:43:04.138579  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9669, role = GRAPH
I20220728 18:43:04.172430  2217 SessionManagerProcessor.cpp:56] Session id '1659004984171310' not found
I20220728 18:43:04.446739  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220728 18:43:04.673228  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220728 18:43:04.705524  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220728 18:43:08.220129  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
I20220728 18:43:14.145514  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9669, role = GRAPH
I20220728 18:43:14.450146  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220728 18:43:14.687127  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220728 18:43:14.706096  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220728 18:43:14.710201  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220728 18:43:18.234091  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
I20220728 18:43:24.152956  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9669, role = GRAPH
I20220728 18:43:24.453629  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220728 18:43:24.591027  2218 SessionManagerProcessor.cpp:56] Session id '1659005004679071' not found
I20220728 18:43:24.701164  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220728 18:43:24.712709  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220728 18:43:24.719484  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220728 18:43:28.247414  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
I20220728 18:43:34.165603  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9669, role = GRAPH
I20220728 18:43:34.457036  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220728 18:43:34.713713  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220728 18:43:34.725132  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220728 18:43:34.732820  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220728 18:43:38.260579  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
I20220728 18:43:44.174152  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9669, role = GRAPH
I20220728 18:43:44.469471  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220728 18:43:44.600823  2215 SessionManagerProcessor.cpp:56] Session id '1659005024698855' not found
I20220728 18:43:44.727001  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220728 18:43:44.728729  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220728 18:43:44.746327  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220728 18:43:48.273838  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
I20220728 18:43:54.177641  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9669, role = GRAPH
I20220728 18:43:54.473960  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220728 18:43:54.609480  2216 SessionManagerProcessor.cpp:56] Session id '1659005034711483' not found
I20220728 18:43:54.740180  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220728 18:43:54.740198  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220728 18:43:54.759573  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220728 18:43:58.287098  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
I20220728 18:44:04.189182  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9669, role = GRAPH
I20220728 18:44:04.252920  2217 SessionManagerProcessor.cpp:56] Session id '1659005044360495' not found
I20220728 18:44:04.485431  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220728 18:44:04.617712  2216 SessionManagerProcessor.cpp:56] Session id '1659005044724149' not found
I20220728 18:44:04.750733  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220728 18:44:04.755695  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220728 18:44:04.772936  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220728 18:44:08.300670  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
I20220728 18:44:14.199694  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9669, role = GRAPH
I20220728 18:44:14.493829  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220728 18:44:14.622117  2216 SessionManagerProcessor.cpp:56] Session id '1659005054733087' not found
I20220728 18:44:14.758337  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220728 18:44:14.769680  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220728 18:44:14.779146  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220728 18:44:18.313896  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
I20220728 18:44:24.212114  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9669, role = GRAPH
I20220728 18:44:24.501230  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220728 18:44:24.634171  2217 SessionManagerProcessor.cpp:56] Session id '1659005064750766' not found
I20220728 18:44:24.767771  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220728 18:44:24.783166  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220728 18:44:24.792565  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220728 18:44:28.327145  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
I20220728 18:44:34.215493  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9669, role = GRAPH
I20220728 18:44:34.512768  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220728 18:44:34.779280  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220728 18:44:34.796456  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220728 18:44:34.805281  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220728 18:44:38.340409  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
I20220728 18:44:44.229171  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9669, role = GRAPH
I20220728 18:44:44.525570  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220728 18:44:44.785801  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220728 18:44:44.809147  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220728 18:44:44.818727  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220728 18:44:48.353806  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
I20220728 18:44:54.234494  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9669, role = GRAPH
I20220728 18:44:54.537967  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220728 18:44:54.789247  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220728 18:44:54.822378  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220728 18:44:54.832134  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220728 18:44:58.367723  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
I20220728 18:45:04.236912  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9669, role = GRAPH
I20220728 18:45:04.334128  2217 SessionManagerProcessor.cpp:56] Session id '1659005104466739' not found
I20220728 18:45:04.550375  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220728 18:45:04.800784  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220728 18:45:04.835597  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220728 18:45:04.844491  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220728 18:45:08.381810  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
I20220728 18:45:14.242434  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9669, role = GRAPH
I20220728 18:45:14.563153  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220728 18:45:14.805328  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220728 18:45:14.848827  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220728 18:45:14.857100  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220728 18:45:18.395035  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
I20220728 18:45:24.254945  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9669, role = GRAPH
I20220728 18:45:24.567044  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220728 18:45:24.818382  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220728 18:45:24.862347  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220728 18:45:24.869792  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220728 18:45:28.409041  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
I20220728 18:45:34.258447  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9669, role = GRAPH
I20220728 18:45:34.573649  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220728 18:45:34.823279  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220728 18:45:34.875329  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220728 18:45:34.876461  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220728 18:45:38.422415  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
I20220728 18:45:44.262864  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9669, role = GRAPH
I20220728 18:45:44.583942  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220728 18:45:44.835779  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220728 18:45:44.887887  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220728 18:45:44.888974  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220728 18:45:48.435487  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
I20220728 18:45:54.270365  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9669, role = GRAPH
I20220728 18:45:54.312741  2216 SessionManagerProcessor.cpp:56] Session id '1659005154467526' not found
I20220728 18:45:54.587347  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220728 18:45:54.848326  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220728 18:45:54.898274  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220728 18:45:54.900763  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220728 18:45:58.448057  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
I20220728 18:46:04.274863  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9669, role = GRAPH
I20220728 18:46:04.592792  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220728 18:46:04.747087  2216 SessionManagerProcessor.cpp:56] Session id '1659005164905831' not found
I20220728 18:46:04.858858  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220728 18:46:04.901415  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220728 18:46:04.914050  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220728 18:46:08.460925  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
I20220728 18:46:14.287307  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9669, role = GRAPH
I20220728 18:46:14.596287  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220728 18:46:14.862354  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220728 18:46:14.914146  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220728 18:46:14.927176  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220728 18:46:18.474215  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
I20220728 18:46:24.298082  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9669, role = GRAPH
I20220728 18:46:24.599723  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220728 18:46:24.765720  2216 SessionManagerProcessor.cpp:56] Session id '1659005184932339' not found
I20220728 18:46:24.874663  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220728 18:46:24.927783  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220728 18:46:24.939889  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220728 18:46:28.488243  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
I20220728 18:46:34.310217  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9669, role = GRAPH
I20220728 18:46:34.346850  2217 SessionManagerProcessor.cpp:56] Session id '1659005194517666' not found
I20220728 18:46:34.612211  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220728 18:46:34.880297  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220728 18:46:34.940519  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220728 18:46:34.952567  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220728 18:46:41.702702  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
I20220728 18:46:48.752094  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9669, role = GRAPH
I20220728 18:46:49.032012  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220728 18:46:49.125145  2215 SessionManagerProcessor.cpp:56] Session id '1659005201884688' not found
I20220728 18:46:49.125293  2218 SessionManagerProcessor.cpp:56] Session id '1659005201134347' not found
I20220728 18:46:49.125335  2215 SessionManagerProcessor.cpp:56] Session id '1659005201507340' not found
I20220728 18:46:49.125344  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220728 18:46:49.218197  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220728 18:46:49.218216  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220728 18:46:56.852330  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
I20220728 18:47:06.276823  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9669, role = GRAPH
I20220728 18:47:06.745326  2215 SessionManagerProcessor.cpp:56] Session id '1659005208374529' not found
I20220728 18:47:06.745390  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220728 18:47:06.839319  2218 SessionManagerProcessor.cpp:56] Session id '1659005212133639' not found
I20220728 18:47:06.839372  2216 SessionManagerProcessor.cpp:56] Session id '1659005207721226' not found
I20220728 18:47:07.497363  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220728 18:47:07.591085  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220728 18:47:07.779369  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220728 18:47:19.090761  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
I20220728 18:47:27.319453  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9669, role = GRAPH
I20220728 18:47:27.792346  2218 SessionManagerProcessor.cpp:56] Session id '1659005224204055' not found
I20220728 18:47:29.487306  2218 SessionManagerProcessor.cpp:56] Session id '1659005223632901' not found
I20220728 18:47:29.775668  2218 SessionManagerProcessor.cpp:56] Session id '1659005224579637' not found
I20220728 18:47:30.529917  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220728 18:47:30.718500  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220728 18:47:30.907570  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220728 18:47:30.907593  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220728 18:47:43.151329  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
I20220728 18:47:56.098512  2217 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9669, role = GRAPH
I20220728 18:47:56.578595  2215 SessionManagerProcessor.cpp:56] Session id '1659005242889731' not found
I20220728 18:47:57.047039  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9669, role = GRAPH
I20220728 18:47:57.140120  2217 SessionManagerProcessor.cpp:56] Session id '1659005238219778' not found
I20220728 18:47:57.140193  2218 SessionManagerProcessor.cpp:56] Session id '1659005240041728' not found
I20220728 18:47:57.233681  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9779, role = STORAGE
I20220728 18:47:57.233729  2215 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.118":9779, role = STORAGE
I20220728 18:47:57.419958  2216 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.116":9669, role = GRAPH
I20220728 18:48:14.620054  2218 HBProcessor.cpp:33] Receive heartbeat from "172.17.141.117":9779, role = STORAGE
7

图里的可用性是meta的可用性吗？这个QPS掉零，延迟上涨，怀疑是哪里死锁了。用pstack看下是不是卡在了哪？

8

之前我们有发现meta 死锁的bug，但是是balance leader触发的。不知道和你这个是不是一回事

9

我这个请求是每次请求很回去新建session去metad做身份认证，我理解就是这个身份认证这个时候不可用了

10

我再集群请求出错的时候，pstack了一下meta的leader进程:

Thread 172 (Thread 0x7f4bd64a2700 (LWP 36268)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 171 (Thread 0x7f4bd5591700 (LWP 36269)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 170 (Thread 0x7f4bd45ff700 (LWP 36270)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 169 (Thread 0x7f4bd41ff700 (LWP 36271)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 168 (Thread 0x7f4bd3bff700 (LWP 36272)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 167 (Thread 0x7f4bd35ff700 (LWP 36273)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 166 (Thread 0x7f4bd2fff700 (LWP 36274)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 165 (Thread 0x7f4bd2cfe700 (LWP 36275)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 164 (Thread 0x7f4bd0bff700 (LWP 36276)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 163 (Thread 0x7f4bd08fe700 (LWP 36277)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 162 (Thread 0x7f4bd07fd700 (LWP 36278)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 161 (Thread 0x7f4bcfbff700 (LWP 36279)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 160 (Thread 0x7f4bcf1ff700 (LWP 36280)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 159 (Thread 0x7f4bce7ff700 (LWP 36281)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 158 (Thread 0x7f4bce4fe700 (LWP 36282)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 157 (Thread 0x7f4bce3fd700 (LWP 36283)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 156 (Thread 0x7f4bcdbff700 (LWP 36284)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 155 (Thread 0x7f4bcdafe700 (LWP 36285)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 154 (Thread 0x7f4bccdff700 (LWP 36286)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 153 (Thread 0x7f4bcc3ff700 (LWP 36287)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 152 (Thread 0x7f4bcc0fe700 (LWP 36288)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 151 (Thread 0x7f4bcbffd700 (LWP 36289)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 150 (Thread 0x7f4bcb5ff700 (LWP 36290)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 149 (Thread 0x7f4bcb1ff700 (LWP 36291)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 148 (Thread 0x7f4bcabff700 (LWP 36292)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 147 (Thread 0x7f4bca5ff700 (LWP 36293)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 146 (Thread 0x7f4bc9fff700 (LWP 36294)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 145 (Thread 0x7f4bc99ff700 (LWP 36295)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 144 (Thread 0x7f4bc98fe700 (LWP 36296)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 143 (Thread 0x7f4bc8dff700 (LWP 36297)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 142 (Thread 0x7f4bc87ff700 (LWP 36298)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 141 (Thread 0x7f4bc86fe700 (LWP 36299)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 140 (Thread 0x7f4bc75ff700 (LWP 36300)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 139 (Thread 0x7f4bc74fe700 (LWP 36301)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 138 (Thread 0x7f4bc73fd700 (LWP 36302)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 137 (Thread 0x7f4bc65ff700 (LWP 36303)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 136 (Thread 0x7f4bc64fe700 (LWP 36304)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f151a in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d995c in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x00000000021268d1 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 135 (Thread 0x7f4bc63fd700 (LWP 36305)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x00000000021268d1 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 134 (Thread 0x7f4bc5bff700 (LWP 36306)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 133 (Thread 0x7f4bc5afe700 (LWP 36307)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x0000000002122518 in bool folly::detail::LifoSemBase<folly::SaturatingSemaphore<true, std::atomic>, std::atomic>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#5  0x000000000211d08e in apache::thrift::concurrency::ThreadManager::Impl::waitOnTask() ()
#6  0x0000000002126714 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#7  0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#8  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 132 (Thread 0x7f4bc49ff700 (LWP 36308)):
#0  0x00007f4bd5691183 in epoll_wait () from /lib64/libc.so.6
#1  0x00000000024ba0dd in ?? ()
#2  0x00000000024b0418 in event_base_loop ()
#3  0x00000000019fcd6d in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()> const&), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::_Bind<void (nebula::thread::GenericWorker::*(nebula::thread::GenericWorker*))()> > > >::_M_run() ()
#4  0x00000000029b2090 in ?? ()
#5  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 131 (Thread 0x7f4bc41fe700 (LWP 36309)):
#0  0x00007f4bd5691183 in epoll_wait () from /lib64/libc.so.6
#1  0x00000000024ba0dd in ?? ()
#2  0x00000000024b0418 in event_base_loop ()
#3  0x00000000019fcd6d in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()> const&), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::_Bind<void (nebula::thread::GenericWorker::*(nebula::thread::GenericWorker*))()> > > >::_M_run() ()
#4  0x00000000029b2090 in ?? ()
#5  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 130 (Thread 0x7f4bc31ff700 (LWP 36310)):
#0  0x00007f4bd5691183 in epoll_wait () from /lib64/libc.so.6
#1  0x00000000024ba0dd in ?? ()
#2  0x00000000024b0418 in event_base_loop ()
#3  0x00000000019fcd6d in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()> const&), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::_Bind<void (nebula::thread::GenericWorker::*(nebula::thread::GenericWorker*))()> > > >::_M_run() ()
#4  0x00000000029b2090 in ?? ()
#5  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 129 (Thread 0x7f4bc25fe700 (LWP 36311)):
#0  0x00007f4bd5691183 in epoll_wait () from /lib64/libc.so.6
#1  0x00000000024ba0dd in ?? ()
#2  0x00000000024b0418 in event_base_loop ()
#3  0x00000000019fcd6d in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()> const&), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::_Bind<void (nebula::thread::GenericWorker::*(nebula::thread::GenericWorker*))()> > > >::_M_run() ()
#4  0x00000000029b2090 in ?? ()
#5  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 128 (Thread 0x7f4bc17ff700 (LWP 36312)):
#0  0x00007f4bd5691183 in epoll_wait () from /lib64/libc.so.6
#1  0x00000000024ba0dd in ?? ()
#2  0x00000000024b0418 in event_base_loop ()
#3  0x00000000019fcd6d in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()> const&), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::_Bind<void (nebula::thread::GenericWorker::*(nebula::thread::GenericWorker*))()> > > >::_M_run() ()
#4  0x00000000029b2090 in ?? ()
#5  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 127 (Thread 0x7f4bc07ff700 (LWP 36313)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x000000000239f619 in _ZZN5folly15observer_detail15ObserverManager14UpdatesManager12CurrentQueueC4EvENKUlvE_clEv ()
#5  0x00000000029b2090 in ?? ()
#6  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 126 (Thread 0x7f4bbfffe700 (LWP 36315)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x000000000239f619 in _ZZN5folly15observer_detail15ObserverManager14UpdatesManager12CurrentQueueC4EvENKUlvE_clEv ()
#5  0x00000000029b2090 in ?? ()
#6  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 125 (Thread 0x7f4bbf3ff700 (LWP 36316)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x000000000239f619 in _ZZN5folly15observer_detail15ObserverManager14UpdatesManager12CurrentQueueC4EvENKUlvE_clEv ()
#5  0x00000000029b2090 in ?? ()
#6  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 124 (Thread 0x7f4bbe9fe700 (LWP 36317)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x000000000239f619 in _ZZN5folly15observer_detail15ObserverManager14UpdatesManager12CurrentQueueC4EvENKUlvE_clEv ()
#5  0x00000000029b2090 in ?? ()
#6  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 123 (Thread 0x7f4bbd9ff700 (LWP 36318)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x00000000010f1529 in folly::detail::FutexResult folly::detail::futexWaitUntil<std::atomic<unsigned int>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, unsigned int) ()
#3  0x00000000011d9834 in bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&) ()
#4  0x00000000023a0804 in _ZZN5folly15observer_detail15ObserverManager14UpdatesManager9NextQueueC4EvENKUlvE_clEv ()
#5  0x00000000029b2090 in ?? ()
#6  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 122 (Thread 0x7f4bbc5ff700 (LWP 36319)):
#0  0x00007f4bd5691183 in epoll_wait () from /lib64/libc.so.6
#1  0x00000000024ba0dd in ?? ()
#2  0x00000000024b0418 in event_base_loop ()
#3  0x00000000023eebfe in folly::EventBase::loopBody(int, bool) ()
#4  0x00000000023ef4ee in folly::EventBase::loop() ()
#5  0x00000000023f2128 in folly::EventBase::loopForever() ()
#6  0x000000000237c279 in folly::IOThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) ()
#7  0x000000000238a7e7 in void folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) ()
#8  0x00000000029b2090 in ?? ()
#9  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 121 (Thread 0x7f4bbb7ff700 (LWP 36320)):
#0  0x00007f4bd5691183 in epoll_wait () from /lib64/libc.so.6
#1  0x00000000024ba0dd in ?? ()
#2  0x00000000024b0418 in event_base_loop ()
#3  0x00000000023eebfe in folly::EventBase::loopBody(int, bool) ()
#4  0x00000000023ef4ee in folly::EventBase::loop() ()
#5  0x00000000023f2128 in folly::EventBase::loopForever() ()
#6  0x000000000237c279 in folly::IOThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) ()
#7  0x000000000238a7e7 in void folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) ()
#8  0x00000000029b2090 in ?? ()
#9  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 120 (Thread 0x7f4bba9ff700 (LWP 36321)):
#0  0x00007f4bd5691183 in epoll_wait () from /lib64/libc.so.6
#1  0x00000000024ba0dd in ?? ()
#2  0x00000000024b0418 in event_base_loop ()
#3  0x00000000023eebfe in folly::EventBase::loopBody(int, bool) ()
#4  0x00000000023ef4ee in folly::EventBase::loop() ()
#5  0x00000000023f2128 in folly::EventBase::loopForever() ()
#6  0x000000000237c279 in folly::IOThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) ()
#7  0x000000000238a7e7 in void folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) ()
#8  0x00000000029b2090 in ?? ()
#9  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 119 (Thread 0x7f4bb9bff700 (LWP 36322)):
#0  0x00007f4bd5691183 in epoll_wait () from /lib64/libc.so.6
#1  0x00000000024ba0dd in ?? ()
#2  0x00000000024b0418 in event_base_loop ()
#3  0x00000000023eebfe in folly::EventBase::loopBody(int, bool) ()
#4  0x00000000023ef4ee in folly::EventBase::loop() ()
#5  0x00000000023f2128 in folly::EventBase::loopForever() ()
#6  0x000000000237c279 in folly::IOThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) ()
#7  0x000000000238a7e7 in void folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) ()
#8  0x00000000029b2090 in ?? ()
#9  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 118 (Thread 0x7f4bb8dff700 (LWP 36323)):
#0  0x00007f4bd5691183 in epoll_wait () from /lib64/libc.so.6
#1  0x00000000024ba0dd in ?? ()
#2  0x00000000024b0418 in event_base_loop ()
#3  0x00000000023eebfe in folly::EventBase::loopBody(int, bool) ()
#4  0x00000000023ef4ee in folly::EventBase::loop() ()
#5  0x00000000023f2128 in folly::EventBase::loopForever() ()
#6  0x000000000237c279 in folly::IOThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) ()
#7  0x000000000238a7e7 in void folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) ()
#8  0x00000000029b2090 in ?? ()
#9  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 117 (Thread 0x7f4bb7fff700 (LWP 36324)):
#0  0x00007f4bd5691183 in epoll_wait () from /lib64/libc.so.6
#1  0x00000000024ba0dd in ?? ()
#2  0x00000000024b0418 in event_base_loop ()
#3  0x00000000023eebfe in folly::EventBase::loopBody(int, bool) ()
#4  0x00000000023ef4ee in folly::EventBase::loop() ()
#5  0x00000000023f2128 in folly::EventBase::loopForever() ()
#6  0x000000000237c279 in folly::IOThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) ()
#7  0x000000000238a7e7 in void folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) ()
#8  0x00000000029b2090 in ?? ()
#9  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 116 (Thread 0x7f4bb71ff700 (LWP 36325)):
#0  0x00007f4bd596e063 in pread64 () from /lib64/libpthread.so.0
#1  0x0000000001b2ddee in nebula::wal::WalFileIterator::WalFileIterator(std::shared_ptr<nebula::wal::FileBasedWal>, long, long) ()
#2  0x0000000001b27a65 in nebula::wal::FileBasedWal::iterator(long, long) ()
#3  0x0000000001b19182 in nebula::raftex::Host::prepareAppendLogRequest() ()
#4  0x0000000001b19b00 in nebula::raftex::Host::appendLogs(folly::EventBase*, long, long, long, long, long) ()
#5  0x0000000001ae1a83 in ?? ()
#6  0x0000000001af2827 in ?? ()
#7  0x00000000023aa64c in ?? ()
#8  0x00000000023f4651 in bool folly::AtomicNotificationQueue<folly::Function<void ()> >::drive<folly::EventBase::FuncRunner&>(folly::EventBase::FuncRunner&) ()
#9  0x00000000023f567d in non-virtual thunk to folly::EventBaseAtomicNotificationQueue<folly::Function<void ()>, folly::EventBase::FuncRunner>::handlerReady(unsigned short) ()
#10 0x00000000024aff65 in ?? ()
#11 0x00000000024b05cf in event_base_loop ()
#12 0x00000000023eebfe in folly::EventBase::loopBody(int, bool) ()
#13 0x00000000023ef4ee in folly::EventBase::loop() ()
#14 0x00000000023f2128 in folly::EventBase::loopForever() ()
#15 0x000000000237c279 in folly::IOThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) ()
#16 0x000000000238a7e7 in void folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) ()
#17 0x00000000029b2090 in ?? ()
#18 0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#19 0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 115 (Thread 0x7f4bb63ff700 (LWP 36326)):
#0  0x00007f4bd5691183 in epoll_wait () from /lib64/libc.so.6
#1  0x00000000024ba0dd in ?? ()
#2  0x00000000024b0418 in event_base_loop ()
#3  0x00000000023eebfe in folly::EventBase::loopBody(int, bool) ()
#4  0x00000000023ef4ee in folly::EventBase::loop() ()
#5  0x00000000023f2128 in folly::EventBase::loopForever() ()
#6  0x000000000237c279 in folly::IOThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) ()
#7  0x000000000238a7e7 in void folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) ()
#8  0x00000000029b2090 in ?? ()
#9  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 114 (Thread 0x7f4bb55ff700 (LWP 36327)):
#0  0x00007f4bd5691183 in epoll_wait () from /lib64/libc.so.6
#1  0x00000000024ba0dd in ?? ()
#2  0x00000000024b0418 in event_base_loop ()
#3  0x00000000023eebfe in folly::EventBase::loopBody(int, bool) ()
#4  0x00000000023ef4ee in folly::EventBase::loop() ()
#5  0x00000000023f2128 in folly::EventBase::loopForever() ()
#6  0x000000000237c279 in folly::IOThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) ()
#7  0x000000000238a7e7 in void folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) ()
#8  0x00000000029b2090 in ?? ()
#9  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 113 (Thread 0x7f4bb47ff700 (LWP 36328)):
#0  0x00007f4bd5691183 in epoll_wait () from /lib64/libc.so.6
#1  0x00000000024ba0dd in ?? ()
#2  0x00000000024b0418 in event_base_loop ()
#3  0x00000000023eebfe in folly::EventBase::loopBody(int, bool) ()
#4  0x00000000023ef4ee in folly::EventBase::loop() ()
#5  0x00000000023f2128 in folly::EventBase::loopForever() ()
#6  0x000000000237c279 in folly::IOThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) ()
#7  0x000000000238a7e7 in void folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) ()
#8  0x00000000029b2090 in ?? ()
#9  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 112 (Thread 0x7f4bb39ff700 (LWP 36329)):
#0  0x00007f4bd5691183 in epoll_wait () from /lib64/libc.so.6
#1  0x00000000024ba0dd in ?? ()
#2  0x00000000024b0418 in event_base_loop ()
#3  0x00000000023eebfe in folly::EventBase::loopBody(int, bool) ()
#4  0x00000000023ef4ee in folly::EventBase::loop() ()
#5  0x00000000023f2128 in folly::EventBase::loopForever() ()
#6  0x000000000237c279 in folly::IOThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) ()
#7  0x000000000238a7e7 in void folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) ()
#8  0x00000000029b2090 in ?? ()
#9  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 111 (Thread 0x7f4bb2bff700 (LWP 36330)):
#0  0x00007f4bd5691183 in epoll_wait () from /lib64/libc.so.6
#1  0x00000000024ba0dd in ?? ()
#2  0x00000000024b0418 in event_base_loop ()
#3  0x00000000023eebfe in folly::EventBase::loopBody(int, bool) ()
#4  0x00000000023ef4ee in folly::EventBase::loop() ()
#5  0x00000000023f2128 in folly::EventBase::loopForever() ()
#6  0x000000000237c279 in folly::IOThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) ()
#7  0x000000000238a7e7 in void folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) ()
#8  0x00000000029b2090 in ?? ()
#9  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 110 (Thread 0x7f4bb1dff700 (LWP 36331)):
#0  0x00007f4bd5691183 in epoll_wait () from /lib64/libc.so.6
#1  0x00000000024ba0dd in ?? ()
#2  0x00000000024b0418 in event_base_loop ()
#3  0x00000000023eebfe in folly::EventBase::loopBody(int, bool) ()
#4  0x00000000023ef4ee in folly::EventBase::loop() ()
#5  0x00000000023f2128 in folly::EventBase::loopForever() ()
#6  0x000000000237c279 in folly::IOThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) ()
#7  0x000000000238a7e7 in void folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) ()
#8  0x00000000029b2090 in ?? ()
#9  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 109 (Thread 0x7f4bb0fff700 (LWP 36332)):
#0  0x00007f4bd5691183 in epoll_wait () from /lib64/libc.so.6
#1  0x00000000024ba0dd in ?? ()
#2  0x00000000024b0418 in event_base_loop ()
#3  0x00000000023eebfe in folly::EventBase::loopBody(int, bool) ()
#4  0x00000000023ef4ee in folly::EventBase::loop() ()
#5  0x00000000023f2128 in folly::EventBase::loopForever() ()
#6  0x000000000237c279 in folly::IOThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) ()
#7  0x000000000238a7e7 in void folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) ()
#8  0x00000000029b2090 in ?? ()
#9  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 108 (Thread 0x7f4bb01ff700 (LWP 36333)):
#0  0x00007f4bd5691183 in epoll_wait () from /lib64/libc.so.6
#1  0x00000000024ba0dd in ?? ()
#2  0x00000000024b0418 in event_base_loop ()
#3  0x00000000023eebfe in folly::EventBase::loopBody(int, bool) ()
#4  0x00000000023ef4ee in folly::EventBase::loop() ()
#5  0x00000000023f2128 in folly::EventBase::loopForever() ()
#6  0x000000000237c279 in folly::IOThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) ()
#7  0x000000000238a7e7 in void folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) ()
#8  0x00000000029b2090 in ?? ()
#9  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 107 (Thread 0x7f4baf3ff700 (LWP 36334)):
#0  0x00007f4bd5691183 in epoll_wait () from /lib64/libc.so.6
#1  0x00000000024ba0dd in ?? ()
#2  0x00000000024b0418 in event_base_loop ()
#3  0x00000000023eebfe in folly::EventBase::loopBody(int, bool) ()
#4  0x00000000023ef4ee in folly::EventBase::loop() ()
#5  0x00000000023f2128 in folly::EventBase::loopForever() ()
#6  0x000000000237c279 in folly::IOThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) ()
#7  0x000000000238a7e7 in void folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) ()
#8  0x00000000029b2090 in ?? ()
#9  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 106 (Thread 0x7f4bae2ff700 (LWP 36335)):
#0  0x00007f4bd5691183 in epoll_wait () from /lib64/libc.so.6
#1  0x00000000024ba0dd in ?? ()
#2  0x00000000024b0418 in event_base_loop ()
#3  0x00000000023eebfe in folly::EventBase::loopBody(int, bool) ()
#4  0x00000000023ef4ee in folly::EventBase::loop() ()
#5  0x00000000023f2128 in folly::EventBase::loopForever() ()
#6  0x000000000237c279 in folly::IOThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) ()
#7  0x000000000238a7e7 in void folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) ()
#8  0x00000000029b2090 in ?? ()
#9  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 105 (Thread 0x7f4bad5ff700 (LWP 36343)):
#0  0x00007f4bd596a995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000000294392c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
#2  0x0000000001f74b86 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) ()
#3  0x0000000001f74eee in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) ()
#4  0x00000000029b2090 in ?? ()
#5  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 104 (Thread 0x7f4bac9ff700 (LWP 36344)):
#0  0x00007f4bd596a995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000000294392c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
#2  0x0000000001f74b86 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) ()
#3  0x0000000001f74eee in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) ()
#4  0x00000000029b2090 in ?? ()
#5  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 103 (Thread 0x7f4ba3df4700 (LWP 36443)):
#0  0x00007f4bd596ad42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000001e98d84 in rocksdb::port::CondVar::TimedWait(unsigned long) ()
#2  0x0000000001ca7323 in rocksdb::Timer::Run() ()
#3  0x00000000029b2090 in ?? ()
#4  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 102 (Thread 0x7f4ba45f5700 (LWP 36445)):
#0  0x00007f4bd565756d in nanosleep () from /lib64/libc.so.6
#1  0x00007f4bd5657404 in sleep () from /lib64/libc.so.6
#2  0x00000000019f6452 in ?? ()
#3  0x00000000019f6261 in ?? ()
#4  0x00000000029b2090 in ?? ()
#5  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 101 (Thread 0x7f4ba4df6700 (LWP 36515)):
#0  0x00007f4bd5691183 in epoll_wait () from /lib64/libc.so.6
#1  0x00000000024ba0dd in ?? ()
#2  0x00000000024b0418 in event_base_loop ()
#3  0x00000000019fcd6d in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()> const&), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::_Bind<void (nebula::thread::GenericWorker::*(nebula::thread::GenericWorker*))()> > > >::_M_run() ()
#4  0x00000000029b2090 in ?? ()
#5  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 100 (Thread 0x7f4ba55f7700 (LWP 36516)):
#0  0x00007f4bd5691183 in epoll_wait () from /lib64/libc.so.6
#1  0x00000000024ba0dd in ?? ()
#2  0x00000000024b0418 in event_base_loop ()
#3  0x00000000019fcd6d in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()> const&), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::_Bind<void (nebula::thread::GenericWorker::*(nebula::thread::GenericWorker*))()> > > >::_M_run() ()
#4  0x00000000029b2090 in ?? ()
#5  0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 99 (Thread 0x7f4baa5ff700 (LWP 36517)):
#0  0x00007f4bd5691183 in epoll_wait () from /lib64/libc.so.6
#1  0x00000000024ba0dd in ?? ()
#2  0x00000000024b0418 in event_base_loop ()
#3  0x00000000019fcd6d in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()> const&), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::_Bind<void (nebula::thread::GenericWorker::*
11

上面有些太多了，有些放不下，我搜了一下关于meta的有些

Thread 85 (Thread 0x7f4bbbdfe700 (LWP 36537)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x0000000001138e5b in bool folly::SharedMutexImpl<false, void, std::atomic, folly::SharedMutexPolicyDefault>::lockExclusiveImpl<folly::SharedMutexImpl<false, void, std::atomic, folly::SharedMutexPolicyDefault>::WaitForever>(unsigned int, folly::SharedMutexImpl<false, void, std::atomic, folly::SharedMutexPolicyDefault>::WaitForever&) ()
#3  0x000000000124362f in nebula::meta::RemoveSessionProcessor::process(nebula::meta::cpp2::RemoveSessionReq const&) ()
#4  0x0000000001101f90 in nebula::meta::MetaServiceHandler::future_removeSession(nebula::meta::cpp2::RemoveSessionReq const&) ()
#5  0x00000000016eac08 in nebula::meta::cpp2::MetaServiceSvIf::async_tm_removeSession(std::unique_ptr<apache::thrift::HandlerCallback<nebula::meta::cpp2::ExecResp>, std::default_delete<apache::thrift::HandlerCallback<nebula::meta::cpp2::ExecResp> > >, nebula::meta::cpp2::RemoveSessionReq const&) ()
#6  0x00000000017858a3 in void nebula::meta::cpp2::MetaServiceAsyncProcessor::process_removeSession<apache::thrift::CompactProtocolReader, apache::thrift::CompactProtocolWriter>(std::unique_ptr<apache::thrift::ResponseChannelRequest, apache::thrift::RequestsRegistry::Deleter>, apache::thrift::SerializedCompressedRequest&&, apache::thrift::Cpp2RequestContext*, folly::EventBase*, apache::thrift::concurrency::ThreadManager*) ()
#7  0x00000000016f7563 in apache::thrift::RequestTask<nebula::meta::cpp2::MetaServiceAsyncProcessor>::run() ()
#8  0x0000000001fc87d7 in virtual thunk to apache::thrift::concurrency::FunctionRunner::run() ()
#9  0x00000000021267d8 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#10 0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#11 0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f4bd5690bad in clone () from /lib64/libc.so.6
Thread 84 (Thread 0x7f4bbaffe700 (LWP 36538)):
#0  0x00007f4bd568aec9 in syscall () from /lib64/libc.so.6
#1  0x0000000001eac3ee in folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int) ()
#2  0x0000000001138e5b in bool folly::SharedMutexImpl<false, void, std::atomic, folly::SharedMutexPolicyDefault>::lockExclusiveImpl<folly::SharedMutexImpl<false, void, std::atomic, folly::SharedMutexPolicyDefault>::WaitForever>(unsigned int, folly::SharedMutexImpl<false, void, std::atomic, folly::SharedMutexPolicyDefault>::WaitForever&) ()
#3  0x0000000001243901 in nebula::meta::CreateSessionProcessor::process(nebula::meta::cpp2::CreateSessionReq const&) ()
#4  0x0000000001102990 in nebula::meta::MetaServiceHandler::future_createSession(nebula::meta::cpp2::CreateSessionReq const&) ()
#5  0x00000000016f0bfb in nebula::meta::cpp2::MetaServiceSvIf::async_tm_createSession(std::unique_ptr<apache::thrift::HandlerCallback<nebula::meta::cpp2::CreateSessionResp>, std::default_delete<apache::thrift::HandlerCallback<nebula::meta::cpp2::CreateSessionResp> > >, nebula::meta::cpp2::CreateSessionReq const&) ()
#6  0x0000000001790558 in void nebula::meta::cpp2::MetaServiceAsyncProcessor::process_createSession<apache::thrift::CompactProtocolReader, apache::thrift::CompactProtocolWriter>(std::unique_ptr<apache::thrift::ResponseChannelRequest, apache::thrift::RequestsRegistry::Deleter>, apache::thrift::SerializedCompressedRequest&&, apache::thrift::Cpp2RequestContext*, folly::EventBase*, apache::thrift::concurrency::ThreadManager*) ()
#7  0x00000000016f7563 in apache::thrift::RequestTask<nebula::meta::cpp2::MetaServiceAsyncProcessor>::run() ()
#8  0x0000000001fc87d7 in virtual thunk to apache::thrift::concurrency::FunctionRunner::run() ()
#9  0x00000000021267d8 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() ()
#10 0x00000000021288de in apache::thrift::concurrency::PthreadThread::threadMain(void*) ()
#11 0x00007f4bd5966e25 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f4bd5690bad in clone () from /lib64/libc.so.6
12

@critical27 疑似Meta处理Session的时候死锁了。。。

13

论坛应该可以上传附件的 可以上传下 或者搜一下LeaderBalance之类的 把那个上传上来

这个bug在master已经修掉了 看这 https://github.com/vesoft-inc/nebula/pull/4416
要是源码编译的 改下就好了

14

我搜metad日志和pstack的结果，都搜不到LeaderBalance之类的。
目前测试发现，吐过getsession是低频率的就不会出现这种情况，比如20
tps，但是tps大了比如200tps就会出现metad不可用。
还有一个问题是，如果是死锁的话，为什么我每次一把宕机的那台重启起来，整个metad又可用了呢，这个能把死锁释放掉吗？

15

你把完整pstack上传下 我看看
另外确认下session用法，是不是每个query都新申请了一个session，这样可能是会把meta打爆。
日志里刷的User does not exist就是大量在创建session，（用户为啥不存在是另一个疑点）

16

pstack.txt (261.3 KB)
完整的pstack我上传了。
我是每一个查询都新申请了一个session，但是我查询完都会release了，我觉得不是meta被打爆，因为正常状态下，没有任意一台meta宕机的情况下，一直发请求是没有问题的。

17

我大概知道为啥了，你可以按我说的去观察下其他卡主的情况时候的pstack：

  1. 大量线程在等CreateSession/RemoveSession的锁
  2. 其中有一个线程拿到锁，执行逻辑，但是在等着拿folly::Baton（大概率也是CreateSession或者是RemoveSession的线程）

这里有些太具体细节了，原理估计一时半会说不清，简单来说就是大量的创建删除session，导致已经拿到锁的那个线程的某个回调被分发到另一个线程上，然而另一个线程又在等待锁，AKA，meta死锁了。

所以不要每个query都创建删除session，每次操作都会往meta发请求，且要求写入数据。

2 个赞
18

BTW，麻烦尽可能详细描述下出现不可用的场景，5副本meta，然后其他正常操作，tps一高就会出现？我们家里没复现出来

19

是这样的。
环境：我们搭建了一个集群，5个meta，3个graph。3个storage。
输入条件：一直有持续的tps请求访问集群，这里分为读写请求，各100tps，一共200tps，这里每一个请求都会去重新获取session、执行读\写命令、释放session（这里每次去重新获取session是因为我的目的就是去测试meta的可用性，因为获取session的身份验证是去访问meta的）。
测试过程：在上述持续请求的基础上，我会kill -9掉那一台meta的leader，并且这台meta不会重新起来。
测试结果：可以看到在kill -9 掉leader的时候，我的请求有一小部分失败了（这个时候meta重新选举leader），然后就这样过了几分钟左右，我的请求去getsession访问meta就一直超时，无法获得结果了。
整个过程可以看下面的图，蓝色的线是tps（这里只展示了写请求，只有100tps），红色是请求的错误率，绿色是请求耗时。
image

20

OK，我们试试，我们家里tps打到3000都没挂