服务器宕机后重启失败，9559起不来

问答 问题
1

  • nebula 版本：3.4.1

    image
    image939×33 2.14 KB

  • 部署方式：单机

  • 安装方式： RPM

  • 是否上生产环境： N

  • 硬件信息

    • 磁盘（ 推荐使用 SSD）190G
    • CPU、内存信息

  • 问题：启动以后 9559 的info日志报很多
    Check root user
    Leader changed, retry
    然后status看9559先 exited，9669随后也exited

  • 相关的 meta / storage / graph info 日志信息

meta info：
Log file created at: 2023/09/12 09:04:26
Running on machine: CM-Agent-204
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20230912 09:04:26.055395 22978 MetaDaemon.cpp:137] localhost = "10.20.31.204":9559
I20230912 09:04:26.057498 22978 NebulaStore.cpp:52] Start the raft service...
I20230912 09:04:26.058020 22978 NebulaSnapshotManager.cpp:25] Send snapshot is rate limited to 10485760 for each part by default
I20230912 09:04:26.081435 22978 RaftexService.cpp:46] Start raft service on 9560
I20230912 09:04:26.081501 22978 NebulaStore.cpp:86] Scan the local path, and init the spaces_
I20230912 09:04:26.081532 22978 NebulaStore.cpp:93] Scan path "/usr/local/nebula/data/meta/nebula/0"
I20230912 09:04:26.081540 22978 NebulaStore.cpp:272] Init data from partManager for "10.20.31.204":9559
I20230912 09:04:26.081555 22978 NebulaStore.cpp:388] Create data space 0
I20230912 09:04:26.097362 22978 RocksEngine.cpp:107] open rocksdb on /usr/local/nebula/data/meta/nebula/0/data
I20230912 09:04:26.097522 23062 EventListener.h:21] Rocksdb start compaction column family: default because of LevelL0FilesNum, status: OK, compacted 5 files into 0, base level is 0, output level is 1
I20230912 09:04:26.099763 23062 EventListener.h:35] Rocksdb compaction completed column family: default because of LevelL0FilesNum, status: OK, compacted 5 files into 1, base level is 0, output level is 1
I20230912 09:04:26.486076 22978 NebulaStore.cpp:451] Space 0, part 0 has been added, asLearner 0
I20230912 09:04:26.486114 22978 NebulaStore.cpp:79] Register handler...
I20230912 09:04:26.486120 22978 MetaDaemonInit.cpp:106] Waiting for the leader elected...
I20230912 09:04:26.486124 22978 MetaDaemonInit.cpp:118] Leader has not been elected, sleep 1s
I20230912 09:04:27.486359 22978 MetaDaemonInit.cpp:153] Get meta version is 4
I20230912 09:04:27.486413 22978 MetaDaemonInit.cpp:169] Nebula store init succeeded, clusterId 6621918225705809625
I20230912 09:04:27.486421 22978 MetaDaemon.cpp:150] Start http service
I20230912 09:04:27.488521 22978 MetaDaemonInit.cpp:226] Starting Meta HTTP Service
I20230912 09:04:27.491614 23160 WebService.cpp:124] Web service started on HTTP[19559]
I20230912 09:04:27.491668 22978 MetaDaemonInit.cpp:192] Check root user
I20230912 09:04:27.491688 22978 MetaDaemonInit.cpp:197] Leader changed, retry
I20230912 09:04:28.491865 22978 MetaDaemonInit.cpp:192] Check root user
I20230912 09:04:28.491926 22978 MetaDaemonInit.cpp:197] Leader changed, retry
I20230912 09:04:29.492066 22978 MetaDaemonInit.cpp:192] Check root user
I20230912 09:04:29.492132 22978 MetaDaemonInit.cpp:197] Leader changed, retry
I20230912 09:04:30.492240 22978 MetaDaemonInit.cpp:192] Check root user
I20230912 09:04:30.492282 22978 MetaDaemonInit.cpp:197] Leader changed, retry
I20230912 09:04:31.492424 22978 MetaDaemonInit.cpp:192] Check root user
I20230912 09:04:31.492458 22978 MetaDaemonInit.cpp:197] Leader changed, retry
I20230912 09:04:32.492565 22978 MetaDaemonInit.cpp:192] Check root user
I20230912 09:04:32.492630 22978 MetaDaemonInit.cpp:197] Leader changed, retry
I20230912 09:04:33.492811 22978 MetaDaemonInit.cpp:192] Check root user
I20230912 09:04:33.492872 22978 MetaDaemonInit.cpp:197] Leader changed, retry
I20230912 09:04:34.493069 22978 MetaDaemonInit.cpp:192] Check root user
I20230912 09:04:34.493135 22978 MetaDaemonInit.cpp:197] Leader changed, retry
I20230912 09:04:35.493343 22978 MetaDaemonInit.cpp:192] Check root user
I20230912 09:04:35.493413 22978 MetaDaemonInit.cpp:197] Leader changed, retry
I20230912 09:04:36.493568 22978 MetaDaemonInit.cpp:192] Check root user
I20230912 09:04:36.493638 22978 MetaDaemonInit.cpp:197] Leader changed, retry
I20230912 09:04:37.494330 22978 MetaDaemonInit.cpp:192] Check root user
I20230912 09:04:37.494362 22978 MetaDaemonInit.cpp:197] Leader changed, retry
I20230912 09:04:38.494457 22978 MetaDaemonInit.cpp:192] Check root user
I20230912 09:04:38.494520 22978 MetaDaemonInit.cpp:197] Leader changed, retry
I20230912 09:04:39.494711 22978 MetaDaemonInit.cpp:192] Check root user
I20230912 09:04:39.494789 22978 MetaDaemonInit.cpp:197] Leader changed, retry
I20230912 09:04:40.494894 22978 MetaDaemonInit.cpp:192] Check root user
I20230912 09:04:40.494953 22978 MetaDaemonInit.cpp:197] Leader changed, retry
I20230912 09:04:41.495055 22978 MetaDaemonInit.cpp:192] Check root user
I20230912 09:04:41.495100 22978 MetaDaemonInit.cpp:197] Leader changed, retry
E20230912 09:04:42.495193 22978 MetaDaemonInit.cpp:183] Retry too many times
E20230912 09:04:42.495730 22978 MetaDaemon.cpp:164] Init god user failed

storage info：
Log file created at: 2023/09/12 09:04:26
Running on machine: CM-Agent-204
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20230912 09:04:26.104765 23106 StorageDaemon.cpp:132] localhost = "10.20.31.204":9779
I20230912 09:04:26.105001 23106 StorageDaemon.cpp:147] data path= /usr/local/nebula/data/storage
I20230912 09:04:26.109006 23106 MetaClient.cpp:80] Create meta client to "10.20.31.203":9559
I20230912 09:04:26.109033 23106 MetaClient.cpp:81] root path: /usr/local/nebula, data path size: 1
I20230912 09:04:26.109066 23106 FileBasedClusterIdMan.cpp:53] Get clusterId: 6621918225705809625
E20230912 09:04:29.155211 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:04:29.155686 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:04:42.160069 23156 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:04:42.160138 23156 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:04:42.160213 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:04:42.160234 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:04:55.165664 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:04:55.165728 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:05:08.178999 23167 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:05:08.179047 23167 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:05:08.179081 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:05:08.179091 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:05:21.183712 23375 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:05:21.183794 23375 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:05:21.183862 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:05:21.183882 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:05:34.190141 23376 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:05:34.190220 23376 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:05:34.190289 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:05:34.190363 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:05:47.197489 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:05:47.197569 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:06:00.203886 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:06:00.203966 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:06:13.209089 23379 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:06:13.209143 23379 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:06:13.209280 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:06:13.209373 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:06:26.216223 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:06:26.216277 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:06:39.222749 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:06:39.222841 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:06:52.227967 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:06:52.228025 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:07:05.235167 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:07:05.235276 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:07:18.240352 23446 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:07:18.240417 23446 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:07:18.240473 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:07:18.240512 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:07:31.246208 23447 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:07:31.246306 23447 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:07:31.246443 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:07:31.246510 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:07:44.253199 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:07:44.253329 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:07:57.259002 23149 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:07:57.259102 23149 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:07:57.259189 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:07:57.259219 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:08:10.266389 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:08:10.266470 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:08:23.272836 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:08:23.272914 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:08:36.279938 23167 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:08:36.280019 23167 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:08:36.280092 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:08:36.280155 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:08:49.285214 23375 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:08:49.285300 23375 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:08:49.285413 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:08:49.285436 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:09:02.290500 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:09:02.290582 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:09:15.294709 23377 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:09:15.294793 23377 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:09:15.294863 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:09:15.294922 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:09:28.300267 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:09:28.300371 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:09:41.305871 23379 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:09:41.305970 23379 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:09:41.306126 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:09:41.306195 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:09:54.311831 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:09:54.311913 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:10:07.317898 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:10:07.317994 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:10:20.324018 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:10:20.324105 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:10:33.329013 23445 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:10:33.329063 23445 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:10:33.329097 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:10:33.329105 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:10:46.335716 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:10:46.335804 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:10:59.341948 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:10:59.342034 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:11:12.348593 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:11:12.348690 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:11:25.354254 23149 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:11:25.354395 23149 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:11:25.354494 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:11:25.354537 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:11:38.359341 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:11:38.359421 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:11:51.366108 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:11:51.366158 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:12:04.372299 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:12:04.372411 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:12:17.378261 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:12:17.378373 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:12:30.384402 23376 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:12:30.384479 23376 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:12:30.384547 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:12:30.384569 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:12:43.388967 23377 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:12:43.389052 23377 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:12:43.389122 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:12:43.389143 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:12:56.395103 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:12:56.395144 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:13:09.400586 23379 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:13:09.400656 23379 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:13:09.400785 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:13:09.400846 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:13:22.405606 23380 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:13:22.405690 23380 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:13:22.405849 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:13:22.405948 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:13:35.411734 23381 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:13:35.411819 23381 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:13:35.411958 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:13:35.411996 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:13:48.417340 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:13:48.417425 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:14:01.423847 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:14:01.423908 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:14:14.429908 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:14:14.429992 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:14:27.435760 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:14:27.435858 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:14:40.441808 23448 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:14:40.441920 23448 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:14:40.441988 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:14:40.442010 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:14:53.447578 23149 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:14:53.447661 23149 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:14:53.447810 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:14:53.447877 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:15:06.453357 23156 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:15:06.453444 23156 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:15:06.453511 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:15:06.453548 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:15:19.459647 23166 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:15:19.459745 23166 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:15:19.459895 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:15:19.459985 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:15:32.465992 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:15:32.466042 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:15:45.471011 23375 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:15:45.471065 23375 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:15:45.471170 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:15:45.471230 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:15:58.477212 23376 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:15:58.477358 23376 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:15:58.477461 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:15:58.477530 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:16:11.483191 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:16:11.483294 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:16:24.489718 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:16:24.489811 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:16:37.495182 23379 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:16:37.495227 23379 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:16:37.495362 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:16:37.495442 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:16:50.500367 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:16:50.500449 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:17:03.506392 23381 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:17:03.506492 23381 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:17:03.506562 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:17:03.506584 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:17:16.512049 23382 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:17:16.512105 23382 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:17:16.512157 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:17:16.512171 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:17:29.517570 23445 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:17:29.517650 23445 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:17:29.517729 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:17:29.517774 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:17:42.524127 23446 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:17:42.524210 23446 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:17:42.524358 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:17:42.524423 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:17:55.528977 23447 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:17:55.529039 23447 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:17:55.529186 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:17:55.529245 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:18:08.535020 23448 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:18:08.535122 23448 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:18:08.535233 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:18:08.535300 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:18:21.539719 23149 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:18:21.539803 23149 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:18:21.539870 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:18:21.539912 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:18:34.544780 23156 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:18:34.544860 23156 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:18:34.545014 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:18:34.545080 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:18:47.550556 23166 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:18:47.550607 23166 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:18:47.550652 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:18:47.550662 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:19:00.556926 23167 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:19:00.557045 23167 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:19:00.557124 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:19:00.557147 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:19:13.562637 23375 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:19:13.562717 23375 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:19:13.562851 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:19:13.562918 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:19:26.568889 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:19:26.568972 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:19:39.575341 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:19:39.575399 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:19:52.579209 23378 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:19:52.579294 23378 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:19:52.579392 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:19:52.579440 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:20:05.585009 23379 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:20:05.585120 23379 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:20:05.585211 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:20:05.585254 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:20:18.590724 23380 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:20:18.590809 23380 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:20:18.590896 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:20:18.590920 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:20:31.595887 23381 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:20:31.595971 23381 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:20:31.596038 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:20:31.596060 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:20:44.601519 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:20:44.601567 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:20:57.606375 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:20:57.606454 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:21:10.612951 23446 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:21:10.613061 23446 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:21:10.613133 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:21:10.613155 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:21:23.619081 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:21:23.619128 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:21:36.624737 23448 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:21:36.624819 23448 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused

。。。.....全是一样的。。。。。。。。。。。
E20230912 09:28:32.809252 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:28:32.809298 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:28:45.816128 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:28:45.816215 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:28:58.821180 23156 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:28:58.821285 23156 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:28:58.821419 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:28:58.821506 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:29:11.827423 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:29:11.827522 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:29:24.833802 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:29:24.833899 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:29:37.840389 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:29:37.840473 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:29:50.843909 23376 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:29:50.843962 23376 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:29:50.844015 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:29:50.844027 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:30:03.850299 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:30:03.850435 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:30:16.854466 23106 MetaClient.cpp:112] Heartbeat failed, status:Machine not existed!
I20230912 09:30:16.854522 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:30:29.860774 23379 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:30:29.860881 23379 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:30:29.860960 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:30:29.860985 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:30:42.866060 23380 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:30:42.866144 23380 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:30:42.866341 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:30:42.866415 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:30:55.872731 23381 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:30:55.872820 23381 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:30:55.872984 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:30:55.873057 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:31:08.878880 23382 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:31:08.878966 23382 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:31:08.879117 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:31:08.879216 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:31:21.884584 23445 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:31:21.884642 23445 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:31:21.884678 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:31:21.884689 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:31:34.890283 23446 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:31:34.890412 23446 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:31:34.890553 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:31:34.890581 23106 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:31:47.895947 23447 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:31:47.896037 23447 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:31:47.896124 23106 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:31:47.896147 23106 MetaClient.cpp:137] Waiting for the metad to be ready!

graphd info 
Log file created at: 2023/09/12 09:04:26
Running on machine: CM-Agent-204
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20230912 09:04:26.080802 23055 GraphDaemon.cpp:122] Starting Graph HTTP Service
I20230912 09:04:26.086426 23059 WebService.cpp:124] Web service started on HTTP[19669]
I20230912 09:04:26.086454 23055 GraphDaemon.cpp:136] Number of networking IO threads: 2
I20230912 09:04:26.086460 23055 GraphDaemon.cpp:145] Number of worker threads: 2
I20230912 09:04:26.087610 23055 MetaClient.cpp:80] Create meta client to "10.20.31.203":9559
I20230912 09:04:26.087633 23055 MetaClient.cpp:81] root path: /usr/local/nebula, data path size: 0
E20230912 09:04:29.107667 23105 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:04:29.108137 23105 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:04:29.108278 23055 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:04:29.108327 23055 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:04:42.114020 23055 MetaClient.cpp:112] Heartbeat failed, status:LeaderChanged: Leader changed!
I20230912 09:04:42.114100 23055 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:04:55.118016 23105 MetaClient.cpp:772] Send request to "10.20.31.204":9559, exceed retry limit
E20230912 09:04:55.118091 23105 MetaClient.cpp:773] RpcResponse exception: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E20230912 09:04:55.118155 23055 MetaClient.cpp:112] Heartbeat failed, status:RPC failure in MetaClient: apache::thrift::transport::TTransportException: Dropping unsent request. Connection closed after: apache::thrift::transport::TTransportException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connect
I20230912 09:04:55.118176 23055 MetaClient.cpp:137] Waiting for the metad to be ready!
E20230912 09:05:08.123874 23055 MetaClient.cpp:112] Heartbeat failed, status:LeaderChanged: Leader changed!
E20230912 09:05:11.160117 23055 MetaClient.cpp:157] LeaderChanged: Leader changed!
E20230912 09:05:11.160195 23055 GraphService.cpp:49] Failed to wait for meta service ready synchronously.
E20230912 09:05:11.160224 23055 GraphServer.cpp:39] Failed to wait for meta service ready synchronously.
E20230912 09:05:11.161458 23055 GraphDaemon.cpp:156] The graph server start failed
2

配置文件：

metad：
########## basics ##########
--daemonize=true
--pid_file=pids/nebula-metad.pid
--timezone_name=UTC+08:00
########## logging ##########
--log_dir=logs
--minloglevel=0
--v=0
--logbufsecs=0
--redirect_stdout=true
--stdout_log_file=metad-stdout.log
--stderr_log_file=metad-stderr.log
--stderrthreshold=3
--timestamp_in_logfile_name=true

########## networking ##########
--meta_server_addrs=10.20.31.203:9559,10.20.31.204:9559
--local_ip=10.20.31.204
--port=9559
--ws_ip=0.0.0.0
--ws_http_port=19559
--ws_storage_http_port=19779

########## storage ##########
--data_path=data/meta

########## Misc #########
--default_parts_num=100
--default_replica_factor=1

--heartbeat_interval_secs=10
--agent_heartbeat_interval_secs=60

···
storaged
···
--daemonize=true
--pid_file=pids/nebula-storaged.pid
--local_config=true
--timezone_name=UTC+08:00
--log_dir=logs
--minloglevel=0
--v=0
--logbufsecs=0
--redirect_stdout=true
--stdout_log_file=storaged-stdout.log
--stderr_log_file=storaged-stderr.log
--stderrthreshold=3
--timestamp_in_logfile_name=true

########## networking ##########
--meta_server_addrs=10.20.31.203:9559,10.20.31.204:9559
--local_ip=10.20.31.204
--port=9779
--ws_ip=0.0.0.0
--ws_http_port=19779
--heartbeat_interval_secs=10

######### Raft #########
--raft_heartbeat_interval_secs=30
--raft_rpc_timeout_ms=500
--wal_ttl=14400

########## Disk ##########
--data_path=data/storage

--minimum_reserved_bytes=268435456

--rocksdb_batch_size=4096
--rocksdb_block_cache=4
--engine_type=rocksdb

--rocksdb_compression=lz4
--rocksdb_compression_per_level=
--enable_rocksdb_statistics=false

--rocksdb_stats_level=kExceptHistogramOrTimers

--enable_rocksdb_prefix_filtering=true
--enable_rocksdb_whole_key_filtering=false

############## rocksdb Options ##############
--rocksdb_db_options={}
--rocksdb_column_family_options={"write_buffer_size":"67108864","max_write_buffer_number":"4","max_bytes_for_level_base":"268435456"}
--rocksdb_block_based_table_options={"block_size":"8192"}

############### misc ####################
--query_concurrently=true
--auto_remove_invalid_space=true
--num_io_threads=16
--num_worker_threads=32
--max_concurrent_subtasks=10
--snapshot_part_rate_limit=10485760
--snapshot_batch_size=1048576
--rebuild_index_part_rate_limit=4194304
--rebuild_index_batch_size=1048576

########## memory tracker ##########
--memory_tracker_limit_ratio=0.8
--memory_tracker_untracked_reserved_memory_mb=50

--memory_tracker_detail_log=false
--memory_tracker_detail_log_interval_ms=60000

--memory_purge_enabled=true
--memory_purge_interval_seconds=10
···

graphd
···
########## basics ##########
--daemonize=true
--pid_file=pids/nebula-graphd.pid
--enable_optimizer=true
--default_charset=utf8
--default_collate=utf8_bin
--local_config=true
--timezone_name=UTC+08:00
########## logging ##########
--log_dir=logs
--minloglevel=0
--v=0
--logbufsecs=0
--redirect_stdout=true
--stdout_log_file=graphd-stdout.log
--stderr_log_file=graphd-stderr.log
--stderrthreshold=3
--timestamp_in_logfile_name=true
########## query ##########
--accept_partial_success=false
--max_allowed_query_size=4194304

########## networking ##########
--meta_server_addrs=10.20.31.203:9559,10.20.31.204:9559
--local_ip=10.20.31.204
--listen_netdev=any
--port=9669
--reuse_port=false
--listen_backlog=1024
--client_idle_timeout_secs=28800
--session_idle_timeout_secs=28800
--num_accept_threads=1
--num_netio_threads=0
--num_worker_threads=0
--ws_ip=0.0.0.0
--ws_http_port=19669
--storage_client_timeout_ms=60000
--slow_query_threshold_us=200000
--ws_meta_http_port=19559

########## authentication ##########
--enable_authorize=true
--auth_type=password

--system_memory_high_watermark_ratio=0.8

########## metrics ##########
--enable_space_level_metrics=false

########## experimental feature ##########
--enable_experimental_feature=false

--enable_data_balance=true

########## session ##########
--max_sessions_per_ip_per_user=300

########## memory tracker ##########
--memory_tracker_limit_ratio=0.8
--memory_tracker_untracked_reserved_memory_mb=50

--memory_tracker_detail_log=false
--memory_tracker_detail_log_interval_ms=60000

--memory_purge_enabled=true
--memory_purge_interval_seconds=10
3

会不会是之前的cluster.id没删？

2 个赞
4

需要删除cluster.id吗，我试试

5

好使了，感谢！这事儿我真不知道