- nebula 版本:v1
- 部署方式(分布式 / 单机 / Docker / DBaaS):分布式docker swarm
您好,我根据链接修改配置文件后重启storaged发现重启不了(具体表现为service状态为0/1)。
想请教一下我的操作是否有误呢?
进入每个storaged容器修改conf文件–>在容器内storaged.service restart
您好,我根据链接修改配置文件后重启storaged发现重启不了(具体表现为service状态为0/1)。
想请教一下我的操作是否有误呢?
进入每个storaged容器修改conf文件–>在容器内storaged.service restart
帮忙贴一下 nebula-storage.ERROR 这个日志看看
重启storaged后发现storaged容器shutdown了,以下是重启容器后进入容器查看到的部分报错
E0226 09:02:59.469498 49 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
E0226 09:02:59.469735 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E0226 09:04:14.851496 60 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
E0226 09:04:14.851734 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E0226 09:05:28.862118 61 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 09:05:28.862330 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 09:06:41.879493 56 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 09:06:41.879763 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 09:07:56.295935 59 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
E0226 09:07:56.296182 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E0226 09:10:09.325799 60 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
E0226 09:10:09.326084 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E0226 09:13:22.351272 51 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
E0226 09:13:22.351549 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E0226 09:15:36.327476 62 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
E0226 09:15:36.327729 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 113 (No route to host): No route to host
E0226 09:16:50.352578 59 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 09:16:50.352819 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 09:18:04.366083 54 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
E0226 09:18:04.366371 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E0226 09:20:18.399309 55 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
E0226 09:20:18.399538 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
E0226 09:22:32.428086 54 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
E0226 09:22:32.428339 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E0226 09:22:47.185667 61 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 09:22:47.185950 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 09:24:01.209154 50 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 09:24:01.209391 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 09:27:15.238364 53 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
E0226 09:27:15.238595 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
E0226 09:28:28.258200 62 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 09:28:28.258541 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 09:29:41.280597 57 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 09:29:41.280845 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 09:29:55.041581 60 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 09:29:55.043113 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 09:32:08.077069 49 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
E0226 09:32:08.077361 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E0226 09:33:23.093101 56 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
E0226 09:33:23.093369 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E0226 09:34:38.104574 59 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
E0226 09:34:38.104910 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
E0226 09:35:53.759485 60 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
E0226 09:35:53.759946 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 113 (No route to host): No route to host
E0226 09:37:08.773808 55 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
E0226 09:37:08.774096 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
E0226 09:37:22.789460 56 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
E0226 09:37:22.789717 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
E0226 09:38:37.817687 47 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
E0226 09:38:37.817931 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
E0226 09:40:51.835671 48 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
E0226 09:40:51.835908 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E0226 09:43:05.861591 55 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 09:43:05.861827 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 09:45:19.871691 52 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
E0226 09:45:19.871946 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E0226 09:45:33.918572 51 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 09:45:33.918781 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 09:45:47.945529 50 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 09:45:47.945778 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 09:46:01.974157 55 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
E0226 09:46:01.974375 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
E0226 09:48:14.986485 62 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 09:48:14.986701 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 09:51:29.013584 53 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
E0226 09:51:29.013887 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E0226 09:55:42.031342 62 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
E0226 09:55:42.031703 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E0226 09:57:57.058209 53 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
E0226 09:57:57.058430 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
E0226 10:00:11.447373 50 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
E0226 10:00:11.447665 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E0226 10:03:25.463773 57 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
E0226 10:03:25.464067 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E0226 10:03:39.479369 52 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
E0226 10:03:39.479686 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
E0226 10:05:53.495853 57 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 10:05:53.496088 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 10:08:06.523102 48 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
E0226 10:08:06.523327 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E0226 10:09:20.660161 61 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
E0226 10:09:20.660457 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E0226 10:11:33.677881 48 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 10:11:33.678144 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 10:12:46.694151 55 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 10:12:46.694386 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 10:14:00.714208 58 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 10:14:00.714488 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 10:15:14.726711 59 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 10:15:14.726943 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 10:18:28.531231 54 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
E0226 10:18:28.531531 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E0226 10:18:43.905601 47 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
E0226 10:18:43.908186 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 113 (No route to host): No route to host
E0226 10:19:58.931308 54 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
E0226 10:19:58.931514 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
E0226 10:21:12.201062 55 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 10:21:12.201532 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 10:23:25.236546 50 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
E0226 10:23:25.236801 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E0226 10:23:40.497426 49 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
E0226 10:23:40.497720 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 113 (No route to host): No route to host
E0226 10:25:54.521294 54 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
E0226 10:25:54.521533 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
E0226 10:28:08.553813 61 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 10:28:08.554034 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 10:30:21.578425 58 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
E0226 10:30:21.578754 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E0226 10:30:34.592592 57 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 10:30:34.594125 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 10:32:47.622642 56 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 10:32:47.622921 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 10:35:00.743880 55 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 10:35:00.744199 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 10:36:14.768903 52 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 10:36:14.769117 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 10:36:29.785229 47 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 10:36:29.785512 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 10:37:44.806664 52 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 10:37:44.806990 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 10:38:59.332628 55 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 10:38:59.333117 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 10:41:12.352087 56 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 10:41:12.352341 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 10:42:26.358616 47 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 10:42:26.358844 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 10:44:40.376031 50 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
E0226 10:44:40.376251 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
E0226 10:44:55.392299 47 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
E0226 10:44:55.392547 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
E0226 10:45:11.075907 62 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
E0226 10:45:11.076177 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
E0226 10:45:24.101656 51 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 10:45:24.101861 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 10:45:39.118888 58 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 10:45:39.119146 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 10:46:54.167310 47 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
E0226 10:46:54.167554 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
E0226 10:49:07.177636 58 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 10:49:07.179134 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 10:50:21.201258 57 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 10:50:21.201486 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 10:52:34.224464 58 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 10:52:34.224668 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 10:54:49.116473 49 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
E0226 10:54:49.116693 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
E0226 10:55:04.132781 62 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 10:55:04.133051 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 10:55:18.148572 53 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 10:55:18.148792 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 10:57:32.176467 52 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 10:57:32.176663 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 11:00:46.215339 49 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
E0226 11:00:46.215596 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
E0226 11:03:00.237874 60 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
E0226 11:03:00.241184 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
E0226 11:03:15.239698 57 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
E0226 11:03:15.239948 63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
E0226 11:03:45.323843 63 MetaClient.cpp:110] Heartbeat failed, status:Unknown code -99
有core文件产生吗
刚刚将docker service rollback新建了storaged容器,已经查看不到之前容器里的日志了。
感觉有可能是 端口被占用了 导致storage没启动起来
当时restart后storage容器就停止运行了,为了进入容器查看log我又重启了容器。
在容器内script/目录下查看了nebula-storaged.service status, 输出大致是running as 1, listening on 44500之类的。我一开始以为是重启成功了,但是ERROR的内容如上条帖子所示,并且在swarm的主节点查看storage进程状态是shutdown。
有没有可能正是我重启了容器导致端口被占用?
还有其他方法能重启服务使修改后的配置文件生效么?
修改配置文件,只能重启服务才会生效, 并且端口号是写入meta里面的,如果修改端口号,即使重启成功了,之前的数据可能也无法访问
使用docker部署方式的重启操作有相关文档吗?
看 log 报错是heartbeat 失败. 这个表示 storage 连不上 meta. 一般有两种可能, 一个是 storage 中的 meta 信息配错了, 第二个是 meta 所在的 server 开了防火墙, 不让 storage 连.
貌似是这个原因
“一个是 storage 中的 meta 信息配错了”
storage容器里的配置文件我只改了我要改的参数,忘记把127.0.0.0改成自己的ip了。
我一会儿试试。
storage INFO 报啥错呢?
status.ok() IO ERROR: While lock file: /data/storage/nebula/17/data/LOCK: Resource temporarily unavailable.
还有除进容器内修改配置文件的方法吗?
感觉docker swarm部署的这种方式进容器修改再重启该容器的方法目前行不通。
出现这个报错的直接原因是有两个 rocksdb 实例去尝试拿同一个 lock.
具体到 nebula 的话, 有很大可能是前一个 storage 还没有被完全 stop, 后一个 storaged 就启动了.
可以确认下当 PS 看不到之后, 再进行 start .
我操作的是同一个storage,按理来说不存在前一个和后一个。
对, 就是同一个 storage 的容易出这个问题, 你调 stop storaged 的时候, 如果出现没有成功 stop, 那么再再次 start 的时候, 就会报错.