修改配置后重启失败

  • nebula 版本:v1
  • 部署方式(分布式 / 单机 / Docker / DBaaS):分布式docker swarm

您好,我根据链接修改配置文件后重启storaged发现重启不了(具体表现为service状态为0/1)。

想请教一下我的操作是否有误呢?

进入每个storaged容器修改conf文件–>在容器内storaged.service restart

帮忙贴一下 nebula-storage.ERROR 这个日志看看

1 个赞

重启storaged后发现storaged容器shutdown了,以下是重启容器后进入容器查看到的部分报错

  E0226 09:02:59.469498    49 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
    E0226 09:02:59.469735    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
    E0226 09:04:14.851496    60 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
    E0226 09:04:14.851734    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
    E0226 09:05:28.862118    61 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 09:05:28.862330    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 09:06:41.879493    56 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 09:06:41.879763    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 09:07:56.295935    59 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
    E0226 09:07:56.296182    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
    E0226 09:10:09.325799    60 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
    E0226 09:10:09.326084    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
    E0226 09:13:22.351272    51 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
    E0226 09:13:22.351549    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
    E0226 09:15:36.327476    62 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
    E0226 09:15:36.327729    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 113 (No route to host): No route to host
    E0226 09:16:50.352578    59 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 09:16:50.352819    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 09:18:04.366083    54 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
    E0226 09:18:04.366371    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
    E0226 09:20:18.399309    55 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
    E0226 09:20:18.399538    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
    E0226 09:22:32.428086    54 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
    E0226 09:22:32.428339    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
    E0226 09:22:47.185667    61 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 09:22:47.185950    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 09:24:01.209154    50 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 09:24:01.209391    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 09:27:15.238364    53 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
    E0226 09:27:15.238595    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
    E0226 09:28:28.258200    62 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 09:28:28.258541    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 09:29:41.280597    57 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 09:29:41.280845    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 09:29:55.041581    60 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 09:29:55.043113    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 09:32:08.077069    49 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
    E0226 09:32:08.077361    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
    E0226 09:33:23.093101    56 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
    E0226 09:33:23.093369    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
    E0226 09:34:38.104574    59 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
    E0226 09:34:38.104910    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
    E0226 09:35:53.759485    60 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
    E0226 09:35:53.759946    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 113 (No route to host): No route to host
    E0226 09:37:08.773808    55 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
    E0226 09:37:08.774096    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
    E0226 09:37:22.789460    56 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
    E0226 09:37:22.789717    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
    E0226 09:38:37.817687    47 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
    E0226 09:38:37.817931    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
    E0226 09:40:51.835671    48 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
    E0226 09:40:51.835908    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
    E0226 09:43:05.861591    55 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 09:43:05.861827    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 09:45:19.871691    52 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
    E0226 09:45:19.871946    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
    E0226 09:45:33.918572    51 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 09:45:33.918781    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 09:45:47.945529    50 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 09:45:47.945778    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 09:46:01.974157    55 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
    E0226 09:46:01.974375    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
    E0226 09:48:14.986485    62 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 09:48:14.986701    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 09:51:29.013584    53 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
    E0226 09:51:29.013887    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
    E0226 09:55:42.031342    62 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
    E0226 09:55:42.031703    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
    E0226 09:57:57.058209    53 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
    E0226 09:57:57.058430    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
    E0226 10:00:11.447373    50 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
    E0226 10:00:11.447665    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
    E0226 10:03:25.463773    57 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
    E0226 10:03:25.464067    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
    E0226 10:03:39.479369    52 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
    E0226 10:03:39.479686    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
    E0226 10:05:53.495853    57 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 10:05:53.496088    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 10:08:06.523102    48 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
    E0226 10:08:06.523327    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
    E0226 10:09:20.660161    61 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
    E0226 10:09:20.660457    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
    E0226 10:11:33.677881    48 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 10:11:33.678144    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 10:12:46.694151    55 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 10:12:46.694386    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 10:14:00.714208    58 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 10:14:00.714488    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 10:15:14.726711    59 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 10:15:14.726943    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 10:18:28.531231    54 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
    E0226 10:18:28.531531    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
    E0226 10:18:43.905601    47 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
    E0226 10:18:43.908186    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 113 (No route to host): No route to host
    E0226 10:19:58.931308    54 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
    E0226 10:19:58.931514    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
    E0226 10:21:12.201062    55 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 10:21:12.201532    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 10:23:25.236546    50 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
    E0226 10:23:25.236801    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
    E0226 10:23:40.497426    49 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
    E0226 10:23:40.497720    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 113 (No route to host): No route to host
    E0226 10:25:54.521294    54 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
    E0226 10:25:54.521533    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
    E0226 10:28:08.553813    61 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 10:28:08.554034    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 10:30:21.578425    58 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
    E0226 10:30:21.578754    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
    E0226 10:30:34.592592    57 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 10:30:34.594125    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 10:32:47.622642    56 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 10:32:47.622921    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 10:35:00.743880    55 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 10:35:00.744199    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 10:36:14.768903    52 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 10:36:14.769117    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 10:36:29.785229    47 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 10:36:29.785512    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 10:37:44.806664    52 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 10:37:44.806990    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 10:38:59.332628    55 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 10:38:59.333117    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 10:41:12.352087    56 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 10:41:12.352341    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 10:42:26.358616    47 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 10:42:26.358844    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 10:44:40.376031    50 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
    E0226 10:44:40.376251    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
    E0226 10:44:55.392299    47 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
    E0226 10:44:55.392547    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
    E0226 10:45:11.075907    62 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
    E0226 10:45:11.076177    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
    E0226 10:45:24.101656    51 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 10:45:24.101861    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 10:45:39.118888    58 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 10:45:39.119146    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 10:46:54.167310    47 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
    E0226 10:46:54.167554    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
    E0226 10:49:07.177636    58 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 10:49:07.179134    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 10:50:21.201258    57 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 10:50:21.201486    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 10:52:34.224464    58 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 10:52:34.224668    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 10:54:49.116473    49 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
    E0226 10:54:49.116693    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
    E0226 10:55:04.132781    62 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 10:55:04.133051    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 10:55:18.148572    53 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 10:55:18.148792    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 10:57:32.176467    52 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 10:57:32.176663    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 11:00:46.215339    49 MetaClient.cpp:524] Send request to [192.168.5.67:45500], exceed retry limit
    E0226 11:00:46.215596    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: write timed out during connection, type = Timed out
    E0226 11:03:00.237874    60 MetaClient.cpp:524] Send request to [192.168.5.72:45500], exceed retry limit
    E0226 11:03:00.241184    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: Timed Out
    E0226 11:03:15.239698    57 MetaClient.cpp:524] Send request to [192.168.5.70:45500], exceed retry limit
    E0226 11:03:15.239948    63 MetaClient.cpp:110] Heartbeat failed, status:RPC failure in MetaClient: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
    E0226 11:03:45.323843    63 MetaClient.cpp:110] Heartbeat failed, status:Unknown code -99

有core文件产生吗

刚刚将docker service rollback新建了storaged容器,已经查看不到之前容器里的日志了。

感觉有可能是 端口被占用了 导致storage没启动起来

当时restart后storage容器就停止运行了,为了进入容器查看log我又重启了容器。
在容器内script/目录下查看了nebula-storaged.service status, 输出大致是running as 1, listening on 44500之类的。我一开始以为是重启成功了,但是ERROR的内容如上条帖子所示,并且在swarm的主节点查看storage进程状态是shutdown。

有没有可能正是我重启了容器导致端口被占用?

还有其他方法能重启服务使修改后的配置文件生效么?

修改配置文件,只能重启服务才会生效, 并且端口号是写入meta里面的,如果修改端口号,即使重启成功了,之前的数据可能也无法访问

使用docker部署方式的重启操作有相关文档吗?

看 log 报错是heartbeat 失败. 这个表示 storage 连不上 meta. 一般有两种可能, 一个是 storage 中的 meta 信息配错了, 第二个是 meta 所在的 server 开了防火墙, 不让 storage 连.

貌似是这个原因

“一个是 storage 中的 meta 信息配错了”

storage容器里的配置文件我只改了我要改的参数,忘记把127.0.0.0改成自己的ip了。
我一会儿试试。

1 个赞

@liuyu85cn 配置文件改好后heartbeat相关的error解决了,但是在主节点查看该进程状态依然是shutdown。

个人觉得看起来像是docker swarm的问题。

storage INFO 报啥错呢?

status.ok() IO ERROR: While lock file: /data/storage/nebula/17/data/LOCK: Resource temporarily unavailable.

还有除进容器内修改配置文件的方法吗?

感觉docker swarm部署的这种方式进容器修改再重启该容器的方法目前行不通。

出现这个报错的直接原因是有两个 rocksdb 实例去尝试拿同一个 lock.
具体到 nebula 的话, 有很大可能是前一个 storage 还没有被完全 stop, 后一个 storaged 就启动了.
可以确认下当 PS 看不到之后, 再进行 start .

我操作的是同一个storage,按理来说不存在前一个和后一个。

对, 就是同一个 storage 的容易出这个问题, 你调 stop storaged 的时候, 如果出现没有成功 stop, 那么再再次 start 的时候, 就会报错.