nebula meta节点一直都是unhealthy

Mario · 2021 年2 月 23 日 01:55

提问参考模版：

nebula 版本：v1.2.0
部署方式（分布式 / 单机 / Docker / DBaaS）：docker单机
硬件信息
- 磁盘：HDD
- CPU、内存信息：4核，16G
问题的具体描述
开发服docker单机部署nebula，出现meta节点一直都是unhealthy状态，看了日志如下

image1529×143 8.83 KB

image1505×92 6.27 KB
这个问题怎么处理，重启很多次meta节点都没用

Shylock-Hg · 2021 年2 月 23 日 02:38

meta配置看一下

Mario · 2021 年2 月 23 日 02:52

services:
  metad0:
    image: vesoft/nebula-metad:v1.2.0
    environment:
      USER: root
      TZ:   "${TZ}"
    command:
      - --meta_server_addrs=172.28.1.1:45500,172.28.1.2:45500,172.28.1.3:45500
      - --local_ip=172.28.1.1
      - --ws_ip=172.28.1.1
      - --port=45500
      - --data_path=/data/meta
      - --log_dir=/logs
      - --v=0
      - --minloglevel=0
    healthcheck:
      test: ["CMD", "curl", "-f", "http://172.28.1.1:11000/status"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s
    ports:
      - "45500:45500"
      - 11000
      - 11002
    volumes:
      - /mnt2/data/meta0:/data/meta
      - /mnt2/logs/meta0:/logs
    networks:
      nebula-net:
        ipv4_address: 172.28.1.1
    restart: on-failure
    cap_add:
      - SYS_PTRACE

metad1:
    image: vesoft/nebula-metad:v1.2.0
    environment:
      USER: root
      TZ:   "${TZ}"
    command:
      - --meta_server_addrs=172.28.1.1:45500,172.28.1.2:45500,172.28.1.3:45500
      - --local_ip=172.28.1.2
      - --ws_ip=172.28.1.2
      - --port=45500
      - --data_path=/data/meta
      - --log_dir=/logs
      - --v=0
      - --minloglevel=0
    healthcheck:
      test: ["CMD", "curl", "-f", "http://172.28.1.2:11000/status"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s
    ports:
      - "45501:45500"
      - 11000
      - 11002
    volumes:
      - /mnt2/data/meta1:/data/meta
      - /mnt2/logs/meta1:/logs
    networks:
      nebula-net:
        ipv4_address: 172.28.1.2
    restart: on-failure
    cap_add:
      - SYS_PTRACE

  metad2:
    image: vesoft/nebula-metad:v1.2.0
    environment:
      USER: root
      TZ:   "${TZ}"
    command:
      - --meta_server_addrs=172.28.1.1:45500,172.28.1.2:45500,172.28.1.3:45500
      - --local_ip=172.28.1.3
      - --ws_ip=172.28.1.3
      - --port=45500
      - --data_path=/data/meta
      - --log_dir=/logs
      - --v=0
      - --minloglevel=0
    healthcheck:
      test: ["CMD", "curl", "-f", "http://172.28.1.3:11000/status"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s
    ports:
      - "45502:45500"
      - 11000
      - 11002
    volumes:
      - /mnt2/data/meta2:/data/meta
      - /mnt2/logs/meta2:/logs
    networks:
      nebula-net:
        ipv4_address: 172.28.1.3
    restart: on-failure
    cap_add:
      - SYS_PTRACE

Mario · 2021 年2 月 23 日 02:52

配置贴上了

Shylock-Hg · 2021 年2 月 23 日 03:49

–meta_server_addrs
只有一个meta，就填一个地址

Mario · 2021 年2 月 23 日 04:30

什么意思，只填一个地址，那meta1和meta2都不需要了？

Shylock-Hg · 2021 年2 月 23 日 05:31

看错了，以为你是单机的

Mario · 2021 年2 月 23 日 07:29

这个问题怎么解决啊

Shylock-Hg · 2021 年2 月 23 日 08:02

吧meta的log都贴一下

Mario · 2021 年2 月 23 日 08:07

meta0，1，2的info日志

Shylock-Hg · 2021 年2 月 23 日 08:21

多贴几行

Mario · 2021 年2 月 24 日 08:42

I0224 08:40:16.585942    50 RaftPart.cpp:1206] [Port: 45501, Space: 0, Part: 0] No one is elected, continue the election
I0224 08:40:16.642772     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:17.529587    47 RaftPart.cpp:1124] [Port: 45501, Space: 0, Part: 0] Sending out an election request (space = 0, part = 0, term = 39, lastLogId = 0, lastLogTerm = 0, candidateIP = 172.28.1.3, candidatePort = 45501)
I0224 08:40:17.642917     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:18.531493    47 RaftPart.cpp:1206] [Port: 45501, Space: 0, Part: 0] No one is elected, continue the election
I0224 08:40:18.643127     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:19.301972    48 RaftPart.cpp:1124] [Port: 45501, Space: 0, Part: 0] Sending out an election request (space = 0, part = 0, term = 40, lastLogId = 0, lastLogTerm = 0, candidateIP = 172.28.1.3, candidatePort = 45501)
I0224 08:40:19.643276     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:20.304126    48 RaftPart.cpp:1206] [Port: 45501, Space: 0, Part: 0] No one is elected, continue the election
I0224 08:40:20.643461     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:21.285346    49 RaftPart.cpp:1124] [Port: 45501, Space: 0, Part: 0] Sending out an election request (space = 0, part = 0, term = 41, lastLogId = 0, lastLogTerm = 0, candidateIP = 172.28.1.3, candidatePort = 45501)
I0224 08:40:21.643627     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:22.287544    49 RaftPart.cpp:1206] [Port: 45501, Space: 0, Part: 0] No one is elected, continue the election
I0224 08:40:22.643805     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:23.644009     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:23.655102    50 RaftPart.cpp:1124] [Port: 45501, Space: 0, Part: 0] Sending out an election request (space = 0, part = 0, term = 42, lastLogId = 0, lastLogTerm = 0, candidateIP = 172.28.1.3, candidatePort = 45501)
I0224 08:40:24.644150     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:24.656946    50 RaftPart.cpp:1206] [Port: 45501, Space: 0, Part: 0] No one is elected, continue the election
I0224 08:40:25.644302     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:25.891319    47 RaftPart.cpp:1124] [Port: 45501, Space: 0, Part: 0] Sending out an election request (space = 0, part = 0, term = 43, lastLogId = 0, lastLogTerm = 0, candidateIP = 172.28.1.3, candidatePort = 45501)
I0224 08:40:26.644465     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:26.893460    47 RaftPart.cpp:1206] [Port: 45501, Space: 0, Part: 0] No one is elected, continue the election
I0224 08:40:27.644627     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:28.644804     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:28.878537    48 RaftPart.cpp:1124] [Port: 45501, Space: 0, Part: 0] Sending out an election request (space = 0, part = 0, term = 44, lastLogId = 0, lastLogTerm = 0, candidateIP = 172.28.1.3, candidatePort = 45501)
I0224 08:40:29.644976     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:29.878109    48 RaftPart.cpp:1206] [Port: 45501, Space: 0, Part: 0] No one is elected, continue the election
I0224 08:40:30.645218     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:31.255642    49 RaftPart.cpp:1124] [Port: 45501, Space: 0, Part: 0] Sending out an election request (space = 0, part = 0, term = 45, lastLogId = 0, lastLogTerm = 0, candidateIP = 172.28.1.3, candidatePort = 45501)
I0224 08:40:31.645336     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:32.255687    49 RaftPart.cpp:1206] [Port: 45501, Space: 0, Part: 0] No one is elected, continue the election
I0224 08:40:32.645484     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:33.240873    50 RaftPart.cpp:1124] [Port: 45501, Space: 0, Part: 0] Sending out an election request (space = 0, part = 0, term = 46, lastLogId = 0, lastLogTerm = 0, candidateIP = 172.28.1.3, candidatePort = 45501)
I0224 08:40:33.645632     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:34.239048    50 RaftPart.cpp:1206] [Port: 45501, Space: 0, Part: 0] No one is elected, continue the election
I0224 08:40:34.645782     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:35.645925     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:35.779754    47 RaftPart.cpp:1124] [Port: 45501, Space: 0, Part: 0] Sending out an election request (space = 0, part = 0, term = 47, lastLogId = 0, lastLogTerm = 0, candidateIP = 172.28.1.3, candidatePort = 45501)
I0224 08:40:36.646076     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:36.781786    47 RaftPart.cpp:1206] [Port: 45501, Space: 0, Part: 0] No one is elected, continue the election
I0224 08:40:37.646265     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:37.705845    48 RaftPart.cpp:1124] [Port: 45501, Space: 0, Part: 0] Sending out an election request (space = 0, part = 0, term = 48, lastLogId = 0, lastLogTerm = 0, candidateIP = 172.28.1.3, candidatePort = 45501)
I0224 08:40:38.646458     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:38.707386    48 RaftPart.cpp:1206] [Port: 45501, Space: 0, Part: 0] No one is elected, continue the election
I0224 08:40:39.286044    49 RaftPart.cpp:1124] [Port: 45501, Space: 0, Part: 0] Sending out an election request (space = 0, part = 0, term = 49, lastLogId = 0, lastLogTerm = 0, candidateIP = 172.28.1.3, candidatePort = 45501)
I0224 08:40:39.646669     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:40.287583    49 RaftPart.cpp:1206] [Port: 45501, Space: 0, Part: 0] No one is elected, continue the election
I0224 08:40:40.646862     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:40.928328    50 RaftPart.cpp:1124] [Port: 45501, Space: 0, Part: 0] Sending out an election request (space = 0, part = 0, term = 50, lastLogId = 0, lastLogTerm = 0, candidateIP = 172.28.1.3, candidatePort = 45501)
I0224 08:40:41.647025     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:41.928426    50 RaftPart.cpp:1206] [Port: 45501, Space: 0, Part: 0] No one is elected, continue the election
I0224 08:40:42.647215     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:43.586294    47 RaftPart.cpp:1124] [Port: 45501, Space: 0, Part: 0] Sending out an election request (space = 0, part = 0, term = 51, lastLogId = 0, lastLogTerm = 0, candidateIP = 172.28.1.3, candidatePort = 45501)
I0224 08:40:43.647488     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:44.588016    47 RaftPart.cpp:1206] [Port: 45501, Space: 0, Part: 0] No one is elected, continue the election
I0224 08:40:44.647673     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:45.377974    48 RaftPart.cpp:1124] [Port: 45501, Space: 0, Part: 0] Sending out an election request (space = 0, part = 0, term = 52, lastLogId = 0, lastLogTerm = 0, candidateIP = 172.28.1.3, candidatePort = 45501)
I0224 08:40:45.647810     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:46.379891    48 RaftPart.cpp:1206] [Port: 45501, Space: 0, Part: 0] No one is elected, continue the election
I0224 08:40:46.647974     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:47.648185     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:48.157814    49 RaftPart.cpp:1124] [Port: 45501, Space: 0, Part: 0] Sending out an election request (space = 0, part = 0, term = 53, lastLogId = 0, lastLogTerm = 0, candidateIP = 172.28.1.3, candidatePort = 45501)
I0224 08:40:48.648377     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:49.159860    49 RaftPart.cpp:1206] [Port: 45501, Space: 0, Part: 0] No one is elected, continue the election
I0224 08:40:49.648563     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:50.355198    50 RaftPart.cpp:1124] [Port: 45501, Space: 0, Part: 0] Sending out an election request (space = 0, part = 0, term = 54, lastLogId = 0, lastLogTerm = 0, candidateIP = 172.28.1.3, candidatePort = 45501)
I0224 08:40:50.648730     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:51.357383    50 RaftPart.cpp:1206] [Port: 45501, Space: 0, Part: 0] No one is elected, continue the election
I0224 08:40:51.648948     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:52.649099     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:53.063297    47 RaftPart.cpp:1124] [Port: 45501, Space: 0, Part: 0] Sending out an election request (space = 0, part = 0, term = 55, lastLogId = 0, lastLogTerm = 0, candidateIP = 172.28.1.3, candidatePort = 45501)
I0224 08:40:53.649329     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:54.065521    47 RaftPart.cpp:1206] [Port: 45501, Space: 0, Part: 0] No one is elected, continue the election
I0224 08:40:54.649492     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:55.190881    48 RaftPart.cpp:1124] [Port: 45501, Space: 0, Part: 0] Sending out an election request (space = 0, part = 0, term = 56, lastLogId = 0, lastLogTerm = 0, candidateIP = 172.28.1.3, candidatePort = 45501)
I0224 08:40:55.649732     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:56.190944    48 RaftPart.cpp:1206] [Port: 45501, Space: 0, Part: 0] No one is elected, continue the election
I0224 08:40:56.649991     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:57.650269     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:57.980063    49 RaftPart.cpp:1124] [Port: 45501, Space: 0, Part: 0] Sending out an election request (space = 0, part = 0, term = 57, lastLogId = 0, lastLogTerm = 0, candidateIP = 172.28.1.3, candidatePort = 45501)
I0224 08:40:58.650429     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:40:58.982218    49 RaftPart.cpp:1206] [Port: 45501, Space: 0, Part: 0] No one is elected, continue the election
I0224 08:40:59.650583     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:41:00.507033    50 RaftPart.cpp:1124] [Port: 45501, Space: 0, Part: 0] Sending out an election request (space = 0, part = 0, term = 58, lastLogId = 0, lastLogTerm = 0, candidateIP = 172.28.1.3, candidatePort = 45501)
I0224 08:41:00.650784     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:41:01.508353    50 RaftPart.cpp:1206] [Port: 45501, Space: 0, Part: 0] No one is elected, continue the election
I0224 08:41:01.651000     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:41:02.651146     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:41:03.058151    47 RaftPart.cpp:1124] [Port: 45501, Space: 0, Part: 0] Sending out an election request (space = 0, part = 0, term = 59, lastLogId = 0, lastLogTerm = 0, candidateIP = 172.28.1.3, candidatePort = 45501)
I0224 08:41:03.651307     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:41:04.059854    47 RaftPart.cpp:1206] [Port: 45501, Space: 0, Part: 0] No one is elected, continue the election
I0224 08:41:04.590459    48 RaftPart.cpp:1124] [Port: 45501, Space: 0, Part: 0] Sending out an election request (space = 0, part = 0, term = 60, lastLogId = 0, lastLogTerm = 0, candidateIP = 172.28.1.3, candidatePort = 45501)
I0224 08:41:04.651473     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:41:05.592214    48 RaftPart.cpp:1206] [Port: 45501, Space: 0, Part: 0] No one is elected, continue the election
I0224 08:41:05.651628     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:41:06.651765     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:41:07.433797    49 RaftPart.cpp:1124] [Port: 45501, Space: 0, Part: 0] Sending out an election request (space = 0, part = 0, term = 61, lastLogId = 0, lastLogTerm = 0, candidateIP = 172.28.1.3, candidatePort = 45501)
I0224 08:41:07.651965     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s
I0224 08:41:08.435485    49 RaftPart.cpp:1206] [Port: 45501, Space: 0, Part: 0] No one is elected, continue the election
I0224 08:41:08.652122     1 MetaDaemon.cpp:106] Leader has not been elected, sleep 1s

Mario · 2021 年2 月 24 日 08:43

这是log，重复出现

Shylock-Hg · 2021 年2 月 24 日 08:45

三个meta的log都看看

Mario · 2021 年2 月 24 日 08:48

一样的，重复的。。。

Shylock-Hg · 2021 年2 月 24 日 09:10

那看看meta间网络连通

Mario · 2021 年2 月 24 日 09:22

怎么看，有什么命令？

Shylock-Hg · 2021 年2 月 24 日 09:24

到容器里ping一下吧

Mario · 2021 年2 月 24 日 09:28

互相ping不通，所以导致选举主节点失败是吧

Mario · 2021 年2 月 24 日 09:29

三个meta节点互相ping不通，怎么处理这个网络问题？