- nebula 版本:v2.0.1
- 部署方式(分布式 / 单机 / Docker / DBaaS):docker-compose
- 是否为线上版本:N
- 问题的具体描述
插入新的vertex,通过全文检索搜索不到,检查es中也没有相关文档,文档数量没有变化 - 相关的 meta / storage / graph info 日志信息
测试环境,启动是通过docker-compose,修改了官方提供的nebula-docker-compose文件,添加了三个listener服务,分别如下:
listener0:
image: vesoft/nebula-storaged:v2-nightly
environment:
USER: root
TZ: "${TZ}"
entrypoint:
- sh
- -c
command:
- /usr/local/nebula/bin/nebula-storaged --flagfile=/usr/local/nebula/etc/nebula-storaged-listener.conf --daemonize=false --local_ip=$$(hostname -i) --ws_ip=$$(hostname -i)
depends_on:
- metad0
- metad1
- metad2
healthcheck:
test: ["CMD", "curl", "-sf", "http://listener0:19779/status"]
interval: 30s
timeout: 10s
retries: 3
start_period: 20s
ports:
- 9779
- 19779
- 19780
volumes:
- ./nebula-storaged-listener.conf:/usr/local/nebula/etc/nebula-storaged-listener.conf
- ./data/listener0:/data/listener
- ./logs/listener0:/logs
networks:
- nebula-net
restart: on-failure
cap_add:
- SYS_PTRACE
listener1:
image: vesoft/nebula-storaged:v2-nightly
environment:
USER: root
TZ: "${TZ}"
entrypoint:
- sh
- -c
command:
- /usr/local/nebula/bin/nebula-storaged --flagfile=/usr/local/nebula/etc/nebula-storaged-listener.conf --daemonize=false --local_ip=$$(hostname -i) --ws_ip=$$(hostname -i)
depends_on:
- metad0
- metad1
- metad2
healthcheck:
test: ["CMD", "curl", "-sf", "http://listener1:19779/status"]
interval: 30s
timeout: 10s
retries: 3
start_period: 20s
ports:
- 9779
- 19779
- 19780
volumes:
- ./nebula-storaged-listener.conf:/usr/local/nebula/etc/nebula-storaged-listener.conf
- ./data/listener1:/data/listener
- ./logs/listener1:/logs
networks:
- nebula-net
restart: on-failure
cap_add:
- SYS_PTRACE
listener2:
image: vesoft/nebula-storaged:v2-nightly
environment:
USER: root
TZ: "${TZ}"
entrypoint:
- sh
- -c
command:
- /usr/local/nebula/bin/nebula-storaged --flagfile=/usr/local/nebula/etc/nebula-storaged-listener.conf --daemonize=false --local_ip=$$(hostname -i) --ws_ip=$$(hostname -i)
depends_on:
- metad0
- metad1
- metad2
healthcheck:
test: ["CMD", "curl", "-sf", "http://listener2:19779/status"]
interval: 30s
timeout: 10s
retries: 3
start_period: 20s
ports:
- 9779
- 19779
- 19780
volumes:
- ./nebula-storaged-listener.conf:/usr/local/nebula/etc/nebula-storaged-listener.conf
- ./data/listener2:/data/listener
- ./logs/listener2:/logs
networks:
- nebula-net
restart: on-failure
cap_add:
- SYS_PTRACE
其中挂载的nebula-storaged-listener.conf
配置文件内容如下:
########## nebula-storaged-listener ###########
########## basics ##########
# Whether to run as a daemon process
--daemonize=false
# The file to host the process id
--pid_file=pids/nebula-storaged-listener.pid
########## logging ##########
# The directory to host logging files
--log_dir=/logs
# Log level, 0, 1, 2, 3 for INFO, WARNING, ERROR, FATAL respectively
--minloglevel=0
# Verbose log level, 1, 2, 3, 4, the higher of the level, the more verbose of the logging
--v=0
# Maximum seconds to buffer the log messages
--logbufsecs=0
# Whether to redirect stdout and stderr to separate output files
--redirect_stdout=true
# Destination filename of stdout and stderr, which will also reside in log_dir.
--stdout_log_file=storaged-stdout.log
--stderr_log_file=storaged-stderr.log
# Copy log messages at or above this level to stderr in addition to logfiles. The numbers of severity levels INFO, WARNING, ERROR, and FATAL are 0, 1, 2, and 3, respectively.
--stderrthreshold=2
########## networking ##########
# Meta server address
--meta_server_addrs=metad0:9559,metad1:9559,metad2:9559
# Local ip
# --local_ip=storagels
# Storage daemon listening port
--port=9779
# HTTP service ip
# --ws_ip=storagels
# HTTP service port
--ws_http_port=19779
# HTTP2 service port
--ws_h2_port=19780
# heartbeat with meta service
--heartbeat_interval_secs=10
########## storage ##########
# Listener wal directory. only one path is allowed.
--listener_path=/data/listener
# This parameter can be ignored for compatibility. let's fill A default value of "data"
--data_path=/data/storage
# The type of part manager, [memory | meta]
--part_man_type=memory
# The default reserved bytes for one batch operation
--rocksdb_batch_size=4096
# The default block cache size used in BlockBasedTable.
# The unit is MB.
--rocksdb_block_cache=4
# The type of storage engine, `rocksdb', `memory', etc.
--engine_type=rocksdb
# The type of part, `simple', `consensus'...
--part_type=simple
检查listener
和text search clients
:
(root@nebula) [basketballplayer]> show listener;
+--------+-----------------+----------------------+-----------+
| PartId | Type | Host | Status |
+--------+-----------------+----------------------+-----------+
| 1 | "ELASTICSEARCH" | ""172.24.0.7":9779" | "OFFLINE" |
+--------+-----------------+----------------------+-----------+
| 2 | "ELASTICSEARCH" | ""172.24.0.9":9779" | "OFFLINE" |
+--------+-----------------+----------------------+-----------+
| 3 | "ELASTICSEARCH" | ""172.24.0.10":9779" | "OFFLINE" |
+--------+-----------------+----------------------+-----------+
| 4 | "ELASTICSEARCH" | ""172.24.0.7":9779" | "OFFLINE" |
+--------+-----------------+----------------------+-----------+
| 5 | "ELASTICSEARCH" | ""172.24.0.9":9779" | "OFFLINE" |
+--------+-----------------+----------------------+-----------+
| 6 | "ELASTICSEARCH" | ""172.24.0.10":9779" | "OFFLINE" |
+--------+-----------------+----------------------+-----------+
| 7 | "ELASTICSEARCH" | ""172.24.0.7":9779" | "OFFLINE" |
+--------+-----------------+----------------------+-----------+
| 8 | "ELASTICSEARCH" | ""172.24.0.9":9779" | "OFFLINE" |
+--------+-----------------+----------------------+-----------+
| 9 | "ELASTICSEARCH" | ""172.24.0.10":9779" | "OFFLINE" |
+--------+-----------------+----------------------+-----------+
| 10 | "ELASTICSEARCH" | ""172.24.0.7":9779" | "OFFLINE" |
+--------+-----------------+----------------------+-----------+
Got 10 rows (time spent 3249/8022 us)
Fri, 23 Apr 2021 19:33:00 CST
(root@nebula) [basketballplayer]> show text search clients;
+-----------------+-------+
| Host | Port |
+-----------------+-------+
| "172.17.10.252" | 9200 |
+-----------------+-------+
| "172.17.10.252" | 19200 |
+-----------------+-------+
| "172.17.10.252" | 29200 |
+-----------------+-------+
Got 3 rows (time spent 3095/5195 us)
Fri, 23 Apr 2021 19:33:04 CST
问题现场截图:
(root@nebula) [basketballplayer]> show tag indexes;
+------------------+
| Names |
+------------------+
| "player_index_0" |
+------------------+
| "player_index_1" |
+------------------+
| "team_index_0" |
+------------------+
Got 3 rows (time spent 2638/12718 us)
Fri, 23 Apr 2021 19:34:45 CST
(root@nebula) [basketballplayer]> describe tag index player_index_1;
+--------+--------------------+
| Field | Type |
+--------+--------------------+
| "name" | "fixed_string(20)" |
+--------+--------------------+
Got 1 rows (time spent 2679/4743 us)
Fri, 23 Apr 2021 19:34:53 CST
(root@nebula) [basketballplayer]> insert vertex player(name,age) values "player300":("Robin Khan", 20);
Execution succeeded (time spent 3360/7458 us)
Fri, 23 Apr 2021 19:35:37 CST
(root@nebula) [basketballplayer]> LOOKUP on player where prefix(player.name, "Robin") yield player.name;
Empty set (time spent 61602/63964 us)
Fri, 23 Apr 2021 19:36:18 CST
同时检查es索引nebula_basketballplayer_tag
文档数量在INSERT VERTEX
前后也并无变化。
检查日志并无特别异常:
# cat logs/*/*.ERROR
Log file created at: 2021/04/23 11:34:42
Running on machine: 693f16cd1f3d
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0423 11:34:42.665920 50 QueryInstance.cpp:107] SyntaxError: syntax error near `;'
Log file created at: 2021/04/23 11:13:01
Running on machine: e6b4d028af7b
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0423 11:13:01.676985 1 GflagsManager.cpp:70] Load gflags json failed
Log file created at: 2021/04/23 11:13:01
Running on machine: ea0e91d16145
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0423 11:13:01.367278 1 GflagsManager.cpp:70] Load gflags json failed
Log file created at: 2021/04/23 11:13:01
Running on machine: 9ca094144a08
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0423 11:13:01.195583 1 GflagsManager.cpp:70] Load gflags json failed
Log file created at: 2021/04/23 11:13:01
Running on machine: 12232694f0a1
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0423 11:13:01.201259 119 RaftPart.cpp:366] [Port: 9560, Space: 0, Part: 0] The partition is not a leader
E0423 11:13:01.201561 119 RaftPart.cpp:686] [Port: 9560, Space: 0, Part: 0] Cannot append logs, clean the buffer
E0423 11:13:01.201614 119 ActiveHostsMan.cpp:248] Get last update time failed, error: E_LEADER_CHANGED
E0423 11:13:01.372473 119 ActiveHostsMan.cpp:248] Get last update time failed, error: E_LEADER_CHANGED
E0423 11:13:03.015368 119 ActiveHostsMan.cpp:248] Get last update time failed, error: E_LEADER_CHANGED
E0423 11:13:03.129184 119 ActiveHostsMan.cpp:248] Get last update time failed, error: E_LEADER_CHANGED
Log file created at: 2021/04/23 11:10:47
Running on machine: efdedeb4f0ca
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0423 11:10:47.272495 118 ListenerProcessor.cpp:24] Add listener failed, listener already exists.
E0423 11:12:48.705444 97 JobDescription.cpp:189] Loading Job Description FailedE_NOT_FOUND
E0423 11:12:48.705497 97 JobManager.cpp:90] [JobManager] load an invalid job from queue 0
Log file created at: 2021/04/23 11:12:59
Running on machine: 70d600f1b6d9
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0423 11:12:59.847100 47 RaftPart.cpp:1142] [Port: 9560, Space: 0, Part: 0] Receive response about askForVote from "metad1":9560, error code is E_UNKNOWN_PART
E0423 11:12:59.847286 47 RaftPart.cpp:1142] [Port: 9560, Space: 0, Part: 0] Receive response about askForVote from "metad0":9560, error code is E_UNKNOWN_PART
E0423 11:13:01.112439 119 ActiveHostsMan.cpp:248] Get last update time failed, error: E_LEADER_CHANGED
E0423 11:13:01.683113 119 RaftPart.cpp:366] [Port: 9560, Space: 0, Part: 0] The partition is not a leader
E0423 11:13:01.683192 119 RaftPart.cpp:686] [Port: 9560, Space: 0, Part: 0] Cannot append logs, clean the buffer
E0423 11:13:01.683248 119 ActiveHostsMan.cpp:248] Get last update time failed, error: E_LEADER_CHANGED
E0423 11:13:01.719718 119 ActiveHostsMan.cpp:248] Get last update time failed, error: E_LEADER_CHANGED
Log file created at: 2021/04/23 11:13:01
Running on machine: 1d63cd7217f5
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0423 11:13:01.519528 1 GflagsManager.cpp:70] Load gflags json failed
Log file created at: 2021/04/23 11:13:01
Running on machine: 46c80a274912
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0423 11:13:01.107072 1 GflagsManager.cpp:70] Load gflags json failed
Log file created at: 2021/04/23 11:13:01
Running on machine: f9ecd84af63f
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0423 11:13:01.714326 1 GflagsManager.cpp:70] Load gflags json failed
# cat logs/*/*.WARNING
Log file created at: 2021/04/23 11:34:42
Running on machine: 693f16cd1f3d
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0423 11:34:42.665920 50 QueryInstance.cpp:107] SyntaxError: syntax error near `;'
Log file created at: 2021/04/23 11:13:01
Running on machine: e6b4d028af7b
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0423 11:13:01.676985 1 GflagsManager.cpp:70] Load gflags json failed
Log file created at: 2021/04/23 11:13:01
Running on machine: ea0e91d16145
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0423 11:13:01.367278 1 GflagsManager.cpp:70] Load gflags json failed
Log file created at: 2021/04/23 11:13:01
Running on machine: 9ca094144a08
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0423 11:13:01.195583 1 GflagsManager.cpp:70] Load gflags json failed
Log file created at: 2021/04/23 11:12:59
Running on machine: 12232694f0a1
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
W0423 11:12:59.846249 43 RaftexService.cpp:181] Cannot find the part 0 in the graph space 0
E0423 11:13:01.201259 119 RaftPart.cpp:366] [Port: 9560, Space: 0, Part: 0] The partition is not a leader
E0423 11:13:01.201561 119 RaftPart.cpp:686] [Port: 9560, Space: 0, Part: 0] Cannot append logs, clean the buffer
E0423 11:13:01.201614 119 ActiveHostsMan.cpp:248] Get last update time failed, error: E_LEADER_CHANGED
E0423 11:13:01.372473 119 ActiveHostsMan.cpp:248] Get last update time failed, error: E_LEADER_CHANGED
E0423 11:13:03.015368 119 ActiveHostsMan.cpp:248] Get last update time failed, error: E_LEADER_CHANGED
E0423 11:13:03.129184 119 ActiveHostsMan.cpp:248] Get last update time failed, error: E_LEADER_CHANGED
Log file created at: 2021/04/23 11:12:59
Running on machine: 7971746efaf2
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
W0423 11:12:59.846189 43 RaftexService.cpp:181] Cannot find the part 0 in the graph space 0
Log file created at: 2021/04/23 11:12:59
Running on machine: 70d600f1b6d9
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0423 11:12:59.847100 47 RaftPart.cpp:1142] [Port: 9560, Space: 0, Part: 0] Receive response about askForVote from "metad1":9560, error code is E_UNKNOWN_PART
E0423 11:12:59.847286 47 RaftPart.cpp:1142] [Port: 9560, Space: 0, Part: 0] Receive response about askForVote from "metad0":9560, error code is E_UNKNOWN_PART
E0423 11:13:01.112439 119 ActiveHostsMan.cpp:248] Get last update time failed, error: E_LEADER_CHANGED
E0423 11:13:01.683113 119 RaftPart.cpp:366] [Port: 9560, Space: 0, Part: 0] The partition is not a leader
E0423 11:13:01.683192 119 RaftPart.cpp:686] [Port: 9560, Space: 0, Part: 0] Cannot append logs, clean the buffer
E0423 11:13:01.683248 119 ActiveHostsMan.cpp:248] Get last update time failed, error: E_LEADER_CHANGED
E0423 11:13:01.719718 119 ActiveHostsMan.cpp:248] Get last update time failed, error: E_LEADER_CHANGED
Log file created at: 2021/04/23 11:13:01
Running on machine: 1d63cd7217f5
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0423 11:13:01.519528 1 GflagsManager.cpp:70] Load gflags json failed
W0423 11:13:01.520450 1 FileBasedClusterIdMan.cpp:46] Open file failed, error No such file or directory
Log file created at: 2021/04/23 11:13:01
Running on machine: 46c80a274912
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0423 11:13:01.107072 1 GflagsManager.cpp:70] Load gflags json failed
W0423 11:13:01.108357 1 FileBasedClusterIdMan.cpp:46] Open file failed, error No such file or directory
Log file created at: 2021/04/23 11:13:01
Running on machine: f9ecd84af63f
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0423 11:13:01.714326 1 GflagsManager.cpp:70] Load gflags json failed
W0423 11:13:01.715508 1 FileBasedClusterIdMan.cpp:46] Open file failed, error No such file or directory