- nebula 版本：2.0GA
- 部署方式：参考https://github.com/vesoft-inc/nebula-docker-compose在虚机上部署了3节点的集群
- 硬件信息
- 磁盘SSD
- 4核32G虚机 * 3
Nebula Graph集群信息：
# docker-compose ps
Name Command State Ports
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
nebula-docker-compose_graphd1_1 /usr/local/nebula/bin/nebu ... Up (healthy) 0.0.0.0:49163->19669/tcp, 0.0.0.0:49162->19670/tcp, 0.0.0.0:49164->9669/tcp
nebula-docker-compose_graphd2_1 /usr/local/nebula/bin/nebu ... Up (healthy) 0.0.0.0:49166->19669/tcp, 0.0.0.0:49165->19670/tcp, 0.0.0.0:49167->9669/tcp
nebula-docker-compose_graphd_1 /usr/local/nebula/bin/nebu ... Up (healthy) 0.0.0.0:49175->19669/tcp, 0.0.0.0:49174->19670/tcp, 0.0.0.0:9669->9669/tcp
nebula-docker-compose_metad0_1 ./bin/nebula-metad --flagf ... Up (healthy) 0.0.0.0:49160->19559/tcp, 0.0.0.0:49159->19560/tcp, 0.0.0.0:49161->9559/tcp,
9560/tcp
nebula-docker-compose_metad1_1 ./bin/nebula-metad --flagf ... Up (healthy) 0.0.0.0:49154->19559/tcp, 0.0.0.0:49153->19560/tcp, 0.0.0.0:49155->9559/tcp,
9560/tcp
nebula-docker-compose_metad2_1 ./bin/nebula-metad --flagf ... Up (healthy) 0.0.0.0:49157->19559/tcp, 0.0.0.0:49156->19560/tcp, 0.0.0.0:49158->9559/tcp,
9560/tcp
nebula-docker-compose_storaged0_1 ./bin/nebula-storaged --fl ... Up (healthy) 0.0.0.0:49177->19779/tcp, 0.0.0.0:49176->19780/tcp, 9777/tcp, 9778/tcp,
0.0.0.0:49178->9779/tcp, 9780/tcp
nebula-docker-compose_storaged1_1 ./bin/nebula-storaged --fl ... Up (healthy) 0.0.0.0:49169->19779/tcp, 0.0.0.0:49168->19780/tcp, 9777/tcp, 9778/tcp,
0.0.0.0:49170->9779/tcp, 9780/tcp
nebula-docker-compose_storaged2_1 ./bin/nebula-storaged --fl ... Up (healthy) 0.0.0.0:49172->19779/tcp, 0.0.0.0:49171->19780/tcp, 9777/tcp, 9778/tcp,
0.0.0.0:49173->9779/tcp, 9780/tcp
Nebula Graph建模Schema：
CREATE SPACE csv(partition_num = 15, replica_factor = 1);
CREATE TAG user(id string);
CREATE EDGE action (startId string, endId string);
随后导入了一批数据。
GraphX连接Nebula Graph并运行PageRank的代码：
import java.text.SimpleDateFormat
import com.facebook.thrift.protocol.TCompactProtocol
import org.apache.spark.graphx.Graph
import com.vesoft.nebula.connector.connector.NebulaDataFrameReader
import com.vesoft.nebula.connector.{NebulaConnectionConfig, ReadNebulaConfig}
import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession
val df = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss")
val sparkConf = new SparkConf
sparkConf
.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.registerKryoClasses(Array[Class[_]](classOf[TCompactProtocol]))
val spark = SparkSession
.builder()
.master("local")
.config(sparkConf)
.getOrCreate()
val config = NebulaConnectionConfig
.builder()
.withMetaAddress("127.0.0.1:9669")
.build()
val nebulaReadVertexConfig = ReadNebulaConfig
.builder()
.withSpace("csv")
.withLabel("user")
.withNoColumn(false)
.build()
val nebulaReadEdgeConfig = ReadNebulaConfig
.builder()
.withSpace("csv")
.withLabel("action")
.withNoColumn(false)
.build()
val vertexRDD = spark.read.nebula(config, nebulaReadVertexConfig).loadVerticesToGraphx()
val edgeRDD = spark.read.nebula(config, nebulaReadEdgeConfig).loadEdgesToGraphx()
val graph = Graph(vertexRDD, edgeRDD)
println("===============edges count================ is " + graph.edges.count())
println("===============Start run pageRank================time is " + df.format(System.currentTimeMillis()))
val ranks = graph.pageRank(0.0001).vertices
println("==============pageRank compute end================time is " + df.format(System.currentTimeMillis()))
println(ranks.top(10).mkString("\n"))
运行报错：
# spark-shell -i NebulaSparkReaderExample.scala
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data1/spark/spark-2.4.7-bin-hadoop2.7/jars/nebula-spark-connector-2.0.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/data1/spark/spark-2.4.7-bin-hadoop2.7/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
21/04/01 01:31:45 WARN Utils: Your hostname, tigergraph9001 resolves to a loopback address: 127.0.0.1; using 10.27.20.112 instead (on interface eth0)
21/04/01 01:31:45 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
21/04/01 01:31:46 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://tigergraph9001:4040
Spark context available as 'sc' (master = local[*], app id = local-1617240714905).
Spark session available as 'spark'.
21/04/01 01:31:59 WARN SparkSession$Builder: Using an existing SparkSession; some spark core configurations may not take effect.
21/04/01 01:31:59 WARN ReadNebulaConfig$: returnCols is empty and your result will contain all properties for user
21/04/01 01:31:59 WARN ReadNebulaConfig$: returnCols is empty and your result will contain all properties for action
com.facebook.thrift.TApplicationException: Method name getSpace not found
at com.facebook.thrift.TApplicationException.read(TApplicationException.java:130)
at com.vesoft.nebula.meta.MetaService$Client.recv_getSpace(MetaService.java:513)
at com.vesoft.nebula.meta.MetaService$Client.getSpace(MetaService.java:488)
at com.vesoft.nebula.client.meta.MetaClient.getSpace(MetaClient.java:131)
at com.vesoft.nebula.client.meta.MetaClient.getTag(MetaClient.java:175)
at com.vesoft.nebula.connector.nebula.MetaProvider.getTag(MetaProvider.scala:37)
at com.vesoft.nebula.connector.reader.NebulaSourceReader.getSchema(NebulaSourceReader.scala:68)
at com.vesoft.nebula.connector.reader.NebulaSourceReader.readSchema(NebulaSourceReader.scala:31)
at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation$.create(DataSourceV2Relation.scala:175)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:223)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:186)
at com.vesoft.nebula.connector.connector.package$NebulaDataFrameReader.loadVerticesToDF(package.scala:121)
at com.vesoft.nebula.connector.connector.package$NebulaDataFrameReader.loadVerticesToGraphx(package.scala:153)
... 65 elided
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.7
/_/
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_60)
Type in expressions to have them evaluated.
Type :help for more information.