spark导入mysql数据报错

com · 2021 年9 月 28 日 05:29

nebula 版本：2.5
部署方式：单机
是否为线上版本：N
硬件信息
- 磁盘（（HDD)

exchange 执行导入命令导入mysql数据报错
spark 版本:2.4.5
mysql版本：5.6
mysql-connector-java版本:mysql-connector-java-5.1.49.jar
尝试在连接配置的数据库后面加上zeroDateTimeBehavior=convertToNull。报错“ The connection property ‘zeroDateTimeBehavior’ only accepts values of the form: ‘exception’, ‘round’ or ‘convertToNull’. The value ‘convertToNull?useUnicode=true’ is not in this set.”

21/09/28 13:03:12 INFO DAGScheduler: Job 0 failed: foreachPartition at VerticesProcessor.scala:274, took 3.169806 s
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost, executor driver): java.sql.SQLException: Value '	13826200823z2kaESWz+C14hM1y5De7VQ==10100086��2552552550255255255255100010000.60000000000000�0000-00-00 00:00:002019-05-21 16:17:22' can not be represented as java.sql.Timestamp
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:898)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:887)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:861)
	at com.mysql.jdbc.ResultSetRow.getTimestampFast(ResultSetRow.java:947)
	at com.mysql.jdbc.BufferRow.getTimestampFast(BufferRow.java:559)
	at com.mysql.jdbc.ResultSetImpl.getTimestampInternal(ResultSetImpl.java:5913)
	at com.mysql

steam · 2021 年9 月 28 日 05:46

exchange 的版本号呢？

再把你的配置信息贴一下，看报错信息好像是你导入的数据格式有问题

com · 2021 年9 月 28 日 06:26

现在我把crtime全都设置为null，不报错了，但是有新的报错。
exchange版本：2.0.1
配置文件

{
     # Spark相关配置
     spark: {
         app: {
            name: Nebula Exchange 2.5.0
         }
         driver: {
              cores: 1
              maxResultSize: 1G
         }
         cores {
              max: 16
         }
     }
    
    # Nebula Graph相关配置
     nebula: {
         address:{
          # 以下为Nebula Graph的Graph服务和Meta服务所在机器的IP地址及端口。
          # 如果有多个地址，格式为 "ip1:port","ip2:port","ip3:port"。
          # 不同地址之间以英文逗号 (,) 隔开。
          graph:["*:9669"]
          meta:["*:9559"]
          }
         # 填写的账号必须拥有Nebula Graph相应图空间的写数据权限。
         user: root
         pswd: nebula
         # 填写Nebula Graph中需要写入数据的图空间名称。
         space: network_space
         connection {
          timeout: 3000
          retry: 3
         }
         execution {
          retry: 3
         }
         error: {
          max: 32
          output: /tmp/errors
         }
         rate: {
          limit: 1024
          timeout: 1000
         }
     }
     # 处理点
     tags: [
         # 设置Tag player相关信息。
         {
          # Nebula Graph中对应的Tag名称。
          name: user
          type: {
          # 指定数据源文件格式，设置为MySQL。
          source: mysql
          # 指定如何将点数据导入Nebula Graph：Client或SST。
          sink: client
          }
        
         host:**
          port: 34409
          database:"groot"
          table:"users00"
          user:"maimai"
          password:"maimai"
          sentence:"select id, realname from users00 where realname!='' limit 1;"
        
          # 在fields里指定player表中的列名称，其对应的value会作为Nebula Graph中指定属性
        。
          # fields和nebula.fields里的配置必须一一对应。
          # 如果需要指定多个列名称，用英文逗号（,）隔开。
          fields: [id, realname]
          nebula.fields: [uid,name]
           # 指定表中某一列数据为Nebula Graph中点VID的来源。
          vertex: {
          field: id
          }
        
          # 单批次写入 Nebula Graph 的数据条数。
          batch: 256
          partition: 32
         }
     ]
 }

报错信息：

Exception in thread "main" com.facebook.thrift.protocol.TProtocolException: The field 'code' has been assigned the invalid value -4
	at com.vesoft.nebula.meta.GetSpaceResp.validate(GetSpaceResp.java:488)
	at com.vesoft.nebula.meta.GetSpaceResp.read(GetSpaceResp.java:398)
	at com.vesoft.nebula.meta.MetaService$getSpace_result.read(MetaService.java:10053)
	at com.vesoft.nebula.meta.MetaService$Client.recv_getSpace(MetaService.java:522)
	at com.vesoft.nebula.meta.MetaService$Client.getSpace(MetaService.java:492)
	at com.vesoft.nebula.client.meta.MetaClient.getSpace(MetaClient.java:131)
	at com.vesoft.nebula.client.meta.MetaClient.getTags(MetaClient.java:144)
	at com.vesoft.nebula.exchange.MetaProvider.getLabelType(MetaProvider.scala:69)
	at com.vesoft.nebula.exchange.utils.NebulaUtils$.getDataSourceFieldType(NebulaUtils.scala:32)
	at com.vesoft.nebula.exchange.processor.VerticesProcessor.process(VerticesProcessor.scala:110)
	at com.vesoft.nebula.exchange.Exchange$$anonfun$main$2.apply(Exchange.scala:145)
	at com.vesoft.nebula.exchange.Exchange$$anonfun$main$2.apply(Exchange.scala:122)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at com.vesoft.nebula.exchange.Exchange$.main(Exchange.scala:122)
	at com.vesoft.nebula.exchange.Exchange.main(Exchange.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

nicole · 2021 年9 月 28 日 07:37

java.sql.SQLException: Value ’ 13826200823z2kaESWz+C14hM1y5De7VQ==10100086��2552552550255255255255100010000.60000000000000�0000-00-00 00:00:002019-05-21 16:17:22’ can not be represented as java.sql.Timestamp
这个是数据格式不对

这个是nebula版本与exchange版本不匹配