exchange导入数据失败

没有啊,这个134是我新建的nebula,并没有其他程序在使用。。

还有,我在日志里找到了这么一行警告:

Unable to infer schema for table rzx_dw.sta_0_graph_czrk from file format ORC (inference mode: INFER_AND_SAVE). Using metastore schema.

不知道你见过这个没?

这是读取ORC的数据,你有ORC的数据源么

没听说过ORC啊
我百度到了这篇博客:hive小操作·关于 spark2.4-读取hive3.1事务表_concealed0的博客-CSDN博客
说是spark无法读取hive事务表
怎么解决也没写。。

你先操作下这个吧,看连接的环境是不是对的。

关于ORC 应该是你的hive 配置的文件存储格式是orc。

spark-sql里show tables是没啥问题的,和beeline里的show tables里看到的一样。
但是在beeline里查询sta_0_graph_czrk表是有数据的,在spark-sql里count sta_0_graph_czrk的结果是0。所以根本问题就是spark无法读取hive表里的数据。。。
关键错误信息是:Unable to infer schema for table rzx_dw.sta_0_graph_czrk from file format ORC (inference mode: INFER_AND_SAVE). Using metastore schema.
但是我不知道你怎么解决。。。

可以去Spark 官网看下,有类似的问题
https://issues.apache.org/jira/browse/SPARK-20799

这个问题我也遇到了,定位到了exchange对应代码是:

// $ nebula-spark-utils/nebula-exchange/src/main/scala/com/vesoft/nebula/exchange/Exchange.scala

// reimport for failed tags and edges
    if (ErrorHandler.existError(configs.errorConfig.errorPath)) {
      val batchSuccess = spark.sparkContext.longAccumulator(s"batchSuccess.reimport")
      val batchFailure = spark.sparkContext.longAccumulator(s"batchFailure.reimport")
      val data         = spark.read.text(configs.errorConfig.errorPath)
      val processor    = new ReloadProcessor(data, configs, batchSuccess, batchFailure)
      processor.process()
      LOG.info(s"batchSuccess.reimport: ${batchSuccess.value}")
      LOG.info(s"batchFailure.reimport: ${batchFailure.value}")
    }

这段代码应该是用于拉起失败的batch的,我还没有看里面的具体逻辑,但从我测试的结果来看,失败的任务信息是会被缓存起来的,即使kill掉对应进程,在重启nebula-exchange时也会拉起之前失败的任务。具体是不是这样还需要看一下代码逻辑。
所以我猜测你的问题可能是由于之前没有修改application.conf,然后exchange跑了一个OCR数据(可能是默认的?),任务失败后(大概率是因为缺少space),失败的batch信息被缓存起来。因此之后再调用exchange时,在跑完当前设置的tags/edges后会拉起之前失败的这个任务,因此出现了Space was not chosen.
#使用spark exchange导入报错
这个问题似乎也可以通过注释重启batch逻辑的代码段解决

2 个赞

space是配置在配置文件里的,有的任务会成功,有的会失败,这就很让人疑惑了

你把这种现象的日志发出来看下,你前面没有switch space的日志是正常的,因为读出来的数据是空的。

我很好奇在是什么情况下,nebula的错误日志会一直打印Space was not chosen??
我刚刚又试了一下,第一次启动任务,nebula又一直打印Space was not chosen.当时我kill掉yarn任务,重启后,正常了一段时间,excuroes的日志会打印switch space,然后又不正常了。nebula错误日志开始疯狂打印Space was not chosen.

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/insight/hdfs5/yarn/local/filecache/12/spark2-hdp-yarn-archive.tar.gz/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/3.0.1.0-187/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
21/06/30 09:36:00 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 35613@hd43.jnwj.com
21/06/30 09:36:00 INFO SignalUtils: Registered signal handler for TERM
21/06/30 09:36:00 INFO SignalUtils: Registered signal handler for HUP
21/06/30 09:36:00 INFO SignalUtils: Registered signal handler for INT
21/06/30 09:36:00 INFO SecurityManager: Changing view acls to: renzixing
21/06/30 09:36:00 INFO SecurityManager: Changing modify acls to: renzixing
21/06/30 09:36:00 INFO SecurityManager: Changing view acls groups to: 
21/06/30 09:36:00 INFO SecurityManager: Changing modify acls groups to: 
21/06/30 09:36:00 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(renzixing); groups with view permissions: Set(); users  with modify permissions: Set(renzixing); groups with modify permissions: Set()
21/06/30 09:36:01 INFO TransportClientFactory: Successfully created connection to /15.48.83.120:46485 after 98 ms (0 ms spent in bootstraps)
21/06/30 09:36:01 INFO SecurityManager: Changing view acls to: renzixing
21/06/30 09:36:01 INFO SecurityManager: Changing modify acls to: renzixing
21/06/30 09:36:01 INFO SecurityManager: Changing view acls groups to: 
21/06/30 09:36:01 INFO SecurityManager: Changing modify acls groups to: 
21/06/30 09:36:01 INFO SecurityManager: SecurityManager: authentication disabled; ui acls enabled; users  with view permissions: Set(renzixing); groups with view permissions: Set(); users  with modify permissions: Set(renzixing); groups with modify permissions: Set()
21/06/30 09:36:01 INFO TransportClientFactory: Successfully created connection to /15.48.83.120:46485 after 1 ms (0 ms spent in bootstraps)
21/06/30 09:36:01 INFO DiskBlockManager: Created local directory at /insight/hdfs1/yarn/local/usercache/renzixing/appcache/application_1622405928064_30190/blockmgr-60e6b4a1-6239-4693-a01f-d9d44739fbcc
21/06/30 09:36:01 INFO DiskBlockManager: Created local directory at /insight/hdfs2/yarn/local/usercache/renzixing/appcache/application_1622405928064_30190/blockmgr-9bb3e732-0c35-43d9-8bc8-86a894876a53
21/06/30 09:36:01 INFO DiskBlockManager: Created local directory at /insight/hdfs3/yarn/local/usercache/renzixing/appcache/application_1622405928064_30190/blockmgr-5b4f87a7-1756-4726-8ad8-527ff72fa657
21/06/30 09:36:01 INFO DiskBlockManager: Created local directory at /insight/hdfs4/yarn/local/usercache/renzixing/appcache/application_1622405928064_30190/blockmgr-2a825c05-1c23-4fb4-9240-821543621edb
21/06/30 09:36:01 INFO DiskBlockManager: Created local directory at /insight/hdfs5/yarn/local/usercache/renzixing/appcache/application_1622405928064_30190/blockmgr-69f13e0e-4075-44c7-9d2e-62a833be9f84
21/06/30 09:36:01 INFO DiskBlockManager: Created local directory at /insight/hdfs7/yarn/local/usercache/renzixing/appcache/application_1622405928064_30190/blockmgr-c5306d1c-117b-4300-a421-a046590d12da
21/06/30 09:36:01 INFO DiskBlockManager: Created local directory at /insight/hdfs8/yarn/local/usercache/renzixing/appcache/application_1622405928064_30190/blockmgr-ad2ae35e-52c4-4de4-a7cb-c3636cd904a0
21/06/30 09:36:01 INFO DiskBlockManager: Created local directory at /insight/hdfs9/yarn/local/usercache/renzixing/appcache/application_1622405928064_30190/blockmgr-915c0b57-6c23-44ec-9cbf-9e6748b48ab5
21/06/30 09:36:01 INFO MemoryStore: MemoryStore started with capacity 2.5 GB
21/06/30 09:36:02 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@15.48.83.120:46485
21/06/30 09:36:02 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
21/06/30 09:36:02 INFO Executor: Starting executor ID 1 on host hd43.jnwj.com
21/06/30 09:36:02 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 38080.
21/06/30 09:36:02 INFO NettyBlockTransferService: Server created on hd43.jnwj.com:38080
21/06/30 09:36:02 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
21/06/30 09:36:02 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(1, hd43.jnwj.com, 38080, None)
21/06/30 09:36:02 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(1, hd43.jnwj.com, 38080, None)
21/06/30 09:36:02 INFO BlockManager: Initialized BlockManager: BlockManagerId(1, hd43.jnwj.com, 38080, None)
21/06/30 09:36:15 INFO CoarseGrainedExecutorBackend: Got assigned task 1
21/06/30 09:36:15 INFO Executor: Running task 0.0 in stage 0.0 (TID 1)
21/06/30 09:36:15 INFO Executor: Fetching spark://15.48.83.120:46485/jars/nebula-exchange-2.0.0.jar with timestamp 1625016938723
21/06/30 09:36:15 INFO TransportClientFactory: Successfully created connection to /15.48.83.120:46485 after 2 ms (0 ms spent in bootstraps)
21/06/30 09:36:15 INFO Utils: Fetching spark://15.48.83.120:46485/jars/nebula-exchange-2.0.0.jar to /insight/hdfs1/yarn/local/usercache/renzixing/appcache/application_1622405928064_30190/spark-cd1a03c7-e231-4843-aaf5-e839f59a2e4d/fetchFileTemp161661024458422517.tmp
21/06/30 09:36:15 INFO Utils: Copying /insight/hdfs1/yarn/local/usercache/renzixing/appcache/application_1622405928064_30190/spark-cd1a03c7-e231-4843-aaf5-e839f59a2e4d/-16795906071625016938723_cache to /insight/hdfs5/yarn/local/usercache/renzixing/appcache/application_1622405928064_30190/container_e31_1622405928064_30190_01_000002/./nebula-exchange-2.0.0.jar
21/06/30 09:36:16 INFO Executor: Adding file:/insight/hdfs5/yarn/local/usercache/renzixing/appcache/application_1622405928064_30190/container_e31_1622405928064_30190_01_000002/./nebula-exchange-2.0.0.jar to class loader
21/06/30 09:36:16 INFO TorrentBroadcast: Started reading broadcast variable 1
21/06/30 09:36:16 INFO TransportClientFactory: Successfully created connection to hd3.jnwj.com/15.48.91.58:40368 after 1 ms (0 ms spent in bootstraps)
21/06/30 09:36:16 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 9.5 KB, free 2.5 GB)
21/06/30 09:36:16 INFO TorrentBroadcast: Reading broadcast variable 1 took 172 ms
21/06/30 09:36:17 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 23.9 KB, free 2.5 GB)
21/06/30 09:36:17 INFO CodeGenerator: Code generated in 366.00077 ms
21/06/30 09:36:17 INFO CodeGenerator: Code generated in 10.294945 ms
21/06/30 09:36:17 INFO CodeGenerator: Code generated in 13.669897 ms
21/06/30 09:36:17 INFO CodeGenerator: Code generated in 20.236844 ms
21/06/30 09:36:17 INFO FileScanRDD: Reading File path: hdfs://mycluster/apps/spark/warehouse/rzx_lmtp.db/sta_0_graph_hzth/000021_0, range: 0-5242557, partition values: [empty row]
21/06/30 09:36:17 INFO TorrentBroadcast: Started reading broadcast variable 0
21/06/30 09:36:17 INFO TransportClientFactory: Successfully created connection to /15.48.83.120:33795 after 1 ms (0 ms spent in bootstraps)
21/06/30 09:36:17 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 47.2 KB, free 2.5 GB)
21/06/30 09:36:17 INFO TorrentBroadcast: Reading broadcast variable 0 took 28 ms
21/06/30 09:36:17 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 638.8 KB, free 2.5 GB)
21/06/30 09:36:18 INFO OrcCodecPool: Got brand-new codec ZLIB
21/06/30 09:36:18 INFO ReaderImpl: Reading ORC rows from hdfs://mycluster/apps/spark/warehouse/rzx_lmtp.db/sta_0_graph_hzth/000021_0 with {include: [true, false, true, false, true, false, true, false, true, true], offset: 0, length: 5242557}
21/06/30 09:36:18 INFO RecordReaderImpl: Reader schema not provided -- using file schema struct<hh:string,src_sfzh:string,src_xm:string,des_sfzh:string,des_xm:string,yhzgx:string,yhzgx_detail:string,source:string,cjsj:string>
21/06/30 09:36:19 INFO FileScanRDD: Reading File path: hdfs://mycluster/apps/spark/warehouse/rzx_lmtp.db/sta_0_graph_hzth/000015_0, range: 0-5216011, partition values: [empty row]
21/06/30 09:36:19 INFO ReaderImpl: Reading ORC rows from hdfs://mycluster/apps/spark/warehouse/rzx_lmtp.db/sta_0_graph_hzth/000015_0 with {include: [true, false, true, false, true, false, true, false, true, true], offset: 0, length: 5216011}
21/06/30 09:36:19 INFO RecordReaderImpl: Reader schema not provided -- using file schema struct<hh:string,src_sfzh:string,src_xm:string,des_sfzh:string,des_xm:string,yhzgx:string,yhzgx_detail:string,source:string,cjsj:string>
21/06/30 09:36:19 INFO FileScanRDD: Reading File path: hdfs://mycluster/apps/spark/warehouse/rzx_lmtp.db/sta_0_graph_hzth/000019_0, range: 0-5207607, partition values: [empty row]
21/06/30 09:36:19 INFO ReaderImpl: Reading ORC rows from hdfs://mycluster/apps/spark/warehouse/rzx_lmtp.db/sta_0_graph_hzth/000019_0 with {include: [true, false, true, false, true, false, true, false, true, true], offset: 0, length: 5207607}
21/06/30 09:36:19 INFO RecordReaderImpl: Reader schema not provided -- using file schema struct<hh:string,src_sfzh:string,src_xm:string,des_sfzh:string,des_xm:string,yhzgx:string,yhzgx_detail:string,source:string,cjsj:string>
21/06/30 09:36:19 INFO FileScanRDD: Reading File path: hdfs://mycluster/apps/spark/warehouse/rzx_lmtp.db/sta_0_graph_hzth/000026_0, range: 0-5126944, partition values: [empty row]
21/06/30 09:36:19 INFO ReaderImpl: Reading ORC rows from hdfs://mycluster/apps/spark/warehouse/rzx_lmtp.db/sta_0_graph_hzth/000026_0 with {include: [true, false, true, false, true, false, true, false, true, true], offset: 0, length: 5126944}
21/06/30 09:36:19 INFO RecordReaderImpl: Reader schema not provided -- using file schema struct<hh:string,src_sfzh:string,src_xm:string,des_sfzh:string,des_xm:string,yhzgx:string,yhzgx_detail:string,source:string,cjsj:string>
21/06/30 09:36:19 INFO FileScanRDD: Reading File path: hdfs://mycluster/apps/spark/warehouse/rzx_lmtp.db/sta_0_graph_hzth/000007_0, range: 0-5029916, partition values: [empty row]
21/06/30 09:36:19 INFO ReaderImpl: Reading ORC rows from hdfs://mycluster/apps/spark/warehouse/rzx_lmtp.db/sta_0_graph_hzth/000007_0 with {include: [true, false, true, false, true, false, true, false, true, true], offset: 0, length: 5029916}
21/06/30 09:36:19 INFO RecordReaderImpl: Reader schema not provided -- using file schema struct<hh:string,src_sfzh:string,src_xm:string,des_sfzh:string,des_xm:string,yhzgx:string,yhzgx_detail:string,source:string,cjsj:string>
21/06/30 09:36:20 INFO FileScanRDD: Reading File path: hdfs://mycluster/apps/spark/warehouse/rzx_lmtp.db/sta_0_graph_hzth/000017_0, range: 0-4934666, partition values: [empty row]
21/06/30 09:36:20 INFO ReaderImpl: Reading ORC rows from hdfs://mycluster/apps/spark/warehouse/rzx_lmtp.db/sta_0_graph_hzth/000017_0 with {include: [true, false, true, false, true, false, true, false, true, true], offset: 0, length: 4934666}
21/06/30 09:36:20 INFO RecordReaderImpl: Reader schema not provided -- using file schema struct<hh:string,src_sfzh:string,src_xm:string,des_sfzh:string,des_xm:string,yhzgx:string,yhzgx_detail:string,source:string,cjsj:string>
21/06/30 09:36:20 INFO FileScanRDD: Reading File path: hdfs://mycluster/apps/spark/warehouse/rzx_lmtp.db/sta_0_graph_hzth/000005_0, range: 0-4929844, partition values: [empty row]
21/06/30 09:36:20 INFO ReaderImpl: Reading ORC rows from hdfs://mycluster/apps/spark/warehouse/rzx_lmtp.db/sta_0_graph_hzth/000005_0 with {include: [true, false, true, false, true, false, true, false, true, true], offset: 0, length: 4929844}
21/06/30 09:36:20 INFO RecordReaderImpl: Reader schema not provided -- using file schema struct<hh:string,src_sfzh:string,src_xm:string,des_sfzh:string,des_xm:string,yhzgx:string,yhzgx_detail:string,source:string,cjsj:string>
21/06/30 09:36:20 INFO FileScanRDD: Reading File path: hdfs://mycluster/apps/spark/warehouse/rzx_lmtp.db/sta_0_graph_hzth/000008_0, range: 0-4911712, partition values: [empty row]
21/06/30 09:36:20 INFO ReaderImpl: Reading ORC rows from hdfs://mycluster/apps/spark/warehouse/rzx_lmtp.db/sta_0_graph_hzth/000008_0 with {include: [true, false, true, false, true, false, true, false, true, true], offset: 0, length: 4911712}
21/06/30 09:36:20 INFO RecordReaderImpl: Reader schema not provided -- using file schema struct<hh:string,src_sfzh:string,src_xm:string,des_sfzh:string,des_xm:string,yhzgx:string,yhzgx_detail:string,source:string,cjsj:string>
21/06/30 09:36:20 INFO FileScanRDD: Reading File path: hdfs://mycluster/apps/spark/warehouse/rzx_lmtp.db/sta_0_graph_hzth/000001_0, range: 0-4903785, partition values: [empty row]
21/06/30 09:36:20 INFO ReaderImpl: Reading ORC rows from hdfs://mycluster/apps/spark/warehouse/rzx_lmtp.db/sta_0_graph_hzth/000001_0 with {include: [true, false, true, false, true, false, true, false, true, true], offset: 0, length: 4903785}
21/06/30 09:36:20 INFO RecordReaderImpl: Reader schema not provided -- using file schema struct<hh:string,src_sfzh:string,src_xm:string,des_sfzh:string,des_xm:string,yhzgx:string,yhzgx_detail:string,source:string,cjsj:string>
21/06/30 09:36:21 INFO FileScanRDD: Reading File path: hdfs://mycluster/apps/spark/warehouse/rzx_lmtp.db/sta_0_graph_hzth/000010_0, range: 0-4901081, partition values: [empty row]
21/06/30 09:36:21 INFO ReaderImpl: Reading ORC rows from hdfs://mycluster/apps/spark/warehouse/rzx_lmtp.db/sta_0_graph_hzth/000010_0 with {include: [true, false, true, false, true, false, true, false, true, true], offset: 0, length: 4901081}
21/06/30 09:36:21 INFO RecordReaderImpl: Reader schema not provided -- using file schema struct<hh:string,src_sfzh:string,src_xm:string,des_sfzh:string,des_xm:string,yhzgx:string,yhzgx_detail:string,source:string,cjsj:string>
21/06/30 09:36:21 INFO FileScanRDD: Reading File path: hdfs://mycluster/apps/spark/warehouse/rzx_lmtp.db/sta_0_graph_hzth/000000_0, range: 0-4893455, partition values: [empty row]
21/06/30 09:36:21 INFO ReaderImpl: Reading ORC rows from hdfs://mycluster/apps/spark/warehouse/rzx_lmtp.db/sta_0_graph_hzth/000000_0 with {include: [true, false, true, false, true, false, true, false, true, true], offset: 0, length: 4893455}
21/06/30 09:36:21 INFO RecordReaderImpl: Reader schema not provided -- using file schema struct<hh:string,src_sfzh:string,src_xm:string,des_sfzh:string,des_xm:string,yhzgx:string,yhzgx_detail:string,source:string,cjsj:string>
21/06/30 09:36:21 INFO FileScanRDD: Reading File path: hdfs://mycluster/apps/spark/warehouse/rzx_lmtp.db/sta_0_graph_hzth/000011_0, range: 0-4891621, partition values: [empty row]
21/06/30 09:36:21 INFO ReaderImpl: Reading ORC rows from hdfs://mycluster/apps/spark/warehouse/rzx_lmtp.db/sta_0_graph_hzth/000011_0 with {include: [true, false, true, false, true, false, true, false, true, true], offset: 0, length: 4891621}
21/06/30 09:36:21 INFO RecordReaderImpl: Reader schema not provided -- using file schema struct<hh:string,src_sfzh:string,src_xm:string,des_sfzh:string,des_xm:string,yhzgx:string,yhzgx_detail:string,source:string,cjsj:string>
21/06/30 09:36:21 INFO FileScanRDD: Reading File path: hdfs://mycluster/apps/spark/warehouse/rzx_lmtp.db/sta_0_graph_hzth/000020_0, range: 0-4885259, partition values: [empty row]
21/06/30 09:36:21 INFO ReaderImpl: Reading ORC rows from hdfs://mycluster/apps/spark/warehouse/rzx_lmtp.db/sta_0_graph_hzth/000020_0 with {include: [true, false, true, false, true, false, true, false, true, true], offset: 0, length: 4885259}
21/06/30 09:36:21 INFO RecordReaderImpl: Reader schema not provided -- using file schema struct<hh:string,src_sfzh:string,src_xm:string,des_sfzh:string,des_xm:string,yhzgx:string,yhzgx_detail:string,source:string,cjsj:string>
21/06/30 09:36:21 INFO FileScanRDD: Reading File path: hdfs://mycluster/apps/spark/warehouse/rzx_lmtp.db/sta_0_graph_hzth/000003_0, range: 0-4822510, partition values: [empty row]
21/06/30 09:36:21 INFO ReaderImpl: Reading ORC rows from hdfs://mycluster/apps/spark/warehouse/rzx_lmtp.db/sta_0_graph_hzth/000003_0 with {include: [true, false, true, false, true, false, true, false, true, true], offset: 0, length: 4822510}
21/06/30 09:36:21 INFO RecordReaderImpl: Reader schema not provided -- using file schema struct<hh:string,src_sfzh:string,src_xm:string,des_sfzh:string,des_xm:string,yhzgx:string,yhzgx_detail:string,source:string,cjsj:string>
21/06/30 09:36:22 INFO FileScanRDD: Reading File path: hdfs://mycluster/apps/spark/warehouse/rzx_lmtp.db/sta_0_graph_hzth/000009_0, range: 0-4786422, partition values: [empty row]
21/06/30 09:36:22 INFO ReaderImpl: Reading ORC rows from hdfs://mycluster/apps/spark/warehouse/rzx_lmtp.db/sta_0_graph_hzth/000009_0 with {include: [true, false, true, false, true, false, true, false, true, true], offset: 0, length: 4786422}
21/06/30 09:36:22 INFO RecordReaderImpl: Reader schema not provided -- using file schema struct<hh:string,src_sfzh:string,src_xm:string,des_sfzh:string,des_xm:string,yhzgx:string,yhzgx_detail:string,source:string,cjsj:string>
21/06/30 09:36:24 INFO Executor: Finished task 0.0 in stage 0.0 (TID 1). 2169 bytes result sent to driver
21/06/30 09:36:24 INFO CoarseGrainedExecutorBackend: Got assigned task 4
21/06/30 09:36:24 INFO Executor: Running task 0.0 in stage 1.0 (TID 4)
21/06/30 09:36:24 INFO MapOutputTrackerWorker: Updating epoch to 1 and clearing cache
21/06/30 09:36:24 INFO TorrentBroadcast: Started reading broadcast variable 2
21/06/30 09:36:24 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 9.7 KB, free 2.5 GB)
21/06/30 09:36:24 INFO TorrentBroadcast: Reading broadcast variable 2 took 19 ms
21/06/30 09:36:24 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 23.4 KB, free 2.5 GB)
21/06/30 09:36:24 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 0, fetching them
21/06/30 09:36:24 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@15.48.83.120:46485)
21/06/30 09:36:24 INFO MapOutputTrackerWorker: Got the output locations
21/06/30 09:36:25 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
21/06/30 09:36:25 INFO ShuffleBlockFetcherIterator: Started 1 remote fetches in 18 ms
21/06/30 09:36:25 INFO CodeGenerator: Code generated in 37.436892 ms
21/06/30 09:36:38 INFO Executor: Finished task 0.0 in stage 1.0 (TID 4). 2986 bytes result sent to driver
21/06/30 09:36:38 INFO CoarseGrainedExecutorBackend: Got assigned task 5
21/06/30 09:36:38 INFO Executor: Running task 0.0 in stage 2.0 (TID 5)
21/06/30 09:36:38 INFO MapOutputTrackerWorker: Updating epoch to 2 and clearing cache
21/06/30 09:36:38 INFO TorrentBroadcast: Started reading broadcast variable 3
21/06/30 09:36:38 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 11.7 KB, free 2.5 GB)
21/06/30 09:36:38 INFO TorrentBroadcast: Reading broadcast variable 3 took 13 ms
21/06/30 09:36:38 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 25.4 KB, free 2.5 GB)
21/06/30 09:36:38 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 1, fetching them
21/06/30 09:36:38 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@15.48.83.120:46485)
21/06/30 09:36:38 INFO MapOutputTrackerWorker: Got the output locations
21/06/30 09:36:38 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
21/06/30 09:36:38 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/30 09:36:39 INFO CodeGenerator: Code generated in 13.230947 ms
21/06/30 09:36:39 INFO NebulaPool: Get connection to 15.48.83.130:9669
21/06/30 09:36:39 INFO GraphProvider: switch space qw2graph3
21/06/30 09:36:39 INFO NebulaGraphClientWriter: Connection to List(15.48.83.130:9559)
21/06/30 09:36:49 INFO EdgeProcessor: spark partition for edge cost time:0-9646
21/06/30 09:36:49 INFO Executor: Finished task 0.0 in stage 2.0 (TID 5). 2905 bytes result sent to driver
21/06/30 09:36:49 INFO CoarseGrainedExecutorBackend: Got assigned task 7
21/06/30 09:36:49 INFO Executor: Running task 2.0 in stage 2.0 (TID 7)
21/06/30 09:36:49 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
21/06/30 09:36:49 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/30 09:36:49 INFO NebulaPool: Get connection to 15.48.83.130:9669
21/06/30 09:36:49 INFO GraphProvider: switch space qw2graph3
21/06/30 09:36:49 INFO NebulaGraphClientWriter: Connection to List(15.48.83.130:9559)
21/06/30 09:36:59 INFO EdgeProcessor: spark partition for edge cost time:2-10356
21/06/30 09:36:59 INFO Executor: Finished task 2.0 in stage 2.0 (TID 7). 2905 bytes result sent to driver
21/06/30 09:36:59 INFO CoarseGrainedExecutorBackend: Got assigned task 9
21/06/30 09:36:59 INFO Executor: Running task 4.0 in stage 2.0 (TID 9)
21/06/30 09:36:59 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
21/06/30 09:36:59 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/30 09:36:59 INFO NebulaPool: Get connection to 15.48.83.130:9669
21/06/30 09:36:59 INFO GraphProvider: switch space qw2graph3
21/06/30 09:36:59 INFO NebulaGraphClientWriter: Connection to List(15.48.83.130:9559)
21/06/30 09:37:09 INFO EdgeProcessor: spark partition for edge cost time:4-9625
21/06/30 09:37:09 INFO Executor: Finished task 4.0 in stage 2.0 (TID 9). 2905 bytes result sent to driver
21/06/30 09:37:09 INFO CoarseGrainedExecutorBackend: Got assigned task 11
21/06/30 09:37:09 INFO Executor: Running task 6.0 in stage 2.0 (TID 11)
21/06/30 09:37:09 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
21/06/30 09:37:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
21/06/30 09:37:09 INFO NebulaPool: Get connection to 15.48.83.130:9669
21/06/30 09:37:09 INFO GraphProvider: switch space qw2graph3
21/06/30 09:37:09 INFO NebulaGraphClientWriter: Connection to List(15.48.83.130:9559)
21/06/30 09:37:19 INFO EdgeProcessor: spark partition for edge cost time:6-9654
21/06/30 09:37:19 INFO Executor: Finished task 6.0 in stage 2.0 (TID 11). 2905 bytes result sent to driver
21/06/30 09:37:19 INFO CoarseGrainedExecutorBackend: Got assigned task 13
21/06/30 09:37:19 INFO Executor: Running task 8.0 in stage 2.0 (TID 13)
21/06/30 09:37:19 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
21/06/30 09:37:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/30 09:37:19 INFO NebulaPool: Get connection to 15.48.83.130:9669
21/06/30 09:37:19 INFO GraphProvider: switch space qw2graph3
21/06/30 09:37:19 INFO NebulaGraphClientWriter: Connection to List(15.48.83.130:9559)
21/06/30 09:37:28 INFO EdgeProcessor: spark partition for edge cost time:8-9447
21/06/30 09:37:28 INFO Executor: Finished task 8.0 in stage 2.0 (TID 13). 2905 bytes result sent to driver
21/06/30 09:37:28 INFO CoarseGrainedExecutorBackend: Got assigned task 15
21/06/30 09:37:28 INFO Executor: Running task 10.0 in stage 2.0 (TID 15)
21/06/30 09:37:28 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
21/06/30 09:37:28 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/30 09:37:28 INFO NebulaPool: Get connection to 15.48.83.130:9669
21/06/30 09:37:28 INFO GraphProvider: switch space qw2graph3
21/06/30 09:37:28 INFO NebulaGraphClientWriter: Connection to List(15.48.83.130:9559)
21/06/30 09:37:38 INFO EdgeProcessor: spark partition for edge cost time:10-9768
21/06/30 09:37:38 INFO Executor: Finished task 10.0 in stage 2.0 (TID 15). 2905 bytes result sent to driver
21/06/30 09:37:38 INFO CoarseGrainedExecutorBackend: Got assigned task 17
21/06/30 09:37:38 INFO Executor: Running task 12.0 in stage 2.0 (TID 17)
21/06/30 09:37:38 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
21/06/30 09:37:38 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/30 09:37:38 INFO NebulaPool: Get connection to 15.48.83.130:9669
21/06/30 09:37:38 INFO GraphProvider: switch space qw2graph3
21/06/30 09:37:38 INFO NebulaGraphClientWriter: Connection to List(15.48.83.130:9559)
21/06/30 09:37:48 INFO EdgeProcessor: spark partition for edge cost time:12-9647
21/06/30 09:37:48 INFO Executor: Finished task 12.0 in stage 2.0 (TID 17). 2905 bytes result sent to driver
21/06/30 09:37:48 INFO CoarseGrainedExecutorBackend: Got assigned task 19
21/06/30 09:37:48 INFO Executor: Running task 14.0 in stage 2.0 (TID 19)
21/06/30 09:37:48 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
21/06/30 09:37:48 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
21/06/30 09:37:48 INFO NebulaPool: Get connection to 15.48.83.130:9669
21/06/30 09:37:48 INFO GraphProvider: switch space qw2graph3
21/06/30 09:37:48 INFO NebulaGraphClientWriter: Connection to List(15.48.83.130:9559)
21/06/30 09:37:57 INFO EdgeProcessor: spark partition for edge cost time:14-9753
21/06/30 09:37:57 INFO Executor: Finished task 14.0 in stage 2.0 (TID 19). 2905 bytes result sent to driver
21/06/30 09:37:57 INFO CoarseGrainedExecutorBackend: Got assigned task 21
21/06/30 09:37:57 INFO Executor: Running task 16.0 in stage 2.0 (TID 21)
21/06/30 09:37:57 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
21/06/30 09:37:57 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/30 09:37:57 INFO NebulaPool: Get connection to 15.48.83.130:9669
21/06/30 09:37:57 INFO GraphProvider: switch space qw2graph3
21/06/30 09:37:57 INFO NebulaGraphClientWriter: Connection to List(15.48.83.130:9559)
21/06/30 09:38:07 INFO EdgeProcessor: spark partition for edge cost time:16-9606
21/06/30 09:38:07 INFO Executor: Finished task 16.0 in stage 2.0 (TID 21). 2905 bytes result sent to driver
21/06/30 09:38:07 INFO CoarseGrainedExecutorBackend: Got assigned task 23
21/06/30 09:38:07 INFO Executor: Running task 18.0 in stage 2.0 (TID 23)
21/06/30 09:38:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
21/06/30 09:38:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/30 09:38:07 INFO NebulaPool: Get connection to 15.48.83.130:9669
21/06/30 09:38:07 INFO GraphProvider: switch space qw2graph3
21/06/30 09:38:07 INFO NebulaGraphClientWriter: Connection to List(15.48.83.130:9559)
21/06/30 09:38:17 INFO EdgeProcessor: spark partition for edge cost time:18-9832
21/06/30 09:38:17 INFO Executor: Finished task 18.0 in stage 2.0 (TID 23). 2905 bytes result sent to driver
21/06/30 09:38:17 INFO CoarseGrainedExecutorBackend: Got assigned task 25
21/06/30 09:38:17 INFO Executor: Running task 20.0 in stage 2.0 (TID 25)
21/06/30 09:38:17 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
21/06/30 09:38:17 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/30 09:38:17 INFO NebulaPool: Get connection to 15.48.83.130:9669
21/06/30 09:38:17 INFO GraphProvider: switch space qw2graph3
21/06/30 09:38:17 INFO NebulaGraphClientWriter: Connection to List(15.48.83.130:9559)
21/06/30 09:38:27 INFO EdgeProcessor: spark partition for edge cost time:20-9796
21/06/30 09:38:27 INFO Executor: Finished task 20.0 in stage 2.0 (TID 25). 2905 bytes result sent to driver
21/06/30 09:38:27 INFO CoarseGrainedExecutorBackend: Got assigned task 27
21/06/30 09:38:27 INFO Executor: Running task 22.0 in stage 2.0 (TID 27)
21/06/30 09:38:27 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
21/06/30 09:38:27 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
21/06/30 09:38:27 INFO NebulaPool: Get connection to 15.48.83.130:9669
21/06/30 09:38:27 INFO GraphProvider: switch space qw2graph3
21/06/30 09:38:27 INFO NebulaGraphClientWriter: Connection to List(15.48.83.130:9559)
21/06/30 09:38:37 INFO EdgeProcessor: spark partition for edge cost time:22-10093
21/06/30 09:38:37 INFO Executor: Finished task 22.0 in stage 2.0 (TID 27). 2905 bytes result sent to driver
21/06/30 09:38:37 INFO CoarseGrainedExecutorBackend: Got assigned task 29
21/06/30 09:38:37 INFO Executor: Running task 24.0 in stage 2.0 (TID 29)
21/06/30 09:38:37 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
21/06/30 09:38:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/30 09:38:37 INFO NebulaPool: Get connection to 15.48.83.130:9669
21/06/30 09:38:37 INFO GraphProvider: switch space qw2graph3
21/06/30 09:38:37 INFO NebulaGraphClientWriter: Connection to List(15.48.83.130:9559)
21/06/30 09:38:47 INFO EdgeProcessor: spark partition for edge cost time:24-9745
21/06/30 09:38:47 INFO Executor: Finished task 24.0 in stage 2.0 (TID 29). 2905 bytes result sent to driver
21/06/30 09:38:47 INFO CoarseGrainedExecutorBackend: Got assigned task 31
21/06/30 09:38:47 INFO Executor: Running task 26.0 in stage 2.0 (TID 31)
21/06/30 09:38:47 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
21/06/30 09:38:47 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/30 09:38:47 INFO NebulaPool: Get connection to 15.48.83.130:9669
21/06/30 09:38:47 INFO GraphProvider: switch space qw2graph3
21/06/30 09:38:47 INFO NebulaGraphClientWriter: Connection to List(15.48.83.130:9559)
21/06/30 09:38:56 INFO EdgeProcessor: spark partition for edge cost time:26-9739
21/06/30 09:38:56 INFO Executor: Finished task 26.0 in stage 2.0 (TID 31). 2905 bytes result sent to driver
21/06/30 09:38:56 INFO CoarseGrainedExecutorBackend: Got assigned task 33
21/06/30 09:38:56 INFO Executor: Running task 28.0 in stage 2.0 (TID 33)
21/06/30 09:38:56 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
21/06/30 09:38:56 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
21/06/30 09:38:56 INFO NebulaPool: Get connection to 15.48.83.130:9669
21/06/30 09:38:56 INFO GraphProvider: switch space qw2graph3
21/06/30 09:38:56 INFO NebulaGraphClientWriter: Connection to List(15.48.83.130:9559)
21/06/30 09:39:06 INFO EdgeProcessor: spark partition for edge cost time:28-9588
21/06/30 09:39:06 INFO Executor: Finished task 28.0 in stage 2.0 (TID 33). 2905 bytes result sent to driver
21/06/30 09:39:06 INFO CoarseGrainedExecutorBackend: Got assigned task 35
21/06/30 09:39:06 INFO Executor: Running task 30.0 in stage 2.0 (TID 35)
21/06/30 09:39:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
21/06/30 09:39:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/06/30 09:39:06 INFO NebulaPool: Get connection to 15.48.83.130:9669
21/06/30 09:39:06 INFO GraphProvider: switch space qw2graph3
21/06/30 09:39:06 INFO NebulaGraphClientWriter: Connection to List(15.48.83.130:9559)
21/06/30 09:39:15 INFO EdgeProcessor: spark partition for edge cost time:30-9275
21/06/30 09:39:15 INFO Executor: Finished task 30.0 in stage 2.0 (TID 35). 2905 bytes result sent to driver
21/06/30 09:39:28 INFO CoarseGrainedExecutorBackend: Got assigned task 37
21/06/30 09:39:28 INFO Executor: Running task 81.0 in stage 3.0 (TID 37)
21/06/30 09:39:28 INFO TorrentBroadcast: Started reading broadcast variable 5
21/06/30 09:39:28 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 8.9 KB, free 2.5 GB)
21/06/30 09:39:28 INFO TorrentBroadcast: Reading broadcast variable 5 took 10 ms
21/06/30 09:39:28 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 17.3 KB, free 2.5 GB)
21/06/30 09:39:28 INFO CodeGenerator: Code generated in 10.954415 ms
21/06/30 09:39:29 INFO CodeGenerator: Code generated in 9.678466 ms
21/06/30 09:39:29 INFO NebulaPool: Get connection to 15.48.83.130:9669
21/06/30 09:39:29 INFO FileScanRDD: Reading File path: hdfs://mycluster/tmp/errors/reload.80, range: 0-90617978, partition values: [empty row]
21/06/30 09:39:29 INFO CodeGenerator: Code generated in 15.382938 ms
21/06/30 09:39:29 INFO TorrentBroadcast: Started reading broadcast variable 4
21/06/30 09:39:29 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 47.0 KB, free 2.5 GB)
21/06/30 09:39:29 INFO TorrentBroadcast: Reading broadcast variable 4 took 12 ms
21/06/30 09:39:29 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 638.8 KB, free 2.5 GB)
21/06/30 09:39:46 INFO Executor: Finished task 81.0 in stage 3.0 (TID 37). 1531 bytes result sent to driver
21/06/30 09:39:46 INFO CoarseGrainedExecutorBackend: Got assigned task 39
21/06/30 09:39:46 INFO Executor: Running task 105.0 in stage 3.0 (TID 39)
21/06/30 09:39:46 INFO NebulaPool: Get connection to 15.48.83.130:9669
21/06/30 09:39:46 INFO FileScanRDD: Reading File path: hdfs://mycluster/tmp/errors/reload.102, range: 0-77906250, partition values: [empty row]
21/06/30 09:40:00 INFO Executor: Finished task 105.0 in stage 3.0 (TID 39). 1488 bytes result sent to driver
21/06/30 09:40:00 INFO CoarseGrainedExecutorBackend: Got assigned task 41
21/06/30 09:40:00 INFO Executor: Running task 131.0 in stage 3.0 (TID 41)
21/06/30 09:40:00 INFO NebulaPool: Get connection to 15.48.83.130:9669
21/06/30 09:40:00 INFO FileScanRDD: Reading File path: hdfs://mycluster/tmp/errors/reload.137, range: 0-73658909, partition values: [empty row]

你看下Exchange中配置的error.path目录下是不是有文件。 导入之后你关注下batchSuccess.${tag}和batcFailure.${tag}的值

1、application.conf里error下只有一个output配置,是默认的/tmp/errors,我没修改。然后服务器/tmp目录下没有errors相关的文件夹或文件。
2、任务跑完之后我再看一下

你如果任务跑完了,可以来这边同步结果哈,谢谢 waynelee5991

最近导入数据 spark 一直报 [empty row] , 排查了好多次 找不到原因 今天下午终于找到问题了,我们 把 error 路径修改了后跑成功了 验证了 上边 那个朋友朋友Thericecookers 所说的 失败的任务信息是会被缓存起来的,即使kill掉对应进程,在重启nebula-exchange时也会拉起之前失败的任务。

2 个赞

@steam