2021-06-03 17:51:03,403 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - ShuffleMapStage 3 (save at package.scala:265) finished in 0.569 s 2021-06-03 17:51:03,403 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - looking for newly runnable stages 2021-06-03 17:51:03,403 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - running: Set() 2021-06-03 17:51:03,403 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - waiting: Set() 2021-06-03 17:51:03,403 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - failed: Set() 2021-06-03 17:51:03,405 INFO (main) [Logging.scala:logInfo(54)] - advisoryTargetPostShuffleInputSize: 67108864, targetPostShuffleInputSize 67108864. 2021-06-03 17:51:03,437 INFO (main) [Logging.scala:logInfo(54)] - Start processing data source writer: com.vesoft.nebula.connector.writer.NebulaDataSourceVertexWriter@6980e66a. The input RDD has 1 partitions. 2021-06-03 17:51:03,440 INFO (main) [Logging.scala:logInfo(54)] - Starting job: save at package.scala:265 2021-06-03 17:51:03,441 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Got job 3 (save at package.scala:265) with 1 output partitions 2021-06-03 17:51:03,441 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Final stage: ResultStage 5 (save at package.scala:265) 2021-06-03 17:51:03,441 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Parents of final stage: List(ShuffleMapStage 4) 2021-06-03 17:51:03,443 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Missing parents: List() 2021-06-03 17:51:03,443 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting ResultStage 5 (MapPartitionsRDD[33] at save at package.scala:265), which has no missing parents 2021-06-03 17:51:03,457 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_5 stored as values in memory (estimated size 42.6 KB, free 909.7 MB) 2021-06-03 17:51:03,462 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_5_piece0 stored as bytes in memory (estimated size 19.1 KB, free 909.7 MB) 2021-06-03 17:51:03,463 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Added broadcast_5_piece0 in memory on bdphdp07jobs14:31798 (size: 19.1 KB, free: 912.1 MB) 2021-06-03 17:51:03,464 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Created broadcast 5 from broadcast at DAGScheduler.scala:1161 2021-06-03 17:51:03,464 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting 1 missing tasks from ResultStage 5 (MapPartitionsRDD[33] at save at package.scala:265) (first 15 tasks are for partitions Vector(0)) 2021-06-03 17:51:03,464 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Adding task set 5.0 with 1 tasks 2021-06-03 17:51:03,466 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Starting task 0.0 in stage 5.0 (TID 81, bdphdp070020, executor 2, partition 0, NODE_LOCAL, 7780 bytes) 2021-06-03 17:51:03,500 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Added broadcast_5_piece0 in memory on bdphdp070020:4576 (size: 19.1 KB, free: 1048.6 MB) 2021-06-03 17:51:03,551 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Asked to send map output locations for shuffle 1 to 10.107.119.55:59388 2021-06-03 17:51:03,908 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Added rdd_30_0 in memory on bdphdp070020:4576 (size: 96.3 KB, free: 1048.5 MB) 2021-06-03 17:51:04,046 WARN (task-result-getter-0) [Logging.scala:logWarning(66)] - Lost task 0.0 in stage 5.0 (TID 81, bdphdp070020, executor 2): com.facebook.thrift.transport.TTransportException: java.net.NoRouteToHostException: No route to host (Host unreachable) at com.facebook.thrift.transport.TSocket.open(TSocket.java:175) at com.vesoft.nebula.client.meta.MetaClient.getClient(MetaClient.java:103) at com.vesoft.nebula.client.meta.MetaClient.doConnect(MetaClient.java:98) at com.vesoft.nebula.client.meta.MetaClient.connect(MetaClient.java:88) at com.vesoft.nebula.connector.nebula.MetaProvider.(MetaProvider.scala:22) at com.vesoft.nebula.connector.writer.NebulaWriter.(NebulaWriter.scala:24) at com.vesoft.nebula.connector.writer.NebulaVertexWriter.(NebulaVertexWriter.scala:19) at com.vesoft.nebula.connector.writer.NebulaVertexWriterFactory.createDataWriter(NebulaSourceWriter.scala:28) at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:113) at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec$$anonfun$doExecute$2.apply(WriteToDataSourceV2Exec.scala:67) at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec$$anonfun$doExecute$2.apply(WriteToDataSourceV2Exec.scala:66) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:121) at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.NoRouteToHostException: No route to host (Host unreachable) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at com.facebook.thrift.transport.TSocket.open(TSocket.java:170) ... 18 more 2021-06-03 17:51:04,047 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 0.1 in stage 5.0 (TID 82, bdphdp070020, executor 2, partition 0, NODE_LOCAL, 7780 bytes) 2021-06-03 17:51:04,068 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Lost task 0.1 in stage 5.0 (TID 82) on bdphdp070020, executor 2: com.facebook.thrift.transport.TTransportException (java.net.NoRouteToHostException: No route to host (Host unreachable)) [duplicate 1] 2021-06-03 17:51:04,070 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 0.2 in stage 5.0 (TID 83, bdphdp070020, executor 2, partition 0, NODE_LOCAL, 7780 bytes) 2021-06-03 17:51:06,095 WARN (task-result-getter-1) [Logging.scala:logWarning(66)] - Lost task 0.2 in stage 5.0 (TID 83, bdphdp070020, executor 2): com.facebook.thrift.transport.TTransportException: java.net.SocketTimeoutException: connect timed out at com.facebook.thrift.transport.TSocket.open(TSocket.java:175) at com.vesoft.nebula.client.meta.MetaClient.getClient(MetaClient.java:103) at com.vesoft.nebula.client.meta.MetaClient.doConnect(MetaClient.java:98) at com.vesoft.nebula.client.meta.MetaClient.connect(MetaClient.java:88) at com.vesoft.nebula.connector.nebula.MetaProvider.(MetaProvider.scala:22) at com.vesoft.nebula.connector.writer.NebulaWriter.(NebulaWriter.scala:24) at com.vesoft.nebula.connector.writer.NebulaVertexWriter.(NebulaVertexWriter.scala:19) at com.vesoft.nebula.connector.writer.NebulaVertexWriterFactory.createDataWriter(NebulaSourceWriter.scala:28) at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:113) at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec$$anonfun$doExecute$2.apply(WriteToDataSourceV2Exec.scala:67) at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec$$anonfun$doExecute$2.apply(WriteToDataSourceV2Exec.scala:66) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:121) at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.SocketTimeoutException: connect timed out at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at com.facebook.thrift.transport.TSocket.open(TSocket.java:170) ... 18 more 2021-06-03 17:51:06,096 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 0.3 in stage 5.0 (TID 84, bdphdp070011, executor 1, partition 0, NODE_LOCAL, 7780 bytes) 2021-06-03 17:51:06,162 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Added broadcast_5_piece0 in memory on bdphdp070011:25356 (size: 19.1 KB, free: 1048.6 MB) 2021-06-03 17:51:06,432 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Lost task 0.3 in stage 5.0 (TID 84) on bdphdp070011, executor 1: com.facebook.thrift.transport.TTransportException (java.net.NoRouteToHostException: No route to host (Host unreachable)) [duplicate 2] 2021-06-03 17:51:06,433 ERROR (task-result-getter-2) [Logging.scala:logError(70)] - Task 0 in stage 5.0 failed 4 times; aborting job 2021-06-03 17:51:06,435 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Removed TaskSet 5.0, whose tasks have all completed, from pool 2021-06-03 17:51:06,437 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Cancelling stage 5 2021-06-03 17:51:06,438 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Killing all running tasks in stage 5: Stage cancelled 2021-06-03 17:51:06,439 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - ResultStage 5 (save at package.scala:265) failed in 2.994 s due to Job aborted due to stage failure: Task 0 in stage 5.0 failed 4 times, most recent failure: Lost task 0.3 in stage 5.0 (TID 84, bdphdp070011, executor 1): com.facebook.thrift.transport.TTransportException: java.net.NoRouteToHostException: No route to host (Host unreachable) at com.facebook.thrift.transport.TSocket.open(TSocket.java:175) at com.vesoft.nebula.client.meta.MetaClient.getClient(MetaClient.java:103) at com.vesoft.nebula.client.meta.MetaClient.doConnect(MetaClient.java:98) at com.vesoft.nebula.client.meta.MetaClient.connect(MetaClient.java:88) at com.vesoft.nebula.connector.nebula.MetaProvider.(MetaProvider.scala:22) at com.vesoft.nebula.connector.writer.NebulaWriter.(NebulaWriter.scala:24) at com.vesoft.nebula.connector.writer.NebulaVertexWriter.(NebulaVertexWriter.scala:19) at com.vesoft.nebula.connector.writer.NebulaVertexWriterFactory.createDataWriter(NebulaSourceWriter.scala:28) at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:113) at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec$$anonfun$doExecute$2.apply(WriteToDataSourceV2Exec.scala:67) at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec$$anonfun$doExecute$2.apply(WriteToDataSourceV2Exec.scala:66) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:121) at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.NoRouteToHostException: No route to host (Host unreachable) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at com.facebook.thrift.transport.TSocket.open(TSocket.java:170) ... 18 more Driver stacktrace: 2021-06-03 17:51:06,440 INFO (main) [Logging.scala:logInfo(54)] - Job 3 failed: save at package.scala:265, took 2.999412 s 2021-06-03 17:51:06,441 ERROR (main) [Logging.scala:logError(70)] - Data source writer com.vesoft.nebula.connector.writer.NebulaDataSourceVertexWriter@6980e66a is aborting. 2021-06-03 17:51:06,441 ERROR (main) [NebulaSourceWriter.scala:abort(70)] - NebulaDataSourceVertexWriter abort 2021-06-03 17:51:06,441 ERROR (main) [Logging.scala:logError(70)] - Data source writer com.vesoft.nebula.connector.writer.NebulaDataSourceVertexWriter@6980e66a aborted. 2021-06-03 17:51:06,445 INFO (Thread-1) [Logging.scala:logInfo(54)] - Invoking stop() from shutdown hook 2021-06-03 17:51:06,451 INFO (Thread-1) [Logging.scala:logInfo(54)] - Stopped Spark web UI at http://bdphdp07jobs14:4040 2021-06-03 17:51:06,454 INFO (YARN application state monitor) [Logging.scala:logInfo(54)] - Interrupting monitor thread 2021-06-03 17:51:06,478 INFO (Thread-1) [Logging.scala:logInfo(54)] - Shutting down all executors 2021-06-03 17:51:06,478 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Asking each executor to shut down 2021-06-03 17:51:06,480 INFO (Thread-1) [Logging.scala:logInfo(54)] - Stopping SchedulerExtensionServices (serviceOption=None, services=List(), started=false) 2021-06-03 17:51:06,481 INFO (Thread-1) [Logging.scala:logInfo(54)] - Stopped 2021-06-03 17:51:06,525 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - MapOutputTrackerMasterEndpoint stopped! 2021-06-03 17:51:06,532 INFO (Thread-1) [Logging.scala:logInfo(54)] - MemoryStore cleared 2021-06-03 17:51:06,533 INFO (Thread-1) [Logging.scala:logInfo(54)] - BlockManager stopped 2021-06-03 17:51:06,533 INFO (Thread-1) [Logging.scala:logInfo(54)] - BlockManagerMaster stopped 2021-06-03 17:51:06,535 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - OutputCommitCoordinator stopped! 2021-06-03 17:51:06,540 INFO (Thread-1) [Logging.scala:logInfo(54)] - Successfully stopped SparkContext 2021-06-03 17:51:06,541 INFO (Thread-1) [Logging.scala:logInfo(54)] - Shutdown hook called 2021-06-03 17:51:06,541 INFO (Thread-1) [Logging.scala:logInfo(54)] - Deleting directory /appcom/logs/spark/tmp/spark-541932f1-cda4-4b14-9a3a-a691fae8c0b3 2021-06-03 17:51:06,544 INFO (Thread-1) [Logging.scala:logInfo(54)] - Deleting directory /tmp/spark-1f8232f9-07e9-4bed-a976-fe76f7670085