22/02/21 08:10:16 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
log4j:WARN No appenders could be found for logger (com.vesoft.nebula.exchange.config.Configs$).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
22/02/21 08:10:16 INFO SparkContext: Running Spark version 2.4.6
22/02/21 08:10:16 INFO SparkContext: Submitted application: com.vesoft.nebula.exchange.Exchange
22/02/21 08:10:17 INFO SecurityManager: Changing view acls to: spark
22/02/21 08:10:17 INFO SecurityManager: Changing modify acls to: spark
22/02/21 08:10:17 INFO SecurityManager: Changing view acls groups to: 
22/02/21 08:10:17 INFO SecurityManager: Changing modify acls groups to:
22/02/21 08:10:17 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(spark); groups with view permissions: Set(); users  with modify permissions: Set(spark); groups with modify permissions: Set()
22/02/21 08:10:17 INFO Utils: Successfully started service 'sparkDriver' on port 32875.
22/02/21 08:10:17 INFO SparkEnv: Registering MapOutputTracker
22/02/21 08:10:17 INFO SparkEnv: Registering BlockManagerMaster
22/02/21 08:10:17 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
22/02/21 08:10:17 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
22/02/21 08:10:17 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-8c9b1781-2541-46a0-842a-8cef4b36e2cb
22/02/21 08:10:17 INFO MemoryStore: MemoryStore started with capacity 366.3 MB
22/02/21 08:10:17 INFO SparkEnv: Registering OutputCommitCoordinator
22/02/21 08:10:17 INFO Utils: Successfully started service 'SparkUI' on port 4040.
22/02/21 08:10:17 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://b213d13690e2:4040
22/02/21 08:10:17 INFO SparkContext: Added JAR file:/tmp/nebula-exchange-2.6.1.jar at spark://b213d13690e2:32875/jars/nebula-exchange-2.6.1.jar with timestamp 1645431017984
22/02/21 08:10:18 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://master:7077...
22/02/21 08:10:18 INFO TransportClientFactory: Successfully created connection to master/172.32.0.3:7077 after 41 ms (0 ms spent in bootstraps)
22/02/21 08:10:18 INFO StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20220221081018-0009
22/02/21 08:10:18 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40461.
22/02/21 08:10:18 INFO NettyBlockTransferService: Server created on b213d13690e2:40461
22/02/21 08:10:18 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
22/02/21 08:10:18 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20220221081018-0009/0 on worker-20220221021658-172.32.0.2-41337 (172.32.0.2:41337) with 1 core(s)
22/02/21 08:10:18 INFO StandaloneSchedulerBackend: Granted executor ID app-20220221081018-0009/0 on hostPort 172.32.0.2:41337 with 1 core(s), 1024.0 MB RAM
22/02/21 08:10:18 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, b213d13690e2, 40461, None)
22/02/21 08:10:18 INFO BlockManagerMasterEndpoint: Registering block manager b213d13690e2:40461 with 366.3 MB RAM, BlockManagerId(driver, b213d13690e2, 40461, None)
22/02/21 08:10:18 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, b213d13690e2, 40461, None)
22/02/21 08:10:18 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, b213d13690e2, 40461, None)
22/02/21 08:10:18 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20220221081018-0009/0 is now RUNNING
22/02/21 08:10:18 INFO StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
22/02/21 08:10:18 INFO Exchange$: Processing Tag player
22/02/21 08:10:18 INFO Exchange$: field keys: name, age
22/02/21 08:10:18 INFO Exchange$: nebula keys: name, age
22/02/21 08:10:18 INFO ClickHouseDriver: Driver registered
22/02/21 08:10:19 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/opt/bitnami/spark/spark-warehouse').
22/02/21 08:10:19 INFO SharedState: Warehouse path is 'file:/opt/bitnami/spark/spark-warehouse'.
22/02/21 08:10:20 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
22/02/21 08:10:20 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.32.0.2:35932) with ID 0
22/02/21 08:10:21 INFO BlockManagerMasterEndpoint: Registering block manager 172.32.0.2:34219 with 366.3 MB RAM, BlockManagerId(0, 172.32.0.2, 34219, None)
22/02/21 08:10:23 INFO CodeGenerator: Code generated in 233.198828 ms
22/02/21 08:10:23 INFO ContextCleaner: Cleaned accumulator 2
22/02/21 08:10:23 INFO SparkContext: Starting job: foreachPartition at VerticesProcessor.scala:276
22/02/21 08:10:23 INFO DAGScheduler: Registering RDD 3 (foreachPartition at VerticesProcessor.scala:276) as input to shuffle 0
22/02/21 08:10:23 INFO DAGScheduler: Got job 0 (foreachPartition at VerticesProcessor.scala:276) with 32 output partitions
22/02/21 08:10:23 INFO DAGScheduler: Final stage: ResultStage 1 (foreachPartition at VerticesProcessor.scala:276)
22/02/21 08:10:23 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0)
22/02/21 08:10:23 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0)
22/02/21 08:10:23 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[3] at foreachPartition at VerticesProcessor.scala:276), which has no missing parents
22/02/21 08:10:24 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 10.1 KB, free 366.3 MB)
22/02/21 08:10:24 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 5.4 KB, free 366.3 MB)
22/02/21 08:10:24 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on b213d13690e2:40461 (size: 5.4 KB, free: 366.3 MB)
22/02/21 08:10:24 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1163
22/02/21 08:10:24 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[3] at foreachPartition at VerticesProcessor.scala:276) (first 15 tasks are for partitions Vector(0))
22/02/21 08:10:24 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
22/02/21 08:10:24 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, 172.32.0.2, executor 0, partition 0, PROCESS_LOCAL, 7683 bytes)
22/02/21 08:10:26 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 172.32.0.2:34219 (size: 5.4 KB, free: 366.3 MB)
22/02/21 08:10:28 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 4114 ms on 172.32.0.2 (executor 0) (1/1)
22/02/21 08:10:28 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
22/02/21 08:10:28 INFO DAGScheduler: ShuffleMapStage 0 (foreachPartition at VerticesProcessor.scala:276) finished in 4.690 s
22/02/21 08:10:28 INFO DAGScheduler: looking for newly runnable stages
22/02/21 08:10:28 INFO DAGScheduler: running: Set()
22/02/21 08:10:28 INFO DAGScheduler: waiting: Set(ResultStage 1)
22/02/21 08:10:28 INFO DAGScheduler: failed: Set()
22/02/21 08:10:28 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[8] at foreachPartition at VerticesProcessor.scala:276), which has no missing parents
22/02/21 08:10:28 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 22.4 KB, free 366.3 MB)
22/02/21 08:10:28 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 11.3 KB, free 366.3 MB)
22/02/21 08:10:28 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on b213d13690e2:40461 (size: 11.3 KB, free: 366.3 MB)
22/02/21 08:10:28 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1163
22/02/21 08:10:28 INFO DAGScheduler: Submitting 32 missing tasks from ResultStage 1 (MapPartitionsRDD[8] at foreachPartition at VerticesProcessor.scala:276) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
22/02/21 08:10:28 INFO TaskSchedulerImpl: Adding task set 1.0 with 32 tasks
22/02/21 08:10:28 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, 172.32.0.2, executor 0, partition 0, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:28 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 172.32.0.2:34219 (size: 11.3 KB, free: 366.3 MB)
22/02/21 08:10:29 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 172.32.0.2:35932
22/02/21 08:10:30 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 2, 172.32.0.2, executor 0, partition 1, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:30 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 1997 ms on 172.32.0.2 (executor 0) (1/32)
22/02/21 08:10:30 INFO TaskSetManager: Starting task 24.0 in stage 1.0 (TID 3, 172.32.0.2, executor 0, partition 24, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:30 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 2) in 155 ms on 172.32.0.2 (executor 0) (2/32)
22/02/21 08:10:31 INFO TaskSetManager: Starting task 25.0 in stage 1.0 (TID 4, 172.32.0.2, executor 0, partition 25, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:31 INFO TaskSetManager: Finished task 24.0 in stage 1.0 (TID 3) in 195 ms on 172.32.0.2 (executor 0) (3/32)
22/02/21 08:10:31 INFO TaskSetManager: Starting task 26.0 in stage 1.0 (TID 5, 172.32.0.2, executor 0, partition 26, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:31 INFO TaskSetManager: Finished task 25.0 in stage 1.0 (TID 4) in 131 ms on 172.32.0.2 (executor 0) (4/32)
22/02/21 08:10:31 INFO TaskSetManager: Starting task 27.0 in stage 1.0 (TID 6, 172.32.0.2, executor 0, partition 27, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:31 INFO TaskSetManager: Finished task 26.0 in stage 1.0 (TID 5) in 145 ms on 172.32.0.2 (executor 0) (5/32)
22/02/21 08:10:31 INFO TaskSetManager: Starting task 28.0 in stage 1.0 (TID 7, 172.32.0.2, executor 0, partition 28, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:31 INFO TaskSetManager: Finished task 27.0 in stage 1.0 (TID 6) in 121 ms on 172.32.0.2 (executor 0) (6/32)
22/02/21 08:10:31 INFO TaskSetManager: Starting task 29.0 in stage 1.0 (TID 8, 172.32.0.2, executor 0, partition 29, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:31 INFO TaskSetManager: Finished task 28.0 in stage 1.0 (TID 7) in 149 ms on 172.32.0.2 (executor 0) (7/32)
22/02/21 08:10:31 INFO TaskSetManager: Starting task 30.0 in stage 1.0 (TID 9, 172.32.0.2, executor 0, partition 30, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:31 INFO TaskSetManager: Finished task 29.0 in stage 1.0 (TID 8) in 111 ms on 172.32.0.2 (executor 0) (8/32)
22/02/21 08:10:31 INFO TaskSetManager: Starting task 31.0 in stage 1.0 (TID 10, 172.32.0.2, executor 0, partition 31, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:31 INFO TaskSetManager: Finished task 30.0 in stage 1.0 (TID 9) in 140 ms on 172.32.0.2 (executor 0) (9/32)
22/02/21 08:10:32 INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID 11, 172.32.0.2, executor 0, partition 2, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:32 INFO TaskSetManager: Finished task 31.0 in stage 1.0 (TID 10) in 185 ms on 172.32.0.2 (executor 0) (10/32)
22/02/21 08:10:32 INFO TaskSetManager: Starting task 3.0 in stage 1.0 (TID 12, 172.32.0.2, executor 0, partition 3, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:32 INFO TaskSetManager: Finished task 2.0 in stage 1.0 (TID 11) in 106 ms on 172.32.0.2 (executor 0) (11/32)
22/02/21 08:10:32 INFO TaskSetManager: Starting task 4.0 in stage 1.0 (TID 13, 172.32.0.2, executor 0, partition 4, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:32 INFO TaskSetManager: Finished task 3.0 in stage 1.0 (TID 12) in 106 ms on 172.32.0.2 (executor 0) (12/32)
22/02/21 08:10:32 INFO TaskSetManager: Starting task 5.0 in stage 1.0 (TID 14, 172.32.0.2, executor 0, partition 5, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:32 INFO TaskSetManager: Finished task 4.0 in stage 1.0 (TID 13) in 125 ms on 172.32.0.2 (executor 0) (13/32)
22/02/21 08:10:32 INFO TaskSetManager: Starting task 6.0 in stage 1.0 (TID 15, 172.32.0.2, executor 0, partition 6, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:32 INFO TaskSetManager: Finished task 5.0 in stage 1.0 (TID 14) in 113 ms on 172.32.0.2 (executor 0) (14/32)
22/02/21 08:10:32 INFO TaskSetManager: Starting task 7.0 in stage 1.0 (TID 16, 172.32.0.2, executor 0, partition 7, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:32 INFO TaskSetManager: Finished task 6.0 in stage 1.0 (TID 15) in 116 ms on 172.32.0.2 (executor 0) (15/32)
22/02/21 08:10:32 INFO TaskSetManager: Starting task 8.0 in stage 1.0 (TID 17, 172.32.0.2, executor 0, partition 8, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:32 INFO TaskSetManager: Finished task 7.0 in stage 1.0 (TID 16) in 125 ms on 172.32.0.2 (executor 0) (16/32)
22/02/21 08:10:32 INFO TaskSetManager: Starting task 9.0 in stage 1.0 (TID 18, 172.32.0.2, executor 0, partition 9, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:32 INFO TaskSetManager: Finished task 8.0 in stage 1.0 (TID 17) in 127 ms on 172.32.0.2 (executor 0) (17/32)
22/02/21 08:10:32 INFO TaskSetManager: Starting task 10.0 in stage 1.0 (TID 19, 172.32.0.2, executor 0, partition 10, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:32 INFO TaskSetManager: Finished task 9.0 in stage 1.0 (TID 18) in 126 ms on 172.32.0.2 (executor 0) (18/32)
22/02/21 08:10:33 INFO TaskSetManager: Starting task 11.0 in stage 1.0 (TID 20, 172.32.0.2, executor 0, partition 11, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:33 INFO TaskSetManager: Finished task 10.0 in stage 1.0 (TID 19) in 119 ms on 172.32.0.2 (executor 0) (19/32)
22/02/21 08:10:33 INFO TaskSetManager: Starting task 12.0 in stage 1.0 (TID 21, 172.32.0.2, executor 0, partition 12, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:33 INFO TaskSetManager: Finished task 11.0 in stage 1.0 (TID 20) in 100 ms on 172.32.0.2 (executor 0) (20/32)
22/02/21 08:10:33 INFO TaskSetManager: Starting task 13.0 in stage 1.0 (TID 22, 172.32.0.2, executor 0, partition 13, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:33 INFO TaskSetManager: Finished task 12.0 in stage 1.0 (TID 21) in 108 ms on 172.32.0.2 (executor 0) (21/32)
22/02/21 08:10:33 INFO TaskSetManager: Starting task 14.0 in stage 1.0 (TID 23, 172.32.0.2, executor 0, partition 14, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:33 INFO TaskSetManager: Finished task 13.0 in stage 1.0 (TID 22) in 139 ms on 172.32.0.2 (executor 0) (22/32)
22/02/21 08:10:33 INFO TaskSetManager: Starting task 15.0 in stage 1.0 (TID 24, 172.32.0.2, executor 0, partition 15, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:33 INFO TaskSetManager: Finished task 14.0 in stage 1.0 (TID 23) in 92 ms on 172.32.0.2 (executor 0) (23/32)
22/02/21 08:10:33 INFO TaskSetManager: Starting task 16.0 in stage 1.0 (TID 25, 172.32.0.2, executor 0, partition 16, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:33 INFO TaskSetManager: Finished task 15.0 in stage 1.0 (TID 24) in 79 ms on 172.32.0.2 (executor 0) (24/32)
22/02/21 08:10:33 INFO TaskSetManager: Starting task 17.0 in stage 1.0 (TID 26, 172.32.0.2, executor 0, partition 17, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:33 INFO TaskSetManager: Finished task 16.0 in stage 1.0 (TID 25) in 87 ms on 172.32.0.2 (executor 0) (25/32)
22/02/21 08:10:33 INFO TaskSetManager: Starting task 18.0 in stage 1.0 (TID 27, 172.32.0.2, executor 0, partition 18, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:33 INFO TaskSetManager: Finished task 17.0 in stage 1.0 (TID 26) in 117 ms on 172.32.0.2 (executor 0) (26/32)
22/02/21 08:10:33 INFO TaskSetManager: Starting task 19.0 in stage 1.0 (TID 28, 172.32.0.2, executor 0, partition 19, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:33 INFO TaskSetManager: Finished task 18.0 in stage 1.0 (TID 27) in 86 ms on 172.32.0.2 (executor 0) (27/32)
22/02/21 08:10:33 INFO TaskSetManager: Starting task 20.0 in stage 1.0 (TID 29, 172.32.0.2, executor 0, partition 20, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:33 INFO TaskSetManager: Finished task 19.0 in stage 1.0 (TID 28) in 79 ms on 172.32.0.2 (executor 0) (28/32)
22/02/21 08:10:34 INFO TaskSetManager: Starting task 21.0 in stage 1.0 (TID 30, 172.32.0.2, executor 0, partition 21, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:34 INFO TaskSetManager: Finished task 20.0 in stage 1.0 (TID 29) in 88 ms on 172.32.0.2 (executor 0) (29/32)
22/02/21 08:10:34 INFO TaskSetManager: Starting task 22.0 in stage 1.0 (TID 31, 172.32.0.2, executor 0, partition 22, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:34 INFO TaskSetManager: Finished task 21.0 in stage 1.0 (TID 30) in 79 ms on 172.32.0.2 (executor 0) (30/32)
22/02/21 08:10:34 INFO TaskSetManager: Starting task 23.0 in stage 1.0 (TID 32, 172.32.0.2, executor 0, partition 23, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:34 INFO TaskSetManager: Finished task 22.0 in stage 1.0 (TID 31) in 87 ms on 172.32.0.2 (executor 0) (31/32)
22/02/21 08:10:34 INFO TaskSetManager: Finished task 23.0 in stage 1.0 (TID 32) in 80 ms on 172.32.0.2 (executor 0) (32/32)
22/02/21 08:10:34 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
22/02/21 08:10:34 INFO DAGScheduler: ResultStage 1 (foreachPartition at VerticesProcessor.scala:276) finished in 5.592 s
22/02/21 08:10:34 INFO DAGScheduler: Job 0 finished: foreachPartition at VerticesProcessor.scala:276, took 10.488850 s
22/02/21 08:10:34 INFO Exchange$: import for tag player cost time: 11.95 s
22/02/21 08:10:34 INFO Exchange$: Client-Import: batchSuccess.player: 10
22/02/21 08:10:34 INFO Exchange$: Client-Import: batchFailure.player: 0
22/02/21 08:10:34 INFO Exchange$: Processing Tag team
22/02/21 08:10:34 INFO Exchange$: field keys: name
22/02/21 08:10:34 INFO Exchange$: nebula keys: name
22/02/21 08:10:34 INFO CodeGenerator: Code generated in 38.481228 ms
22/02/21 08:10:34 INFO SparkContext: Starting job: foreachPartition at VerticesProcessor.scala:276
22/02/21 08:10:34 INFO DAGScheduler: Registering RDD 12 (foreachPartition at VerticesProcessor.scala:276) as input to shuffle 1
22/02/21 08:10:34 INFO DAGScheduler: Got job 1 (foreachPartition at VerticesProcessor.scala:276) with 32 output partitions
22/02/21 08:10:34 INFO DAGScheduler: Final stage: ResultStage 3 (foreachPartition at VerticesProcessor.scala:276)
22/02/21 08:10:34 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 2)
22/02/21 08:10:34 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 2)
22/02/21 08:10:34 INFO DAGScheduler: Submitting ShuffleMapStage 2 (MapPartitionsRDD[12] at foreachPartition at VerticesProcessor.scala:276), which has no missing parents
22/02/21 08:10:34 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 9.8 KB, free 366.2 MB)
22/02/21 08:10:34 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 5.3 KB, free 366.2 MB)
22/02/21 08:10:34 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on b213d13690e2:40461 (size: 5.3 KB, free: 366.3 MB)
22/02/21 08:10:34 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1163
22/02/21 08:10:34 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 2 (MapPartitionsRDD[12] at foreachPartition at VerticesProcessor.scala:276) (first 15 tasks are for partitions Vector(0))
22/02/21 08:10:34 INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks
22/02/21 08:10:34 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 33, 172.32.0.2, executor 0, partition 0, PROCESS_LOCAL, 7683 bytes)
22/02/21 08:10:34 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 172.32.0.2:34219 (size: 5.3 KB, free: 366.3 MB)
22/02/21 08:10:34 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 33) in 103 ms on 172.32.0.2 (executor 0) (1/1)
22/02/21 08:10:34 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool
22/02/21 08:10:34 INFO DAGScheduler: ShuffleMapStage 2 (foreachPartition at VerticesProcessor.scala:276) finished in 0.122 s
22/02/21 08:10:34 INFO DAGScheduler: looking for newly runnable stages
22/02/21 08:10:34 INFO DAGScheduler: running: Set()
22/02/21 08:10:34 INFO DAGScheduler: waiting: Set(ResultStage 3)
22/02/21 08:10:34 INFO DAGScheduler: failed: Set()
22/02/21 08:10:34 INFO DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD[17] at foreachPartition at VerticesProcessor.scala:276), which has no missing parents
22/02/21 08:10:34 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 22.0 KB, free 366.2 MB)
22/02/21 08:10:34 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 11.1 KB, free 366.2 MB)
22/02/21 08:10:34 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on b213d13690e2:40461 (size: 11.1 KB, free: 366.3 MB)
22/02/21 08:10:34 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1163
22/02/21 08:10:34 INFO DAGScheduler: Submitting 32 missing tasks from ResultStage 3 (MapPartitionsRDD[17] at foreachPartition at VerticesProcessor.scala:276) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
22/02/21 08:10:34 INFO TaskSchedulerImpl: Adding task set 3.0 with 32 tasks
22/02/21 08:10:34 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 34, 172.32.0.2, executor 0, partition 0, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:34 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on 172.32.0.2:34219 (size: 11.1 KB, free: 366.3 MB)
22/02/21 08:10:34 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 172.32.0.2:35932
22/02/21 08:10:34 INFO TaskSetManager: Starting task 24.0 in stage 3.0 (TID 35, 172.32.0.2, executor 0, partition 24, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:34 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 34) in 178 ms on 172.32.0.2 (executor 0) (1/32)
22/02/21 08:10:35 INFO TaskSetManager: Starting task 25.0 in stage 3.0 (TID 36, 172.32.0.2, executor 0, partition 25, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:35 INFO TaskSetManager: Finished task 24.0 in stage 3.0 (TID 35) in 118 ms on 172.32.0.2 (executor 0) (2/32)
22/02/21 08:10:35 INFO TaskSetManager: Starting task 26.0 in stage 3.0 (TID 37, 172.32.0.2, executor 0, partition 26, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:35 INFO TaskSetManager: Finished task 25.0 in stage 3.0 (TID 36) in 110 ms on 172.32.0.2 (executor 0) (3/32)
22/02/21 08:10:35 INFO TaskSetManager: Starting task 27.0 in stage 3.0 (TID 38, 172.32.0.2, executor 0, partition 27, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:35 INFO TaskSetManager: Finished task 26.0 in stage 3.0 (TID 37) in 109 ms on 172.32.0.2 (executor 0) (4/32)
22/02/21 08:10:35 INFO TaskSetManager: Starting task 28.0 in stage 3.0 (TID 39, 172.32.0.2, executor 0, partition 28, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:35 INFO TaskSetManager: Finished task 27.0 in stage 3.0 (TID 38) in 109 ms on 172.32.0.2 (executor 0) (5/32)
22/02/21 08:10:35 INFO TaskSetManager: Starting task 29.0 in stage 3.0 (TID 40, 172.32.0.2, executor 0, partition 29, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:35 INFO TaskSetManager: Finished task 28.0 in stage 3.0 (TID 39) in 109 ms on 172.32.0.2 (executor 0) (6/32)
22/02/21 08:10:35 INFO TaskSetManager: Starting task 30.0 in stage 3.0 (TID 41, 172.32.0.2, executor 0, partition 30, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:35 INFO TaskSetManager: Finished task 29.0 in stage 3.0 (TID 40) in 105 ms on 172.32.0.2 (executor 0) (7/32)
22/02/21 08:10:35 INFO TaskSetManager: Starting task 31.0 in stage 3.0 (TID 42, 172.32.0.2, executor 0, partition 31, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:35 INFO TaskSetManager: Finished task 30.0 in stage 3.0 (TID 41) in 109 ms on 172.32.0.2 (executor 0) (8/32)
22/02/21 08:10:35 INFO TaskSetManager: Starting task 1.0 in stage 3.0 (TID 43, 172.32.0.2, executor 0, partition 1, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:35 INFO TaskSetManager: Finished task 31.0 in stage 3.0 (TID 42) in 102 ms on 172.32.0.2 (executor 0) (9/32)
22/02/21 08:10:35 INFO TaskSetManager: Starting task 2.0 in stage 3.0 (TID 44, 172.32.0.2, executor 0, partition 2, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:35 INFO TaskSetManager: Finished task 1.0 in stage 3.0 (TID 43) in 80 ms on 172.32.0.2 (executor 0) (10/32)
22/02/21 08:10:36 INFO TaskSetManager: Starting task 3.0 in stage 3.0 (TID 45, 172.32.0.2, executor 0, partition 3, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:36 INFO TaskSetManager: Finished task 2.0 in stage 3.0 (TID 44) in 83 ms on 172.32.0.2 (executor 0) (11/32)
22/02/21 08:10:36 INFO TaskSetManager: Starting task 4.0 in stage 3.0 (TID 46, 172.32.0.2, executor 0, partition 4, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:36 INFO TaskSetManager: Finished task 3.0 in stage 3.0 (TID 45) in 77 ms on 172.32.0.2 (executor 0) (12/32)
22/02/21 08:10:36 INFO TaskSetManager: Starting task 5.0 in stage 3.0 (TID 47, 172.32.0.2, executor 0, partition 5, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:36 INFO TaskSetManager: Finished task 4.0 in stage 3.0 (TID 46) in 81 ms on 172.32.0.2 (executor 0) (13/32)
22/02/21 08:10:36 INFO TaskSetManager: Starting task 6.0 in stage 3.0 (TID 48, 172.32.0.2, executor 0, partition 6, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:36 INFO TaskSetManager: Finished task 5.0 in stage 3.0 (TID 47) in 80 ms on 172.32.0.2 (executor 0) (14/32)
22/02/21 08:10:36 INFO TaskSetManager: Starting task 7.0 in stage 3.0 (TID 49, 172.32.0.2, executor 0, partition 7, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:36 INFO TaskSetManager: Finished task 6.0 in stage 3.0 (TID 48) in 74 ms on 172.32.0.2 (executor 0) (15/32)
22/02/21 08:10:36 INFO TaskSetManager: Starting task 8.0 in stage 3.0 (TID 50, 172.32.0.2, executor 0, partition 8, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:36 INFO TaskSetManager: Finished task 7.0 in stage 3.0 (TID 49) in 81 ms on 172.32.0.2 (executor 0) (16/32)
22/02/21 08:10:36 INFO TaskSetManager: Starting task 9.0 in stage 3.0 (TID 51, 172.32.0.2, executor 0, partition 9, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:36 INFO TaskSetManager: Finished task 8.0 in stage 3.0 (TID 50) in 127 ms on 172.32.0.2 (executor 0) (17/32)
22/02/21 08:10:36 INFO TaskSetManager: Starting task 10.0 in stage 3.0 (TID 52, 172.32.0.2, executor 0, partition 10, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:36 INFO TaskSetManager: Finished task 9.0 in stage 3.0 (TID 51) in 80 ms on 172.32.0.2 (executor 0) (18/32)
22/02/21 08:10:36 INFO TaskSetManager: Starting task 11.0 in stage 3.0 (TID 53, 172.32.0.2, executor 0, partition 11, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:36 INFO TaskSetManager: Finished task 10.0 in stage 3.0 (TID 52) in 78 ms on 172.32.0.2 (executor 0) (19/32)
22/02/21 08:10:36 INFO TaskSetManager: Starting task 12.0 in stage 3.0 (TID 54, 172.32.0.2, executor 0, partition 12, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:36 INFO TaskSetManager: Finished task 11.0 in stage 3.0 (TID 53) in 89 ms on 172.32.0.2 (executor 0) (20/32)
22/02/21 08:10:36 INFO TaskSetManager: Starting task 13.0 in stage 3.0 (TID 55, 172.32.0.2, executor 0, partition 13, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:36 INFO TaskSetManager: Finished task 12.0 in stage 3.0 (TID 54) in 76 ms on 172.32.0.2 (executor 0) (21/32)
22/02/21 08:10:36 INFO TaskSetManager: Starting task 14.0 in stage 3.0 (TID 56, 172.32.0.2, executor 0, partition 14, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:36 INFO TaskSetManager: Finished task 13.0 in stage 3.0 (TID 55) in 78 ms on 172.32.0.2 (executor 0) (22/32)
22/02/21 08:10:36 INFO TaskSetManager: Starting task 15.0 in stage 3.0 (TID 57, 172.32.0.2, executor 0, partition 15, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:36 INFO TaskSetManager: Finished task 14.0 in stage 3.0 (TID 56) in 74 ms on 172.32.0.2 (executor 0) (23/32)
22/02/21 08:10:37 INFO TaskSetManager: Starting task 16.0 in stage 3.0 (TID 58, 172.32.0.2, executor 0, partition 16, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:37 INFO TaskSetManager: Finished task 15.0 in stage 3.0 (TID 57) in 80 ms on 172.32.0.2 (executor 0) (24/32)
22/02/21 08:10:37 INFO TaskSetManager: Starting task 17.0 in stage 3.0 (TID 59, 172.32.0.2, executor 0, partition 17, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:37 INFO TaskSetManager: Finished task 16.0 in stage 3.0 (TID 58) in 76 ms on 172.32.0.2 (executor 0) (25/32)
22/02/21 08:10:37 INFO TaskSetManager: Starting task 18.0 in stage 3.0 (TID 60, 172.32.0.2, executor 0, partition 18, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:37 INFO TaskSetManager: Finished task 17.0 in stage 3.0 (TID 59) in 73 ms on 172.32.0.2 (executor 0) (26/32)
22/02/21 08:10:37 INFO TaskSetManager: Starting task 19.0 in stage 3.0 (TID 61, 172.32.0.2, executor 0, partition 19, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:37 INFO TaskSetManager: Finished task 18.0 in stage 3.0 (TID 60) in 80 ms on 172.32.0.2 (executor 0) (27/32)
22/02/21 08:10:37 INFO TaskSetManager: Starting task 20.0 in stage 3.0 (TID 62, 172.32.0.2, executor 0, partition 20, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:37 INFO TaskSetManager: Finished task 19.0 in stage 3.0 (TID 61) in 76 ms on 172.32.0.2 (executor 0) (28/32)
22/02/21 08:10:37 INFO TaskSetManager: Starting task 21.0 in stage 3.0 (TID 63, 172.32.0.2, executor 0, partition 21, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:37 INFO TaskSetManager: Finished task 20.0 in stage 3.0 (TID 62) in 74 ms on 172.32.0.2 (executor 0) (29/32)
22/02/21 08:10:37 INFO TaskSetManager: Starting task 22.0 in stage 3.0 (TID 64, 172.32.0.2, executor 0, partition 22, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:37 INFO TaskSetManager: Finished task 21.0 in stage 3.0 (TID 63) in 73 ms on 172.32.0.2 (executor 0) (30/32)
22/02/21 08:10:37 INFO TaskSetManager: Starting task 23.0 in stage 3.0 (TID 65, 172.32.0.2, executor 0, partition 23, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:37 INFO TaskSetManager: Finished task 22.0 in stage 3.0 (TID 64) in 87 ms on 172.32.0.2 (executor 0) (31/32)
22/02/21 08:10:37 INFO TaskSetManager: Finished task 23.0 in stage 3.0 (TID 65) in 146 ms on 172.32.0.2 (executor 0) (32/32)
22/02/21 08:10:37 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool
22/02/21 08:10:37 INFO DAGScheduler: ResultStage 3 (foreachPartition at VerticesProcessor.scala:276) finished in 2.953 s
22/02/21 08:10:37 INFO DAGScheduler: Job 1 finished: foreachPartition at VerticesProcessor.scala:276, took 3.099664 s
22/02/21 08:10:37 INFO Exchange$: import for tag team cost time: 3.32 s
22/02/21 08:10:37 INFO Exchange$: Client-Import: batchSuccess.team: 9
22/02/21 08:10:37 INFO Exchange$: Client-Import: batchFailure.team: 0
22/02/21 08:10:37 INFO Exchange$: Processing Edge follow
22/02/21 08:10:37 INFO Exchange$: field keys: degree
22/02/21 08:10:37 INFO Exchange$: nebula keys: degree
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 85
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 14
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 3
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 66
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 24
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 16
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 97
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 75
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 19
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 87
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 81
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 12
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 5
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 32
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 22
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 9
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 28
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 56
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 38
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 65
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 74
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 11
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 63
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 105
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 86
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 6
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 91
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 96
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 15
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 29
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 39
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 50
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 59
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 10
22/02/21 08:10:39 INFO ContextCleaner: Cleaned shuffle 0
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 7
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 25
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 33
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 36
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 0
22/02/21 08:10:39 INFO ContextCleaner: Cleaned shuffle 1
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 70
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 21
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 54
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 64
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 30
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 100
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 27
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 35
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 47
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 53
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 34
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 44
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 101
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 31
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 110
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 93
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 108
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 113
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 102
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 107
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 8
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 41
22/02/21 08:10:39 INFO CodeGenerator: Code generated in 37.152623 ms
22/02/21 08:10:39 INFO BlockManagerInfo: Removed broadcast_0_piece0 on b213d13690e2:40461 in memory (size: 5.4 KB, free: 366.3 MB)
22/02/21 08:10:39 INFO BlockManagerInfo: Removed broadcast_0_piece0 on 172.32.0.2:34219 in memory (size: 5.4 KB, free: 366.3 MB)
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 42
22/02/21 08:10:39 INFO BlockManagerInfo: Removed broadcast_2_piece0 on b213d13690e2:40461 in memory (size: 5.3 KB, free: 366.3 MB)
22/02/21 08:10:39 INFO BlockManagerInfo: Removed broadcast_2_piece0 on 172.32.0.2:34219 in memory (size: 5.3 KB, free: 366.3 MB)
22/02/21 08:10:39 INFO SparkContext: Starting job: foreachPartition at EdgeProcessor.scala:350
22/02/21 08:10:39 INFO DAGScheduler: Registering RDD 21 (foreachPartition at EdgeProcessor.scala:350) as input to shuffle 2
22/02/21 08:10:39 INFO DAGScheduler: Got job 2 (foreachPartition at EdgeProcessor.scala:350) with 32 output partitions
22/02/21 08:10:39 INFO DAGScheduler: Final stage: ResultStage 5 (foreachPartition at EdgeProcessor.scala:350)
22/02/21 08:10:39 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 4)
22/02/21 08:10:39 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 4)
22/02/21 08:10:39 INFO DAGScheduler: Submitting ShuffleMapStage 4 (MapPartitionsRDD[21] at foreachPartition at EdgeProcessor.scala:350), which has no missing parents
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 104
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 77
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 61
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 99
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 46
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 18
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 80
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 20
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 52
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 72
22/02/21 08:10:39 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 10.1 KB, free 366.2 MB)
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 92
22/02/21 08:10:39 INFO BlockManagerInfo: Removed broadcast_1_piece0 on b213d13690e2:40461 in memory (size: 11.3 KB, free: 366.3 MB)
22/02/21 08:10:39 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 5.3 KB, free 366.3 MB)
22/02/21 08:10:39 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on b213d13690e2:40461 (size: 5.3 KB, free: 366.3 MB)
22/02/21 08:10:39 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1163
22/02/21 08:10:39 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 4 (MapPartitionsRDD[21] at foreachPartition at EdgeProcessor.scala:350) (first 15 tasks are for partitions Vector(0))
22/02/21 08:10:39 INFO TaskSchedulerImpl: Adding task set 4.0 with 1 tasks
22/02/21 08:10:39 INFO BlockManagerInfo: Removed broadcast_1_piece0 on 172.32.0.2:34219 in memory (size: 11.3 KB, free: 366.3 MB)
22/02/21 08:10:39 INFO TaskSetManager: Starting task 0.0 in stage 4.0 (TID 66, 172.32.0.2, executor 0, partition 0, PROCESS_LOCAL, 7683 bytes)
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 109
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 55
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 112
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 89
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 23
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 48
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 79
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 1
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 69
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 13
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 67
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 103
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 51
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 37
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 94
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 68
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 78
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 60
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 83
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 82
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 73
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 40
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 43
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 45
22/02/21 08:10:39 INFO BlockManagerInfo: Removed broadcast_3_piece0 on b213d13690e2:40461 in memory (size: 11.1 KB, free: 366.3 MB)
22/02/21 08:10:39 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on 172.32.0.2:34219 (size: 5.3 KB, free: 366.3 MB)
22/02/21 08:10:39 INFO BlockManagerInfo: Removed broadcast_3_piece0 on 172.32.0.2:34219 in memory (size: 11.1 KB, free: 366.3 MB)
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 71
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 57
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 62
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 58
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 95
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 90
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 111
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 84
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 98
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 76
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 88
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 17
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 106
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 49
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 26
22/02/21 08:10:39 INFO ContextCleaner: Cleaned accumulator 4
22/02/21 08:10:39 INFO TaskSetManager: Finished task 0.0 in stage 4.0 (TID 66) in 158 ms on 172.32.0.2 (executor 0) (1/1)
22/02/21 08:10:39 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool
22/02/21 08:10:39 INFO DAGScheduler: ShuffleMapStage 4 (foreachPartition at EdgeProcessor.scala:350) finished in 0.181 s
22/02/21 08:10:39 INFO DAGScheduler: looking for newly runnable stages
22/02/21 08:10:39 INFO DAGScheduler: running: Set()
22/02/21 08:10:39 INFO DAGScheduler: waiting: Set(ResultStage 5)
22/02/21 08:10:39 INFO DAGScheduler: failed: Set()
22/02/21 08:10:39 INFO DAGScheduler: Submitting ResultStage 5 (MapPartitionsRDD[26] at foreachPartition at EdgeProcessor.scala:350), which has no missing parents
22/02/21 08:10:39 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 22.0 KB, free 366.3 MB)
22/02/21 08:10:39 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 11.0 KB, free 366.3 MB)
22/02/21 08:10:39 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on b213d13690e2:40461 (size: 11.0 KB, free: 366.3 MB)
22/02/21 08:10:39 INFO SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1163
22/02/21 08:10:39 INFO DAGScheduler: Submitting 32 missing tasks from ResultStage 5 (MapPartitionsRDD[26] at foreachPartition at EdgeProcessor.scala:350) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
22/02/21 08:10:39 INFO TaskSchedulerImpl: Adding task set 5.0 with 32 tasks
22/02/21 08:10:39 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID 67, 172.32.0.2, executor 0, partition 0, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:39 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on 172.32.0.2:34219 (size: 11.0 KB, free: 366.3 MB)
22/02/21 08:10:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 172.32.0.2:35932
22/02/21 08:10:40 INFO TaskSetManager: Starting task 24.0 in stage 5.0 (TID 68, 172.32.0.2, executor 0, partition 24, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:40 INFO TaskSetManager: Finished task 0.0 in stage 5.0 (TID 67) in 218 ms on 172.32.0.2 (executor 0) (1/32)
22/02/21 08:10:40 INFO TaskSetManager: Starting task 25.0 in stage 5.0 (TID 69, 172.32.0.2, executor 0, partition 25, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:40 INFO TaskSetManager: Finished task 24.0 in stage 5.0 (TID 68) in 133 ms on 172.32.0.2 (executor 0) (2/32)
22/02/21 08:10:40 INFO TaskSetManager: Starting task 26.0 in stage 5.0 (TID 70, 172.32.0.2, executor 0, partition 26, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:40 INFO TaskSetManager: Finished task 25.0 in stage 5.0 (TID 69) in 143 ms on 172.32.0.2 (executor 0) (3/32)
22/02/21 08:10:40 INFO TaskSetManager: Starting task 27.0 in stage 5.0 (TID 71, 172.32.0.2, executor 0, partition 27, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:40 INFO TaskSetManager: Finished task 26.0 in stage 5.0 (TID 70) in 133 ms on 172.32.0.2 (executor 0) (4/32)
22/02/21 08:10:40 INFO TaskSetManager: Starting task 28.0 in stage 5.0 (TID 72, 172.32.0.2, executor 0, partition 28, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:40 INFO TaskSetManager: Finished task 27.0 in stage 5.0 (TID 71) in 141 ms on 172.32.0.2 (executor 0) (5/32)
22/02/21 08:10:40 INFO TaskSetManager: Starting task 29.0 in stage 5.0 (TID 73, 172.32.0.2, executor 0, partition 29, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:40 INFO TaskSetManager: Finished task 28.0 in stage 5.0 (TID 72) in 150 ms on 172.32.0.2 (executor 0) (6/32)
22/02/21 08:10:40 INFO TaskSetManager: Starting task 30.0 in stage 5.0 (TID 74, 172.32.0.2, executor 0, partition 30, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:40 INFO TaskSetManager: Finished task 29.0 in stage 5.0 (TID 73) in 137 ms on 172.32.0.2 (executor 0) (7/32)
22/02/21 08:10:41 INFO TaskSetManager: Starting task 31.0 in stage 5.0 (TID 75, 172.32.0.2, executor 0, partition 31, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:41 INFO TaskSetManager: Finished task 30.0 in stage 5.0 (TID 74) in 122 ms on 172.32.0.2 (executor 0) (8/32)
22/02/21 08:10:41 INFO TaskSetManager: Starting task 1.0 in stage 5.0 (TID 76, 172.32.0.2, executor 0, partition 1, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:41 INFO TaskSetManager: Finished task 31.0 in stage 5.0 (TID 75) in 133 ms on 172.32.0.2 (executor 0) (9/32)
22/02/21 08:10:41 INFO TaskSetManager: Starting task 2.0 in stage 5.0 (TID 77, 172.32.0.2, executor 0, partition 2, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:41 INFO TaskSetManager: Finished task 1.0 in stage 5.0 (TID 76) in 88 ms on 172.32.0.2 (executor 0) (10/32)
22/02/21 08:10:41 INFO TaskSetManager: Starting task 3.0 in stage 5.0 (TID 78, 172.32.0.2, executor 0, partition 3, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:41 INFO TaskSetManager: Finished task 2.0 in stage 5.0 (TID 77) in 113 ms on 172.32.0.2 (executor 0) (11/32)
22/02/21 08:10:41 INFO TaskSetManager: Starting task 4.0 in stage 5.0 (TID 79, 172.32.0.2, executor 0, partition 4, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:41 INFO TaskSetManager: Finished task 3.0 in stage 5.0 (TID 78) in 80 ms on 172.32.0.2 (executor 0) (12/32)
22/02/21 08:10:41 INFO TaskSetManager: Starting task 5.0 in stage 5.0 (TID 80, 172.32.0.2, executor 0, partition 5, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:41 INFO TaskSetManager: Finished task 4.0 in stage 5.0 (TID 79) in 74 ms on 172.32.0.2 (executor 0) (13/32)
22/02/21 08:10:41 INFO TaskSetManager: Starting task 6.0 in stage 5.0 (TID 81, 172.32.0.2, executor 0, partition 6, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:41 INFO TaskSetManager: Finished task 5.0 in stage 5.0 (TID 80) in 70 ms on 172.32.0.2 (executor 0) (14/32)
22/02/21 08:10:41 INFO TaskSetManager: Starting task 7.0 in stage 5.0 (TID 82, 172.32.0.2, executor 0, partition 7, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:41 INFO TaskSetManager: Finished task 6.0 in stage 5.0 (TID 81) in 82 ms on 172.32.0.2 (executor 0) (15/32)
22/02/21 08:10:41 INFO TaskSetManager: Starting task 8.0 in stage 5.0 (TID 83, 172.32.0.2, executor 0, partition 8, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:41 INFO TaskSetManager: Finished task 7.0 in stage 5.0 (TID 82) in 73 ms on 172.32.0.2 (executor 0) (16/32)
22/02/21 08:10:41 INFO TaskSetManager: Starting task 9.0 in stage 5.0 (TID 84, 172.32.0.2, executor 0, partition 9, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:41 INFO TaskSetManager: Finished task 8.0 in stage 5.0 (TID 83) in 67 ms on 172.32.0.2 (executor 0) (17/32)
22/02/21 08:10:41 INFO TaskSetManager: Starting task 10.0 in stage 5.0 (TID 85, 172.32.0.2, executor 0, partition 10, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:41 INFO TaskSetManager: Finished task 9.0 in stage 5.0 (TID 84) in 74 ms on 172.32.0.2 (executor 0) (18/32)
22/02/21 08:10:41 INFO TaskSetManager: Starting task 11.0 in stage 5.0 (TID 86, 172.32.0.2, executor 0, partition 11, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:41 INFO TaskSetManager: Finished task 10.0 in stage 5.0 (TID 85) in 82 ms on 172.32.0.2 (executor 0) (19/32)
22/02/21 08:10:42 INFO TaskSetManager: Starting task 12.0 in stage 5.0 (TID 87, 172.32.0.2, executor 0, partition 12, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:42 INFO TaskSetManager: Finished task 11.0 in stage 5.0 (TID 86) in 91 ms on 172.32.0.2 (executor 0) (20/32)
22/02/21 08:10:42 INFO TaskSetManager: Starting task 13.0 in stage 5.0 (TID 88, 172.32.0.2, executor 0, partition 13, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:42 INFO TaskSetManager: Finished task 12.0 in stage 5.0 (TID 87) in 74 ms on 172.32.0.2 (executor 0) (21/32)
22/02/21 08:10:42 INFO TaskSetManager: Starting task 14.0 in stage 5.0 (TID 89, 172.32.0.2, executor 0, partition 14, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:42 INFO TaskSetManager: Finished task 13.0 in stage 5.0 (TID 88) in 77 ms on 172.32.0.2 (executor 0) (22/32)
22/02/21 08:10:42 INFO TaskSetManager: Starting task 15.0 in stage 5.0 (TID 90, 172.32.0.2, executor 0, partition 15, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:42 INFO TaskSetManager: Finished task 14.0 in stage 5.0 (TID 89) in 72 ms on 172.32.0.2 (executor 0) (23/32)
22/02/21 08:10:42 INFO TaskSetManager: Starting task 16.0 in stage 5.0 (TID 91, 172.32.0.2, executor 0, partition 16, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:42 INFO TaskSetManager: Finished task 15.0 in stage 5.0 (TID 90) in 75 ms on 172.32.0.2 (executor 0) (24/32)
22/02/21 08:10:42 INFO TaskSetManager: Starting task 17.0 in stage 5.0 (TID 92, 172.32.0.2, executor 0, partition 17, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:42 INFO TaskSetManager: Finished task 16.0 in stage 5.0 (TID 91) in 69 ms on 172.32.0.2 (executor 0) (25/32)
22/02/21 08:10:42 INFO TaskSetManager: Starting task 18.0 in stage 5.0 (TID 93, 172.32.0.2, executor 0, partition 18, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:42 INFO TaskSetManager: Finished task 17.0 in stage 5.0 (TID 92) in 72 ms on 172.32.0.2 (executor 0) (26/32)
22/02/21 08:10:42 INFO TaskSetManager: Starting task 19.0 in stage 5.0 (TID 94, 172.32.0.2, executor 0, partition 19, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:42 INFO TaskSetManager: Finished task 18.0 in stage 5.0 (TID 93) in 87 ms on 172.32.0.2 (executor 0) (27/32)
22/02/21 08:10:42 INFO TaskSetManager: Starting task 20.0 in stage 5.0 (TID 95, 172.32.0.2, executor 0, partition 20, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:42 INFO TaskSetManager: Finished task 19.0 in stage 5.0 (TID 94) in 83 ms on 172.32.0.2 (executor 0) (28/32)
22/02/21 08:10:42 INFO TaskSetManager: Starting task 21.0 in stage 5.0 (TID 96, 172.32.0.2, executor 0, partition 21, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:42 INFO TaskSetManager: Finished task 20.0 in stage 5.0 (TID 95) in 83 ms on 172.32.0.2 (executor 0) (29/32)
22/02/21 08:10:42 INFO TaskSetManager: Starting task 22.0 in stage 5.0 (TID 97, 172.32.0.2, executor 0, partition 22, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:42 INFO TaskSetManager: Finished task 21.0 in stage 5.0 (TID 96) in 83 ms on 172.32.0.2 (executor 0) (30/32)
22/02/21 08:10:42 INFO TaskSetManager: Starting task 23.0 in stage 5.0 (TID 98, 172.32.0.2, executor 0, partition 23, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:42 INFO TaskSetManager: Finished task 22.0 in stage 5.0 (TID 97) in 76 ms on 172.32.0.2 (executor 0) (31/32)
22/02/21 08:10:42 INFO TaskSetManager: Finished task 23.0 in stage 5.0 (TID 98) in 81 ms on 172.32.0.2 (executor 0) (32/32)
22/02/21 08:10:42 INFO TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool
22/02/21 08:10:42 INFO DAGScheduler: ResultStage 5 (foreachPartition at EdgeProcessor.scala:350) finished in 3.138 s
22/02/21 08:10:42 INFO DAGScheduler: Job 2 finished: foreachPartition at EdgeProcessor.scala:350, took 3.334442 s
22/02/21 08:10:42 INFO Exchange$: import for edge follow cost time: 5.17 s
22/02/21 08:10:42 INFO Exchange$: Client-Import: batchSuccess.follow: 9
22/02/21 08:10:42 INFO Exchange$: Client-Import: batchFailure.follow: 0
22/02/21 08:10:42 INFO Exchange$: Processing Edge serve
22/02/21 08:10:42 INFO Exchange$: field keys: start_year, end_year
22/02/21 08:10:42 INFO Exchange$: nebula keys: start_year, end_year
22/02/21 08:10:43 INFO CodeGenerator: Code generated in 19.736682 ms
22/02/21 08:10:43 INFO SparkContext: Starting job: foreachPartition at EdgeProcessor.scala:350
22/02/21 08:10:43 INFO DAGScheduler: Registering RDD 30 (foreachPartition at EdgeProcessor.scala:350) as input to shuffle 3
22/02/21 08:10:43 INFO DAGScheduler: Got job 3 (foreachPartition at EdgeProcessor.scala:350) with 32 output partitions
22/02/21 08:10:43 INFO DAGScheduler: Final stage: ResultStage 7 (foreachPartition at EdgeProcessor.scala:350)
22/02/21 08:10:43 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 6)
22/02/21 08:10:43 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 6)
22/02/21 08:10:43 INFO DAGScheduler: Submitting ShuffleMapStage 6 (MapPartitionsRDD[30] at foreachPartition at EdgeProcessor.scala:350), which has no missing parents
22/02/21 08:10:43 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 10.4 KB, free 366.2 MB)
22/02/21 08:10:43 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 5.4 KB, free 366.2 MB)
22/02/21 08:10:43 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on b213d13690e2:40461 (size: 5.4 KB, free: 366.3 MB)
22/02/21 08:10:43 INFO SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:1163
22/02/21 08:10:43 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 6 (MapPartitionsRDD[30] at foreachPartition at EdgeProcessor.scala:350) (first 15 tasks are for partitions Vector(0))
22/02/21 08:10:43 INFO TaskSchedulerImpl: Adding task set 6.0 with 1 tasks
22/02/21 08:10:43 INFO TaskSetManager: Starting task 0.0 in stage 6.0 (TID 99, 172.32.0.2, executor 0, partition 0, PROCESS_LOCAL, 7683 bytes)
22/02/21 08:10:43 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on 172.32.0.2:34219 (size: 5.4 KB, free: 366.3 MB)
22/02/21 08:10:43 INFO TaskSetManager: Finished task 0.0 in stage 6.0 (TID 99) in 145 ms on 172.32.0.2 (executor 0) (1/1)
22/02/21 08:10:43 INFO TaskSchedulerImpl: Removed TaskSet 6.0, whose tasks have all completed, from pool 
22/02/21 08:10:43 INFO DAGScheduler: ShuffleMapStage 6 (foreachPartition at EdgeProcessor.scala:350) finished in 0.168 s
22/02/21 08:10:43 INFO DAGScheduler: looking for newly runnable stages
22/02/21 08:10:43 INFO DAGScheduler: running: Set()
22/02/21 08:10:43 INFO DAGScheduler: waiting: Set(ResultStage 7)
22/02/21 08:10:43 INFO DAGScheduler: failed: Set()
22/02/21 08:10:43 INFO DAGScheduler: Submitting ResultStage 7 (MapPartitionsRDD[35] at foreachPartition at EdgeProcessor.scala:350), which has no missing parents
22/02/21 08:10:43 INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 22.4 KB, free 366.2 MB)
22/02/21 08:10:43 INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 11.1 KB, free 366.2 MB)
22/02/21 08:10:43 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on b213d13690e2:40461 (size: 11.1 KB, free: 366.3 MB)
22/02/21 08:10:43 INFO SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1163
22/02/21 08:10:43 INFO DAGScheduler: Submitting 32 missing tasks from ResultStage 7 (MapPartitionsRDD[35] at foreachPartition at EdgeProcessor.scala:350) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
22/02/21 08:10:43 INFO TaskSchedulerImpl: Adding task set 7.0 with 32 tasks
22/02/21 08:10:43 INFO TaskSetManager: Starting task 24.0 in stage 7.0 (TID 100, 172.32.0.2, executor 0, partition 24, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:43 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on 172.32.0.2:34219 (size: 11.1 KB, free: 366.3 MB)
22/02/21 08:10:43 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 3 to 172.32.0.2:35932
22/02/21 08:10:43 INFO TaskSetManager: Starting task 25.0 in stage 7.0 (TID 101, 172.32.0.2, executor 0, partition 25, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:43 INFO TaskSetManager: Finished task 24.0 in stage 7.0 (TID 100) in 152 ms on 172.32.0.2 (executor 0) (1/32)
22/02/21 08:10:43 INFO TaskSetManager: Starting task 26.0 in stage 7.0 (TID 102, 172.32.0.2, executor 0, partition 26, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:43 INFO TaskSetManager: Finished task 25.0 in stage 7.0 (TID 101) in 106 ms on 172.32.0.2 (executor 0) (2/32)
22/02/21 08:10:43 INFO TaskSetManager: Starting task 27.0 in stage 7.0 (TID 103, 172.32.0.2, executor 0, partition 27, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:43 INFO TaskSetManager: Finished task 26.0 in stage 7.0 (TID 102) in 107 ms on 172.32.0.2 (executor 0) (3/32)
22/02/21 08:10:43 INFO TaskSetManager: Starting task 28.0 in stage 7.0 (TID 104, 172.32.0.2, executor 0, partition 28, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:43 INFO TaskSetManager: Finished task 27.0 in stage 7.0 (TID 103) in 120 ms on 172.32.0.2 (executor 0) (4/32)
22/02/21 08:10:43 INFO TaskSetManager: Starting task 29.0 in stage 7.0 (TID 105, 172.32.0.2, executor 0, partition 29, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:43 INFO TaskSetManager: Finished task 28.0 in stage 7.0 (TID 104) in 95 ms on 172.32.0.2 (executor 0) (5/32)
22/02/21 08:10:43 INFO TaskSetManager: Starting task 30.0 in stage 7.0 (TID 106, 172.32.0.2, executor 0, partition 30, NODE_LOCAL, 7771 bytes)
22/02/21 08:10:44 INFO TaskSetManager: Finished task 29.0 in stage 7.0 (TID 105) in 90 ms on 172.32.0.2 (executor 0) (6/32)
22/02/21 08:10:44 INFO TaskSetManager: Starting task 0.0 in stage 7.0 (TID 107, 172.32.0.2, executor 0, partition 0, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:44 INFO TaskSetManager: Finished task 30.0 in stage 7.0 (TID 106) in 90 ms on 172.32.0.2 (executor 0) (7/32)
22/02/21 08:10:44 INFO TaskSetManager: Starting task 1.0 in stage 7.0 (TID 108, 172.32.0.2, executor 0, partition 1, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:44 INFO TaskSetManager: Finished task 0.0 in stage 7.0 (TID 107) in 73 ms on 172.32.0.2 (executor 0) (8/32)
22/02/21 08:10:44 INFO TaskSetManager: Starting task 2.0 in stage 7.0 (TID 109, 172.32.0.2, executor 0, partition 2, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:44 INFO TaskSetManager: Finished task 1.0 in stage 7.0 (TID 108) in 70 ms on 172.32.0.2 (executor 0) (9/32)
22/02/21 08:10:44 INFO TaskSetManager: Starting task 3.0 in stage 7.0 (TID 110, 172.32.0.2, executor 0, partition 3, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:44 INFO TaskSetManager: Finished task 2.0 in stage 7.0 (TID 109) in 71 ms on 172.32.0.2 (executor 0) (10/32)
22/02/21 08:10:44 INFO TaskSetManager: Starting task 4.0 in stage 7.0 (TID 111, 172.32.0.2, executor 0, partition 4, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:44 INFO TaskSetManager: Finished task 3.0 in stage 7.0 (TID 110) in 70 ms on 172.32.0.2 (executor 0) (11/32)
22/02/21 08:10:44 INFO TaskSetManager: Starting task 5.0 in stage 7.0 (TID 112, 172.32.0.2, executor 0, partition 5, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:44 INFO TaskSetManager: Finished task 4.0 in stage 7.0 (TID 111) in 102 ms on 172.32.0.2 (executor 0) (12/32)
22/02/21 08:10:44 INFO TaskSetManager: Starting task 6.0 in stage 7.0 (TID 113, 172.32.0.2, executor 0, partition 6, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:44 INFO TaskSetManager: Finished task 5.0 in stage 7.0 (TID 112) in 70 ms on 172.32.0.2 (executor 0) (13/32)
22/02/21 08:10:44 INFO TaskSetManager: Starting task 7.0 in stage 7.0 (TID 114, 172.32.0.2, executor 0, partition 7, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:44 INFO TaskSetManager: Finished task 6.0 in stage 7.0 (TID 113) in 80 ms on 172.32.0.2 (executor 0) (14/32)
22/02/21 08:10:44 INFO TaskSetManager: Starting task 8.0 in stage 7.0 (TID 115, 172.32.0.2, executor 0, partition 8, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:44 INFO TaskSetManager: Finished task 7.0 in stage 7.0 (TID 114) in 76 ms on 172.32.0.2 (executor 0) (15/32)
22/02/21 08:10:44 INFO TaskSetManager: Starting task 9.0 in stage 7.0 (TID 116, 172.32.0.2, executor 0, partition 9, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:44 INFO TaskSetManager: Finished task 8.0 in stage 7.0 (TID 115) in 79 ms on 172.32.0.2 (executor 0) (16/32)
22/02/21 08:10:44 INFO TaskSetManager: Starting task 10.0 in stage 7.0 (TID 117, 172.32.0.2, executor 0, partition 10, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:44 INFO TaskSetManager: Finished task 9.0 in stage 7.0 (TID 116) in 75 ms on 172.32.0.2 (executor 0) (17/32)
22/02/21 08:10:44 INFO TaskSetManager: Starting task 11.0 in stage 7.0 (TID 118, 172.32.0.2, executor 0, partition 11, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:44 INFO TaskSetManager: Finished task 10.0 in stage 7.0 (TID 117) in 75 ms on 172.32.0.2 (executor 0) (18/32)
22/02/21 08:10:44 INFO TaskSetManager: Starting task 12.0 in stage 7.0 (TID 119, 172.32.0.2, executor 0, partition 12, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:44 INFO TaskSetManager: Finished task 11.0 in stage 7.0 (TID 118) in 72 ms on 172.32.0.2 (executor 0) (19/32)
22/02/21 08:10:45 INFO TaskSetManager: Starting task 13.0 in stage 7.0 (TID 120, 172.32.0.2, executor 0, partition 13, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:45 INFO TaskSetManager: Finished task 12.0 in stage 7.0 (TID 119) in 80 ms on 172.32.0.2 (executor 0) (20/32)
22/02/21 08:10:45 INFO TaskSetManager: Starting task 14.0 in stage 7.0 (TID 121, 172.32.0.2, executor 0, partition 14, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:45 INFO TaskSetManager: Finished task 13.0 in stage 7.0 (TID 120) in 71 ms on 172.32.0.2 (executor 0) (21/32)
22/02/21 08:10:45 INFO TaskSetManager: Starting task 15.0 in stage 7.0 (TID 122, 172.32.0.2, executor 0, partition 15, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:45 INFO TaskSetManager: Finished task 14.0 in stage 7.0 (TID 121) in 69 ms on 172.32.0.2 (executor 0) (22/32)
22/02/21 08:10:45 INFO TaskSetManager: Starting task 16.0 in stage 7.0 (TID 123, 172.32.0.2, executor 0, partition 16, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:45 INFO TaskSetManager: Finished task 15.0 in stage 7.0 (TID 122) in 70 ms on 172.32.0.2 (executor 0) (23/32)
22/02/21 08:10:45 INFO TaskSetManager: Starting task 17.0 in stage 7.0 (TID 124, 172.32.0.2, executor 0, partition 17, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:45 INFO TaskSetManager: Finished task 16.0 in stage 7.0 (TID 123) in 69 ms on 172.32.0.2 (executor 0) (24/32)
22/02/21 08:10:45 INFO TaskSetManager: Starting task 18.0 in stage 7.0 (TID 125, 172.32.0.2, executor 0, partition 18, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:45 INFO TaskSetManager: Finished task 17.0 in stage 7.0 (TID 124) in 71 ms on 172.32.0.2 (executor 0) (25/32)
22/02/21 08:10:45 INFO TaskSetManager: Starting task 19.0 in stage 7.0 (TID 126, 172.32.0.2, executor 0, partition 19, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:45 INFO TaskSetManager: Finished task 18.0 in stage 7.0 (TID 125) in 73 ms on 172.32.0.2 (executor 0) (26/32)
22/02/21 08:10:45 INFO TaskSetManager: Starting task 20.0 in stage 7.0 (TID 127, 172.32.0.2, executor 0, partition 20, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:45 INFO TaskSetManager: Finished task 19.0 in stage 7.0 (TID 126) in 72 ms on 172.32.0.2 (executor 0) (27/32)
22/02/21 08:10:45 INFO TaskSetManager: Starting task 21.0 in stage 7.0 (TID 128, 172.32.0.2, executor 0, partition 21, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:45 INFO TaskSetManager: Finished task 20.0 in stage 7.0 (TID 127) in 76 ms on 172.32.0.2 (executor 0) (28/32)
22/02/21 08:10:45 INFO TaskSetManager: Starting task 22.0 in stage 7.0 (TID 129, 172.32.0.2, executor 0, partition 22, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:45 INFO TaskSetManager: Finished task 21.0 in stage 7.0 (TID 128) in 68 ms on 172.32.0.2 (executor 0) (29/32)
22/02/21 08:10:45 INFO TaskSetManager: Starting task 23.0 in stage 7.0 (TID 130, 172.32.0.2, executor 0, partition 23, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:45 INFO TaskSetManager: Finished task 22.0 in stage 7.0 (TID 129) in 91 ms on 172.32.0.2 (executor 0) (30/32)
22/02/21 08:10:45 INFO TaskSetManager: Starting task 31.0 in stage 7.0 (TID 131, 172.32.0.2, executor 0, partition 31, PROCESS_LOCAL, 7771 bytes)
22/02/21 08:10:45 INFO TaskSetManager: Finished task 23.0 in stage 7.0 (TID 130) in 69 ms on 172.32.0.2 (executor 0) (31/32)
22/02/21 08:10:45 INFO TaskSetManager: Finished task 31.0 in stage 7.0 (TID 131) in 77 ms on 172.32.0.2 (executor 0) (32/32)
22/02/21 08:10:45 INFO TaskSchedulerImpl: Removed TaskSet 7.0, whose tasks have all completed, from pool
22/02/21 08:10:45 INFO DAGScheduler: ResultStage 7 (foreachPartition at EdgeProcessor.scala:350) finished in 2.634 s
22/02/21 08:10:45 INFO DAGScheduler: Job 3 finished: foreachPartition at EdgeProcessor.scala:350, took 2.814434 s
22/02/21 08:10:45 INFO Exchange$: import for edge serve cost time: 2.94 s
22/02/21 08:10:45 INFO Exchange$: Client-Import: batchSuccess.serve: 7
22/02/21 08:10:45 INFO Exchange$: Client-Import: batchFailure.serve: 0
22/02/21 08:10:45 INFO SparkUI: Stopped Spark web UI at http://b213d13690e2:4040
22/02/21 08:10:45 INFO StandaloneSchedulerBackend: Shutting down all executors
22/02/21 08:10:45 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
22/02/21 08:10:46 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
22/02/21 08:10:46 INFO MemoryStore: MemoryStore cleared
22/02/21 08:10:46 INFO BlockManager: BlockManager stopped
22/02/21 08:10:46 INFO BlockManagerMaster: BlockManagerMaster stopped
22/02/21 08:10:46 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
22/02/21 08:10:46 INFO SparkContext: Successfully stopped SparkContext
22/02/21 08:10:46 INFO ShutdownHookManager: Shutdown hook called
22/02/21 08:10:46 INFO ShutdownHookManager: Deleting directory /tmp/spark-8ec17941-a20d-481e-a035-2cf1ef879941
22/02/21 08:10:46 INFO ShutdownHookManager: Deleting directory /tmp/spark-8128bb4a-11ab-4064-b710-1e170ef8483f