+ spark-submit --class com.vesoft.nebula.exchange.Exchange --master yarn --queue RT --name risk_topology_user_phone --num-executors 10 --executor-cores 4 --executor-memory 40G --driver-memory 4G --conf spark.driver.extraJavaOptions=-Dfile.encoding=utf-8 --conf spark.executor.extraJavaOptions=-Dfile.encoding=utf-8 --conf spark.network.timeout=240000 /root/work/nebula-sst/conf/nebula-exchange-2.6.3.jar -c /root/work/nebula-sst/topology_user_and_phone.conf 22/04/28 17:28:54 INFO [main] Configs$: DataBase Config com.vesoft.nebula.exchange.config.DataBaseConfigEntry@5555f04f 22/04/28 17:28:54 INFO [main] Configs$: User Config com.vesoft.nebula.exchange.config.UserConfigEntry@63f5d64b 22/04/28 17:28:54 INFO [main] Configs$: Connection Config Some(Config(SimpleConfigObject({"timeout":300000}))) 22/04/28 17:28:54 INFO [main] Configs$: Execution Config com.vesoft.nebula.exchange.config.ExecutionConfigEntry@c757796a 22/04/28 17:28:54 INFO [main] Configs$: Source Config File source path: hdfs://hadoop-cluster/home/risk_manager/nebula_data_v2/vt_user/day=2022-04-24/part-*, separator: Some(,), header: Some(false) 22/04/28 17:28:54 INFO [main] Configs$: Sink Config File source path: hdfs://hadoop-cluster/home/risk_manager/nebula_data_v2/vt_user/day=2022-04-24/part-*, separator: Some(,), header: Some(false) 22/04/28 17:28:54 INFO [main] Configs$: name vt_user batch 256 22/04/28 17:28:54 INFO [main] Configs$: Tag Config: Tag name: vt_user, source: File source path: hdfs://hadoop-cluster/home/risk_manager/nebula_data_v2/vt_user/day=2022-04-24/part-*, separator: Some(,), header: Some(false), sink: File sink: from /tmp to hdfs://hadoop-cluster/home/risk_manager/nebula_sst_v2/topology_user_and_phone/day=2022-04-24, vertex field: _c0, vertex policy: None, batch: 256, partition: 32. 22/04/28 17:28:54 INFO [main] Exchange$: Config Configs(com.vesoft.nebula.exchange.config.DataBaseConfigEntry@5555f04f,com.vesoft.nebula.exchange.config.UserConfigEntry@63f5d64b,com.vesoft.nebula.exchange.config.ConnectionConfigEntry@a89e2e42,com.vesoft.nebula.exchange.config.ExecutionConfigEntry@c757796a,com.vesoft.nebula.exchange.config.ErrorConfigEntry@d70781fc,com.vesoft.nebula.exchange.config.RateConfigEntry@9553c360,com.vesoft.nebula.exchange.config.SslConfigEntry@745bf6f4,,List(Tag name: vt_user, source: File source path: hdfs://hadoop-cluster/home/risk_manager/nebula_data_v2/vt_user/day=2022-04-24/part-*, separator: Some(,), header: Some(false), sink: File sink: from /tmp to hdfs://hadoop-cluster/home/risk_manager/nebula_sst_v2/topology_user_and_phone/day=2022-04-24, vertex field: _c0, vertex policy: None, batch: 256, partition: 32.),List(),None) 22/04/28 17:28:54 INFO [main] SparkContext: Running Spark version 2.4.5 22/04/28 17:28:54 WARN [main] SparkConf: Total executor cores: 10 is not divisible by cores per executor: 4, the left cores: 2 will not be allocated 22/04/28 17:28:54 INFO [main] SparkContext: Submitted application: risk_topology_user_phone 22/04/28 17:28:54 INFO [main] SecurityManager: Changing view acls to: root,* 22/04/28 17:28:54 INFO [main] SecurityManager: Changing modify acls to: root 22/04/28 17:28:54 INFO [main] SecurityManager: Changing view acls groups to: 22/04/28 17:28:54 INFO [main] SecurityManager: Changing modify acls groups to: 22/04/28 17:28:54 INFO [main] SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root, *); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set() 22/04/28 17:28:55 INFO [main] Utils: Successfully started service 'sparkDriver' on port 45261. 22/04/28 17:28:55 INFO [main] SparkEnv: Registering MapOutputTracker 22/04/28 17:28:55 INFO [main] SparkEnv: Registering BlockManagerMaster 22/04/28 17:28:55 INFO [main] BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 22/04/28 17:28:55 INFO [main] BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up 22/04/28 17:28:55 INFO [main] DiskBlockManager: Created local directory at /tmp/blockmgr-537132ee-9d85-4889-9879-8b86b7b74b0c 22/04/28 17:28:55 INFO [main] MemoryStore: MemoryStore started with capacity 2004.6 MB 22/04/28 17:28:55 INFO [main] SparkEnv: Registering OutputCommitCoordinator 22/04/28 17:28:55 INFO [main] Utils: Successfully started service 'SparkUI' on port 4040. 22/04/28 17:28:55 INFO [main] SparkUI: Bound SparkUI to 0.0.0.0, and started at http://172.16.67.122:4040 22/04/28 17:28:55 INFO [main] SparkContext: Added JAR file:/root/work/alpha-risk-topology/risk-nebula-sst/conf/nebula-exchange-2.6.3.jar at spark://172.16.67.122:45261/jars/nebula-exchange-2.6.3.jar with timestamp 1651138135454 22/04/28 17:28:56 INFO [main] Client: Requesting a new application from cluster with 22 NodeManagers 22/04/28 17:28:56 INFO [main] Client: Verifying our application has not requested more than the maximum memory capability of the cluster (254976 MB per container) 22/04/28 17:28:56 INFO [main] Client: Will allocate AM container, with 896 MB memory including 384 MB overhead 22/04/28 17:28:56 INFO [main] Client: Setting up container launch context for our AM 22/04/28 17:28:56 INFO [main] Client: Setting up the launch environment for our AM container 22/04/28 17:28:56 INFO [main] Client: Preparing resources for our AM container 22/04/28 17:28:56 WARN [main] Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. 22/04/28 17:29:00 INFO [main] Client: Uploading resource file:/tmp/spark-d452e02e-bbec-496c-8492-2be605d831df/__spark_libs__8832634010354430098.zip -> hdfs://hadoop-cluster/user/root/.sparkStaging/application_1607584549220_196043/__spark_libs__8832634010354430098.zip 22/04/28 17:29:04 INFO [main] Client: Uploading resource file:/etc/ecm/spark-conf/hive-site.xml -> hdfs://hadoop-cluster/user/root/.sparkStaging/application_1607584549220_196043/hive-site.xml 22/04/28 17:29:05 INFO [main] Client: Uploading resource file:/tmp/spark-d452e02e-bbec-496c-8492-2be605d831df/__spark_conf__4581093488647528842.zip -> hdfs://hadoop-cluster/user/root/.sparkStaging/application_1607584549220_196043/__spark_conf__.zip 22/04/28 17:29:06 INFO [main] Client: EMR SmartData using Java binary at JAVA_HOME. 22/04/28 17:29:06 INFO [main] SecurityManager: Changing view acls to: root,* 22/04/28 17:29:06 INFO [main] SecurityManager: Changing modify acls to: root 22/04/28 17:29:06 INFO [main] SecurityManager: Changing view acls groups to: 22/04/28 17:29:06 INFO [main] SecurityManager: Changing modify acls groups to: 22/04/28 17:29:06 INFO [main] SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root, *); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set() 22/04/28 17:29:07 INFO [main] Client: Submitting application application_1607584549220_196043 to ResourceManager 22/04/28 17:29:07 INFO [main] YarnClientImpl: Submitted application application_1607584549220_196043 22/04/28 17:29:07 INFO [main] SchedulerExtensionServices: Starting Yarn extension services with app application_1607584549220_196043 and attemptId None 22/04/28 17:29:08 INFO [main] Client: Application report for application_1607584549220_196043 (state: ACCEPTED) 22/04/28 17:29:08 INFO [main] Client: client token: N/A diagnostics: AM container is launched, waiting for AM container to Register with RM ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: RT start time: 1651138147104 final status: UNDEFINED tracking URL: http://emr-header-1.cluster-196425:20888/proxy/application_1607584549220_196043/ user: root 22/04/28 17:29:09 INFO [main] Client: Application report for application_1607584549220_196043 (state: ACCEPTED) 22/04/28 17:29:10 INFO [main] Client: Application report for application_1607584549220_196043 (state: ACCEPTED) 22/04/28 17:29:11 INFO [main] Client: Application report for application_1607584549220_196043 (state: ACCEPTED) 22/04/28 17:29:12 INFO [dispatcher-event-loop-3] YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> emr-header-1.cluster-196425, PROXY_URI_BASES -> http://emr-header-1.cluster-196425:20888/proxy/application_1607584549220_196043, RM_HA_URLS -> emr-header-2.cluster-196425:8088,emr-header-1.cluster-196425:8088), /proxy/application_1607584549220_196043 22/04/28 17:29:12 INFO [main] Client: Application report for application_1607584549220_196043 (state: RUNNING) 22/04/28 17:29:12 INFO [main] Client: client token: N/A diagnostics: N/A ApplicationMaster host: 172.16.68.34 ApplicationMaster RPC port: -1 queue: RT start time: 1651138147104 final status: UNDEFINED tracking URL: http://emr-header-1.cluster-196425:20888/proxy/application_1607584549220_196043/ user: root 22/04/28 17:29:12 INFO [main] YarnClientSchedulerBackend: Application application_1607584549220_196043 has started running. 22/04/28 17:29:12 INFO [main] Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 41394. 22/04/28 17:29:12 INFO [main] NettyBlockTransferService: Server created on 172.16.67.122:41394 22/04/28 17:29:12 INFO [main] BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 22/04/28 17:29:12 INFO [dispatcher-event-loop-1] YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark-client://YarnAM) 22/04/28 17:29:12 INFO [main] BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 172.16.67.122, 41394, None) 22/04/28 17:29:12 INFO [dispatcher-event-loop-3] BlockManagerMasterEndpoint: Registering block manager 172.16.67.122:41394 with 2004.6 MB RAM, BlockManagerId(driver, 172.16.67.122, 41394, None) 22/04/28 17:29:12 INFO [main] BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 172.16.67.122, 41394, None) 22/04/28 17:29:12 INFO [main] BlockManager: external shuffle service port = 7337 22/04/28 17:29:12 INFO [main] BlockManager: Initialized BlockManager: BlockManagerId(driver, 172.16.67.122, 41394, None) 22/04/28 17:29:12 INFO [main] JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /metrics/json. 22/04/28 17:29:12 INFO [main] EventLoggingListener: Logging events to hdfs://hadoop-cluster/spark-history/application_1607584549220_196043 22/04/28 17:29:16 INFO [dispatcher-event-loop-0] YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.16.72.56:49634) with ID 1 22/04/28 17:29:16 INFO [dispatcher-event-loop-0] BlockManagerMasterEndpoint: Registering block manager emr-worker-8.cluster-196425:33263 with 21.2 GB RAM, BlockManagerId(1, emr-worker-8.cluster-196425, 33263, None) 22/04/28 17:29:16 INFO [dispatcher-event-loop-0] YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.16.72.52:59280) with ID 6 22/04/28 17:29:16 INFO [dispatcher-event-loop-2] YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.16.72.52:59338) with ID 4 22/04/28 17:29:16 INFO [dispatcher-event-loop-3] BlockManagerMasterEndpoint: Registering block manager emr-worker-6.cluster-196425:37495 with 21.2 GB RAM, BlockManagerId(6, emr-worker-6.cluster-196425, 37495, None) 22/04/28 17:29:16 INFO [dispatcher-event-loop-1] YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.16.67.172:46370) with ID 5 22/04/28 17:29:16 INFO [dispatcher-event-loop-2] BlockManagerMasterEndpoint: Registering block manager emr-worker-6.cluster-196425:41171 with 21.2 GB RAM, BlockManagerId(4, emr-worker-6.cluster-196425, 41171, None) 22/04/28 17:29:16 INFO [dispatcher-event-loop-0] YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.16.67.172:46372) with ID 2 22/04/28 17:29:16 INFO [dispatcher-event-loop-3] BlockManagerMasterEndpoint: Registering block manager emr-worker-19.cluster-196425:41635 with 21.2 GB RAM, BlockManagerId(5, emr-worker-19.cluster-196425, 41635, None) 22/04/28 17:29:17 INFO [dispatcher-event-loop-0] BlockManagerMasterEndpoint: Registering block manager emr-worker-19.cluster-196425:42145 with 21.2 GB RAM, BlockManagerId(2, emr-worker-19.cluster-196425, 42145, None) 22/04/28 17:29:17 INFO [dispatcher-event-loop-0] YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.16.72.52:60224) with ID 7 22/04/28 17:29:17 INFO [dispatcher-event-loop-3] BlockManagerMasterEndpoint: Registering block manager emr-worker-6.cluster-196425:36207 with 21.2 GB RAM, BlockManagerId(7, emr-worker-6.cluster-196425, 36207, None) 22/04/28 17:29:17 INFO [dispatcher-event-loop-1] YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.16.72.54:56452) with ID 3 22/04/28 17:29:18 INFO [dispatcher-event-loop-3] BlockManagerMasterEndpoint: Registering block manager emr-worker-9.cluster-196425:46825 with 21.2 GB RAM, BlockManagerId(3, emr-worker-9.cluster-196425, 46825, None) 22/04/28 17:29:25 INFO [main] YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after waiting maxRegisteredResourcesWaitingTime: 30000(ms) 22/04/28 17:29:25 INFO [main] Exchange$: Processing Tag vt_user 22/04/28 17:29:25 INFO [main] Exchange$: field keys: _c1, _c2, _c4, _c5 22/04/28 17:29:25 INFO [main] Exchange$: nebula keys: gender, age, u_state, c_time 22/04/28 17:29:25 INFO [main] Exchange$: Loading CSV files from hdfs://hadoop-cluster/home/risk_manager/nebula_data_v2/vt_user/day=2022-04-24/part-* 22/04/28 17:29:25 INFO [main] SharedState: loading hive config file: file:/etc/ecm/spark-conf/hive-site.xml 22/04/28 17:29:25 INFO [main] SharedState: spark.sql.warehouse.dir is not set, but hive.metastore.warehouse.dir is set. Setting spark.sql.warehouse.dir to the value of hive.metastore.warehouse.dir ('/user/hive/warehouse'). 22/04/28 17:29:25 INFO [main] SharedState: Warehouse path is '/user/hive/warehouse'. 22/04/28 17:29:25 INFO [main] JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL. 22/04/28 17:29:25 INFO [main] JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/json. 22/04/28 17:29:25 INFO [main] JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/execution. 22/04/28 17:29:25 INFO [main] JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/execution/json. 22/04/28 17:29:25 INFO [main] JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /static/sql. 22/04/28 17:29:25 INFO [main] JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /streamingsql. 22/04/28 17:29:25 INFO [main] JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /streamingsql/json. 22/04/28 17:29:25 INFO [main] JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /streamingsql/statistics. 22/04/28 17:29:25 INFO [main] JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /streamingsql/statistics/json. 22/04/28 17:29:25 INFO [main] JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /static/sql. 22/04/28 17:29:25 INFO [main] JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /streamingsql/query/kill. 22/04/28 17:29:25 INFO [main] JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /streamingsql/query/restart. 22/04/28 17:29:26 INFO [main] StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint 22/04/28 17:29:26 INFO [main] InMemoryFileIndex: Listing leaf files and directories in parallel under 100 paths. The first several paths are: hdfs://hadoop-cluster/home/risk_manager/nebula_data_v2/vt_user/day=2022-04-24/part-00000, hdfs://hadoop-cluster/home/risk_manager/nebula_data_v2/vt_user/day=2022-04-24/part-00001, hdfs://hadoop-cluster/home/risk_manager/nebula_data_v2/vt_user/day=2022-04-24/part-00002, hdfs://hadoop-cluster/home/risk_manager/nebula_data_v2/vt_user/day=2022-04-24/part-00003, hdfs://hadoop-cluster/home/risk_manager/nebula_data_v2/vt_user/day=2022-04-24/part-00004, hdfs://hadoop-cluster/home/risk_manager/nebula_data_v2/vt_user/day=2022-04-24/part-00005, hdfs://hadoop-cluster/home/risk_manager/nebula_data_v2/vt_user/day=2022-04-24/part-00006, hdfs://hadoop-cluster/home/risk_manager/nebula_data_v2/vt_user/day=2022-04-24/part-00007, hdfs://hadoop-cluster/home/risk_manager/nebula_data_v2/vt_user/day=2022-04-24/part-00008, hdfs://hadoop-cluster/home/risk_manager/nebula_data_v2/vt_user/day=2022-04-24/part-00009. 22/04/28 17:29:26 INFO [main] SparkContext: Starting job: csv at FileBaseReader.scala:86 22/04/28 17:29:26 INFO [dag-scheduler-event-loop] DAGScheduler: Got job 0 (csv at FileBaseReader.scala:86) with 100 output partitions 22/04/28 17:29:26 INFO [dag-scheduler-event-loop] DAGScheduler: Final stage: ResultStage 0 (csv at FileBaseReader.scala:86) 22/04/28 17:29:26 INFO [dag-scheduler-event-loop] DAGScheduler: Parents of final stage: List() 22/04/28 17:29:26 INFO [dag-scheduler-event-loop] DAGScheduler: Missing parents: List() 22/04/28 17:29:26 INFO [dag-scheduler-event-loop] DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[2] at csv at FileBaseReader.scala:86), which has no missing parents 22/04/28 17:29:26 INFO [dag-scheduler-event-loop] MemoryStore: Block broadcast_0 stored as values in memory (estimated size 114.5 KB, free 2004.5 MB) 22/04/28 17:29:26 INFO [dag-scheduler-event-loop] MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 43.4 KB, free 2004.4 MB) 22/04/28 17:29:26 INFO [dispatcher-event-loop-3] BlockManagerInfo: Added broadcast_0_piece0 in memory on 172.16.67.122:41394 (size: 43.4 KB, free: 2004.6 MB) 22/04/28 17:29:26 INFO [dag-scheduler-event-loop] SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1163 22/04/28 17:29:26 INFO [dag-scheduler-event-loop] DAGScheduler: Submitting 100 missing tasks from ResultStage 0 (MapPartitionsRDD[2] at csv at FileBaseReader.scala:86) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14)) 22/04/28 17:29:26 INFO [dag-scheduler-event-loop] YarnScheduler: Adding task set 0.0 with 100 tasks 22/04/28 17:29:26 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, emr-worker-6.cluster-196425, executor 7, partition 0, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:26 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, emr-worker-9.cluster-196425, executor 3, partition 1, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:26 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, emr-worker-6.cluster-196425, executor 6, partition 2, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:26 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, emr-worker-6.cluster-196425, executor 4, partition 3, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:26 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 4.0 in stage 0.0 (TID 4, emr-worker-8.cluster-196425, executor 1, partition 4, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:26 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 5.0 in stage 0.0 (TID 5, emr-worker-19.cluster-196425, executor 2, partition 5, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:26 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 6.0 in stage 0.0 (TID 6, emr-worker-19.cluster-196425, executor 5, partition 6, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:26 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 7.0 in stage 0.0 (TID 7, emr-worker-6.cluster-196425, executor 7, partition 7, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:26 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 8.0 in stage 0.0 (TID 8, emr-worker-9.cluster-196425, executor 3, partition 8, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:26 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 9.0 in stage 0.0 (TID 9, emr-worker-6.cluster-196425, executor 6, partition 9, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:26 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 10.0 in stage 0.0 (TID 10, emr-worker-6.cluster-196425, executor 4, partition 10, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:26 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 11.0 in stage 0.0 (TID 11, emr-worker-8.cluster-196425, executor 1, partition 11, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:26 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 12.0 in stage 0.0 (TID 12, emr-worker-19.cluster-196425, executor 2, partition 12, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:26 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 13.0 in stage 0.0 (TID 13, emr-worker-19.cluster-196425, executor 5, partition 13, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:26 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 14.0 in stage 0.0 (TID 14, emr-worker-6.cluster-196425, executor 7, partition 14, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:26 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 15.0 in stage 0.0 (TID 15, emr-worker-9.cluster-196425, executor 3, partition 15, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:26 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 16.0 in stage 0.0 (TID 16, emr-worker-6.cluster-196425, executor 6, partition 16, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:26 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 17.0 in stage 0.0 (TID 17, emr-worker-6.cluster-196425, executor 4, partition 17, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:26 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 18.0 in stage 0.0 (TID 18, emr-worker-8.cluster-196425, executor 1, partition 18, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:26 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 19.0 in stage 0.0 (TID 19, emr-worker-19.cluster-196425, executor 2, partition 19, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:26 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 20.0 in stage 0.0 (TID 20, emr-worker-19.cluster-196425, executor 5, partition 20, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:26 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 21.0 in stage 0.0 (TID 21, emr-worker-6.cluster-196425, executor 7, partition 21, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:26 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 22.0 in stage 0.0 (TID 22, emr-worker-9.cluster-196425, executor 3, partition 22, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:26 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 23.0 in stage 0.0 (TID 23, emr-worker-6.cluster-196425, executor 6, partition 23, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:26 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 24.0 in stage 0.0 (TID 24, emr-worker-6.cluster-196425, executor 4, partition 24, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:26 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 25.0 in stage 0.0 (TID 25, emr-worker-8.cluster-196425, executor 1, partition 25, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:26 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 26.0 in stage 0.0 (TID 26, emr-worker-19.cluster-196425, executor 2, partition 26, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:26 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 27.0 in stage 0.0 (TID 27, emr-worker-19.cluster-196425, executor 5, partition 27, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:28 INFO [dispatcher-event-loop-2] BlockManagerInfo: Added broadcast_0_piece0 in memory on emr-worker-6.cluster-196425:37495 (size: 43.4 KB, free: 21.2 GB) 22/04/28 17:29:28 INFO [dispatcher-event-loop-3] BlockManagerInfo: Added broadcast_0_piece0 in memory on emr-worker-9.cluster-196425:46825 (size: 43.4 KB, free: 21.2 GB) 22/04/28 17:29:28 INFO [dispatcher-event-loop-2] BlockManagerInfo: Added broadcast_0_piece0 in memory on emr-worker-8.cluster-196425:33263 (size: 43.4 KB, free: 21.2 GB) 22/04/28 17:29:28 INFO [dispatcher-event-loop-3] BlockManagerInfo: Added broadcast_0_piece0 in memory on emr-worker-6.cluster-196425:41171 (size: 43.4 KB, free: 21.2 GB) 22/04/28 17:29:28 INFO [dispatcher-event-loop-0] BlockManagerInfo: Added broadcast_0_piece0 in memory on emr-worker-19.cluster-196425:41635 (size: 43.4 KB, free: 21.2 GB) 22/04/28 17:29:28 INFO [dispatcher-event-loop-0] BlockManagerInfo: Added broadcast_0_piece0 in memory on emr-worker-6.cluster-196425:36207 (size: 43.4 KB, free: 21.2 GB) 22/04/28 17:29:29 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 28.0 in stage 0.0 (TID 28, emr-worker-6.cluster-196425, executor 6, partition 28, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 29.0 in stage 0.0 (TID 29, emr-worker-6.cluster-196425, executor 6, partition 29, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 30.0 in stage 0.0 (TID 30, emr-worker-6.cluster-196425, executor 6, partition 30, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 31.0 in stage 0.0 (TID 31, emr-worker-6.cluster-196425, executor 6, partition 31, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 32.0 in stage 0.0 (TID 32, emr-worker-8.cluster-196425, executor 1, partition 32, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 33.0 in stage 0.0 (TID 33, emr-worker-8.cluster-196425, executor 1, partition 33, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 34.0 in stage 0.0 (TID 34, emr-worker-8.cluster-196425, executor 1, partition 34, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 35.0 in stage 0.0 (TID 35, emr-worker-8.cluster-196425, executor 1, partition 35, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 9.0 in stage 0.0 (TID 9) in 2545 ms on emr-worker-6.cluster-196425 (executor 6) (1/100) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 4.0 in stage 0.0 (TID 4) in 2568 ms on emr-worker-8.cluster-196425 (executor 1) (2/100) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 18.0 in stage 0.0 (TID 18) in 2519 ms on emr-worker-8.cluster-196425 (executor 1) (3/100) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 25.0 in stage 0.0 (TID 25) in 2497 ms on emr-worker-8.cluster-196425 (executor 1) (4/100) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 11.0 in stage 0.0 (TID 11) in 2543 ms on emr-worker-8.cluster-196425 (executor 1) (5/100) 22/04/28 17:29:29 INFO [task-result-getter-2] TaskSetManager: Finished task 23.0 in stage 0.0 (TID 23) in 2510 ms on emr-worker-6.cluster-196425 (executor 6) (6/100) 22/04/28 17:29:29 INFO [task-result-getter-3] TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 2583 ms on emr-worker-6.cluster-196425 (executor 6) (7/100) 22/04/28 17:29:29 INFO [task-result-getter-1] TaskSetManager: Finished task 16.0 in stage 0.0 (TID 16) in 2532 ms on emr-worker-6.cluster-196425 (executor 6) (8/100) 22/04/28 17:29:29 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 36.0 in stage 0.0 (TID 36, emr-worker-9.cluster-196425, executor 3, partition 36, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 37.0 in stage 0.0 (TID 37, emr-worker-9.cluster-196425, executor 3, partition 37, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 38.0 in stage 0.0 (TID 38, emr-worker-9.cluster-196425, executor 3, partition 38, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 39.0 in stage 0.0 (TID 39, emr-worker-9.cluster-196425, executor 3, partition 39, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 40.0 in stage 0.0 (TID 40, emr-worker-6.cluster-196425, executor 6, partition 40, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 41.0 in stage 0.0 (TID 41, emr-worker-6.cluster-196425, executor 6, partition 41, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 42.0 in stage 0.0 (TID 42, emr-worker-6.cluster-196425, executor 6, partition 42, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 43.0 in stage 0.0 (TID 43, emr-worker-6.cluster-196425, executor 6, partition 43, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 44.0 in stage 0.0 (TID 44, emr-worker-8.cluster-196425, executor 1, partition 44, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 45.0 in stage 0.0 (TID 45, emr-worker-8.cluster-196425, executor 1, partition 45, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 46.0 in stage 0.0 (TID 46, emr-worker-8.cluster-196425, executor 1, partition 46, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 47.0 in stage 0.0 (TID 47, emr-worker-8.cluster-196425, executor 1, partition 47, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 48.0 in stage 0.0 (TID 48, emr-worker-6.cluster-196425, executor 6, partition 48, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [task-result-getter-3] TaskSetManager: Finished task 22.0 in stage 0.0 (TID 22) in 2570 ms on emr-worker-9.cluster-196425 (executor 3) (9/100) 22/04/28 17:29:29 INFO [task-result-getter-3] TaskSetManager: Finished task 31.0 in stage 0.0 (TID 31) in 88 ms on emr-worker-6.cluster-196425 (executor 6) (10/100) 22/04/28 17:29:29 INFO [task-result-getter-3] TaskSetManager: Finished task 28.0 in stage 0.0 (TID 28) in 107 ms on emr-worker-6.cluster-196425 (executor 6) (11/100) 22/04/28 17:29:29 INFO [task-result-getter-3] TaskSetManager: Finished task 29.0 in stage 0.0 (TID 29) in 102 ms on emr-worker-6.cluster-196425 (executor 6) (12/100) 22/04/28 17:29:29 INFO [task-result-getter-3] TaskSetManager: Finished task 30.0 in stage 0.0 (TID 30) in 96 ms on emr-worker-6.cluster-196425 (executor 6) (13/100) 22/04/28 17:29:29 INFO [task-result-getter-1] TaskSetManager: Finished task 15.0 in stage 0.0 (TID 15) in 2595 ms on emr-worker-9.cluster-196425 (executor 3) (14/100) 22/04/28 17:29:29 INFO [task-result-getter-1] TaskSetManager: Finished task 35.0 in stage 0.0 (TID 35) in 74 ms on emr-worker-8.cluster-196425 (executor 1) (15/100) 22/04/28 17:29:29 INFO [task-result-getter-1] TaskSetManager: Finished task 33.0 in stage 0.0 (TID 33) in 81 ms on emr-worker-8.cluster-196425 (executor 1) (16/100) 22/04/28 17:29:29 INFO [task-result-getter-1] TaskSetManager: Finished task 32.0 in stage 0.0 (TID 32) in 85 ms on emr-worker-8.cluster-196425 (executor 1) (17/100) 22/04/28 17:29:29 INFO [task-result-getter-1] TaskSetManager: Finished task 40.0 in stage 0.0 (TID 40) in 30 ms on emr-worker-6.cluster-196425 (executor 6) (18/100) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 8.0 in stage 0.0 (TID 8) in 2621 ms on emr-worker-9.cluster-196425 (executor 3) (19/100) 22/04/28 17:29:29 INFO [task-result-getter-2] TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 2653 ms on emr-worker-9.cluster-196425 (executor 3) (20/100) 22/04/28 17:29:29 INFO [task-result-getter-3] TaskSetManager: Finished task 34.0 in stage 0.0 (TID 34) in 84 ms on emr-worker-8.cluster-196425 (executor 1) (21/100) 22/04/28 17:29:29 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 49.0 in stage 0.0 (TID 49, emr-worker-6.cluster-196425, executor 6, partition 49, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [task-result-getter-1] TaskSetManager: Finished task 43.0 in stage 0.0 (TID 43) in 30 ms on emr-worker-6.cluster-196425 (executor 6) (22/100) 22/04/28 17:29:29 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 50.0 in stage 0.0 (TID 50, emr-worker-8.cluster-196425, executor 1, partition 50, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 51.0 in stage 0.0 (TID 51, emr-worker-6.cluster-196425, executor 6, partition 51, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 52.0 in stage 0.0 (TID 52, emr-worker-6.cluster-196425, executor 6, partition 52, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 53.0 in stage 0.0 (TID 53, emr-worker-8.cluster-196425, executor 1, partition 53, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 54.0 in stage 0.0 (TID 54, emr-worker-8.cluster-196425, executor 1, partition 54, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 55.0 in stage 0.0 (TID 55, emr-worker-8.cluster-196425, executor 1, partition 55, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 56.0 in stage 0.0 (TID 56, emr-worker-9.cluster-196425, executor 3, partition 56, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 57.0 in stage 0.0 (TID 57, emr-worker-9.cluster-196425, executor 3, partition 57, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 58.0 in stage 0.0 (TID 58, emr-worker-9.cluster-196425, executor 3, partition 58, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 59.0 in stage 0.0 (TID 59, emr-worker-9.cluster-196425, executor 3, partition 59, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 60.0 in stage 0.0 (TID 60, emr-worker-6.cluster-196425, executor 6, partition 60, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 61.0 in stage 0.0 (TID 61, emr-worker-8.cluster-196425, executor 1, partition 61, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 62.0 in stage 0.0 (TID 62, emr-worker-6.cluster-196425, executor 6, partition 62, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 63.0 in stage 0.0 (TID 63, emr-worker-8.cluster-196425, executor 1, partition 63, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 64.0 in stage 0.0 (TID 64, emr-worker-8.cluster-196425, executor 1, partition 64, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 65.0 in stage 0.0 (TID 65, emr-worker-8.cluster-196425, executor 1, partition 65, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [task-result-getter-2] TaskSetManager: Finished task 41.0 in stage 0.0 (TID 41) in 72 ms on emr-worker-6.cluster-196425 (executor 6) (23/100) 22/04/28 17:29:29 INFO [task-result-getter-2] TaskSetManager: Finished task 44.0 in stage 0.0 (TID 44) in 61 ms on emr-worker-8.cluster-196425 (executor 1) (24/100) 22/04/28 17:29:29 INFO [task-result-getter-2] TaskSetManager: Finished task 46.0 in stage 0.0 (TID 46) in 57 ms on emr-worker-8.cluster-196425 (executor 1) (25/100) 22/04/28 17:29:29 INFO [task-result-getter-1] TaskSetManager: Finished task 47.0 in stage 0.0 (TID 47) in 54 ms on emr-worker-8.cluster-196425 (executor 1) (26/100) 22/04/28 17:29:29 INFO [task-result-getter-2] TaskSetManager: Finished task 39.0 in stage 0.0 (TID 39) in 79 ms on emr-worker-9.cluster-196425 (executor 3) (27/100) 22/04/28 17:29:29 INFO [task-result-getter-1] TaskSetManager: Finished task 36.0 in stage 0.0 (TID 36) in 88 ms on emr-worker-9.cluster-196425 (executor 3) (28/100) 22/04/28 17:29:29 INFO [task-result-getter-2] TaskSetManager: Finished task 37.0 in stage 0.0 (TID 37) in 85 ms on emr-worker-9.cluster-196425 (executor 3) (29/100) 22/04/28 17:29:29 INFO [task-result-getter-1] TaskSetManager: Finished task 38.0 in stage 0.0 (TID 38) in 82 ms on emr-worker-9.cluster-196425 (executor 3) (30/100) 22/04/28 17:29:29 INFO [task-result-getter-1] TaskSetManager: Finished task 50.0 in stage 0.0 (TID 50) in 36 ms on emr-worker-8.cluster-196425 (executor 1) (31/100) 22/04/28 17:29:29 INFO [task-result-getter-1] TaskSetManager: Finished task 49.0 in stage 0.0 (TID 49) in 39 ms on emr-worker-6.cluster-196425 (executor 6) (32/100) 22/04/28 17:29:29 INFO [task-result-getter-1] TaskSetManager: Finished task 54.0 in stage 0.0 (TID 54) in 28 ms on emr-worker-8.cluster-196425 (executor 1) (33/100) 22/04/28 17:29:29 INFO [task-result-getter-1] TaskSetManager: Finished task 53.0 in stage 0.0 (TID 53) in 31 ms on emr-worker-8.cluster-196425 (executor 1) (34/100) 22/04/28 17:29:29 INFO [task-result-getter-1] TaskSetManager: Finished task 55.0 in stage 0.0 (TID 55) in 27 ms on emr-worker-8.cluster-196425 (executor 1) (35/100) 22/04/28 17:29:29 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 66.0 in stage 0.0 (TID 66, emr-worker-6.cluster-196425, executor 6, partition 66, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 67.0 in stage 0.0 (TID 67, emr-worker-6.cluster-196425, executor 6, partition 67, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 45.0 in stage 0.0 (TID 45) in 68 ms on emr-worker-8.cluster-196425 (executor 1) (36/100) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 51.0 in stage 0.0 (TID 51) in 41 ms on emr-worker-6.cluster-196425 (executor 6) (37/100) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 68.0 in stage 0.0 (TID 68, emr-worker-9.cluster-196425, executor 3, partition 68, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 69.0 in stage 0.0 (TID 69, emr-worker-9.cluster-196425, executor 3, partition 69, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 70.0 in stage 0.0 (TID 70, emr-worker-9.cluster-196425, executor 3, partition 70, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 71.0 in stage 0.0 (TID 71, emr-worker-9.cluster-196425, executor 3, partition 71, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 72.0 in stage 0.0 (TID 72, emr-worker-8.cluster-196425, executor 1, partition 72, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [task-result-getter-3] TaskSetManager: Finished task 42.0 in stage 0.0 (TID 42) in 86 ms on emr-worker-6.cluster-196425 (executor 6) (38/100) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 73.0 in stage 0.0 (TID 73, emr-worker-19.cluster-196425, executor 5, partition 73, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 74.0 in stage 0.0 (TID 74, emr-worker-19.cluster-196425, executor 5, partition 74, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 75.0 in stage 0.0 (TID 75, emr-worker-19.cluster-196425, executor 5, partition 75, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 76.0 in stage 0.0 (TID 76, emr-worker-19.cluster-196425, executor 5, partition 76, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 77.0 in stage 0.0 (TID 77, emr-worker-6.cluster-196425, executor 6, partition 77, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [task-result-getter-1] TaskSetManager: Finished task 52.0 in stage 0.0 (TID 52) in 60 ms on emr-worker-6.cluster-196425 (executor 6) (39/100) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 78.0 in stage 0.0 (TID 78, emr-worker-6.cluster-196425, executor 6, partition 78, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 79.0 in stage 0.0 (TID 79, emr-worker-8.cluster-196425, executor 1, partition 79, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 57.0 in stage 0.0 (TID 57) in 59 ms on emr-worker-9.cluster-196425 (executor 3) (40/100) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 59.0 in stage 0.0 (TID 59) in 55 ms on emr-worker-9.cluster-196425 (executor 3) (41/100) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 61.0 in stage 0.0 (TID 61) in 52 ms on emr-worker-8.cluster-196425 (executor 1) (42/100) 22/04/28 17:29:29 INFO [task-result-getter-1] TaskSetManager: Finished task 58.0 in stage 0.0 (TID 58) in 57 ms on emr-worker-9.cluster-196425 (executor 3) (43/100) 22/04/28 17:29:29 INFO [task-result-getter-2] TaskSetManager: Finished task 48.0 in stage 0.0 (TID 48) in 91 ms on emr-worker-6.cluster-196425 (executor 6) (44/100) 22/04/28 17:29:29 INFO [task-result-getter-1] TaskSetManager: Finished task 20.0 in stage 0.0 (TID 20) in 2666 ms on emr-worker-19.cluster-196425 (executor 5) (45/100) 22/04/28 17:29:29 INFO [task-result-getter-2] TaskSetManager: Finished task 27.0 in stage 0.0 (TID 27) in 2646 ms on emr-worker-19.cluster-196425 (executor 5) (46/100) 22/04/28 17:29:29 INFO [task-result-getter-2] TaskSetManager: Finished task 62.0 in stage 0.0 (TID 62) in 51 ms on emr-worker-6.cluster-196425 (executor 6) (47/100) 22/04/28 17:29:29 INFO [task-result-getter-2] TaskSetManager: Finished task 60.0 in stage 0.0 (TID 60) in 55 ms on emr-worker-6.cluster-196425 (executor 6) (48/100) 22/04/28 17:29:29 INFO [task-result-getter-2] TaskSetManager: Finished task 63.0 in stage 0.0 (TID 63) in 49 ms on emr-worker-8.cluster-196425 (executor 1) (49/100) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 6.0 in stage 0.0 (TID 6) in 2715 ms on emr-worker-19.cluster-196425 (executor 5) (50/100) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 80.0 in stage 0.0 (TID 80, emr-worker-8.cluster-196425, executor 1, partition 80, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 81.0 in stage 0.0 (TID 81, emr-worker-8.cluster-196425, executor 1, partition 81, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 82.0 in stage 0.0 (TID 82, emr-worker-9.cluster-196425, executor 3, partition 82, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [task-result-getter-3] TaskSetManager: Finished task 56.0 in stage 0.0 (TID 56) in 74 ms on emr-worker-9.cluster-196425 (executor 3) (51/100) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 83.0 in stage 0.0 (TID 83, emr-worker-6.cluster-196425, executor 6, partition 83, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 84.0 in stage 0.0 (TID 84, emr-worker-6.cluster-196425, executor 6, partition 84, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 85.0 in stage 0.0 (TID 85, emr-worker-9.cluster-196425, executor 3, partition 85, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 86.0 in stage 0.0 (TID 86, emr-worker-9.cluster-196425, executor 3, partition 86, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 65.0 in stage 0.0 (TID 65) in 64 ms on emr-worker-8.cluster-196425 (executor 1) (52/100) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 66.0 in stage 0.0 (TID 66) in 58 ms on emr-worker-6.cluster-196425 (executor 6) (53/100) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 67.0 in stage 0.0 (TID 67) in 55 ms on emr-worker-6.cluster-196425 (executor 6) (54/100) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 70.0 in stage 0.0 (TID 70) in 48 ms on emr-worker-9.cluster-196425 (executor 3) (55/100) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 69.0 in stage 0.0 (TID 69) in 51 ms on emr-worker-9.cluster-196425 (executor 3) (56/100) 22/04/28 17:29:29 INFO [task-result-getter-1] TaskSetManager: Finished task 13.0 in stage 0.0 (TID 13) in 2711 ms on emr-worker-19.cluster-196425 (executor 5) (57/100) 22/04/28 17:29:29 INFO [task-result-getter-2] TaskSetManager: Finished task 64.0 in stage 0.0 (TID 64) in 68 ms on emr-worker-8.cluster-196425 (executor 1) (58/100) 22/04/28 17:29:29 INFO [task-result-getter-3] TaskSetManager: Finished task 68.0 in stage 0.0 (TID 68) in 54 ms on emr-worker-9.cluster-196425 (executor 3) (59/100) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 87.0 in stage 0.0 (TID 87, emr-worker-9.cluster-196425, executor 3, partition 87, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 88.0 in stage 0.0 (TID 88, emr-worker-8.cluster-196425, executor 1, partition 88, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 89.0 in stage 0.0 (TID 89, emr-worker-6.cluster-196425, executor 4, partition 89, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 90.0 in stage 0.0 (TID 90, emr-worker-6.cluster-196425, executor 4, partition 90, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 91.0 in stage 0.0 (TID 91, emr-worker-6.cluster-196425, executor 4, partition 91, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 92.0 in stage 0.0 (TID 92, emr-worker-6.cluster-196425, executor 4, partition 92, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 93.0 in stage 0.0 (TID 93, emr-worker-6.cluster-196425, executor 6, partition 93, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 94.0 in stage 0.0 (TID 94, emr-worker-8.cluster-196425, executor 1, partition 94, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 95.0 in stage 0.0 (TID 95, emr-worker-6.cluster-196425, executor 6, partition 95, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 96.0 in stage 0.0 (TID 96, emr-worker-8.cluster-196425, executor 1, partition 96, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 97.0 in stage 0.0 (TID 97, emr-worker-8.cluster-196425, executor 1, partition 97, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 98.0 in stage 0.0 (TID 98, emr-worker-9.cluster-196425, executor 3, partition 98, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 99.0 in stage 0.0 (TID 99, emr-worker-9.cluster-196425, executor 3, partition 99, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:29 INFO [task-result-getter-1] TaskSetManager: Finished task 72.0 in stage 0.0 (TID 72) in 82 ms on emr-worker-8.cluster-196425 (executor 1) (60/100) 22/04/28 17:29:29 INFO [task-result-getter-3] TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 2787 ms on emr-worker-6.cluster-196425 (executor 4) (61/100) 22/04/28 17:29:29 INFO [task-result-getter-1] TaskSetManager: Finished task 24.0 in stage 0.0 (TID 24) in 2715 ms on emr-worker-6.cluster-196425 (executor 4) (62/100) 22/04/28 17:29:29 INFO [task-result-getter-3] TaskSetManager: Finished task 17.0 in stage 0.0 (TID 17) in 2737 ms on emr-worker-6.cluster-196425 (executor 4) (63/100) 22/04/28 17:29:29 INFO [task-result-getter-1] TaskSetManager: Finished task 77.0 in stage 0.0 (TID 77) in 73 ms on emr-worker-6.cluster-196425 (executor 6) (64/100) 22/04/28 17:29:29 INFO [task-result-getter-3] TaskSetManager: Finished task 79.0 in stage 0.0 (TID 79) in 70 ms on emr-worker-8.cluster-196425 (executor 1) (65/100) 22/04/28 17:29:29 INFO [task-result-getter-3] TaskSetManager: Finished task 80.0 in stage 0.0 (TID 80) in 57 ms on emr-worker-8.cluster-196425 (executor 1) (66/100) 22/04/28 17:29:29 INFO [task-result-getter-3] TaskSetManager: Finished task 81.0 in stage 0.0 (TID 81) in 55 ms on emr-worker-8.cluster-196425 (executor 1) (67/100) 22/04/28 17:29:29 INFO [task-result-getter-3] TaskSetManager: Finished task 82.0 in stage 0.0 (TID 82) in 53 ms on emr-worker-9.cluster-196425 (executor 3) (68/100) 22/04/28 17:29:29 INFO [task-result-getter-3] TaskSetManager: Finished task 85.0 in stage 0.0 (TID 85) in 46 ms on emr-worker-9.cluster-196425 (executor 3) (69/100) 22/04/28 17:29:29 INFO [task-result-getter-3] TaskSetManager: Finished task 83.0 in stage 0.0 (TID 83) in 51 ms on emr-worker-6.cluster-196425 (executor 6) (70/100) 22/04/28 17:29:29 INFO [task-result-getter-2] TaskSetManager: Finished task 10.0 in stage 0.0 (TID 10) in 2763 ms on emr-worker-6.cluster-196425 (executor 4) (71/100) 22/04/28 17:29:29 INFO [task-result-getter-1] TaskSetManager: Finished task 78.0 in stage 0.0 (TID 78) in 77 ms on emr-worker-6.cluster-196425 (executor 6) (72/100) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 71.0 in stage 0.0 (TID 71) in 94 ms on emr-worker-9.cluster-196425 (executor 3) (73/100) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 86.0 in stage 0.0 (TID 86) in 52 ms on emr-worker-9.cluster-196425 (executor 3) (74/100) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 87.0 in stage 0.0 (TID 87) in 47 ms on emr-worker-9.cluster-196425 (executor 3) (75/100) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 76.0 in stage 0.0 (TID 76) in 85 ms on emr-worker-19.cluster-196425 (executor 5) (76/100) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 75.0 in stage 0.0 (TID 75) in 88 ms on emr-worker-19.cluster-196425 (executor 5) (77/100) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 88.0 in stage 0.0 (TID 88) in 46 ms on emr-worker-8.cluster-196425 (executor 1) (78/100) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 94.0 in stage 0.0 (TID 94) in 27 ms on emr-worker-8.cluster-196425 (executor 1) (79/100) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 96.0 in stage 0.0 (TID 96) in 24 ms on emr-worker-8.cluster-196425 (executor 1) (80/100) 22/04/28 17:29:29 INFO [task-result-getter-1] TaskSetManager: Finished task 73.0 in stage 0.0 (TID 73) in 92 ms on emr-worker-19.cluster-196425 (executor 5) (81/100) 22/04/28 17:29:29 INFO [task-result-getter-2] TaskSetManager: Finished task 74.0 in stage 0.0 (TID 74) in 90 ms on emr-worker-19.cluster-196425 (executor 5) (82/100) 22/04/28 17:29:29 INFO [task-result-getter-3] TaskSetManager: Finished task 84.0 in stage 0.0 (TID 84) in 58 ms on emr-worker-6.cluster-196425 (executor 6) (83/100) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 93.0 in stage 0.0 (TID 93) in 41 ms on emr-worker-6.cluster-196425 (executor 6) (84/100) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 95.0 in stage 0.0 (TID 95) in 34 ms on emr-worker-6.cluster-196425 (executor 6) (85/100) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 98.0 in stage 0.0 (TID 98) in 27 ms on emr-worker-9.cluster-196425 (executor 3) (86/100) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 97.0 in stage 0.0 (TID 97) in 30 ms on emr-worker-8.cluster-196425 (executor 1) (87/100) 22/04/28 17:29:29 INFO [task-result-getter-1] TaskSetManager: Finished task 99.0 in stage 0.0 (TID 99) in 33 ms on emr-worker-9.cluster-196425 (executor 3) (88/100) 22/04/28 17:29:29 INFO [task-result-getter-3] TaskSetManager: Finished task 89.0 in stage 0.0 (TID 89) in 66 ms on emr-worker-6.cluster-196425 (executor 4) (89/100) 22/04/28 17:29:29 INFO [task-result-getter-2] TaskSetManager: Finished task 91.0 in stage 0.0 (TID 91) in 61 ms on emr-worker-6.cluster-196425 (executor 4) (90/100) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 90.0 in stage 0.0 (TID 90) in 66 ms on emr-worker-6.cluster-196425 (executor 4) (91/100) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 92.0 in stage 0.0 (TID 92) in 62 ms on emr-worker-6.cluster-196425 (executor 4) (92/100) 22/04/28 17:29:29 INFO [dispatcher-event-loop-3] BlockManagerInfo: Added broadcast_0_piece0 in memory on emr-worker-19.cluster-196425:42145 (size: 43.4 KB, free: 21.2 GB) 22/04/28 17:29:29 INFO [task-result-getter-3] TaskSetManager: Finished task 14.0 in stage 0.0 (TID 14) in 2936 ms on emr-worker-6.cluster-196425 (executor 7) (93/100) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 7.0 in stage 0.0 (TID 7) in 2963 ms on emr-worker-6.cluster-196425 (executor 7) (94/100) 22/04/28 17:29:29 INFO [task-result-getter-0] TaskSetManager: Finished task 21.0 in stage 0.0 (TID 21) in 2915 ms on emr-worker-6.cluster-196425 (executor 7) (95/100) 22/04/28 17:29:29 INFO [task-result-getter-2] TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 3006 ms on emr-worker-6.cluster-196425 (executor 7) (96/100) 22/04/28 17:29:30 INFO [task-result-getter-1] TaskSetManager: Finished task 19.0 in stage 0.0 (TID 19) in 3943 ms on emr-worker-19.cluster-196425 (executor 2) (97/100) 22/04/28 17:29:30 INFO [task-result-getter-3] TaskSetManager: Finished task 5.0 in stage 0.0 (TID 5) in 3992 ms on emr-worker-19.cluster-196425 (executor 2) (98/100) 22/04/28 17:29:30 INFO [task-result-getter-0] TaskSetManager: Finished task 26.0 in stage 0.0 (TID 26) in 3922 ms on emr-worker-19.cluster-196425 (executor 2) (99/100) 22/04/28 17:29:30 INFO [task-result-getter-2] TaskSetManager: Finished task 12.0 in stage 0.0 (TID 12) in 3966 ms on emr-worker-19.cluster-196425 (executor 2) (100/100) 22/04/28 17:29:30 INFO [dag-scheduler-event-loop] DAGScheduler: ResultStage 0 (csv at FileBaseReader.scala:86) finished in 4.380 s 22/04/28 17:29:30 INFO [task-result-getter-2] YarnScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool 22/04/28 17:29:30 INFO [main] DAGScheduler: Job 0 finished: csv at FileBaseReader.scala:86, took 4.429510 s 22/04/28 17:29:30 INFO [main] InMemoryFileIndex: It took 4691 ms to list leaf files for 100 paths. 22/04/28 17:29:30 INFO [main] InMemoryFileIndex: Listing leaf files and directories in parallel under 100 paths. The first several paths are: hdfs://hadoop-cluster/home/risk_manager/nebula_data_v2/vt_user/day=2022-04-24/part-00000, hdfs://hadoop-cluster/home/risk_manager/nebula_data_v2/vt_user/day=2022-04-24/part-00001, hdfs://hadoop-cluster/home/risk_manager/nebula_data_v2/vt_user/day=2022-04-24/part-00002, hdfs://hadoop-cluster/home/risk_manager/nebula_data_v2/vt_user/day=2022-04-24/part-00003, hdfs://hadoop-cluster/home/risk_manager/nebula_data_v2/vt_user/day=2022-04-24/part-00004, hdfs://hadoop-cluster/home/risk_manager/nebula_data_v2/vt_user/day=2022-04-24/part-00005, hdfs://hadoop-cluster/home/risk_manager/nebula_data_v2/vt_user/day=2022-04-24/part-00006, hdfs://hadoop-cluster/home/risk_manager/nebula_data_v2/vt_user/day=2022-04-24/part-00007, hdfs://hadoop-cluster/home/risk_manager/nebula_data_v2/vt_user/day=2022-04-24/part-00008, hdfs://hadoop-cluster/home/risk_manager/nebula_data_v2/vt_user/day=2022-04-24/part-00009. 22/04/28 17:29:31 INFO [main] SparkContext: Starting job: csv at FileBaseReader.scala:86 22/04/28 17:29:31 INFO [dag-scheduler-event-loop] DAGScheduler: Got job 1 (csv at FileBaseReader.scala:86) with 100 output partitions 22/04/28 17:29:31 INFO [dag-scheduler-event-loop] DAGScheduler: Final stage: ResultStage 1 (csv at FileBaseReader.scala:86) 22/04/28 17:29:31 INFO [dag-scheduler-event-loop] DAGScheduler: Parents of final stage: List() 22/04/28 17:29:31 INFO [dag-scheduler-event-loop] DAGScheduler: Missing parents: List() 22/04/28 17:29:31 INFO [dag-scheduler-event-loop] DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[5] at csv at FileBaseReader.scala:86), which has no missing parents 22/04/28 17:29:31 INFO [dag-scheduler-event-loop] MemoryStore: Block broadcast_1 stored as values in memory (estimated size 114.5 KB, free 2004.3 MB) 22/04/28 17:29:31 INFO [dag-scheduler-event-loop] MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 43.5 KB, free 2004.3 MB) 22/04/28 17:29:31 INFO [dispatcher-event-loop-1] BlockManagerInfo: Added broadcast_1_piece0 in memory on 172.16.67.122:41394 (size: 43.5 KB, free: 2004.5 MB) 22/04/28 17:29:31 INFO [dag-scheduler-event-loop] SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1163 22/04/28 17:29:31 INFO [dag-scheduler-event-loop] DAGScheduler: Submitting 100 missing tasks from ResultStage 1 (MapPartitionsRDD[5] at csv at FileBaseReader.scala:86) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14)) 22/04/28 17:29:31 INFO [dag-scheduler-event-loop] YarnScheduler: Adding task set 1.0 with 100 tasks 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 0.0 in stage 1.0 (TID 100, emr-worker-6.cluster-196425, executor 6, partition 0, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 1.0 in stage 1.0 (TID 101, emr-worker-6.cluster-196425, executor 7, partition 1, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 2.0 in stage 1.0 (TID 102, emr-worker-8.cluster-196425, executor 1, partition 2, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 3.0 in stage 1.0 (TID 103, emr-worker-19.cluster-196425, executor 2, partition 3, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 4.0 in stage 1.0 (TID 104, emr-worker-9.cluster-196425, executor 3, partition 4, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 5.0 in stage 1.0 (TID 105, emr-worker-6.cluster-196425, executor 4, partition 5, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 6.0 in stage 1.0 (TID 106, emr-worker-19.cluster-196425, executor 5, partition 6, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 7.0 in stage 1.0 (TID 107, emr-worker-6.cluster-196425, executor 6, partition 7, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 8.0 in stage 1.0 (TID 108, emr-worker-6.cluster-196425, executor 7, partition 8, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 9.0 in stage 1.0 (TID 109, emr-worker-8.cluster-196425, executor 1, partition 9, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 10.0 in stage 1.0 (TID 110, emr-worker-19.cluster-196425, executor 2, partition 10, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 11.0 in stage 1.0 (TID 111, emr-worker-9.cluster-196425, executor 3, partition 11, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 12.0 in stage 1.0 (TID 112, emr-worker-6.cluster-196425, executor 4, partition 12, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 13.0 in stage 1.0 (TID 113, emr-worker-19.cluster-196425, executor 5, partition 13, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 14.0 in stage 1.0 (TID 114, emr-worker-6.cluster-196425, executor 6, partition 14, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 15.0 in stage 1.0 (TID 115, emr-worker-6.cluster-196425, executor 7, partition 15, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 16.0 in stage 1.0 (TID 116, emr-worker-8.cluster-196425, executor 1, partition 16, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 17.0 in stage 1.0 (TID 117, emr-worker-19.cluster-196425, executor 2, partition 17, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 18.0 in stage 1.0 (TID 118, emr-worker-9.cluster-196425, executor 3, partition 18, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 19.0 in stage 1.0 (TID 119, emr-worker-6.cluster-196425, executor 4, partition 19, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 20.0 in stage 1.0 (TID 120, emr-worker-19.cluster-196425, executor 5, partition 20, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 21.0 in stage 1.0 (TID 121, emr-worker-6.cluster-196425, executor 6, partition 21, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 22.0 in stage 1.0 (TID 122, emr-worker-6.cluster-196425, executor 7, partition 22, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 23.0 in stage 1.0 (TID 123, emr-worker-8.cluster-196425, executor 1, partition 23, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 24.0 in stage 1.0 (TID 124, emr-worker-19.cluster-196425, executor 2, partition 24, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 25.0 in stage 1.0 (TID 125, emr-worker-9.cluster-196425, executor 3, partition 25, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 26.0 in stage 1.0 (TID 126, emr-worker-6.cluster-196425, executor 4, partition 26, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 27.0 in stage 1.0 (TID 127, emr-worker-19.cluster-196425, executor 5, partition 27, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] BlockManagerInfo: Added broadcast_1_piece0 in memory on emr-worker-6.cluster-196425:37495 (size: 43.5 KB, free: 21.2 GB) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] BlockManagerInfo: Added broadcast_1_piece0 in memory on emr-worker-19.cluster-196425:41635 (size: 43.5 KB, free: 21.2 GB) 22/04/28 17:29:31 INFO [dispatcher-event-loop-2] BlockManagerInfo: Added broadcast_1_piece0 in memory on emr-worker-9.cluster-196425:46825 (size: 43.5 KB, free: 21.2 GB) 22/04/28 17:29:31 INFO [dispatcher-event-loop-2] BlockManagerInfo: Added broadcast_1_piece0 in memory on emr-worker-8.cluster-196425:33263 (size: 43.5 KB, free: 21.2 GB) 22/04/28 17:29:31 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 28.0 in stage 1.0 (TID 128, emr-worker-6.cluster-196425, executor 6, partition 28, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 29.0 in stage 1.0 (TID 129, emr-worker-6.cluster-196425, executor 6, partition 29, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 30.0 in stage 1.0 (TID 130, emr-worker-6.cluster-196425, executor 6, partition 30, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-3] BlockManagerInfo: Added broadcast_1_piece0 in memory on emr-worker-6.cluster-196425:41171 (size: 43.5 KB, free: 21.2 GB) 22/04/28 17:29:31 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 31.0 in stage 1.0 (TID 131, emr-worker-6.cluster-196425, executor 6, partition 31, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [task-result-getter-1] TaskSetManager: Finished task 21.0 in stage 1.0 (TID 121) in 67 ms on emr-worker-6.cluster-196425 (executor 6) (1/100) 22/04/28 17:29:31 INFO [task-result-getter-2] TaskSetManager: Finished task 14.0 in stage 1.0 (TID 114) in 80 ms on emr-worker-6.cluster-196425 (executor 6) (2/100) 22/04/28 17:29:31 INFO [task-result-getter-0] TaskSetManager: Finished task 0.0 in stage 1.0 (TID 100) in 103 ms on emr-worker-6.cluster-196425 (executor 6) (3/100) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 32.0 in stage 1.0 (TID 132, emr-worker-8.cluster-196425, executor 1, partition 32, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 7.0 in stage 1.0 (TID 107) in 95 ms on emr-worker-6.cluster-196425 (executor 6) (4/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 16.0 in stage 1.0 (TID 116) in 81 ms on emr-worker-8.cluster-196425 (executor 1) (5/100) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 33.0 in stage 1.0 (TID 133, emr-worker-19.cluster-196425, executor 5, partition 33, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 34.0 in stage 1.0 (TID 134, emr-worker-19.cluster-196425, executor 5, partition 34, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 35.0 in stage 1.0 (TID 135, emr-worker-19.cluster-196425, executor 5, partition 35, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 36.0 in stage 1.0 (TID 136, emr-worker-19.cluster-196425, executor 5, partition 36, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 37.0 in stage 1.0 (TID 137, emr-worker-8.cluster-196425, executor 1, partition 37, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-1] BlockManagerInfo: Added broadcast_1_piece0 in memory on emr-worker-6.cluster-196425:36207 (size: 43.5 KB, free: 21.2 GB) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 38.0 in stage 1.0 (TID 138, emr-worker-8.cluster-196425, executor 1, partition 38, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 39.0 in stage 1.0 (TID 139, emr-worker-8.cluster-196425, executor 1, partition 39, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [task-result-getter-1] TaskSetManager: Finished task 6.0 in stage 1.0 (TID 106) in 111 ms on emr-worker-19.cluster-196425 (executor 5) (6/100) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 40.0 in stage 1.0 (TID 140, emr-worker-6.cluster-196425, executor 6, partition 40, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 41.0 in stage 1.0 (TID 141, emr-worker-9.cluster-196425, executor 3, partition 41, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 42.0 in stage 1.0 (TID 142, emr-worker-9.cluster-196425, executor 3, partition 42, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-1] BlockManagerInfo: Added broadcast_1_piece0 in memory on emr-worker-19.cluster-196425:42145 (size: 43.5 KB, free: 21.2 GB) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 43.0 in stage 1.0 (TID 143, emr-worker-9.cluster-196425, executor 3, partition 43, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 44.0 in stage 1.0 (TID 144, emr-worker-9.cluster-196425, executor 3, partition 44, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 45.0 in stage 1.0 (TID 145, emr-worker-6.cluster-196425, executor 4, partition 45, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 46.0 in stage 1.0 (TID 146, emr-worker-6.cluster-196425, executor 4, partition 46, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 47.0 in stage 1.0 (TID 147, emr-worker-6.cluster-196425, executor 4, partition 47, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 48.0 in stage 1.0 (TID 148, emr-worker-6.cluster-196425, executor 4, partition 48, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 49.0 in stage 1.0 (TID 149, emr-worker-8.cluster-196425, executor 1, partition 49, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 50.0 in stage 1.0 (TID 150, emr-worker-6.cluster-196425, executor 6, partition 50, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 51.0 in stage 1.0 (TID 151, emr-worker-19.cluster-196425, executor 5, partition 51, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 52.0 in stage 1.0 (TID 152, emr-worker-6.cluster-196425, executor 6, partition 52, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 53.0 in stage 1.0 (TID 153, emr-worker-6.cluster-196425, executor 6, partition 53, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 54.0 in stage 1.0 (TID 154, emr-worker-19.cluster-196425, executor 5, partition 54, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 55.0 in stage 1.0 (TID 155, emr-worker-19.cluster-196425, executor 5, partition 55, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 56.0 in stage 1.0 (TID 156, emr-worker-8.cluster-196425, executor 1, partition 56, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 57.0 in stage 1.0 (TID 157, emr-worker-8.cluster-196425, executor 1, partition 57, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 58.0 in stage 1.0 (TID 158, emr-worker-8.cluster-196425, executor 1, partition 58, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 59.0 in stage 1.0 (TID 159, emr-worker-19.cluster-196425, executor 5, partition 59, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 60.0 in stage 1.0 (TID 160, emr-worker-6.cluster-196425, executor 6, partition 60, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [task-result-getter-2] TaskSetManager: Finished task 13.0 in stage 1.0 (TID 113) in 166 ms on emr-worker-19.cluster-196425 (executor 5) (7/100) 22/04/28 17:29:31 INFO [task-result-getter-2] TaskSetManager: Finished task 2.0 in stage 1.0 (TID 102) in 184 ms on emr-worker-8.cluster-196425 (executor 1) (8/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 23.0 in stage 1.0 (TID 123) in 149 ms on emr-worker-8.cluster-196425 (executor 1) (9/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 28.0 in stage 1.0 (TID 128) in 100 ms on emr-worker-6.cluster-196425 (executor 6) (10/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 18.0 in stage 1.0 (TID 118) in 159 ms on emr-worker-9.cluster-196425 (executor 3) (11/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 4.0 in stage 1.0 (TID 104) in 182 ms on emr-worker-9.cluster-196425 (executor 3) (12/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 11.0 in stage 1.0 (TID 111) in 171 ms on emr-worker-9.cluster-196425 (executor 3) (13/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 25.0 in stage 1.0 (TID 125) in 147 ms on emr-worker-9.cluster-196425 (executor 3) (14/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 12.0 in stage 1.0 (TID 112) in 169 ms on emr-worker-6.cluster-196425 (executor 4) (15/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 26.0 in stage 1.0 (TID 126) in 147 ms on emr-worker-6.cluster-196425 (executor 4) (16/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 5.0 in stage 1.0 (TID 105) in 182 ms on emr-worker-6.cluster-196425 (executor 4) (17/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 19.0 in stage 1.0 (TID 119) in 159 ms on emr-worker-6.cluster-196425 (executor 4) (18/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 32.0 in stage 1.0 (TID 132) in 86 ms on emr-worker-8.cluster-196425 (executor 1) (19/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 29.0 in stage 1.0 (TID 129) in 99 ms on emr-worker-6.cluster-196425 (executor 6) (20/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 33.0 in stage 1.0 (TID 133) in 84 ms on emr-worker-19.cluster-196425 (executor 5) (21/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 30.0 in stage 1.0 (TID 130) in 96 ms on emr-worker-6.cluster-196425 (executor 6) (22/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 31.0 in stage 1.0 (TID 131) in 92 ms on emr-worker-6.cluster-196425 (executor 6) (23/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 34.0 in stage 1.0 (TID 134) in 82 ms on emr-worker-19.cluster-196425 (executor 5) (24/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 36.0 in stage 1.0 (TID 136) in 79 ms on emr-worker-19.cluster-196425 (executor 5) (25/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 38.0 in stage 1.0 (TID 138) in 75 ms on emr-worker-8.cluster-196425 (executor 1) (26/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 37.0 in stage 1.0 (TID 137) in 77 ms on emr-worker-8.cluster-196425 (executor 1) (27/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 39.0 in stage 1.0 (TID 139) in 72 ms on emr-worker-8.cluster-196425 (executor 1) (28/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 35.0 in stage 1.0 (TID 135) in 82 ms on emr-worker-19.cluster-196425 (executor 5) (29/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 40.0 in stage 1.0 (TID 140) in 71 ms on emr-worker-6.cluster-196425 (executor 6) (30/100) 22/04/28 17:29:31 INFO [task-result-getter-0] TaskSetManager: Finished task 27.0 in stage 1.0 (TID 127) in 151 ms on emr-worker-19.cluster-196425 (executor 5) (31/100) 22/04/28 17:29:31 INFO [task-result-getter-2] TaskSetManager: Finished task 9.0 in stage 1.0 (TID 109) in 182 ms on emr-worker-8.cluster-196425 (executor 1) (32/100) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 61.0 in stage 1.0 (TID 161, emr-worker-6.cluster-196425, executor 7, partition 61, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 62.0 in stage 1.0 (TID 162, emr-worker-6.cluster-196425, executor 7, partition 62, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 63.0 in stage 1.0 (TID 163, emr-worker-6.cluster-196425, executor 7, partition 63, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 64.0 in stage 1.0 (TID 164, emr-worker-6.cluster-196425, executor 7, partition 64, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 65.0 in stage 1.0 (TID 165, emr-worker-9.cluster-196425, executor 3, partition 65, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 66.0 in stage 1.0 (TID 166, emr-worker-9.cluster-196425, executor 3, partition 66, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [task-result-getter-1] TaskSetManager: Finished task 20.0 in stage 1.0 (TID 120) in 178 ms on emr-worker-19.cluster-196425 (executor 5) (33/100) 22/04/28 17:29:31 INFO [task-result-getter-1] TaskSetManager: Finished task 1.0 in stage 1.0 (TID 101) in 209 ms on emr-worker-6.cluster-196425 (executor 7) (34/100) 22/04/28 17:29:31 INFO [task-result-getter-1] TaskSetManager: Finished task 42.0 in stage 1.0 (TID 142) in 83 ms on emr-worker-9.cluster-196425 (executor 3) (35/100) 22/04/28 17:29:31 INFO [task-result-getter-1] TaskSetManager: Finished task 41.0 in stage 1.0 (TID 141) in 85 ms on emr-worker-9.cluster-196425 (executor 3) (36/100) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 67.0 in stage 1.0 (TID 167, emr-worker-9.cluster-196425, executor 3, partition 67, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 68.0 in stage 1.0 (TID 168, emr-worker-9.cluster-196425, executor 3, partition 68, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 69.0 in stage 1.0 (TID 169, emr-worker-6.cluster-196425, executor 4, partition 69, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 70.0 in stage 1.0 (TID 170, emr-worker-6.cluster-196425, executor 4, partition 70, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 71.0 in stage 1.0 (TID 171, emr-worker-19.cluster-196425, executor 2, partition 71, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 72.0 in stage 1.0 (TID 172, emr-worker-19.cluster-196425, executor 2, partition 72, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 73.0 in stage 1.0 (TID 173, emr-worker-19.cluster-196425, executor 2, partition 73, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [task-result-getter-2] TaskSetManager: Finished task 8.0 in stage 1.0 (TID 108) in 216 ms on emr-worker-6.cluster-196425 (executor 7) (37/100) 22/04/28 17:29:31 INFO [task-result-getter-2] TaskSetManager: Finished task 44.0 in stage 1.0 (TID 144) in 95 ms on emr-worker-9.cluster-196425 (executor 3) (38/100) 22/04/28 17:29:31 INFO [task-result-getter-2] TaskSetManager: Finished task 46.0 in stage 1.0 (TID 146) in 92 ms on emr-worker-6.cluster-196425 (executor 4) (39/100) 22/04/28 17:29:31 INFO [task-result-getter-0] TaskSetManager: Finished task 22.0 in stage 1.0 (TID 122) in 192 ms on emr-worker-6.cluster-196425 (executor 7) (40/100) 22/04/28 17:29:31 INFO [task-result-getter-0] TaskSetManager: Finished task 10.0 in stage 1.0 (TID 110) in 214 ms on emr-worker-19.cluster-196425 (executor 2) (41/100) 22/04/28 17:29:31 INFO [task-result-getter-0] TaskSetManager: Finished task 3.0 in stage 1.0 (TID 103) in 225 ms on emr-worker-19.cluster-196425 (executor 2) (42/100) 22/04/28 17:29:31 INFO [task-result-getter-0] TaskSetManager: Finished task 24.0 in stage 1.0 (TID 124) in 190 ms on emr-worker-19.cluster-196425 (executor 2) (43/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 15.0 in stage 1.0 (TID 115) in 205 ms on emr-worker-6.cluster-196425 (executor 7) (44/100) 22/04/28 17:29:31 INFO [task-result-getter-1] TaskSetManager: Finished task 43.0 in stage 1.0 (TID 143) in 100 ms on emr-worker-9.cluster-196425 (executor 3) (45/100) 22/04/28 17:29:31 INFO [task-result-getter-2] TaskSetManager: Finished task 45.0 in stage 1.0 (TID 145) in 96 ms on emr-worker-6.cluster-196425 (executor 4) (46/100) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 74.0 in stage 1.0 (TID 174, emr-worker-8.cluster-196425, executor 1, partition 74, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 75.0 in stage 1.0 (TID 175, emr-worker-19.cluster-196425, executor 2, partition 75, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 76.0 in stage 1.0 (TID 176, emr-worker-6.cluster-196425, executor 4, partition 76, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 77.0 in stage 1.0 (TID 177, emr-worker-6.cluster-196425, executor 4, partition 77, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 78.0 in stage 1.0 (TID 178, emr-worker-6.cluster-196425, executor 6, partition 78, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 79.0 in stage 1.0 (TID 179, emr-worker-6.cluster-196425, executor 6, partition 79, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 80.0 in stage 1.0 (TID 180, emr-worker-6.cluster-196425, executor 6, partition 80, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 81.0 in stage 1.0 (TID 181, emr-worker-19.cluster-196425, executor 5, partition 81, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 82.0 in stage 1.0 (TID 182, emr-worker-19.cluster-196425, executor 5, partition 82, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 83.0 in stage 1.0 (TID 183, emr-worker-19.cluster-196425, executor 5, partition 83, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 84.0 in stage 1.0 (TID 184, emr-worker-8.cluster-196425, executor 1, partition 84, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 85.0 in stage 1.0 (TID 185, emr-worker-8.cluster-196425, executor 1, partition 85, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 86.0 in stage 1.0 (TID 186, emr-worker-8.cluster-196425, executor 1, partition 86, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 87.0 in stage 1.0 (TID 187, emr-worker-6.cluster-196425, executor 6, partition 87, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 88.0 in stage 1.0 (TID 188, emr-worker-19.cluster-196425, executor 5, partition 88, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 89.0 in stage 1.0 (TID 189, emr-worker-6.cluster-196425, executor 7, partition 89, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 90.0 in stage 1.0 (TID 190, emr-worker-6.cluster-196425, executor 7, partition 90, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 17.0 in stage 1.0 (TID 117) in 247 ms on emr-worker-19.cluster-196425 (executor 2) (47/100) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 91.0 in stage 1.0 (TID 191, emr-worker-6.cluster-196425, executor 7, partition 91, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [task-result-getter-1] TaskSetManager: Finished task 47.0 in stage 1.0 (TID 147) in 138 ms on emr-worker-6.cluster-196425 (executor 4) (48/100) 22/04/28 17:29:31 INFO [task-result-getter-1] TaskSetManager: Finished task 52.0 in stage 1.0 (TID 152) in 122 ms on emr-worker-6.cluster-196425 (executor 6) (49/100) 22/04/28 17:29:31 INFO [task-result-getter-1] TaskSetManager: Finished task 53.0 in stage 1.0 (TID 153) in 121 ms on emr-worker-6.cluster-196425 (executor 6) (50/100) 22/04/28 17:29:31 INFO [task-result-getter-1] TaskSetManager: Finished task 51.0 in stage 1.0 (TID 151) in 127 ms on emr-worker-19.cluster-196425 (executor 5) (51/100) 22/04/28 17:29:31 INFO [task-result-getter-1] TaskSetManager: Finished task 54.0 in stage 1.0 (TID 154) in 120 ms on emr-worker-19.cluster-196425 (executor 5) (52/100) 22/04/28 17:29:31 INFO [task-result-getter-1] TaskSetManager: Finished task 55.0 in stage 1.0 (TID 155) in 115 ms on emr-worker-19.cluster-196425 (executor 5) (53/100) 22/04/28 17:29:31 INFO [task-result-getter-1] TaskSetManager: Finished task 57.0 in stage 1.0 (TID 157) in 102 ms on emr-worker-8.cluster-196425 (executor 1) (54/100) 22/04/28 17:29:31 INFO [task-result-getter-1] TaskSetManager: Finished task 58.0 in stage 1.0 (TID 158) in 100 ms on emr-worker-8.cluster-196425 (executor 1) (55/100) 22/04/28 17:29:31 INFO [task-result-getter-1] TaskSetManager: Finished task 56.0 in stage 1.0 (TID 156) in 114 ms on emr-worker-8.cluster-196425 (executor 1) (56/100) 22/04/28 17:29:31 INFO [task-result-getter-1] TaskSetManager: Finished task 60.0 in stage 1.0 (TID 160) in 93 ms on emr-worker-6.cluster-196425 (executor 6) (57/100) 22/04/28 17:29:31 INFO [task-result-getter-1] TaskSetManager: Finished task 59.0 in stage 1.0 (TID 159) in 95 ms on emr-worker-19.cluster-196425 (executor 5) (58/100) 22/04/28 17:29:31 INFO [task-result-getter-1] TaskSetManager: Finished task 62.0 in stage 1.0 (TID 162) in 79 ms on emr-worker-6.cluster-196425 (executor 7) (59/100) 22/04/28 17:29:31 INFO [task-result-getter-1] TaskSetManager: Finished task 63.0 in stage 1.0 (TID 163) in 82 ms on emr-worker-6.cluster-196425 (executor 7) (60/100) 22/04/28 17:29:31 INFO [task-result-getter-1] TaskSetManager: Finished task 61.0 in stage 1.0 (TID 161) in 86 ms on emr-worker-6.cluster-196425 (executor 7) (61/100) 22/04/28 17:29:31 INFO [task-result-getter-2] TaskSetManager: Finished task 48.0 in stage 1.0 (TID 148) in 142 ms on emr-worker-6.cluster-196425 (executor 4) (62/100) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 92.0 in stage 1.0 (TID 192, emr-worker-6.cluster-196425, executor 7, partition 92, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 93.0 in stage 1.0 (TID 193, emr-worker-9.cluster-196425, executor 3, partition 93, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 94.0 in stage 1.0 (TID 194, emr-worker-9.cluster-196425, executor 3, partition 94, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 95.0 in stage 1.0 (TID 195, emr-worker-6.cluster-196425, executor 4, partition 95, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [task-result-getter-0] TaskSetManager: Finished task 49.0 in stage 1.0 (TID 149) in 156 ms on emr-worker-8.cluster-196425 (executor 1) (63/100) 22/04/28 17:29:31 INFO [task-result-getter-0] TaskSetManager: Finished task 66.0 in stage 1.0 (TID 166) in 92 ms on emr-worker-9.cluster-196425 (executor 3) (64/100) 22/04/28 17:29:31 INFO [task-result-getter-0] TaskSetManager: Finished task 69.0 in stage 1.0 (TID 169) in 83 ms on emr-worker-6.cluster-196425 (executor 4) (65/100) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 96.0 in stage 1.0 (TID 196, emr-worker-6.cluster-196425, executor 4, partition 96, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 97.0 in stage 1.0 (TID 197, emr-worker-9.cluster-196425, executor 3, partition 97, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 98.0 in stage 1.0 (TID 198, emr-worker-9.cluster-196425, executor 3, partition 98, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [task-result-getter-2] TaskSetManager: Finished task 65.0 in stage 1.0 (TID 165) in 102 ms on emr-worker-9.cluster-196425 (executor 3) (66/100) 22/04/28 17:29:31 INFO [task-result-getter-2] TaskSetManager: Finished task 67.0 in stage 1.0 (TID 167) in 93 ms on emr-worker-9.cluster-196425 (executor 3) (67/100) 22/04/28 17:29:31 INFO [task-result-getter-2] TaskSetManager: Finished task 68.0 in stage 1.0 (TID 168) in 92 ms on emr-worker-9.cluster-196425 (executor 3) (68/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 50.0 in stage 1.0 (TID 150) in 160 ms on emr-worker-6.cluster-196425 (executor 6) (69/100) 22/04/28 17:29:31 INFO [task-result-getter-1] TaskSetManager: Finished task 64.0 in stage 1.0 (TID 164) in 106 ms on emr-worker-6.cluster-196425 (executor 7) (70/100) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 99.0 in stage 1.0 (TID 199, emr-worker-8.cluster-196425, executor 1, partition 99, PROCESS_LOCAL, 7822 bytes) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 76.0 in stage 1.0 (TID 176) in 79 ms on emr-worker-6.cluster-196425 (executor 4) (71/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 72.0 in stage 1.0 (TID 172) in 90 ms on emr-worker-19.cluster-196425 (executor 2) (72/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 77.0 in stage 1.0 (TID 177) in 73 ms on emr-worker-6.cluster-196425 (executor 4) (73/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 75.0 in stage 1.0 (TID 175) in 81 ms on emr-worker-19.cluster-196425 (executor 2) (74/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 71.0 in stage 1.0 (TID 171) in 93 ms on emr-worker-19.cluster-196425 (executor 2) (75/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 83.0 in stage 1.0 (TID 183) in 60 ms on emr-worker-19.cluster-196425 (executor 5) (76/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 82.0 in stage 1.0 (TID 182) in 62 ms on emr-worker-19.cluster-196425 (executor 5) (77/100) 22/04/28 17:29:31 INFO [task-result-getter-2] TaskSetManager: Finished task 74.0 in stage 1.0 (TID 174) in 85 ms on emr-worker-8.cluster-196425 (executor 1) (78/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 81.0 in stage 1.0 (TID 181) in 68 ms on emr-worker-19.cluster-196425 (executor 5) (79/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 84.0 in stage 1.0 (TID 184) in 62 ms on emr-worker-8.cluster-196425 (executor 1) (80/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 85.0 in stage 1.0 (TID 185) in 60 ms on emr-worker-8.cluster-196425 (executor 1) (81/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 86.0 in stage 1.0 (TID 186) in 56 ms on emr-worker-8.cluster-196425 (executor 1) (82/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 89.0 in stage 1.0 (TID 189) in 49 ms on emr-worker-6.cluster-196425 (executor 7) (83/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 88.0 in stage 1.0 (TID 188) in 51 ms on emr-worker-19.cluster-196425 (executor 5) (84/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 90.0 in stage 1.0 (TID 190) in 48 ms on emr-worker-6.cluster-196425 (executor 7) (85/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 91.0 in stage 1.0 (TID 191) in 46 ms on emr-worker-6.cluster-196425 (executor 7) (86/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 78.0 in stage 1.0 (TID 178) in 77 ms on emr-worker-6.cluster-196425 (executor 6) (87/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 79.0 in stage 1.0 (TID 179) in 74 ms on emr-worker-6.cluster-196425 (executor 6) (88/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 80.0 in stage 1.0 (TID 180) in 73 ms on emr-worker-6.cluster-196425 (executor 6) (89/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 92.0 in stage 1.0 (TID 192) in 29 ms on emr-worker-6.cluster-196425 (executor 7) (90/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 87.0 in stage 1.0 (TID 187) in 56 ms on emr-worker-6.cluster-196425 (executor 6) (91/100) 22/04/28 17:29:31 INFO [task-result-getter-1] TaskSetManager: Finished task 73.0 in stage 1.0 (TID 173) in 96 ms on emr-worker-19.cluster-196425 (executor 2) (92/100) 22/04/28 17:29:31 INFO [task-result-getter-0] TaskSetManager: Finished task 70.0 in stage 1.0 (TID 170) in 101 ms on emr-worker-6.cluster-196425 (executor 4) (93/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 93.0 in stage 1.0 (TID 193) in 31 ms on emr-worker-9.cluster-196425 (executor 3) (94/100) 22/04/28 17:29:31 INFO [task-result-getter-1] TaskSetManager: Finished task 94.0 in stage 1.0 (TID 194) in 31 ms on emr-worker-9.cluster-196425 (executor 3) (95/100) 22/04/28 17:29:31 INFO [task-result-getter-2] TaskSetManager: Finished task 96.0 in stage 1.0 (TID 196) in 27 ms on emr-worker-6.cluster-196425 (executor 4) (96/100) 22/04/28 17:29:31 INFO [task-result-getter-2] TaskSetManager: Finished task 95.0 in stage 1.0 (TID 195) in 31 ms on emr-worker-6.cluster-196425 (executor 4) (97/100) 22/04/28 17:29:31 INFO [task-result-getter-3] TaskSetManager: Finished task 99.0 in stage 1.0 (TID 199) in 20 ms on emr-worker-8.cluster-196425 (executor 1) (98/100) 22/04/28 17:29:31 INFO [task-result-getter-1] TaskSetManager: Finished task 97.0 in stage 1.0 (TID 197) in 30 ms on emr-worker-9.cluster-196425 (executor 3) (99/100) 22/04/28 17:29:31 INFO [task-result-getter-2] TaskSetManager: Finished task 98.0 in stage 1.0 (TID 198) in 38 ms on emr-worker-9.cluster-196425 (executor 3) (100/100) 22/04/28 17:29:31 INFO [task-result-getter-2] YarnScheduler: Removed TaskSet 1.0, whose tasks have all completed, from pool 22/04/28 17:29:31 INFO [dag-scheduler-event-loop] DAGScheduler: ResultStage 1 (csv at FileBaseReader.scala:86) finished in 0.367 s 22/04/28 17:29:31 INFO [main] DAGScheduler: Job 1 finished: csv at FileBaseReader.scala:86, took 0.382394 s 22/04/28 17:29:31 INFO [main] InMemoryFileIndex: It took 445 ms to list leaf files for 100 paths. 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 47 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 40 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 42 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 1 22/04/28 17:29:31 INFO [dispatcher-event-loop-3] BlockManagerInfo: Removed broadcast_0_piece0 on 172.16.67.122:41394 in memory (size: 43.4 KB, free: 2004.6 MB) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] BlockManagerInfo: Removed broadcast_0_piece0 on emr-worker-8.cluster-196425:33263 in memory (size: 43.4 KB, free: 21.2 GB) 22/04/28 17:29:31 INFO [dispatcher-event-loop-1] BlockManagerInfo: Removed broadcast_0_piece0 on emr-worker-6.cluster-196425:37495 in memory (size: 43.4 KB, free: 21.2 GB) 22/04/28 17:29:31 INFO [dispatcher-event-loop-1] BlockManagerInfo: Removed broadcast_0_piece0 on emr-worker-6.cluster-196425:36207 in memory (size: 43.4 KB, free: 21.2 GB) 22/04/28 17:29:31 INFO [dispatcher-event-loop-1] BlockManagerInfo: Removed broadcast_0_piece0 on emr-worker-6.cluster-196425:41171 in memory (size: 43.4 KB, free: 21.2 GB) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] BlockManagerInfo: Removed broadcast_0_piece0 on emr-worker-19.cluster-196425:42145 in memory (size: 43.4 KB, free: 21.2 GB) 22/04/28 17:29:31 INFO [dispatcher-event-loop-1] BlockManagerInfo: Removed broadcast_0_piece0 on emr-worker-9.cluster-196425:46825 in memory (size: 43.4 KB, free: 21.2 GB) 22/04/28 17:29:31 INFO [dispatcher-event-loop-2] BlockManagerInfo: Removed broadcast_0_piece0 on emr-worker-19.cluster-196425:41635 in memory (size: 43.4 KB, free: 21.2 GB) 22/04/28 17:29:31 INFO [dispatcher-event-loop-1] BlockManagerInfo: Removed broadcast_1_piece0 on 172.16.67.122:41394 in memory (size: 43.5 KB, free: 2004.6 MB) 22/04/28 17:29:31 INFO [dispatcher-event-loop-1] BlockManagerInfo: Removed broadcast_1_piece0 on emr-worker-6.cluster-196425:36207 in memory (size: 43.5 KB, free: 21.2 GB) 22/04/28 17:29:31 INFO [dispatcher-event-loop-1] BlockManagerInfo: Removed broadcast_1_piece0 on emr-worker-6.cluster-196425:41171 in memory (size: 43.5 KB, free: 21.2 GB) 22/04/28 17:29:31 INFO [dispatcher-event-loop-3] BlockManagerInfo: Removed broadcast_1_piece0 on emr-worker-6.cluster-196425:37495 in memory (size: 43.5 KB, free: 21.2 GB) 22/04/28 17:29:31 INFO [dispatcher-event-loop-2] BlockManagerInfo: Removed broadcast_1_piece0 on emr-worker-19.cluster-196425:41635 in memory (size: 43.5 KB, free: 21.2 GB) 22/04/28 17:29:31 INFO [dispatcher-event-loop-0] BlockManagerInfo: Removed broadcast_1_piece0 on emr-worker-8.cluster-196425:33263 in memory (size: 43.5 KB, free: 21.2 GB) 22/04/28 17:29:31 INFO [dispatcher-event-loop-1] BlockManagerInfo: Removed broadcast_1_piece0 on emr-worker-19.cluster-196425:42145 in memory (size: 43.5 KB, free: 21.2 GB) 22/04/28 17:29:31 INFO [dispatcher-event-loop-1] BlockManagerInfo: Removed broadcast_1_piece0 on emr-worker-9.cluster-196425:46825 in memory (size: 43.5 KB, free: 21.2 GB) 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 19 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 45 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 27 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 14 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 8 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 21 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 11 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 9 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 30 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 48 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 0 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 49 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 25 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 3 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 16 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 29 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 6 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 31 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 20 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 17 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 22 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 46 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 24 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 28 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 7 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 15 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 5 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 13 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 26 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 41 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 35 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 36 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 44 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 2 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 18 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 10 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 4 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 34 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 23 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 12 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 38 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 37 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 33 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 32 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 43 22/04/28 17:29:31 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 39 22/04/28 17:29:32 INFO [main] FileSourceStrategy: Pruning directories with: 22/04/28 17:29:32 INFO [main] FileSourceStrategy: Post-Scan Filters: (length(trim(value#0, None)) > 0) 22/04/28 17:29:32 INFO [main] FileSourceStrategy: Output Data Schema: struct 22/04/28 17:29:32 INFO [main] FileSourceScanExec: Pushed Filters: 22/04/28 17:29:32 INFO [main] CodeGenerator: Code generated in 192.593036 ms 22/04/28 17:29:32 INFO [main] CodeGenerator: Code generated in 18.495732 ms 22/04/28 17:29:32 INFO [main] MemoryStore: Block broadcast_2 stored as values in memory (estimated size 476.6 KB, free 2004.1 MB) 22/04/28 17:29:32 INFO [main] MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 41.9 KB, free 2004.1 MB) 22/04/28 17:29:32 INFO [dispatcher-event-loop-0] BlockManagerInfo: Added broadcast_2_piece0 in memory on 172.16.67.122:41394 (size: 41.9 KB, free: 2004.6 MB) 22/04/28 17:29:32 INFO [main] SparkContext: Created broadcast 2 from csv at FileBaseReader.scala:86 22/04/28 17:29:32 INFO [main] FileSourceScanExec: ---1--with dpp size:1 22/04/28 17:29:32 INFO [main] FileSourceScanExec: s---2----with out dppfiles:1 22/04/28 17:29:32 INFO [main] FileSourceScanExec: Planning scan with bin packing, max size: 134217728 bytes, open cost is considered as scanning 4194304 bytes, using packing strategy: efficiency-oriented. 22/04/28 17:29:32 INFO [main] GPLNativeCodeLoader: Loaded native gpl library from the embedded binaries 22/04/28 17:29:32 INFO [main] LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 97184efe294f64a51a4c5c172cbc22146103da53] 22/04/28 17:29:33 INFO [main] SparkContext: Starting job: csv at FileBaseReader.scala:86 22/04/28 17:29:33 INFO [dag-scheduler-event-loop] DAGScheduler: Got job 2 (csv at FileBaseReader.scala:86) with 1 output partitions 22/04/28 17:29:33 INFO [dag-scheduler-event-loop] DAGScheduler: Final stage: ResultStage 2 (csv at FileBaseReader.scala:86) 22/04/28 17:29:33 INFO [dag-scheduler-event-loop] DAGScheduler: Parents of final stage: List() 22/04/28 17:29:33 INFO [dag-scheduler-event-loop] DAGScheduler: Missing parents: List() 22/04/28 17:29:33 INFO [dag-scheduler-event-loop] DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[9] at csv at FileBaseReader.scala:86), which has no missing parents 22/04/28 17:29:33 INFO [dag-scheduler-event-loop] MemoryStore: Block broadcast_3 stored as values in memory (estimated size 8.9 KB, free 2004.1 MB) 22/04/28 17:29:33 INFO [dag-scheduler-event-loop] MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 4.6 KB, free 2004.1 MB) 22/04/28 17:29:33 INFO [dispatcher-event-loop-2] BlockManagerInfo: Added broadcast_3_piece0 in memory on 172.16.67.122:41394 (size: 4.6 KB, free: 2004.6 MB) 22/04/28 17:29:33 INFO [dag-scheduler-event-loop] SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1163 22/04/28 17:29:33 INFO [dag-scheduler-event-loop] DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (MapPartitionsRDD[9] at csv at FileBaseReader.scala:86) (first 15 tasks are for partitions Vector(0)) 22/04/28 17:29:33 INFO [dag-scheduler-event-loop] YarnScheduler: Adding task set 2.0 with 1 tasks 22/04/28 17:29:33 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 0.0 in stage 2.0 (TID 200, emr-worker-6.cluster-196425, executor 6, partition 0, RACK_LOCAL, 8429 bytes) 22/04/28 17:29:33 INFO [dispatcher-event-loop-2] BlockManagerInfo: Added broadcast_3_piece0 in memory on emr-worker-6.cluster-196425:37495 (size: 4.6 KB, free: 21.2 GB) 22/04/28 17:29:33 INFO [dispatcher-event-loop-3] BlockManagerInfo: Added broadcast_2_piece0 in memory on emr-worker-6.cluster-196425:37495 (size: 41.9 KB, free: 21.2 GB) 22/04/28 17:29:33 INFO [task-result-getter-0] TaskSetManager: Finished task 0.0 in stage 2.0 (TID 200) in 535 ms on emr-worker-6.cluster-196425 (executor 6) (1/1) 22/04/28 17:29:33 INFO [task-result-getter-0] YarnScheduler: Removed TaskSet 2.0, whose tasks have all completed, from pool 22/04/28 17:29:33 INFO [dag-scheduler-event-loop] DAGScheduler: ResultStage 2 (csv at FileBaseReader.scala:86) finished in 0.567 s 22/04/28 17:29:33 INFO [main] DAGScheduler: Job 2 finished: csv at FileBaseReader.scala:86, took 0.574365 s 22/04/28 17:29:33 INFO [main] SparkSQLQueryListener: head is called 22/04/28 17:29:33 INFO [main] SparkSQLQueryListener: Spark user root executed on 1651138173605 with spark sql successfully. 22/04/28 17:29:33 INFO [main] FileSourceStrategy: Pruning directories with: 22/04/28 17:29:33 INFO [main] FileSourceStrategy: Post-Scan Filters: 22/04/28 17:29:33 INFO [main] FileSourceStrategy: Output Data Schema: struct 22/04/28 17:29:33 INFO [main] FileSourceScanExec: Pushed Filters: 22/04/28 17:29:33 INFO [main] CodeGenerator: Code generated in 8.050851 ms 22/04/28 17:29:33 INFO [main] MemoryStore: Block broadcast_4 stored as values in memory (estimated size 476.6 KB, free 2003.6 MB) 22/04/28 17:29:33 INFO [main] MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 41.9 KB, free 2003.6 MB) 22/04/28 17:29:33 INFO [dispatcher-event-loop-0] BlockManagerInfo: Added broadcast_4_piece0 in memory on 172.16.67.122:41394 (size: 41.9 KB, free: 2004.5 MB) 22/04/28 17:29:33 INFO [main] SparkContext: Created broadcast 4 from csv at FileBaseReader.scala:86 22/04/28 17:29:33 INFO [main] FileSourceScanExec: ---1--with dpp size:1 22/04/28 17:29:33 INFO [main] FileSourceScanExec: s---2----with out dppfiles:1 22/04/28 17:29:33 INFO [main] FileSourceScanExec: Planning scan with bin packing, max size: 134217728 bytes, open cost is considered as scanning 4194304 bytes, using packing strategy: efficiency-oriented. 22/04/28 17:29:33 INFO [main] FileSourceStrategy: Pruning directories with: 22/04/28 17:29:33 INFO [main] FileSourceStrategy: Post-Scan Filters: 22/04/28 17:29:33 INFO [main] FileSourceStrategy: Output Data Schema: struct<_c0: string, _c1: string, _c2: string, _c3: string, _c4: string ... 1 more field> 22/04/28 17:29:33 INFO [main] FileSourceScanExec: Pushed Filters: 22/04/28 17:29:34 INFO [adaptive-query-stage-1] MemoryStore: Block broadcast_5 stored as values in memory (estimated size 476.6 KB, free 2003.1 MB) 22/04/28 17:29:34 INFO [adaptive-query-stage-1] MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 41.9 KB, free 2003.1 MB) 22/04/28 17:29:34 INFO [dispatcher-event-loop-1] BlockManagerInfo: Added broadcast_5_piece0 in memory on 172.16.67.122:41394 (size: 41.9 KB, free: 2004.5 MB) 22/04/28 17:29:34 INFO [adaptive-query-stage-1] SparkContext: Created broadcast 5 from run at ThreadPoolExecutor.java:1149 22/04/28 17:29:34 INFO [adaptive-query-stage-1] FileSourceScanExec: ---1--with dpp size:1 22/04/28 17:29:34 INFO [adaptive-query-stage-1] FileSourceScanExec: s---2----with out dppfiles:1 22/04/28 17:29:34 INFO [adaptive-query-stage-1] FileSourceScanExec: Planning scan with bin packing, max size: 134217728 bytes, open cost is considered as scanning 4194304 bytes, using packing strategy: efficiency-oriented. 22/04/28 17:29:34 INFO [dag-scheduler-event-loop] DAGScheduler: Registering RDD 19 (run at ThreadPoolExecutor.java:1149) as input to shuffle 0 22/04/28 17:29:34 INFO [dag-scheduler-event-loop] DAGScheduler: Got map stage job 3 (run at ThreadPoolExecutor.java:1149) with 50 output partitions 22/04/28 17:29:34 INFO [dag-scheduler-event-loop] DAGScheduler: Final stage: ShuffleMapStage 3 (run at ThreadPoolExecutor.java:1149) 22/04/28 17:29:34 INFO [dag-scheduler-event-loop] DAGScheduler: Parents of final stage: List() 22/04/28 17:29:34 INFO [dag-scheduler-event-loop] DAGScheduler: Missing parents: List() 22/04/28 17:29:34 INFO [dag-scheduler-event-loop] DAGScheduler: Submitting ShuffleMapStage 3 (MapPartitionsRDD[19] at run at ThreadPoolExecutor.java:1149), which has no missing parents 22/04/28 17:29:34 INFO [dag-scheduler-event-loop] MemoryStore: Block broadcast_6 stored as values in memory (estimated size 12.4 KB, free 2003.1 MB) 22/04/28 17:29:34 INFO [dag-scheduler-event-loop] MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 6.6 KB, free 2003.0 MB) 22/04/28 17:29:34 INFO [dispatcher-event-loop-3] BlockManagerInfo: Added broadcast_6_piece0 in memory on 172.16.67.122:41394 (size: 6.6 KB, free: 2004.5 MB) 22/04/28 17:29:34 INFO [dag-scheduler-event-loop] SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:1163 22/04/28 17:29:34 INFO [dag-scheduler-event-loop] DAGScheduler: Submitting 50 missing tasks from ShuffleMapStage 3 (MapPartitionsRDD[19] at run at ThreadPoolExecutor.java:1149) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14)) 22/04/28 17:29:34 INFO [dag-scheduler-event-loop] YarnScheduler: Adding task set 3.0 with 50 tasks 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 0.0 in stage 3.0 (TID 201, emr-worker-6.cluster-196425, executor 7, partition 0, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 1.0 in stage 3.0 (TID 202, emr-worker-8.cluster-196425, executor 1, partition 1, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 2.0 in stage 3.0 (TID 203, emr-worker-6.cluster-196425, executor 4, partition 2, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 3.0 in stage 3.0 (TID 204, emr-worker-19.cluster-196425, executor 5, partition 3, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 4.0 in stage 3.0 (TID 205, emr-worker-6.cluster-196425, executor 6, partition 4, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 5.0 in stage 3.0 (TID 206, emr-worker-19.cluster-196425, executor 2, partition 5, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 6.0 in stage 3.0 (TID 207, emr-worker-9.cluster-196425, executor 3, partition 6, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 7.0 in stage 3.0 (TID 208, emr-worker-6.cluster-196425, executor 7, partition 7, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 8.0 in stage 3.0 (TID 209, emr-worker-8.cluster-196425, executor 1, partition 8, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 9.0 in stage 3.0 (TID 210, emr-worker-6.cluster-196425, executor 4, partition 9, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 10.0 in stage 3.0 (TID 211, emr-worker-19.cluster-196425, executor 5, partition 10, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 11.0 in stage 3.0 (TID 212, emr-worker-6.cluster-196425, executor 6, partition 11, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 12.0 in stage 3.0 (TID 213, emr-worker-19.cluster-196425, executor 2, partition 12, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 13.0 in stage 3.0 (TID 214, emr-worker-9.cluster-196425, executor 3, partition 13, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 14.0 in stage 3.0 (TID 215, emr-worker-6.cluster-196425, executor 7, partition 14, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 15.0 in stage 3.0 (TID 216, emr-worker-8.cluster-196425, executor 1, partition 15, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 16.0 in stage 3.0 (TID 217, emr-worker-6.cluster-196425, executor 4, partition 16, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 17.0 in stage 3.0 (TID 218, emr-worker-19.cluster-196425, executor 5, partition 17, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 18.0 in stage 3.0 (TID 219, emr-worker-6.cluster-196425, executor 6, partition 18, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 19.0 in stage 3.0 (TID 220, emr-worker-19.cluster-196425, executor 2, partition 19, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 20.0 in stage 3.0 (TID 221, emr-worker-9.cluster-196425, executor 3, partition 20, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 21.0 in stage 3.0 (TID 222, emr-worker-6.cluster-196425, executor 7, partition 21, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 22.0 in stage 3.0 (TID 223, emr-worker-8.cluster-196425, executor 1, partition 22, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 23.0 in stage 3.0 (TID 224, emr-worker-6.cluster-196425, executor 4, partition 23, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 24.0 in stage 3.0 (TID 225, emr-worker-19.cluster-196425, executor 5, partition 24, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 25.0 in stage 3.0 (TID 226, emr-worker-6.cluster-196425, executor 6, partition 25, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 26.0 in stage 3.0 (TID 227, emr-worker-19.cluster-196425, executor 2, partition 26, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 27.0 in stage 3.0 (TID 228, emr-worker-9.cluster-196425, executor 3, partition 27, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] BlockManagerInfo: Added broadcast_6_piece0 in memory on emr-worker-6.cluster-196425:41171 (size: 6.6 KB, free: 21.2 GB) 22/04/28 17:29:34 INFO [dispatcher-event-loop-3] BlockManagerInfo: Added broadcast_6_piece0 in memory on emr-worker-6.cluster-196425:36207 (size: 6.6 KB, free: 21.2 GB) 22/04/28 17:29:34 INFO [dispatcher-event-loop-0] BlockManagerInfo: Added broadcast_6_piece0 in memory on emr-worker-9.cluster-196425:46825 (size: 6.6 KB, free: 21.2 GB) 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] BlockManagerInfo: Added broadcast_6_piece0 in memory on emr-worker-19.cluster-196425:42145 (size: 6.6 KB, free: 21.2 GB) 22/04/28 17:29:34 INFO [dispatcher-event-loop-1] BlockManagerInfo: Added broadcast_6_piece0 in memory on emr-worker-6.cluster-196425:37495 (size: 6.6 KB, free: 21.2 GB) 22/04/28 17:29:34 INFO [dispatcher-event-loop-3] BlockManagerInfo: Added broadcast_6_piece0 in memory on emr-worker-19.cluster-196425:41635 (size: 6.6 KB, free: 21.2 GB) 22/04/28 17:29:34 INFO [dispatcher-event-loop-0] BlockManagerInfo: Added broadcast_6_piece0 in memory on emr-worker-8.cluster-196425:33263 (size: 6.6 KB, free: 21.2 GB) 22/04/28 17:29:34 INFO [dispatcher-event-loop-1] BlockManagerInfo: Added broadcast_5_piece0 in memory on emr-worker-6.cluster-196425:37495 (size: 41.9 KB, free: 21.2 GB) 22/04/28 17:29:34 INFO [dispatcher-event-loop-0] BlockManagerInfo: Added broadcast_5_piece0 in memory on emr-worker-8.cluster-196425:33263 (size: 41.9 KB, free: 21.2 GB) 22/04/28 17:29:34 INFO [dispatcher-event-loop-3] BlockManagerInfo: Added broadcast_5_piece0 in memory on emr-worker-6.cluster-196425:36207 (size: 41.9 KB, free: 21.2 GB) 22/04/28 17:29:34 INFO [dispatcher-event-loop-3] BlockManagerInfo: Added broadcast_5_piece0 in memory on emr-worker-9.cluster-196425:46825 (size: 41.9 KB, free: 21.2 GB) 22/04/28 17:29:34 INFO [dispatcher-event-loop-3] BlockManagerInfo: Added broadcast_5_piece0 in memory on emr-worker-19.cluster-196425:42145 (size: 41.9 KB, free: 21.2 GB) 22/04/28 17:29:34 INFO [dispatcher-event-loop-2] BlockManagerInfo: Added broadcast_5_piece0 in memory on emr-worker-6.cluster-196425:41171 (size: 41.9 KB, free: 21.2 GB) 22/04/28 17:29:34 INFO [dispatcher-event-loop-1] BlockManagerInfo: Added broadcast_5_piece0 in memory on emr-worker-19.cluster-196425:41635 (size: 41.9 KB, free: 21.2 GB) 22/04/28 17:29:43 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 28.0 in stage 3.0 (TID 229, emr-worker-8.cluster-196425, executor 1, partition 28, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:43 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 29.0 in stage 3.0 (TID 230, emr-worker-8.cluster-196425, executor 1, partition 29, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:43 INFO [task-result-getter-3] TaskSetManager: Finished task 22.0 in stage 3.0 (TID 223) in 9558 ms on emr-worker-8.cluster-196425 (executor 1) (1/50) 22/04/28 17:29:43 INFO [task-result-getter-1] TaskSetManager: Finished task 15.0 in stage 3.0 (TID 216) in 9559 ms on emr-worker-8.cluster-196425 (executor 1) (2/50) 22/04/28 17:29:44 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 30.0 in stage 3.0 (TID 231, emr-worker-8.cluster-196425, executor 1, partition 30, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:44 INFO [task-result-getter-2] TaskSetManager: Finished task 8.0 in stage 3.0 (TID 209) in 9934 ms on emr-worker-8.cluster-196425 (executor 1) (3/50) 22/04/28 17:29:44 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 31.0 in stage 3.0 (TID 232, emr-worker-6.cluster-196425, executor 6, partition 31, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:44 INFO [task-result-getter-0] TaskSetManager: Finished task 25.0 in stage 3.0 (TID 226) in 10053 ms on emr-worker-6.cluster-196425 (executor 6) (4/50) 22/04/28 17:29:44 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 32.0 in stage 3.0 (TID 233, emr-worker-8.cluster-196425, executor 1, partition 32, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:44 INFO [task-result-getter-3] TaskSetManager: Finished task 1.0 in stage 3.0 (TID 202) in 10064 ms on emr-worker-8.cluster-196425 (executor 1) (5/50) 22/04/28 17:29:44 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 33.0 in stage 3.0 (TID 234, emr-worker-6.cluster-196425, executor 6, partition 33, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:44 INFO [task-result-getter-1] TaskSetManager: Finished task 4.0 in stage 3.0 (TID 205) in 10155 ms on emr-worker-6.cluster-196425 (executor 6) (6/50) 22/04/28 17:29:44 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 34.0 in stage 3.0 (TID 235, emr-worker-6.cluster-196425, executor 6, partition 34, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:44 INFO [task-result-getter-2] TaskSetManager: Finished task 18.0 in stage 3.0 (TID 219) in 10229 ms on emr-worker-6.cluster-196425 (executor 6) (7/50) 22/04/28 17:29:44 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 35.0 in stage 3.0 (TID 236, emr-worker-6.cluster-196425, executor 4, partition 35, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:44 INFO [task-result-getter-0] TaskSetManager: Finished task 23.0 in stage 3.0 (TID 224) in 10257 ms on emr-worker-6.cluster-196425 (executor 4) (8/50) 22/04/28 17:29:44 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 36.0 in stage 3.0 (TID 237, emr-worker-19.cluster-196425, executor 2, partition 36, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:44 INFO [task-result-getter-3] TaskSetManager: Finished task 26.0 in stage 3.0 (TID 227) in 10552 ms on emr-worker-19.cluster-196425 (executor 2) (9/50) 22/04/28 17:29:44 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 37.0 in stage 3.0 (TID 238, emr-worker-19.cluster-196425, executor 2, partition 37, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:44 INFO [task-result-getter-1] TaskSetManager: Finished task 5.0 in stage 3.0 (TID 206) in 10566 ms on emr-worker-19.cluster-196425 (executor 2) (10/50) 22/04/28 17:29:44 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 38.0 in stage 3.0 (TID 239, emr-worker-6.cluster-196425, executor 7, partition 38, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:44 INFO [task-result-getter-2] TaskSetManager: Finished task 0.0 in stage 3.0 (TID 201) in 10610 ms on emr-worker-6.cluster-196425 (executor 7) (11/50) 22/04/28 17:29:44 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 39.0 in stage 3.0 (TID 240, emr-worker-19.cluster-196425, executor 2, partition 39, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:44 INFO [task-result-getter-0] TaskSetManager: Finished task 19.0 in stage 3.0 (TID 220) in 10609 ms on emr-worker-19.cluster-196425 (executor 2) (12/50) 22/04/28 17:29:44 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 40.0 in stage 3.0 (TID 241, emr-worker-6.cluster-196425, executor 7, partition 40, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:44 INFO [task-result-getter-3] TaskSetManager: Finished task 7.0 in stage 3.0 (TID 208) in 10647 ms on emr-worker-6.cluster-196425 (executor 7) (13/50) 22/04/28 17:29:44 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 41.0 in stage 3.0 (TID 242, emr-worker-19.cluster-196425, executor 2, partition 41, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:44 INFO [task-result-getter-1] TaskSetManager: Finished task 12.0 in stage 3.0 (TID 213) in 10725 ms on emr-worker-19.cluster-196425 (executor 2) (14/50) 22/04/28 17:29:44 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 42.0 in stage 3.0 (TID 243, emr-worker-6.cluster-196425, executor 7, partition 42, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:44 INFO [task-result-getter-2] TaskSetManager: Finished task 21.0 in stage 3.0 (TID 222) in 10733 ms on emr-worker-6.cluster-196425 (executor 7) (15/50) 22/04/28 17:29:44 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 43.0 in stage 3.0 (TID 244, emr-worker-19.cluster-196425, executor 5, partition 43, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:44 INFO [task-result-getter-0] TaskSetManager: Finished task 24.0 in stage 3.0 (TID 225) in 10744 ms on emr-worker-19.cluster-196425 (executor 5) (16/50) 22/04/28 17:29:44 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 44.0 in stage 3.0 (TID 245, emr-worker-6.cluster-196425, executor 7, partition 44, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:44 INFO [task-result-getter-3] TaskSetManager: Finished task 14.0 in stage 3.0 (TID 215) in 10797 ms on emr-worker-6.cluster-196425 (executor 7) (17/50) 22/04/28 17:29:45 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 45.0 in stage 3.0 (TID 246, emr-worker-6.cluster-196425, executor 4, partition 45, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:45 INFO [task-result-getter-1] TaskSetManager: Finished task 16.0 in stage 3.0 (TID 217) in 10968 ms on emr-worker-6.cluster-196425 (executor 4) (18/50) 22/04/28 17:29:45 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 46.0 in stage 3.0 (TID 247, emr-worker-6.cluster-196425, executor 4, partition 46, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:45 INFO [task-result-getter-2] TaskSetManager: Finished task 2.0 in stage 3.0 (TID 203) in 11114 ms on emr-worker-6.cluster-196425 (executor 4) (19/50) 22/04/28 17:29:45 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 47.0 in stage 3.0 (TID 248, emr-worker-19.cluster-196425, executor 5, partition 47, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:45 INFO [task-result-getter-0] TaskSetManager: Finished task 17.0 in stage 3.0 (TID 218) in 11179 ms on emr-worker-19.cluster-196425 (executor 5) (20/50) 22/04/28 17:29:45 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 48.0 in stage 3.0 (TID 249, emr-worker-6.cluster-196425, executor 6, partition 48, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:45 INFO [task-result-getter-3] TaskSetManager: Finished task 11.0 in stage 3.0 (TID 212) in 11211 ms on emr-worker-6.cluster-196425 (executor 6) (21/50) 22/04/28 17:29:45 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 49.0 in stage 3.0 (TID 250, emr-worker-6.cluster-196425, executor 4, partition 49, RACK_LOCAL, 8418 bytes) 22/04/28 17:29:45 INFO [task-result-getter-1] TaskSetManager: Finished task 9.0 in stage 3.0 (TID 210) in 11215 ms on emr-worker-6.cluster-196425 (executor 4) (22/50) 22/04/28 17:29:45 INFO [task-result-getter-2] TaskSetManager: Finished task 3.0 in stage 3.0 (TID 204) in 11245 ms on emr-worker-19.cluster-196425 (executor 5) (23/50) 22/04/28 17:29:45 INFO [task-result-getter-0] TaskSetManager: Finished task 10.0 in stage 3.0 (TID 211) in 11303 ms on emr-worker-19.cluster-196425 (executor 5) (24/50) 22/04/28 17:29:46 INFO [task-result-getter-3] TaskSetManager: Finished task 13.0 in stage 3.0 (TID 214) in 12748 ms on emr-worker-9.cluster-196425 (executor 3) (25/50) 22/04/28 17:29:46 INFO [task-result-getter-1] TaskSetManager: Finished task 27.0 in stage 3.0 (TID 228) in 12873 ms on emr-worker-9.cluster-196425 (executor 3) (26/50) 22/04/28 17:29:47 INFO [task-result-getter-2] TaskSetManager: Finished task 20.0 in stage 3.0 (TID 221) in 12895 ms on emr-worker-9.cluster-196425 (executor 3) (27/50) 22/04/28 17:29:47 INFO [task-result-getter-0] TaskSetManager: Finished task 6.0 in stage 3.0 (TID 207) in 12948 ms on emr-worker-9.cluster-196425 (executor 3) (28/50) 22/04/28 17:29:51 INFO [task-result-getter-3] TaskSetManager: Finished task 29.0 in stage 3.0 (TID 230) in 7656 ms on emr-worker-8.cluster-196425 (executor 1) (29/50) 22/04/28 17:29:51 INFO [task-result-getter-1] TaskSetManager: Finished task 28.0 in stage 3.0 (TID 229) in 7767 ms on emr-worker-8.cluster-196425 (executor 1) (30/50) 22/04/28 17:29:51 INFO [task-result-getter-2] TaskSetManager: Finished task 30.0 in stage 3.0 (TID 231) in 7760 ms on emr-worker-8.cluster-196425 (executor 1) (31/50) 22/04/28 17:29:51 INFO [task-result-getter-0] TaskSetManager: Finished task 32.0 in stage 3.0 (TID 233) in 7762 ms on emr-worker-8.cluster-196425 (executor 1) (32/50) 22/04/28 17:29:52 INFO [task-result-getter-3] TaskSetManager: Finished task 35.0 in stage 3.0 (TID 236) in 7812 ms on emr-worker-6.cluster-196425 (executor 4) (33/50) 22/04/28 17:29:52 INFO [task-result-getter-1] TaskSetManager: Finished task 31.0 in stage 3.0 (TID 232) in 8105 ms on emr-worker-6.cluster-196425 (executor 6) (34/50) 22/04/28 17:29:52 INFO [task-result-getter-2] TaskSetManager: Finished task 33.0 in stage 3.0 (TID 234) in 8017 ms on emr-worker-6.cluster-196425 (executor 6) (35/50) 22/04/28 17:29:52 INFO [task-result-getter-0] TaskSetManager: Finished task 43.0 in stage 3.0 (TID 244) in 7735 ms on emr-worker-19.cluster-196425 (executor 5) (36/50) 22/04/28 17:29:52 INFO [task-result-getter-3] TaskSetManager: Finished task 38.0 in stage 3.0 (TID 239) in 7899 ms on emr-worker-6.cluster-196425 (executor 7) (37/50) 22/04/28 17:29:52 INFO [task-result-getter-1] TaskSetManager: Finished task 42.0 in stage 3.0 (TID 243) in 7836 ms on emr-worker-6.cluster-196425 (executor 7) (38/50) 22/04/28 17:29:52 INFO [task-result-getter-2] TaskSetManager: Finished task 44.0 in stage 3.0 (TID 245) in 7834 ms on emr-worker-6.cluster-196425 (executor 7) (39/50) 22/04/28 17:29:52 INFO [task-result-getter-0] TaskSetManager: Finished task 34.0 in stage 3.0 (TID 235) in 8531 ms on emr-worker-6.cluster-196425 (executor 6) (40/50) 22/04/28 17:29:52 INFO [task-result-getter-3] TaskSetManager: Finished task 45.0 in stage 3.0 (TID 246) in 7803 ms on emr-worker-6.cluster-196425 (executor 4) (41/50) 22/04/28 17:29:52 INFO [task-result-getter-1] TaskSetManager: Finished task 47.0 in stage 3.0 (TID 248) in 7598 ms on emr-worker-19.cluster-196425 (executor 5) (42/50) 22/04/28 17:29:52 INFO [task-result-getter-2] TaskSetManager: Finished task 36.0 in stage 3.0 (TID 237) in 8320 ms on emr-worker-19.cluster-196425 (executor 2) (43/50) 22/04/28 17:29:53 INFO [task-result-getter-0] TaskSetManager: Finished task 49.0 in stage 3.0 (TID 250) in 7739 ms on emr-worker-6.cluster-196425 (executor 4) (44/50) 22/04/28 17:29:53 INFO [task-result-getter-3] TaskSetManager: Finished task 37.0 in stage 3.0 (TID 238) in 8417 ms on emr-worker-19.cluster-196425 (executor 2) (45/50) 22/04/28 17:29:53 INFO [task-result-getter-1] TaskSetManager: Finished task 39.0 in stage 3.0 (TID 240) in 8389 ms on emr-worker-19.cluster-196425 (executor 2) (46/50) 22/04/28 17:29:53 INFO [task-result-getter-2] TaskSetManager: Finished task 40.0 in stage 3.0 (TID 241) in 8370 ms on emr-worker-6.cluster-196425 (executor 7) (47/50) 22/04/28 17:29:53 INFO [task-result-getter-0] TaskSetManager: Finished task 41.0 in stage 3.0 (TID 242) in 8315 ms on emr-worker-19.cluster-196425 (executor 2) (48/50) 22/04/28 17:29:53 INFO [task-result-getter-3] TaskSetManager: Finished task 46.0 in stage 3.0 (TID 247) in 7996 ms on emr-worker-6.cluster-196425 (executor 4) (49/50) 22/04/28 17:29:53 INFO [task-result-getter-1] TaskSetManager: Finished task 48.0 in stage 3.0 (TID 249) in 8267 ms on emr-worker-6.cluster-196425 (executor 6) (50/50) 22/04/28 17:29:53 INFO [task-result-getter-1] YarnScheduler: Removed TaskSet 3.0, whose tasks have all completed, from pool 22/04/28 17:29:53 INFO [dag-scheduler-event-loop] DAGScheduler: ShuffleMapStage 3 (run at ThreadPoolExecutor.java:1149) finished in 19.536 s 22/04/28 17:29:53 INFO [dag-scheduler-event-loop] DAGScheduler: looking for newly runnable stages 22/04/28 17:29:53 INFO [dag-scheduler-event-loop] DAGScheduler: running: Set() 22/04/28 17:29:53 INFO [dag-scheduler-event-loop] DAGScheduler: waiting: Set() 22/04/28 17:29:53 INFO [dag-scheduler-event-loop] DAGScheduler: failed: Set() 22/04/28 17:29:53 INFO [adaptive-query-stage-0] CodeGenerator: Code generated in 19.024511 ms 22/04/28 17:29:53 INFO [dag-scheduler-event-loop] DAGScheduler: Registering RDD 24 (run at ThreadPoolExecutor.java:1149) as input to shuffle 1 22/04/28 17:29:53 INFO [dag-scheduler-event-loop] DAGScheduler: Got map stage job 4 (run at ThreadPoolExecutor.java:1149) with 32 output partitions 22/04/28 17:29:53 INFO [dag-scheduler-event-loop] DAGScheduler: Final stage: ShuffleMapStage 5 (run at ThreadPoolExecutor.java:1149) 22/04/28 17:29:53 INFO [dag-scheduler-event-loop] DAGScheduler: Parents of final stage: List(ShuffleMapStage 4) 22/04/28 17:29:53 INFO [dag-scheduler-event-loop] DAGScheduler: Missing parents: List() 22/04/28 17:29:53 INFO [dag-scheduler-event-loop] DAGScheduler: Submitting ShuffleMapStage 5 (MapPartitionsRDD[24] at run at ThreadPoolExecutor.java:1149), which has no missing parents 22/04/28 17:29:53 INFO [dag-scheduler-event-loop] MemoryStore: Block broadcast_7 stored as values in memory (estimated size 32.0 KB, free 2003.0 MB) 22/04/28 17:29:53 INFO [dag-scheduler-event-loop] MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 11.8 KB, free 2003.0 MB) 22/04/28 17:29:53 INFO [dispatcher-event-loop-2] BlockManagerInfo: Added broadcast_7_piece0 in memory on 172.16.67.122:41394 (size: 11.8 KB, free: 2004.5 MB) 22/04/28 17:29:53 INFO [dag-scheduler-event-loop] SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1163 22/04/28 17:29:53 INFO [dag-scheduler-event-loop] DAGScheduler: Submitting 32 missing tasks from ShuffleMapStage 5 (MapPartitionsRDD[24] at run at ThreadPoolExecutor.java:1149) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14)) 22/04/28 17:29:53 INFO [dag-scheduler-event-loop] YarnScheduler: Adding task set 5.0 with 32 tasks 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 62 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 112 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 53 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 114 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 129 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 0.0 in stage 5.0 (TID 251, emr-worker-6.cluster-196425, executor 6, partition 0, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 1.0 in stage 5.0 (TID 252, emr-worker-8.cluster-196425, executor 1, partition 1, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 2.0 in stage 5.0 (TID 253, emr-worker-6.cluster-196425, executor 4, partition 2, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 3.0 in stage 5.0 (TID 254, emr-worker-6.cluster-196425, executor 7, partition 3, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 4.0 in stage 5.0 (TID 255, emr-worker-19.cluster-196425, executor 2, partition 4, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 5.0 in stage 5.0 (TID 256, emr-worker-9.cluster-196425, executor 3, partition 5, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 6.0 in stage 5.0 (TID 257, emr-worker-19.cluster-196425, executor 5, partition 6, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 7.0 in stage 5.0 (TID 258, emr-worker-6.cluster-196425, executor 6, partition 7, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 8.0 in stage 5.0 (TID 259, emr-worker-8.cluster-196425, executor 1, partition 8, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 9.0 in stage 5.0 (TID 260, emr-worker-6.cluster-196425, executor 4, partition 9, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 10.0 in stage 5.0 (TID 261, emr-worker-6.cluster-196425, executor 7, partition 10, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 11.0 in stage 5.0 (TID 262, emr-worker-19.cluster-196425, executor 2, partition 11, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 12.0 in stage 5.0 (TID 263, emr-worker-9.cluster-196425, executor 3, partition 12, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 13.0 in stage 5.0 (TID 264, emr-worker-19.cluster-196425, executor 5, partition 13, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 14.0 in stage 5.0 (TID 265, emr-worker-6.cluster-196425, executor 6, partition 14, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 15.0 in stage 5.0 (TID 266, emr-worker-8.cluster-196425, executor 1, partition 15, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 16.0 in stage 5.0 (TID 267, emr-worker-6.cluster-196425, executor 4, partition 16, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 17.0 in stage 5.0 (TID 268, emr-worker-6.cluster-196425, executor 7, partition 17, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 18.0 in stage 5.0 (TID 269, emr-worker-19.cluster-196425, executor 2, partition 18, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 19.0 in stage 5.0 (TID 270, emr-worker-9.cluster-196425, executor 3, partition 19, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 20.0 in stage 5.0 (TID 271, emr-worker-19.cluster-196425, executor 5, partition 20, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:29:53 INFO [dispatcher-event-loop-2] BlockManagerInfo: Removed broadcast_6_piece0 on emr-worker-8.cluster-196425:33263 in memory (size: 6.6 KB, free: 21.2 GB) 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 21.0 in stage 5.0 (TID 272, emr-worker-6.cluster-196425, executor 6, partition 21, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 22.0 in stage 5.0 (TID 273, emr-worker-8.cluster-196425, executor 1, partition 22, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 23.0 in stage 5.0 (TID 274, emr-worker-6.cluster-196425, executor 4, partition 23, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 24.0 in stage 5.0 (TID 275, emr-worker-6.cluster-196425, executor 7, partition 24, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:29:53 INFO [dispatcher-event-loop-2] BlockManagerInfo: Removed broadcast_6_piece0 on emr-worker-6.cluster-196425:36207 in memory (size: 6.6 KB, free: 21.2 GB) 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 25.0 in stage 5.0 (TID 276, emr-worker-19.cluster-196425, executor 2, partition 25, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 26.0 in stage 5.0 (TID 277, emr-worker-9.cluster-196425, executor 3, partition 26, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 27.0 in stage 5.0 (TID 278, emr-worker-19.cluster-196425, executor 5, partition 27, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:29:53 INFO [dispatcher-event-loop-2] BlockManagerInfo: Removed broadcast_6_piece0 on emr-worker-19.cluster-196425:42145 in memory (size: 6.6 KB, free: 21.2 GB) 22/04/28 17:29:53 INFO [dispatcher-event-loop-2] BlockManagerInfo: Removed broadcast_6_piece0 on emr-worker-6.cluster-196425:41171 in memory (size: 6.6 KB, free: 21.2 GB) 22/04/28 17:29:53 INFO [dispatcher-event-loop-2] BlockManagerInfo: Removed broadcast_6_piece0 on emr-worker-9.cluster-196425:46825 in memory (size: 6.6 KB, free: 21.2 GB) 22/04/28 17:29:53 INFO [dispatcher-event-loop-2] BlockManagerInfo: Removed broadcast_6_piece0 on emr-worker-6.cluster-196425:37495 in memory (size: 6.6 KB, free: 21.2 GB) 22/04/28 17:29:53 INFO [dispatcher-event-loop-2] BlockManagerInfo: Removed broadcast_6_piece0 on emr-worker-19.cluster-196425:41635 in memory (size: 6.6 KB, free: 21.2 GB) 22/04/28 17:29:53 INFO [dispatcher-event-loop-0] BlockManagerInfo: Removed broadcast_6_piece0 on 172.16.67.122:41394 in memory (size: 6.6 KB, free: 2004.5 MB) 22/04/28 17:29:53 INFO [dispatcher-event-loop-1] BlockManagerInfo: Added broadcast_7_piece0 in memory on emr-worker-8.cluster-196425:33263 (size: 11.8 KB, free: 21.2 GB) 22/04/28 17:29:53 INFO [dispatcher-event-loop-2] BlockManagerInfo: Added broadcast_7_piece0 in memory on emr-worker-19.cluster-196425:41635 (size: 11.8 KB, free: 21.2 GB) 22/04/28 17:29:53 INFO [dispatcher-event-loop-2] BlockManagerInfo: Added broadcast_7_piece0 in memory on emr-worker-19.cluster-196425:42145 (size: 11.8 KB, free: 21.2 GB) 22/04/28 17:29:53 INFO [dispatcher-event-loop-2] BlockManagerInfo: Added broadcast_7_piece0 in memory on emr-worker-6.cluster-196425:37495 (size: 11.8 KB, free: 21.2 GB) 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] BlockManagerInfo: Added broadcast_7_piece0 in memory on emr-worker-9.cluster-196425:46825 (size: 11.8 KB, free: 21.2 GB) 22/04/28 17:29:53 INFO [dispatcher-event-loop-0] BlockManagerInfo: Added broadcast_7_piece0 in memory on emr-worker-6.cluster-196425:41171 (size: 11.8 KB, free: 21.2 GB) 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 111 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 81 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 119 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 133 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 74 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 52 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 90 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 127 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 51 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 61 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 70 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 115 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 117 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 54 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 79 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 72 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 76 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 77 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 113 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 132 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 126 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 67 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 110 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 71 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 123 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 58 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 69 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 84 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] BlockManagerInfo: Removed broadcast_2_piece0 on 172.16.67.122:41394 in memory (size: 41.9 KB, free: 2004.5 MB) 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] BlockManagerInfo: Removed broadcast_2_piece0 on emr-worker-6.cluster-196425:37495 in memory (size: 41.9 KB, free: 21.2 GB) 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] BlockManagerInfo: Added broadcast_7_piece0 in memory on emr-worker-6.cluster-196425:36207 (size: 11.8 KB, free: 21.2 GB) 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 80 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 109 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 75 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 60 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 56 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 78 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 55 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 128 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 130 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 120 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 122 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 125 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 118 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 50 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 65 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 116 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 82 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 63 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 73 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 68 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 124 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 88 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 83 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 131 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 66 22/04/28 17:29:53 INFO [dispatcher-event-loop-1] BlockManagerInfo: Removed broadcast_3_piece0 on 172.16.67.122:41394 in memory (size: 4.6 KB, free: 2004.5 MB) 22/04/28 17:29:53 INFO [dispatcher-event-loop-1] BlockManagerInfo: Removed broadcast_3_piece0 on emr-worker-6.cluster-196425:37495 in memory (size: 4.6 KB, free: 21.2 GB) 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 85 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] BlockManagerInfo: Removed broadcast_4_piece0 on 172.16.67.122:41394 in memory (size: 41.9 KB, free: 2004.5 MB) 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 59 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 121 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 57 22/04/28 17:29:53 INFO [Spark Context Cleaner] ContextCleaner: Cleaned accumulator 64 22/04/28 17:29:53 INFO [dispatcher-event-loop-1] MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 172.16.72.56:49634 22/04/28 17:29:53 INFO [dispatcher-event-loop-2] MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 172.16.67.172:46372 22/04/28 17:29:53 INFO [dispatcher-event-loop-0] MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 172.16.72.54:56452 22/04/28 17:29:53 INFO [dispatcher-event-loop-3] MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 172.16.67.172:46370 22/04/28 17:29:53 INFO [dispatcher-event-loop-1] MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 172.16.72.52:59280 22/04/28 17:29:53 INFO [dispatcher-event-loop-2] MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 172.16.72.52:59338 22/04/28 17:29:53 INFO [dispatcher-event-loop-0] MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 172.16.72.52:60224 22/04/28 17:30:04 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 28.0 in stage 5.0 (TID 279, emr-worker-8.cluster-196425, executor 1, partition 28, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:30:04 INFO [task-result-getter-2] TaskSetManager: Finished task 1.0 in stage 5.0 (TID 252) in 10704 ms on emr-worker-8.cluster-196425 (executor 1) (1/32) 22/04/28 17:30:04 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 29.0 in stage 5.0 (TID 280, emr-worker-8.cluster-196425, executor 1, partition 29, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:30:04 INFO [task-result-getter-0] TaskSetManager: Finished task 22.0 in stage 5.0 (TID 273) in 10703 ms on emr-worker-8.cluster-196425 (executor 1) (2/32) 22/04/28 17:30:04 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 30.0 in stage 5.0 (TID 281, emr-worker-8.cluster-196425, executor 1, partition 30, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:30:04 INFO [task-result-getter-3] TaskSetManager: Finished task 8.0 in stage 5.0 (TID 259) in 10749 ms on emr-worker-8.cluster-196425 (executor 1) (3/32) 22/04/28 17:30:04 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 31.0 in stage 5.0 (TID 282, emr-worker-8.cluster-196425, executor 1, partition 31, PROCESS_LOCAL, 7769 bytes) 22/04/28 17:30:04 INFO [task-result-getter-1] TaskSetManager: Finished task 15.0 in stage 5.0 (TID 266) in 10757 ms on emr-worker-8.cluster-196425 (executor 1) (4/32) 22/04/28 17:30:05 INFO [task-result-getter-2] TaskSetManager: Finished task 21.0 in stage 5.0 (TID 272) in 11463 ms on emr-worker-6.cluster-196425 (executor 6) (5/32) 22/04/28 17:30:05 INFO [task-result-getter-0] TaskSetManager: Finished task 7.0 in stage 5.0 (TID 258) in 11511 ms on emr-worker-6.cluster-196425 (executor 6) (6/32) 22/04/28 17:30:05 INFO [task-result-getter-3] TaskSetManager: Finished task 0.0 in stage 5.0 (TID 251) in 11544 ms on emr-worker-6.cluster-196425 (executor 6) (7/32) 22/04/28 17:30:05 INFO [task-result-getter-1] TaskSetManager: Finished task 14.0 in stage 5.0 (TID 265) in 11565 ms on emr-worker-6.cluster-196425 (executor 6) (8/32) 22/04/28 17:30:05 INFO [task-result-getter-2] TaskSetManager: Finished task 10.0 in stage 5.0 (TID 261) in 11656 ms on emr-worker-6.cluster-196425 (executor 7) (9/32) 22/04/28 17:30:05 INFO [task-result-getter-0] TaskSetManager: Finished task 11.0 in stage 5.0 (TID 262) in 11687 ms on emr-worker-19.cluster-196425 (executor 2) (10/32) 22/04/28 17:30:05 INFO [task-result-getter-3] TaskSetManager: Finished task 16.0 in stage 5.0 (TID 267) in 11731 ms on emr-worker-6.cluster-196425 (executor 4) (11/32) 22/04/28 17:30:05 INFO [task-result-getter-1] TaskSetManager: Finished task 2.0 in stage 5.0 (TID 253) in 11734 ms on emr-worker-6.cluster-196425 (executor 4) (12/32) 22/04/28 17:30:05 INFO [task-result-getter-2] TaskSetManager: Finished task 9.0 in stage 5.0 (TID 260) in 11734 ms on emr-worker-6.cluster-196425 (executor 4) (13/32) 22/04/28 17:30:05 INFO [task-result-getter-0] TaskSetManager: Finished task 25.0 in stage 5.0 (TID 276) in 11818 ms on emr-worker-19.cluster-196425 (executor 2) (14/32) 22/04/28 17:30:05 INFO [task-result-getter-3] TaskSetManager: Finished task 4.0 in stage 5.0 (TID 255) in 11833 ms on emr-worker-19.cluster-196425 (executor 2) (15/32) 22/04/28 17:30:05 INFO [task-result-getter-1] TaskSetManager: Finished task 3.0 in stage 5.0 (TID 254) in 11837 ms on emr-worker-6.cluster-196425 (executor 7) (16/32) 22/04/28 17:30:05 INFO [task-result-getter-2] TaskSetManager: Finished task 17.0 in stage 5.0 (TID 268) in 11847 ms on emr-worker-6.cluster-196425 (executor 7) (17/32) 22/04/28 17:30:05 INFO [task-result-getter-0] TaskSetManager: Finished task 24.0 in stage 5.0 (TID 275) in 11909 ms on emr-worker-6.cluster-196425 (executor 7) (18/32) 22/04/28 17:30:05 INFO [task-result-getter-3] TaskSetManager: Finished task 23.0 in stage 5.0 (TID 274) in 11911 ms on emr-worker-6.cluster-196425 (executor 4) (19/32) 22/04/28 17:30:05 INFO [task-result-getter-1] TaskSetManager: Finished task 18.0 in stage 5.0 (TID 269) in 12002 ms on emr-worker-19.cluster-196425 (executor 2) (20/32) 22/04/28 17:30:06 INFO [task-result-getter-2] TaskSetManager: Finished task 6.0 in stage 5.0 (TID 257) in 12299 ms on emr-worker-19.cluster-196425 (executor 5) (21/32) 22/04/28 17:30:06 INFO [task-result-getter-0] TaskSetManager: Finished task 13.0 in stage 5.0 (TID 264) in 12382 ms on emr-worker-19.cluster-196425 (executor 5) (22/32) 22/04/28 17:30:06 INFO [task-result-getter-3] TaskSetManager: Finished task 20.0 in stage 5.0 (TID 271) in 12473 ms on emr-worker-19.cluster-196425 (executor 5) (23/32) 22/04/28 17:30:06 INFO [task-result-getter-1] TaskSetManager: Finished task 27.0 in stage 5.0 (TID 278) in 12646 ms on emr-worker-19.cluster-196425 (executor 5) (24/32) 22/04/28 17:30:07 INFO [task-result-getter-2] TaskSetManager: Finished task 5.0 in stage 5.0 (TID 256) in 13538 ms on emr-worker-9.cluster-196425 (executor 3) (25/32) 22/04/28 17:30:07 INFO [task-result-getter-0] TaskSetManager: Finished task 19.0 in stage 5.0 (TID 270) in 13543 ms on emr-worker-9.cluster-196425 (executor 3) (26/32) 22/04/28 17:30:07 INFO [task-result-getter-3] TaskSetManager: Finished task 26.0 in stage 5.0 (TID 277) in 13796 ms on emr-worker-9.cluster-196425 (executor 3) (27/32) 22/04/28 17:30:07 INFO [task-result-getter-1] TaskSetManager: Finished task 12.0 in stage 5.0 (TID 263) in 13815 ms on emr-worker-9.cluster-196425 (executor 3) (28/32) 22/04/28 17:30:13 INFO [task-result-getter-2] TaskSetManager: Finished task 30.0 in stage 5.0 (TID 281) in 8554 ms on emr-worker-8.cluster-196425 (executor 1) (29/32) 22/04/28 17:30:13 INFO [task-result-getter-0] TaskSetManager: Finished task 31.0 in stage 5.0 (TID 282) in 8737 ms on emr-worker-8.cluster-196425 (executor 1) (30/32) 22/04/28 17:30:13 INFO [task-result-getter-3] TaskSetManager: Finished task 28.0 in stage 5.0 (TID 279) in 8862 ms on emr-worker-8.cluster-196425 (executor 1) (31/32) 22/04/28 17:30:13 INFO [task-result-getter-1] TaskSetManager: Finished task 29.0 in stage 5.0 (TID 280) in 8876 ms on emr-worker-8.cluster-196425 (executor 1) (32/32) 22/04/28 17:30:13 INFO [task-result-getter-1] YarnScheduler: Removed TaskSet 5.0, whose tasks have all completed, from pool 22/04/28 17:30:13 INFO [dag-scheduler-event-loop] DAGScheduler: ShuffleMapStage 5 (run at ThreadPoolExecutor.java:1149) finished in 19.641 s 22/04/28 17:30:13 INFO [dag-scheduler-event-loop] DAGScheduler: looking for newly runnable stages 22/04/28 17:30:13 INFO [dag-scheduler-event-loop] DAGScheduler: running: Set() 22/04/28 17:30:13 INFO [dag-scheduler-event-loop] DAGScheduler: waiting: Set() 22/04/28 17:30:13 INFO [dag-scheduler-event-loop] DAGScheduler: failed: Set() 22/04/28 17:30:13 INFO [main] ExchangeCoordinator: advisoryTargetPostShuffleInputSize: 67108864, targetPostShuffleInputSize 67108864. 22/04/28 17:30:13 INFO [main] CodeGenerator: Code generated in 21.014288 ms 22/04/28 17:30:13 INFO [main] SparkContext: Starting job: foreachPartition at VerticesProcessor.scala:181 22/04/28 17:30:13 INFO [dag-scheduler-event-loop] DAGScheduler: Got job 5 (foreachPartition at VerticesProcessor.scala:181) with 63 output partitions 22/04/28 17:30:13 INFO [dag-scheduler-event-loop] DAGScheduler: Final stage: ResultStage 8 (foreachPartition at VerticesProcessor.scala:181) 22/04/28 17:30:13 INFO [dag-scheduler-event-loop] DAGScheduler: Parents of final stage: List(ShuffleMapStage 7) 22/04/28 17:30:13 INFO [dag-scheduler-event-loop] DAGScheduler: Missing parents: List() 22/04/28 17:30:13 INFO [dag-scheduler-event-loop] DAGScheduler: Submitting ResultStage 8 (MapPartitionsRDD[34] at foreachPartition at VerticesProcessor.scala:181), which has no missing parents 22/04/28 17:30:13 INFO [dag-scheduler-event-loop] MemoryStore: Block broadcast_8 stored as values in memory (estimated size 68.8 KB, free 2004.0 MB) 22/04/28 17:30:13 INFO [dag-scheduler-event-loop] MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 29.9 KB, free 2004.0 MB) 22/04/28 17:30:13 INFO [dispatcher-event-loop-2] BlockManagerInfo: Added broadcast_8_piece0 in memory on 172.16.67.122:41394 (size: 29.9 KB, free: 2004.5 MB) 22/04/28 17:30:13 INFO [dag-scheduler-event-loop] SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:1163 22/04/28 17:30:13 INFO [dag-scheduler-event-loop] DAGScheduler: Submitting 63 missing tasks from ResultStage 8 (MapPartitionsRDD[34] at foreachPartition at VerticesProcessor.scala:181) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14)) 22/04/28 17:30:13 INFO [dag-scheduler-event-loop] YarnScheduler: Adding task set 8.0 with 63 tasks 22/04/28 17:30:13 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 0.0 in stage 8.0 (TID 283, emr-worker-8.cluster-196425, executor 1, partition 0, NODE_LOCAL, 7780 bytes) 22/04/28 17:30:13 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 1.0 in stage 8.0 (TID 284, emr-worker-8.cluster-196425, executor 1, partition 1, NODE_LOCAL, 7780 bytes) 22/04/28 17:30:13 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 2.0 in stage 8.0 (TID 285, emr-worker-8.cluster-196425, executor 1, partition 2, NODE_LOCAL, 7780 bytes) 22/04/28 17:30:13 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 3.0 in stage 8.0 (TID 286, emr-worker-8.cluster-196425, executor 1, partition 3, NODE_LOCAL, 7780 bytes) 22/04/28 17:30:13 INFO [dispatcher-event-loop-3] BlockManagerInfo: Added broadcast_8_piece0 in memory on emr-worker-8.cluster-196425:33263 (size: 29.9 KB, free: 21.2 GB) 22/04/28 17:30:13 INFO [dispatcher-event-loop-2] MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 172.16.72.56:49634 22/04/28 17:30:17 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 4.0 in stage 8.0 (TID 287, emr-worker-6.cluster-196425, executor 7, partition 4, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:17 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 5.0 in stage 8.0 (TID 288, emr-worker-19.cluster-196425, executor 5, partition 5, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:17 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 6.0 in stage 8.0 (TID 289, emr-worker-6.cluster-196425, executor 6, partition 6, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:17 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 7.0 in stage 8.0 (TID 290, emr-worker-9.cluster-196425, executor 3, partition 7, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:17 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 8.0 in stage 8.0 (TID 291, emr-worker-19.cluster-196425, executor 2, partition 8, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:17 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 9.0 in stage 8.0 (TID 292, emr-worker-6.cluster-196425, executor 4, partition 9, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:17 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 10.0 in stage 8.0 (TID 293, emr-worker-6.cluster-196425, executor 7, partition 10, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:17 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 11.0 in stage 8.0 (TID 294, emr-worker-19.cluster-196425, executor 5, partition 11, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:17 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 12.0 in stage 8.0 (TID 295, emr-worker-6.cluster-196425, executor 6, partition 12, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:17 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 13.0 in stage 8.0 (TID 296, emr-worker-9.cluster-196425, executor 3, partition 13, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:17 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 14.0 in stage 8.0 (TID 297, emr-worker-19.cluster-196425, executor 2, partition 14, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:17 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 15.0 in stage 8.0 (TID 298, emr-worker-6.cluster-196425, executor 4, partition 15, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:17 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 16.0 in stage 8.0 (TID 299, emr-worker-6.cluster-196425, executor 7, partition 16, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:17 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 17.0 in stage 8.0 (TID 300, emr-worker-19.cluster-196425, executor 5, partition 17, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:17 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 18.0 in stage 8.0 (TID 301, emr-worker-6.cluster-196425, executor 6, partition 18, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:17 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 19.0 in stage 8.0 (TID 302, emr-worker-9.cluster-196425, executor 3, partition 19, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:17 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 20.0 in stage 8.0 (TID 303, emr-worker-19.cluster-196425, executor 2, partition 20, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:17 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 21.0 in stage 8.0 (TID 304, emr-worker-6.cluster-196425, executor 4, partition 21, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:17 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 22.0 in stage 8.0 (TID 305, emr-worker-6.cluster-196425, executor 7, partition 22, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:17 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 23.0 in stage 8.0 (TID 306, emr-worker-19.cluster-196425, executor 5, partition 23, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:17 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 24.0 in stage 8.0 (TID 307, emr-worker-6.cluster-196425, executor 6, partition 24, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:17 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 25.0 in stage 8.0 (TID 308, emr-worker-9.cluster-196425, executor 3, partition 25, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:17 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 26.0 in stage 8.0 (TID 309, emr-worker-19.cluster-196425, executor 2, partition 26, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:17 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 27.0 in stage 8.0 (TID 310, emr-worker-6.cluster-196425, executor 4, partition 27, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:17 INFO [dispatcher-event-loop-2] BlockManagerInfo: Added broadcast_8_piece0 in memory on emr-worker-6.cluster-196425:36207 (size: 29.9 KB, free: 21.2 GB) 22/04/28 17:30:17 INFO [dispatcher-event-loop-0] BlockManagerInfo: Added broadcast_8_piece0 in memory on emr-worker-9.cluster-196425:46825 (size: 29.9 KB, free: 21.2 GB) 22/04/28 17:30:17 INFO [dispatcher-event-loop-3] BlockManagerInfo: Added broadcast_8_piece0 in memory on emr-worker-19.cluster-196425:42145 (size: 29.9 KB, free: 21.2 GB) 22/04/28 17:30:17 INFO [dispatcher-event-loop-1] BlockManagerInfo: Added broadcast_8_piece0 in memory on emr-worker-6.cluster-196425:37495 (size: 29.9 KB, free: 21.2 GB) 22/04/28 17:30:17 INFO [dispatcher-event-loop-2] BlockManagerInfo: Added broadcast_8_piece0 in memory on emr-worker-6.cluster-196425:41171 (size: 29.9 KB, free: 21.2 GB) 22/04/28 17:30:17 INFO [dispatcher-event-loop-0] BlockManagerInfo: Added broadcast_8_piece0 in memory on emr-worker-19.cluster-196425:41635 (size: 29.9 KB, free: 21.2 GB) 22/04/28 17:30:17 INFO [dispatcher-event-loop-3] MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 172.16.72.52:60224 22/04/28 17:30:17 INFO [dispatcher-event-loop-1] MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 172.16.72.52:59338 22/04/28 17:30:17 INFO [dispatcher-event-loop-2] MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 172.16.72.54:56452 22/04/28 17:30:17 INFO [dispatcher-event-loop-0] MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 172.16.72.52:59280 22/04/28 17:30:17 INFO [dispatcher-event-loop-3] MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 172.16.67.172:46370 22/04/28 17:30:17 INFO [dispatcher-event-loop-1] MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 172.16.67.172:46372 22/04/28 17:30:32 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 28.0 in stage 8.0 (TID 311, emr-worker-8.cluster-196425, executor 1, partition 28, NODE_LOCAL, 7780 bytes) 22/04/28 17:30:32 INFO [task-result-getter-2] TaskSetManager: Finished task 2.0 in stage 8.0 (TID 285) in 18820 ms on emr-worker-8.cluster-196425 (executor 1) (1/63) 22/04/28 17:30:33 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 29.0 in stage 8.0 (TID 312, emr-worker-8.cluster-196425, executor 1, partition 29, NODE_LOCAL, 7780 bytes) 22/04/28 17:30:33 INFO [task-result-getter-0] TaskSetManager: Finished task 3.0 in stage 8.0 (TID 286) in 19510 ms on emr-worker-8.cluster-196425 (executor 1) (2/63) 22/04/28 17:30:33 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 30.0 in stage 8.0 (TID 313, emr-worker-8.cluster-196425, executor 1, partition 30, NODE_LOCAL, 7780 bytes) 22/04/28 17:30:33 INFO [task-result-getter-3] TaskSetManager: Finished task 0.0 in stage 8.0 (TID 283) in 20096 ms on emr-worker-8.cluster-196425 (executor 1) (3/63) 22/04/28 17:30:35 INFO [task-result-getter-1] TaskSetManager: Finished task 21.0 in stage 8.0 (TID 304) in 18324 ms on emr-worker-6.cluster-196425 (executor 4) (4/63) 22/04/28 17:30:35 INFO [task-result-getter-2] TaskSetManager: Finished task 16.0 in stage 8.0 (TID 299) in 18426 ms on emr-worker-6.cluster-196425 (executor 7) (5/63) 22/04/28 17:30:36 INFO [task-result-getter-0] TaskSetManager: Finished task 27.0 in stage 8.0 (TID 310) in 18906 ms on emr-worker-6.cluster-196425 (executor 4) (6/63) 22/04/28 17:30:36 INFO [task-result-getter-3] TaskSetManager: Finished task 18.0 in stage 8.0 (TID 301) in 19104 ms on emr-worker-6.cluster-196425 (executor 6) (7/63) 22/04/28 17:30:36 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 31.0 in stage 8.0 (TID 314, emr-worker-6.cluster-196425, executor 7, partition 31, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:36 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 32.0 in stage 8.0 (TID 315, emr-worker-6.cluster-196425, executor 7, partition 32, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:36 INFO [task-result-getter-1] TaskSetManager: Finished task 10.0 in stage 8.0 (TID 293) in 19541 ms on emr-worker-6.cluster-196425 (executor 7) (8/63) 22/04/28 17:30:36 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 33.0 in stage 8.0 (TID 316, emr-worker-6.cluster-196425, executor 6, partition 33, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:36 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 34.0 in stage 8.0 (TID 317, emr-worker-6.cluster-196425, executor 6, partition 34, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:36 INFO [task-result-getter-2] TaskSetManager: Finished task 6.0 in stage 8.0 (TID 289) in 19852 ms on emr-worker-6.cluster-196425 (executor 6) (9/63) 22/04/28 17:30:37 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 35.0 in stage 8.0 (TID 318, emr-worker-6.cluster-196425, executor 4, partition 35, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:37 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 36.0 in stage 8.0 (TID 319, emr-worker-6.cluster-196425, executor 4, partition 36, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:37 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 37.0 in stage 8.0 (TID 320, emr-worker-6.cluster-196425, executor 7, partition 37, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:37 INFO [task-result-getter-0] TaskSetManager: Finished task 22.0 in stage 8.0 (TID 305) in 20377 ms on emr-worker-6.cluster-196425 (executor 7) (10/63) 22/04/28 17:30:37 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 38.0 in stage 8.0 (TID 321, emr-worker-19.cluster-196425, executor 2, partition 38, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:37 INFO [task-result-getter-3] TaskSetManager: Finished task 8.0 in stage 8.0 (TID 291) in 20475 ms on emr-worker-19.cluster-196425 (executor 2) (11/63) 22/04/28 17:30:38 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 39.0 in stage 8.0 (TID 322, emr-worker-19.cluster-196425, executor 2, partition 39, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:38 INFO [task-result-getter-1] TaskSetManager: Finished task 20.0 in stage 8.0 (TID 303) in 21090 ms on emr-worker-19.cluster-196425 (executor 2) (12/63) 22/04/28 17:30:38 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 40.0 in stage 8.0 (TID 323, emr-worker-19.cluster-196425, executor 5, partition 40, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:38 INFO [task-result-getter-2] TaskSetManager: Finished task 11.0 in stage 8.0 (TID 294) in 21357 ms on emr-worker-19.cluster-196425 (executor 5) (13/63) 22/04/28 17:30:39 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 41.0 in stage 8.0 (TID 324, emr-worker-19.cluster-196425, executor 2, partition 41, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:39 INFO [task-result-getter-0] TaskSetManager: Finished task 26.0 in stage 8.0 (TID 309) in 21963 ms on emr-worker-19.cluster-196425 (executor 2) (14/63) 22/04/28 17:30:39 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 42.0 in stage 8.0 (TID 325, emr-worker-19.cluster-196425, executor 5, partition 42, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:39 INFO [task-result-getter-3] TaskSetManager: Finished task 17.0 in stage 8.0 (TID 300) in 22237 ms on emr-worker-19.cluster-196425 (executor 5) (15/63) 22/04/28 17:30:40 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 43.0 in stage 8.0 (TID 326, emr-worker-6.cluster-196425, executor 4, partition 43, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:40 INFO [task-result-getter-1] TaskSetManager: Finished task 9.0 in stage 8.0 (TID 292) in 23433 ms on emr-worker-6.cluster-196425 (executor 4) (16/63) 22/04/28 17:30:43 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 44.0 in stage 8.0 (TID 327, emr-worker-6.cluster-196425, executor 6, partition 44, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:43 INFO [task-result-getter-2] TaskSetManager: Finished task 24.0 in stage 8.0 (TID 307) in 26326 ms on emr-worker-6.cluster-196425 (executor 6) (17/63) 22/04/28 17:30:43 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 45.0 in stage 8.0 (TID 328, emr-worker-8.cluster-196425, executor 1, partition 45, NODE_LOCAL, 7780 bytes) 22/04/28 17:30:43 INFO [task-result-getter-0] TaskSetManager: Finished task 1.0 in stage 8.0 (TID 284) in 30256 ms on emr-worker-8.cluster-196425 (executor 1) (18/63) 22/04/28 17:30:43 INFO [task-result-getter-3] TaskSetManager: Finished task 13.0 in stage 8.0 (TID 296) in 26857 ms on emr-worker-9.cluster-196425 (executor 3) (19/63) 22/04/28 17:30:44 INFO [task-result-getter-1] TaskSetManager: Finished task 15.0 in stage 8.0 (TID 298) in 27042 ms on emr-worker-6.cluster-196425 (executor 4) (20/63) 22/04/28 17:30:46 INFO [task-result-getter-2] TaskSetManager: Finished task 12.0 in stage 8.0 (TID 295) in 28910 ms on emr-worker-6.cluster-196425 (executor 6) (21/63) 22/04/28 17:30:47 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 46.0 in stage 8.0 (TID 329, emr-worker-9.cluster-196425, executor 3, partition 46, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:47 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 47.0 in stage 8.0 (TID 330, emr-worker-9.cluster-196425, executor 3, partition 47, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:47 INFO [task-result-getter-0] TaskSetManager: Finished task 25.0 in stage 8.0 (TID 308) in 29943 ms on emr-worker-9.cluster-196425 (executor 3) (22/63) 22/04/28 17:30:47 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 48.0 in stage 8.0 (TID 331, emr-worker-6.cluster-196425, executor 4, partition 48, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:47 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 49.0 in stage 8.0 (TID 332, emr-worker-6.cluster-196425, executor 6, partition 49, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:48 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 50.0 in stage 8.0 (TID 333, emr-worker-6.cluster-196425, executor 7, partition 50, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:48 INFO [task-result-getter-3] TaskSetManager: Finished task 4.0 in stage 8.0 (TID 287) in 30898 ms on emr-worker-6.cluster-196425 (executor 7) (23/63) 22/04/28 17:30:49 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 51.0 in stage 8.0 (TID 334, emr-worker-8.cluster-196425, executor 1, partition 51, NODE_LOCAL, 7780 bytes) 22/04/28 17:30:49 INFO [task-result-getter-1] TaskSetManager: Finished task 29.0 in stage 8.0 (TID 312) in 16927 ms on emr-worker-8.cluster-196425 (executor 1) (24/63) 22/04/28 17:30:53 INFO [dispatcher-event-loop-2] TaskSetManager: Starting task 52.0 in stage 8.0 (TID 335, emr-worker-6.cluster-196425, executor 7, partition 52, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:53 INFO [task-result-getter-2] TaskSetManager: Finished task 32.0 in stage 8.0 (TID 315) in 17317 ms on emr-worker-6.cluster-196425 (executor 7) (25/63) 22/04/28 17:30:54 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 53.0 in stage 8.0 (TID 336, emr-worker-19.cluster-196425, executor 5, partition 53, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:54 INFO [task-result-getter-0] TaskSetManager: Finished task 23.0 in stage 8.0 (TID 306) in 37022 ms on emr-worker-19.cluster-196425 (executor 5) (26/63) 22/04/28 17:30:54 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 54.0 in stage 8.0 (TID 337, emr-worker-6.cluster-196425, executor 7, partition 54, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:54 INFO [task-result-getter-3] TaskSetManager: Finished task 31.0 in stage 8.0 (TID 314) in 17569 ms on emr-worker-6.cluster-196425 (executor 7) (27/63) 22/04/28 17:30:54 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 55.0 in stage 8.0 (TID 338, emr-worker-6.cluster-196425, executor 6, partition 55, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:54 INFO [task-result-getter-1] TaskSetManager: Finished task 33.0 in stage 8.0 (TID 316) in 17314 ms on emr-worker-6.cluster-196425 (executor 6) (28/63) 22/04/28 17:30:55 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 56.0 in stage 8.0 (TID 339, emr-worker-6.cluster-196425, executor 4, partition 56, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:55 INFO [task-result-getter-2] TaskSetManager: Finished task 36.0 in stage 8.0 (TID 319) in 17980 ms on emr-worker-6.cluster-196425 (executor 4) (29/63) 22/04/28 17:30:57 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 57.0 in stage 8.0 (TID 340, emr-worker-6.cluster-196425, executor 4, partition 57, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:57 INFO [task-result-getter-0] TaskSetManager: Finished task 43.0 in stage 8.0 (TID 326) in 16483 ms on emr-worker-6.cluster-196425 (executor 4) (30/63) 22/04/28 17:30:57 INFO [dispatcher-event-loop-1] TaskSetManager: Starting task 58.0 in stage 8.0 (TID 341, emr-worker-19.cluster-196425, executor 5, partition 58, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:57 INFO [task-result-getter-3] TaskSetManager: Finished task 42.0 in stage 8.0 (TID 325) in 18177 ms on emr-worker-19.cluster-196425 (executor 5) (31/63) 22/04/28 17:30:58 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 59.0 in stage 8.0 (TID 342, emr-worker-19.cluster-196425, executor 5, partition 59, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:58 INFO [task-result-getter-1] TaskSetManager: Finished task 5.0 in stage 8.0 (TID 288) in 41005 ms on emr-worker-19.cluster-196425 (executor 5) (32/63) 22/04/28 17:30:58 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 60.0 in stage 8.0 (TID 343, emr-worker-19.cluster-196425, executor 5, partition 60, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:58 INFO [task-result-getter-2] TaskSetManager: Finished task 40.0 in stage 8.0 (TID 323) in 20467 ms on emr-worker-19.cluster-196425 (executor 5) (33/63) 22/04/28 17:30:59 INFO [dispatcher-event-loop-0] TaskSetManager: Starting task 61.0 in stage 8.0 (TID 344, emr-worker-9.cluster-196425, executor 3, partition 61, RACK_LOCAL, 7780 bytes) 22/04/28 17:30:59 INFO [task-result-getter-0] TaskSetManager: Finished task 19.0 in stage 8.0 (TID 302) in 42142 ms on emr-worker-9.cluster-196425 (executor 3) (34/63) 22/04/28 17:31:00 INFO [dispatcher-event-loop-3] TaskSetManager: Starting task 62.0 in stage 8.0 (TID 345, emr-worker-19.cluster-196425, executor 2, partition 62, RACK_LOCAL, 7780 bytes) 22/04/28 17:31:00 INFO [task-result-getter-3] TaskSetManager: Finished task 41.0 in stage 8.0 (TID 324) in 21466 ms on emr-worker-19.cluster-196425 (executor 2) (35/63) 22/04/28 17:31:00 INFO [task-result-getter-1] TaskSetManager: Finished task 45.0 in stage 8.0 (TID 328) in 17124 ms on emr-worker-8.cluster-196425 (executor 1) (36/63) 22/04/28 17:31:04 INFO [task-result-getter-2] TaskSetManager: Finished task 50.0 in stage 8.0 (TID 333) in 16777 ms on emr-worker-6.cluster-196425 (executor 7) (37/63) 22/04/28 17:31:07 INFO [task-result-getter-0] TaskSetManager: Finished task 51.0 in stage 8.0 (TID 334) in 17291 ms on emr-worker-8.cluster-196425 (executor 1) (38/63) 22/04/28 17:31:07 INFO [task-result-getter-3] TaskSetManager: Finished task 46.0 in stage 8.0 (TID 329) in 20809 ms on emr-worker-9.cluster-196425 (executor 3) (39/63) 22/04/28 17:31:07 INFO [task-result-getter-1] TaskSetManager: Finished task 47.0 in stage 8.0 (TID 330) in 20907 ms on emr-worker-9.cluster-196425 (executor 3) (40/63) 22/04/28 17:31:12 INFO [task-result-getter-2] TaskSetManager: Finished task 54.0 in stage 8.0 (TID 337) in 18135 ms on emr-worker-6.cluster-196425 (executor 7) (41/63) 22/04/28 17:31:12 INFO [task-result-getter-0] TaskSetManager: Finished task 53.0 in stage 8.0 (TID 336) in 18469 ms on emr-worker-19.cluster-196425 (executor 5) (42/63) 22/04/28 17:31:12 INFO [task-result-getter-3] TaskSetManager: Finished task 56.0 in stage 8.0 (TID 339) in 17787 ms on emr-worker-6.cluster-196425 (executor 4) (43/63) 22/04/28 17:31:14 INFO [task-result-getter-1] TaskSetManager: Finished task 30.0 in stage 8.0 (TID 313) in 40717 ms on emr-worker-8.cluster-196425 (executor 1) (44/63) 22/04/28 17:31:15 INFO [task-result-getter-2] TaskSetManager: Finished task 58.0 in stage 8.0 (TID 341) in 17674 ms on emr-worker-19.cluster-196425 (executor 5) (45/63) 22/04/28 17:31:16 INFO [task-result-getter-0] TaskSetManager: Finished task 28.0 in stage 8.0 (TID 311) in 43881 ms on emr-worker-8.cluster-196425 (executor 1) (46/63) 22/04/28 17:31:18 INFO [task-result-getter-3] TaskSetManager: Finished task 14.0 in stage 8.0 (TID 297) in 60977 ms on emr-worker-19.cluster-196425 (executor 2) (47/63) 22/04/28 17:31:18 INFO [task-result-getter-1] TaskSetManager: Finished task 34.0 in stage 8.0 (TID 317) in 41171 ms on emr-worker-6.cluster-196425 (executor 6) (48/63) 22/04/28 17:31:19 INFO [task-result-getter-2] TaskSetManager: Finished task 61.0 in stage 8.0 (TID 344) in 20648 ms on emr-worker-9.cluster-196425 (executor 3) (49/63) 22/04/28 17:31:19 INFO [task-result-getter-0] TaskSetManager: Finished task 35.0 in stage 8.0 (TID 318) in 42862 ms on emr-worker-6.cluster-196425 (executor 4) (50/63) 22/04/28 17:31:50 INFO [task-result-getter-3] TaskSetManager: Finished task 38.0 in stage 8.0 (TID 321) in 73265 ms on emr-worker-19.cluster-196425 (executor 2) (51/63) 22/04/28 17:31:53 INFO [task-result-getter-1] TaskSetManager: Finished task 7.0 in stage 8.0 (TID 290) in 96478 ms on emr-worker-9.cluster-196425 (executor 3) (52/63) 22/04/28 17:31:54 INFO [task-result-getter-2] TaskSetManager: Finished task 49.0 in stage 8.0 (TID 332) in 66906 ms on emr-worker-6.cluster-196425 (executor 6) (53/63) 22/04/28 17:31:55 INFO [task-result-getter-0] TaskSetManager: Finished task 48.0 in stage 8.0 (TID 331) in 68122 ms on emr-worker-6.cluster-196425 (executor 4) (54/63) 22/04/28 17:32:07 INFO [task-result-getter-3] TaskSetManager: Finished task 44.0 in stage 8.0 (TID 327) in 84327 ms on emr-worker-6.cluster-196425 (executor 6) (55/63) 22/04/28 17:32:08 INFO [task-result-getter-1] TaskSetManager: Finished task 60.0 in stage 8.0 (TID 343) in 69804 ms on emr-worker-19.cluster-196425 (executor 5) (56/63) 22/04/28 17:32:47 INFO [task-result-getter-2] TaskSetManager: Finished task 39.0 in stage 8.0 (TID 322) in 129247 ms on emr-worker-19.cluster-196425 (executor 2) (57/63) 22/04/28 17:32:59 INFO [task-result-getter-0] TaskSetManager: Finished task 37.0 in stage 8.0 (TID 320) in 141908 ms on emr-worker-6.cluster-196425 (executor 7) (58/63) 22/04/28 17:33:01 INFO [task-result-getter-3] TaskSetManager: Finished task 52.0 in stage 8.0 (TID 335) in 127408 ms on emr-worker-6.cluster-196425 (executor 7) (59/63) 22/04/28 17:33:15 INFO [task-result-getter-1] TaskSetManager: Finished task 59.0 in stage 8.0 (TID 342) in 137455 ms on emr-worker-19.cluster-196425 (executor 5) (60/63) 22/04/28 17:33:17 INFO [task-result-getter-2] TaskSetManager: Finished task 57.0 in stage 8.0 (TID 340) in 140317 ms on emr-worker-6.cluster-196425 (executor 4) (61/63) 22/04/28 17:33:19 INFO [task-result-getter-0] TaskSetManager: Finished task 55.0 in stage 8.0 (TID 338) in 144874 ms on emr-worker-6.cluster-196425 (executor 6) (62/63) 22/04/28 17:33:57 INFO [task-result-getter-3] TaskSetManager: Finished task 62.0 in stage 8.0 (TID 345) in 177415 ms on emr-worker-19.cluster-196425 (executor 2) (63/63) 22/04/28 17:33:57 INFO [task-result-getter-3] YarnScheduler: Removed TaskSet 8.0, whose tasks have all completed, from pool 22/04/28 17:33:57 INFO [dag-scheduler-event-loop] DAGScheduler: ResultStage 8 (foreachPartition at VerticesProcessor.scala:181) finished in 224.497 s 22/04/28 17:33:57 INFO [main] DAGScheduler: Job 5 finished: foreachPartition at VerticesProcessor.scala:181, took 224.512205 s 22/04/28 17:33:57 INFO [main] Exchange$: import for tag vt_user cost time: 264.25 s 22/04/28 17:33:57 INFO [main] Exchange$: SST-Import: failure.vt_user: 0 22/04/28 17:33:57 WARN [main] Exchange$: Edge is not defined 22/04/28 17:33:57 INFO [main] SparkUI: Stopped Spark web UI at http://172.16.67.122:4040 22/04/28 17:33:57 INFO [YARN application state monitor] YarnClientSchedulerBackend: Interrupting monitor thread 22/04/28 17:33:58 INFO [main] YarnClientSchedulerBackend: Shutting down all executors 22/04/28 17:33:58 INFO [dispatcher-event-loop-2] YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down 22/04/28 17:33:58 INFO [main] SchedulerExtensionServices: Stopping SchedulerExtensionServices (serviceOption=None, services=List(), started=false) 22/04/28 17:33:58 INFO [main] YarnClientSchedulerBackend: Stopped 22/04/28 17:34:03 INFO [dispatcher-event-loop-3] MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 22/04/28 17:34:03 INFO [main] MemoryStore: MemoryStore cleared 22/04/28 17:34:03 INFO [main] BlockManager: BlockManager stopped 22/04/28 17:34:03 INFO [main] BlockManagerMaster: BlockManagerMaster stopped 22/04/28 17:34:03 INFO [dispatcher-event-loop-1] OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 22/04/28 17:34:03 INFO [main] SparkContext: Successfully stopped SparkContext 22/04/28 17:34:04 INFO [pool-1-thread-1] ShutdownHookManager: Shutdown hook called 22/04/28 17:34:04 INFO [pool-1-thread-1] ShutdownHookManager: Deleting directory /tmp/spark-d452e02e-bbec-496c-8492-2be605d831df 22/04/28 17:34:04 INFO [pool-1-thread-1] ShutdownHookManager: Deleting directory /tmp/spark-ba9ecda7-c1f6-4cb6-97d5-fdc57e0f90f4 + '[' 0 -ne 0 ']'