2021-06-07 16:46:33,576 WARN (main) [Logging.scala:logWarning(66)] - The configuration key 'spark.yarn.driver.memoryOverhead' has been deprecated as of Spark 2.3 and may be removed in the future. Please use the new key 'spark.driver.memoryOverhead' instead. ----------neubla的配置信息如下所示：---------- edgeSql:select id_no_applyer_tag_id, id_no_contactor_tag_id, rn as degree, ID_NO_APPLYER, ID_NO_CONTACTOR, CREATED_DATETIME, LAST_MODIFIED_DATETIME from( select applyer.id_no as id_no_applyer_tag_id, contack.id_no as id_no_contactor_tag_id, applyer.id_no ID_NO_APPLYER, contack.id_no ID_NO_CONTACTOR, unix_timestamp(nvl(contack.created_datetime,to_timestamp('9999-12-31 23:59:99','yyyy-MM-dd HH:mm:ss'))) as CREATED_DATETIME, unix_timestamp(nvl(contack.last_modified_datetime,to_timestamp('9999-12-31 23:59:99','yyyy-MM-dd HH:mm:ss'))) as LAST_MODIFIED_DATETIME, row_number() over(partition by applyer.id_no,contack.id_no order by contack.last_modified_datetime desc) as rn from MERGE_TM_APPLY_INFO_ZAPS_DCN as applyer inner join MERGE_TM_CONTACT_ZAPS_ZDCN as contack on applyer.app_no=contack.app_no where applyer.ds='20210427' and contack.ds='20210427' and applyer.id_no is not null and contack.id_no is not null )t where rn=1 edgeName:APPLYER2CONTACTOR srcIdField:id_no_applyer_tag_id dstIdField:id_no_contactor_tag_id batch：2000 ------------------------------------------- 2021-06-07 16:46:35,176 WARN (main) [Logging.scala:logWarning(66)] - The configuration key 'spark.yarn.executor.memoryOverhead' has been deprecated as of Spark 2.3 and may be removed in the future. Please use the new key 'spark.executor.memoryOverhead' instead. 2021-06-07 16:46:35,177 WARN (main) [Logging.scala:logWarning(66)] - The configuration key 'spark.yarn.driver.memoryOverhead' has been deprecated as of Spark 2.3 and may be removed in the future. Please use the new key 'spark.driver.memoryOverhead' instead. 2021-06-07 16:46:35,190 INFO (main) [Logging.scala:logInfo(54)] - Running Spark version 2.4.3 2021-06-07 16:46:35,201 WARN (main) [Logging.scala:logWarning(66)] - Note that spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone/kubernetes and LOCAL_DIRS in YARN). 2021-06-07 16:46:35,205 WARN (main) [Logging.scala:logWarning(66)] - Detected deprecated memory fraction settings: [spark.storage.memoryFraction]. As of Spark 1.6, execution and storage memory management are unified. All memory fractions used in the old model are now deprecated and no longer read. If you wish to use the old memory management, you may explicitly enable `spark.memory.useLegacyMode` (not recommended). 2021-06-07 16:46:35,209 INFO (main) [Logging.scala:logInfo(54)] - Submitted application: hive2nebula 2021-06-07 16:46:35,245 INFO (main) [Logging.scala:logInfo(54)] - Changing view acls to: hduser0334 2021-06-07 16:46:35,245 INFO (main) [Logging.scala:logInfo(54)] - Changing modify acls to: hduser0334 2021-06-07 16:46:35,245 INFO (main) [Logging.scala:logInfo(54)] - Changing view acls groups to: 2021-06-07 16:46:35,246 INFO (main) [Logging.scala:logInfo(54)] - Changing modify acls groups to: 2021-06-07 16:46:35,246 INFO (main) [Logging.scala:logInfo(54)] - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hduser0334); groups with view permissions: Set(); users with modify permissions: Set(hduser0334); groups with modify permissions: Set() 2021-06-07 16:46:35,441 INFO (main) [Logging.scala:logInfo(54)] - Successfully started service 'sparkDriver' on port 32832. 2021-06-07 16:46:35,457 INFO (main) [Logging.scala:logInfo(54)] - Registering MapOutputTracker 2021-06-07 16:46:35,469 INFO (main) [Logging.scala:logInfo(54)] - Registering BlockManagerMaster 2021-06-07 16:46:35,472 INFO (main) [Logging.scala:logInfo(54)] - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 2021-06-07 16:46:35,473 INFO (main) [Logging.scala:logInfo(54)] - BlockManagerMasterEndpoint up 2021-06-07 16:46:35,479 INFO (main) [Logging.scala:logInfo(54)] - Created local directory at /appcom/logs/spark/tmp/blockmgr-66aa398a-9c04-42b2-a584-ba2b8b772237 2021-06-07 16:46:35,489 INFO (main) [Logging.scala:logInfo(54)] - MemoryStore started with capacity 366.3 MB 2021-06-07 16:46:35,533 INFO (main) [Logging.scala:logInfo(54)] - Registering OutputCommitCoordinator 2021-06-07 16:46:35,682 INFO (main) [Logging.scala:logInfo(54)] - Successfully started service 'SparkUI' on port 4040. 2021-06-07 16:46:35,716 INFO (main) [Logging.scala:logInfo(54)] - Bound SparkUI to 0.0.0.0, and started at http://bdphdp07jobs14:4040 2021-06-07 16:46:35,739 INFO (main) [Logging.scala:logInfo(54)] - Added JAR viewfs:///apps-data/hduser0334/nebula-exchange/nebula-sync-2.0.1.jar at viewfs:///apps-data/hduser0334/nebula-exchange/nebula-sync-2.0.1.jar with timestamp 1623055595738 2021-06-07 16:46:35,942 INFO (main) [ConfiguredRMFailoverProxyProvider.java:performFailover(100)] - Failing over to rm2 2021-06-07 16:46:35,953 INFO (main) [Logging.scala:logInfo(54)] - Requesting a new application from cluster with 22 NodeManagers 2021-06-07 16:46:35,988 INFO (main) [Logging.scala:logInfo(54)] - Verifying our application has not requested more than the maximum memory capability of the cluster (20480 MB per container) 2021-06-07 16:46:35,988 INFO (main) [Logging.scala:logInfo(54)] - Will allocate AM container, with 1408 MB memory including 384 MB overhead 2021-06-07 16:46:35,989 INFO (main) [Logging.scala:logInfo(54)] - Setting up container launch context for our AM 2021-06-07 16:46:35,990 INFO (main) [Logging.scala:logInfo(54)] - Setting up the launch environment for our AM container 2021-06-07 16:46:35,995 INFO (main) [Logging.scala:logInfo(54)] - Preparing resources for our AM container 2021-06-07 16:46:36,015 WARN (main) [Logging.scala:logWarning(66)] - Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. 2021-06-07 16:46:37,950 INFO (main) [Logging.scala:logInfo(54)] - Uploading resource file:/appcom/logs/spark/tmp/spark-6f7f1690-5162-48e6-a862-359e0ab0c653/__spark_libs__1272496730700104025.zip -> viewfs://bdphdp07/tmp/hduser0334/spark-stagingdir/.sparkStaging/application_1616590772587_981117/__spark_libs__1272496730700104025.zip 2021-06-07 16:46:45,310 INFO (main) [Logging.scala:logInfo(54)] - Uploading resource file:/appcom/logs/spark/tmp/spark-6f7f1690-5162-48e6-a862-359e0ab0c653/__spark_conf__6038093923542428052.zip -> viewfs://bdphdp07/tmp/hduser0334/spark-stagingdir/.sparkStaging/application_1616590772587_981117/__spark_conf__.zip 2021-06-07 16:46:45,422 INFO (main) [Logging.scala:logInfo(54)] - Changing view acls to: hduser0334 2021-06-07 16:46:45,422 INFO (main) [Logging.scala:logInfo(54)] - Changing modify acls to: hduser0334 2021-06-07 16:46:45,423 INFO (main) [Logging.scala:logInfo(54)] - Changing view acls groups to: 2021-06-07 16:46:45,423 INFO (main) [Logging.scala:logInfo(54)] - Changing modify acls groups to: 2021-06-07 16:46:45,423 INFO (main) [Logging.scala:logInfo(54)] - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hduser0334); groups with view permissions: Set(); users with modify permissions: Set(hduser0334); groups with modify permissions: Set() 2021-06-07 16:46:46,115 INFO (main) [Logging.scala:logInfo(54)] - Submitting application application_1616590772587_981117 to ResourceManager 2021-06-07 16:46:46,142 INFO (main) [YarnClientImpl.java:submitApplication(273)] - Submitted application application_1616590772587_981117 2021-06-07 16:46:46,144 INFO (main) [Logging.scala:logInfo(54)] - Starting Yarn extension services with app application_1616590772587_981117 and attemptId None 2021-06-07 16:46:47,151 INFO (main) [Logging.scala:logInfo(54)] - Application report for application_1616590772587_981117 (state: ACCEPTED) 2021-06-07 16:46:47,154 INFO (main) [Logging.scala:logInfo(54)] - client token: N/A diagnostics: N/A ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: root.queue_3399_01 start time: 1623055606128 final status: UNDEFINED tracking URL: http://bdphdp070002:8088/proxy/application_1616590772587_981117/ user: hduser0334 2021-06-07 16:46:48,161 INFO (main) [Logging.scala:logInfo(54)] - Application report for application_1616590772587_981117 (state: ACCEPTED) 2021-06-07 16:46:49,165 INFO (main) [Logging.scala:logInfo(54)] - Application report for application_1616590772587_981117 (state: ACCEPTED) 2021-06-07 16:46:50,168 INFO (main) [Logging.scala:logInfo(54)] - Application report for application_1616590772587_981117 (state: ACCEPTED) 2021-06-07 16:46:51,172 INFO (main) [Logging.scala:logInfo(54)] - Application report for application_1616590772587_981117 (state: ACCEPTED) 2021-06-07 16:46:51,976 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> bdphdp070003,bdphdp070002, PROXY_URI_BASES -> http://bdphdp070003:8088/proxy/application_1616590772587_981117,http://bdphdp070002:8088/proxy/application_1616590772587_981117, RM_HA_URLS -> bdphdp070003:8088,bdphdp070002:8088), /proxy/application_1616590772587_981117 2021-06-07 16:46:51,977 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /jobs, /jobs/json, /jobs/job, /jobs/job/json, /stages, /stages/json, /stages/stage, /stages/stage/json, /stages/pool, /stages/pool/json, /storage, /storage/json, /storage/rdd, /storage/rdd/json, /environment, /environment/json, /executors, /executors/json, /executors/threadDump, /executors/threadDump/json, /static, /, /api, /jobs/job/kill, /stages/stage/kill. 2021-06-07 16:46:52,120 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - ApplicationMaster registered as NettyRpcEndpointRef(spark-client://YarnAM) 2021-06-07 16:46:52,175 INFO (main) [Logging.scala:logInfo(54)] - Application report for application_1616590772587_981117 (state: RUNNING) 2021-06-07 16:46:52,176 INFO (main) [Logging.scala:logInfo(54)] - client token: N/A diagnostics: N/A ApplicationMaster host: 10.107.108.135 ApplicationMaster RPC port: -1 queue: root.queue_3399_01 start time: 1623055606128 final status: UNDEFINED tracking URL: http://bdphdp070002:8088/proxy/application_1616590772587_981117/ user: hduser0334 2021-06-07 16:46:52,177 INFO (main) [Logging.scala:logInfo(54)] - Application application_1616590772587_981117 has started running. 2021-06-07 16:46:52,182 INFO (main) [Logging.scala:logInfo(54)] - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 16197. 2021-06-07 16:46:52,182 INFO (main) [Logging.scala:logInfo(54)] - Server created on bdphdp07jobs14:16197 2021-06-07 16:46:52,183 INFO (main) [Logging.scala:logInfo(54)] - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 2021-06-07 16:46:52,202 INFO (main) [Logging.scala:logInfo(54)] - Registering BlockManager BlockManagerId(driver, bdphdp07jobs14, 16197, None) 2021-06-07 16:46:52,204 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Registering block manager bdphdp07jobs14:16197 with 366.3 MB RAM, BlockManagerId(driver, bdphdp07jobs14, 16197, None) 2021-06-07 16:46:52,206 INFO (main) [Logging.scala:logInfo(54)] - Registered BlockManager BlockManagerId(driver, bdphdp07jobs14, 16197, None) 2021-06-07 16:46:52,207 INFO (main) [Logging.scala:logInfo(54)] - Initialized BlockManager: BlockManagerId(driver, bdphdp07jobs14, 16197, None) 2021-06-07 16:46:52,296 INFO (main) [Logging.scala:logInfo(54)] - Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /metrics/json. 2021-06-07 16:46:52,417 INFO (main) [Logging.scala:logInfo(54)] - Logging events to viewfs://bdphdp07/sparkeventlogs/application_1616590772587_981117.lz4 2021-06-07 16:46:56,281 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.107.119.198:36674) with ID 2 2021-06-07 16:46:56,393 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Registering block manager bdphdp070025:18606 with 1048.8 MB RAM, BlockManagerId(2, bdphdp070025, 18606, None) 2021-06-07 16:46:56,899 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.107.97.188:62930) with ID 1 2021-06-07 16:46:56,931 INFO (main) [Logging.scala:logInfo(54)] - SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8 2021-06-07 16:46:56,946 INFO (main) [Logging.scala:logInfo(54)] - loading hive config file: file:/appcom/config/spark-config/spark-submit/hive-site.xml 2021-06-07 16:46:56,968 INFO (main) [Logging.scala:logInfo(54)] - spark.sql.warehouse.dir is not set, but hive.metastore.warehouse.dir is set. Setting spark.sql.warehouse.dir to the value of hive.metastore.warehouse.dir ('/user/hive/warehouse/hadoop'). 2021-06-07 16:46:56,968 INFO (main) [Logging.scala:logInfo(54)] - Warehouse path is '/user/hive/warehouse/hadoop'. 2021-06-07 16:46:56,975 INFO (main) [Logging.scala:logInfo(54)] - Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL. 2021-06-07 16:46:56,976 INFO (main) [Logging.scala:logInfo(54)] - Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/json. 2021-06-07 16:46:56,977 INFO (main) [Logging.scala:logInfo(54)] - Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/execution. 2021-06-07 16:46:56,977 INFO (main) [Logging.scala:logInfo(54)] - Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/execution/json. 2021-06-07 16:46:56,978 INFO (main) [Logging.scala:logInfo(54)] - Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /static/sql. 2021-06-07 16:46:57,037 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Registering block manager bdphdp070016:35542 with 1048.8 MB RAM, BlockManagerId(1, bdphdp070016, 35542, None) 2021-06-07 16:46:57,358 INFO (main) [Logging.scala:logInfo(54)] - Registered StateStoreCoordinator endpoint 2021-06-07 16:46:57,363 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: use ccpd_z_ods_safe 2021-06-07 16:46:57,363 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: use ccpd_z_ods_safe 2021-06-07 16:46:57,735 INFO (main) [Logging.scala:logInfo(54)] - Initializing HiveMetastoreConnection version 1.2.1 using Spark classes. 2021-06-07 16:46:58,043 WARN (main) [HiveConf.java:initialize(2762)] - HiveConf of name hive.empty.txt.generate does not exist 2021-06-07 16:46:58,043 WARN (main) [HiveConf.java:initialize(2762)] - HiveConf of name hive.empty.orc.generate does not exist 2021-06-07 16:46:58,044 WARN (main) [HiveConf.java:initialize(2762)] - HiveConf of name hive.strict.checks.type.safety does not exist 2021-06-07 16:46:58,044 WARN (main) [HiveConf.java:initialize(2762)] - HiveConf of name hive.compare.function.adapter does not exist 2021-06-07 16:46:58,044 WARN (main) [HiveConf.java:initialize(2762)] - HiveConf of name hive.exec.schema.evolution does not exist 2021-06-07 16:46:58,044 WARN (main) [HiveConf.java:initialize(2762)] - HiveConf of name hive.decimal.trailing.zero.switch does not exist 2021-06-07 16:46:58,045 INFO (main) [HiveConf.java:setupJDOConnectionPassword(2800)] - set up jdo connection password successful 2021-06-07 16:46:58,135 INFO (main) [HiveMetaStoreClient.java:open(377)] - Trying to connect to metastore with URI thrift://bdphdp070029:9083 2021-06-07 16:46:58,169 INFO (main) [HiveMetaStoreClient.java:open(494)] - Connected to metastore. 2021-06-07 16:47:20,151 INFO (main) [SessionState.java:createPath(641)] - Created local directory: /appcom/tmp/hive/7df4d976-551c-4785-ba3e-770f465f4f06_resources 2021-06-07 16:47:20,162 INFO (main) [SessionState.java:createPath(641)] - Created HDFS directory: /tmp/hduser0334/hive-hduser0334/hduser0334/7df4d976-551c-4785-ba3e-770f465f4f06 2021-06-07 16:47:20,166 INFO (main) [SessionState.java:createPath(641)] - Created local directory: /appcom/tmp/hive/hduser0334/7df4d976-551c-4785-ba3e-770f465f4f06 2021-06-07 16:47:20,177 INFO (main) [SessionState.java:createPath(641)] - Created HDFS directory: /tmp/hduser0334/hive-hduser0334/hduser0334/7df4d976-551c-4785-ba3e-770f465f4f06/_tmp_space.db 2021-06-07 16:47:20,182 INFO (main) [Logging.scala:logInfo(54)] - Warehouse location for Hive client (version 1.2.2) is /user/hive/warehouse/hadoop 2021-06-07 16:47:20,269 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: select id_no_applyer_tag_id, id_no_contactor_tag_id, rn as degree, ID_NO_APPLYER, ID_NO_CONTACTOR, CREATED_DATETIME, LAST_MODIFIED_DATETIME from( select applyer.id_no as id_no_applyer_tag_id, contack.id_no as id_no_contactor_tag_id, applyer.id_no ID_NO_APPLYER, contack.id_no ID_NO_CONTACTOR, unix_timestamp(nvl(contack.created_datetime,to_timestamp('9999-12-31 23:59:99','yyyy-MM-dd HH:mm:ss'))) as CREATED_DATETIME, unix_timestamp(nvl(contack.last_modified_datetime,to_timestamp('9999-12-31 23:59:99','yyyy-MM-dd HH:mm:ss'))) as LAST_MODIFIED_DATETIME, row_number() over(partition by applyer.id_no,contack.id_no order by contack.last_modified_datetime desc) as rn from MERGE_TM_APPLY_INFO_ZAPS_DCN as applyer inner join MERGE_TM_CONTACT_ZAPS_ZDCN as contack on applyer.app_no=contack.app_no where applyer.ds='20210427' and contack.ds='20210427' and applyer.id_no is not null and contack.id_no is not null )t where rn=1 2021-06-07 16:47:20,269 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: select id_no_applyer_tag_id, id_no_contactor_tag_id, rn as degree, ID_NO_APPLYER, ID_NO_CONTACTOR, CREATED_DATETIME, LAST_MODIFIED_DATETIME from( select applyer.id_no as id_no_applyer_tag_id, contack.id_no as id_no_contactor_tag_id, applyer.id_no ID_NO_APPLYER, contack.id_no ID_NO_CONTACTOR, unix_timestamp(nvl(contack.created_datetime,to_timestamp('9999-12-31 23:59:99','yyyy-MM-dd HH:mm:ss'))) as CREATED_DATETIME, unix_timestamp(nvl(contack.last_modified_datetime,to_timestamp('9999-12-31 23:59:99','yyyy-MM-dd HH:mm:ss'))) as LAST_MODIFIED_DATETIME, row_number() over(partition by applyer.id_no,contack.id_no order by contack.last_modified_datetime desc) as rn from MERGE_TM_APPLY_INFO_ZAPS_DCN as applyer inner join MERGE_TM_CONTACT_ZAPS_ZDCN as contack on applyer.app_no=contack.app_no where applyer.ds='20210427' and contack.ds='20210427' and applyer.id_no is not null and contack.id_no is not null )t where rn=1 2021-06-07 16:47:20,577 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,582 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,583 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,583 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,583 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,584 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,584 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,585 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,585 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,586 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,586 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,586 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,587 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,587 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,587 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,587 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,588 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,588 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,589 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,589 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,589 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,590 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,590 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,590 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,591 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,591 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,591 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,592 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,592 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,593 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,593 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,594 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,594 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,594 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,595 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: int 2021-06-07 16:47:20,595 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,595 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,596 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,596 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: int 2021-06-07 16:47:20,596 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,597 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: int 2021-06-07 16:47:20,597 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: timestamp 2021-06-07 16:47:20,597 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: timestamp 2021-06-07 16:47:20,598 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,598 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,598 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,599 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,599 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,599 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,600 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,600 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,600 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,601 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,601 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,601 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,602 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,602 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,602 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,602 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,603 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,603 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,603 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,604 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,604 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,604 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,605 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,605 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,605 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,605 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,606 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,606 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,606 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,607 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,607 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,607 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,607 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,608 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,608 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,608 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,609 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,609 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,609 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,609 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,610 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,610 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,610 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: int 2021-06-07 16:47:20,610 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,611 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,611 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,611 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,612 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,612 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,612 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,612 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,613 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,614 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,614 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,622 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: array 2021-06-07 16:47:20,827 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,827 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,827 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,828 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,828 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,828 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,829 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: timestamp 2021-06-07 16:47:20,829 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: timestamp 2021-06-07 16:47:20,829 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,829 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,830 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,830 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,830 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,830 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,831 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:21,263 WARN (main) [Logging.scala:logWarning(66)] - Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.debug.maxToStringFields' in SparkEnv.conf. 2021-06-07 16:47:21,471 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 1 2021-06-07 16:47:21,472 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 4 2021-06-07 16:47:21,472 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 0 2021-06-07 16:47:21,472 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 2 2021-06-07 16:47:21,472 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 3 2021-06-07 16:47:21,472 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 5 2021-06-07 16:47:21,569 INFO (main) [Logging.scala:logInfo(54)] - Code generated in 191.542498 ms 2021-06-07 16:47:21,645 INFO (main) [Logging.scala:logInfo(54)] - Code generated in 21.677265 ms 2021-06-07 16:47:21,669 INFO (main) [Logging.scala:logInfo(54)] - Code generated in 15.050502 ms 2021-06-07 16:47:21,701 INFO (main) [Logging.scala:logInfo(54)] - Code generated in 20.598301 ms 2021-06-07 16:47:21,712 INFO (main) [Logging.scala:logInfo(54)] - Code generated in 8.807863 ms 2021-06-07 16:47:21,720 INFO (main) [Logging.scala:logInfo(54)] - Code generated in 5.98673 ms 2021-06-07 16:47:21,916 INFO (main) [Logging.scala:logInfo(54)] - Block broadcast_0 stored as values in memory (estimated size 1163.6 KB, free 365.2 MB) 2021-06-07 16:47:22,079 INFO (main) [Logging.scala:logInfo(54)] - Block broadcast_0_piece0 stored as bytes in memory (estimated size 71.2 KB, free 365.1 MB) 2021-06-07 16:47:22,081 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Added broadcast_0_piece0 in memory on bdphdp07jobs14:16197 (size: 71.2 KB, free: 366.2 MB) 2021-06-07 16:47:22,083 INFO (main) [Logging.scala:logInfo(54)] - Created broadcast 0 from 2021-06-07 16:47:22,406 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,407 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,407 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,407 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,407 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,408 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,408 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,408 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,409 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,409 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,409 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,409 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,410 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,410 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,410 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,411 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,411 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,411 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,411 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,412 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,412 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,412 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,413 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,413 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,413 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,414 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,414 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,414 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,414 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,415 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,415 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,415 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,416 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,416 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,416 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: int 2021-06-07 16:47:22,417 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,417 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,417 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,417 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: int 2021-06-07 16:47:22,418 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,418 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: int 2021-06-07 16:47:22,418 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: timestamp 2021-06-07 16:47:22,419 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: timestamp 2021-06-07 16:47:22,419 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,419 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,419 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,420 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,420 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,420 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,421 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,421 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,421 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,421 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,422 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,422 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,422 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,423 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,423 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,423 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,423 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,424 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,424 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,424 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,425 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,425 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,425 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,425 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,426 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,426 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,426 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,426 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,427 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,427 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,427 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,427 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,428 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,428 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,428 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,429 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,429 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,429 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,429 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,429 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,430 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,430 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,430 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: int 2021-06-07 16:47:22,431 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,431 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,431 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,431 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,432 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,432 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,432 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,432 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,433 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,433 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,433 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,967 WARN (main) [HiveConf.java:initialize(2762)] - HiveConf of name hive.empty.txt.generate does not exist 2021-06-07 16:47:22,968 WARN (main) [HiveConf.java:initialize(2762)] - HiveConf of name hive.empty.orc.generate does not exist 2021-06-07 16:47:22,968 WARN (main) [HiveConf.java:initialize(2762)] - HiveConf of name hive.strict.checks.type.safety does not exist 2021-06-07 16:47:22,968 WARN (main) [HiveConf.java:initialize(2762)] - HiveConf of name hive.compare.function.adapter does not exist 2021-06-07 16:47:22,969 WARN (main) [HiveConf.java:initialize(2762)] - HiveConf of name hive.exec.schema.evolution does not exist 2021-06-07 16:47:22,969 WARN (main) [HiveConf.java:initialize(2762)] - HiveConf of name hive.decimal.trailing.zero.switch does not exist 2021-06-07 16:47:22,970 INFO (main) [HiveConf.java:setupJDOConnectionPassword(2800)] - set up jdo connection password successful 2021-06-07 16:47:23,115 INFO (main) [FileInputFormat.java:listStatus(249)] - Total input paths to process : 1 2021-06-07 16:47:23,131 INFO (main) [FileInputFormat.java:listStatus(249)] - Total input paths to process : 1 2021-06-07 16:47:23,290 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Registering RDD 9 (show at NebulaSparkEdgeWriter.scala:83) 2021-06-07 16:47:23,290 INFO (main) [Logging.scala:logInfo(54)] - Code generated in 149.759248 ms 2021-06-07 16:47:23,296 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Got map stage job 0 (show at NebulaSparkEdgeWriter.scala:83) with 40 output partitions 2021-06-07 16:47:23,297 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Final stage: ShuffleMapStage 0 (show at NebulaSparkEdgeWriter.scala:83) 2021-06-07 16:47:23,298 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Parents of final stage: List() 2021-06-07 16:47:23,299 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Missing parents: List() 2021-06-07 16:47:23,303 INFO (main) [Logging.scala:logInfo(54)] - Block broadcast_1 stored as values in memory (estimated size 1158.3 KB, free 364.0 MB) 2021-06-07 16:47:23,303 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting ShuffleMapStage 0 (MapPartitionsRDD[9] at show at NebulaSparkEdgeWriter.scala:83), which has no missing parents 2021-06-07 16:47:23,337 INFO (main) [Logging.scala:logInfo(54)] - Block broadcast_1_piece0 stored as bytes in memory (estimated size 70.5 KB, free 363.9 MB) 2021-06-07 16:47:23,338 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Added broadcast_1_piece0 in memory on bdphdp07jobs14:16197 (size: 70.5 KB, free: 366.2 MB) 2021-06-07 16:47:23,339 INFO (main) [Logging.scala:logInfo(54)] - Created broadcast 1 from 2021-06-07 16:47:23,350 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_2 stored as values in memory (estimated size 21.6 KB, free 363.9 MB) 2021-06-07 16:47:23,363 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_2_piece0 stored as bytes in memory (estimated size 8.5 KB, free 363.9 MB) 2021-06-07 16:47:23,364 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Added broadcast_2_piece0 in memory on bdphdp07jobs14:16197 (size: 8.5 KB, free: 366.2 MB) 2021-06-07 16:47:23,365 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Created broadcast 2 from broadcast at DAGScheduler.scala:1161 2021-06-07 16:47:23,377 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting 40 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[9] at show at NebulaSparkEdgeWriter.scala:83) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14)) 2021-06-07 16:47:23,378 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Adding task set 0.0 with 40 tasks 2021-06-07 16:47:23,496 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Starting task 0.0 in stage 0.0 (TID 0, bdphdp070016, executor 1, partition 0, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:23,498 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Starting task 1.0 in stage 0.0 (TID 1, bdphdp070025, executor 2, partition 1, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:23,498 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Starting task 2.0 in stage 0.0 (TID 2, bdphdp070016, executor 1, partition 2, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:23,499 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Starting task 3.0 in stage 0.0 (TID 3, bdphdp070025, executor 2, partition 3, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:23,605 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:23,606 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:23,606 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:23,606 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:23,607 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:23,607 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:23,607 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: timestamp 2021-06-07 16:47:23,607 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: timestamp 2021-06-07 16:47:23,608 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:23,608 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:23,608 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:23,609 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:23,609 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:23,609 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:23,610 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:23,978 INFO (main) [FileInputFormat.java:listStatus(249)] - Total input paths to process : 1 2021-06-07 16:47:23,980 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Registering RDD 16 (show at NebulaSparkEdgeWriter.scala:83) 2021-06-07 16:47:23,981 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Got map stage job 1 (show at NebulaSparkEdgeWriter.scala:83) with 20 output partitions 2021-06-07 16:47:23,981 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Final stage: ShuffleMapStage 1 (show at NebulaSparkEdgeWriter.scala:83) 2021-06-07 16:47:23,981 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Parents of final stage: List() 2021-06-07 16:47:23,981 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Missing parents: List() 2021-06-07 16:47:23,981 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting ShuffleMapStage 1 (MapPartitionsRDD[16] at show at NebulaSparkEdgeWriter.scala:83), which has no missing parents 2021-06-07 16:47:23,987 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_3 stored as values in memory (estimated size 14.9 KB, free 363.9 MB) 2021-06-07 16:47:23,993 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_3_piece0 stored as bytes in memory (estimated size 7.2 KB, free 363.8 MB) 2021-06-07 16:47:23,994 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Added broadcast_3_piece0 in memory on bdphdp07jobs14:16197 (size: 7.2 KB, free: 366.1 MB) 2021-06-07 16:47:23,994 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Created broadcast 3 from broadcast at DAGScheduler.scala:1161 2021-06-07 16:47:23,995 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting 20 missing tasks from ShuffleMapStage 1 (MapPartitionsRDD[16] at show at NebulaSparkEdgeWriter.scala:83) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14)) 2021-06-07 16:47:23,996 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Adding task set 1.0 with 20 tasks 2021-06-07 16:47:26,250 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Added broadcast_2_piece0 in memory on bdphdp070016:35542 (size: 8.5 KB, free: 1048.8 MB) 2021-06-07 16:47:26,642 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Added broadcast_0_piece0 in memory on bdphdp070016:35542 (size: 71.2 KB, free: 1048.7 MB) 2021-06-07 16:47:26,659 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Added broadcast_2_piece0 in memory on bdphdp070025:18606 (size: 8.5 KB, free: 1048.8 MB) 2021-06-07 16:47:27,067 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Added broadcast_0_piece0 in memory on bdphdp070025:18606 (size: 71.2 KB, free: 1048.7 MB) 2021-06-07 16:47:27,503 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 4.0 in stage 0.0 (TID 4, bdphdp070016, executor 1, partition 4, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,505 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 5.0 in stage 0.0 (TID 5, bdphdp070016, executor 1, partition 5, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,532 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 2.0 in stage 0.0 (TID 2) in 4033 ms on bdphdp070016 (executor 1) (1/40) 2021-06-07 16:47:27,533 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 0.0 in stage 0.0 (TID 0) in 4048 ms on bdphdp070016 (executor 1) (2/40) 2021-06-07 16:47:27,584 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Starting task 6.0 in stage 0.0 (TID 6, bdphdp070016, executor 1, partition 6, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,587 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 5.0 in stage 0.0 (TID 5) in 83 ms on bdphdp070016 (executor 1) (3/40) 2021-06-07 16:47:27,589 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Starting task 7.0 in stage 0.0 (TID 7, bdphdp070016, executor 1, partition 7, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,592 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 4.0 in stage 0.0 (TID 4) in 90 ms on bdphdp070016 (executor 1) (4/40) 2021-06-07 16:47:27,640 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 8.0 in stage 0.0 (TID 8, bdphdp070016, executor 1, partition 8, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,641 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 6.0 in stage 0.0 (TID 6) in 57 ms on bdphdp070016 (executor 1) (5/40) 2021-06-07 16:47:27,645 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 9.0 in stage 0.0 (TID 9, bdphdp070016, executor 1, partition 9, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,646 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 7.0 in stage 0.0 (TID 7) in 56 ms on bdphdp070016 (executor 1) (6/40) 2021-06-07 16:47:27,699 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Starting task 10.0 in stage 0.0 (TID 10, bdphdp070016, executor 1, partition 10, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,700 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 9.0 in stage 0.0 (TID 9) in 55 ms on bdphdp070016 (executor 1) (7/40) 2021-06-07 16:47:27,700 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Starting task 11.0 in stage 0.0 (TID 11, bdphdp070016, executor 1, partition 11, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,701 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 8.0 in stage 0.0 (TID 8) in 61 ms on bdphdp070016 (executor 1) (8/40) 2021-06-07 16:47:27,748 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 12.0 in stage 0.0 (TID 12, bdphdp070016, executor 1, partition 12, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,749 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 10.0 in stage 0.0 (TID 10) in 50 ms on bdphdp070016 (executor 1) (9/40) 2021-06-07 16:47:27,761 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 13.0 in stage 0.0 (TID 13, bdphdp070016, executor 1, partition 13, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,761 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 11.0 in stage 0.0 (TID 11) in 61 ms on bdphdp070016 (executor 1) (10/40) 2021-06-07 16:47:27,814 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 14.0 in stage 0.0 (TID 14, bdphdp070016, executor 1, partition 14, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,814 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 12.0 in stage 0.0 (TID 12) in 66 ms on bdphdp070016 (executor 1) (11/40) 2021-06-07 16:47:27,819 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Starting task 15.0 in stage 0.0 (TID 15, bdphdp070016, executor 1, partition 15, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,820 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 13.0 in stage 0.0 (TID 13) in 60 ms on bdphdp070016 (executor 1) (12/40) 2021-06-07 16:47:27,856 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 16.0 in stage 0.0 (TID 16, bdphdp070016, executor 1, partition 16, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,857 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 14.0 in stage 0.0 (TID 14) in 43 ms on bdphdp070016 (executor 1) (13/40) 2021-06-07 16:47:27,863 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 17.0 in stage 0.0 (TID 17, bdphdp070016, executor 1, partition 17, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,864 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 15.0 in stage 0.0 (TID 15) in 46 ms on bdphdp070016 (executor 1) (14/40) 2021-06-07 16:47:27,901 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 18.0 in stage 0.0 (TID 18, bdphdp070016, executor 1, partition 18, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,901 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 16.0 in stage 0.0 (TID 16) in 45 ms on bdphdp070016 (executor 1) (15/40) 2021-06-07 16:47:27,910 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Starting task 19.0 in stage 0.0 (TID 19, bdphdp070016, executor 1, partition 19, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,910 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 17.0 in stage 0.0 (TID 17) in 47 ms on bdphdp070016 (executor 1) (16/40) 2021-06-07 16:47:27,949 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 20.0 in stage 0.0 (TID 20, bdphdp070016, executor 1, partition 20, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,950 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 18.0 in stage 0.0 (TID 18) in 49 ms on bdphdp070016 (executor 1) (17/40) 2021-06-07 16:47:27,953 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 21.0 in stage 0.0 (TID 21, bdphdp070025, executor 2, partition 21, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,953 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 1.0 in stage 0.0 (TID 1) in 4456 ms on bdphdp070025 (executor 2) (18/40) 2021-06-07 16:47:27,954 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 22.0 in stage 0.0 (TID 22, bdphdp070025, executor 2, partition 22, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,954 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 3.0 in stage 0.0 (TID 3) in 4456 ms on bdphdp070025 (executor 2) (19/40) 2021-06-07 16:47:27,955 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Starting task 23.0 in stage 0.0 (TID 23, bdphdp070016, executor 1, partition 23, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,956 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 19.0 in stage 0.0 (TID 19) in 47 ms on bdphdp070016 (executor 1) (20/40) 2021-06-07 16:47:27,989 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 24.0 in stage 0.0 (TID 24, bdphdp070016, executor 1, partition 24, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,989 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 20.0 in stage 0.0 (TID 20) in 40 ms on bdphdp070016 (executor 1) (21/40) 2021-06-07 16:47:27,990 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 25.0 in stage 0.0 (TID 25, bdphdp070016, executor 1, partition 25, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,990 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 23.0 in stage 0.0 (TID 23) in 35 ms on bdphdp070016 (executor 1) (22/40) 2021-06-07 16:47:28,006 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 26.0 in stage 0.0 (TID 26, bdphdp070025, executor 2, partition 26, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:28,006 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 22.0 in stage 0.0 (TID 22) in 53 ms on bdphdp070025 (executor 2) (23/40) 2021-06-07 16:47:28,007 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 27.0 in stage 0.0 (TID 27, bdphdp070025, executor 2, partition 27, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:28,007 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 21.0 in stage 0.0 (TID 21) in 55 ms on bdphdp070025 (executor 2) (24/40) 2021-06-07 16:47:28,026 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 28.0 in stage 0.0 (TID 28, bdphdp070016, executor 1, partition 28, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:28,026 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 25.0 in stage 0.0 (TID 25) in 36 ms on bdphdp070016 (executor 1) (25/40) 2021-06-07 16:47:28,027 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 29.0 in stage 0.0 (TID 29, bdphdp070016, executor 1, partition 29, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:28,027 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 24.0 in stage 0.0 (TID 24) in 39 ms on bdphdp070016 (executor 1) (26/40) 2021-06-07 16:47:28,045 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Starting task 30.0 in stage 0.0 (TID 30, bdphdp070025, executor 2, partition 30, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:28,045 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 27.0 in stage 0.0 (TID 27) in 38 ms on bdphdp070025 (executor 2) (27/40) 2021-06-07 16:47:28,046 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 31.0 in stage 0.0 (TID 31, bdphdp070025, executor 2, partition 31, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:28,047 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 26.0 in stage 0.0 (TID 26) in 41 ms on bdphdp070025 (executor 2) (28/40) 2021-06-07 16:47:28,064 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 32.0 in stage 0.0 (TID 32, bdphdp070016, executor 1, partition 32, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:28,064 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 29.0 in stage 0.0 (TID 29) in 37 ms on bdphdp070016 (executor 1) (29/40) 2021-06-07 16:47:28,065 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 33.0 in stage 0.0 (TID 33, bdphdp070016, executor 1, partition 33, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:28,065 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 28.0 in stage 0.0 (TID 28) in 40 ms on bdphdp070016 (executor 1) (30/40) 2021-06-07 16:47:28,080 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Starting task 34.0 in stage 0.0 (TID 34, bdphdp070025, executor 2, partition 34, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:28,080 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 31.0 in stage 0.0 (TID 31) in 34 ms on bdphdp070025 (executor 2) (31/40) 2021-06-07 16:47:28,085 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 35.0 in stage 0.0 (TID 35, bdphdp070025, executor 2, partition 35, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:28,085 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 30.0 in stage 0.0 (TID 30) in 40 ms on bdphdp070025 (executor 2) (32/40) 2021-06-07 16:47:28,115 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 36.0 in stage 0.0 (TID 36, bdphdp070025, executor 2, partition 36, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:28,116 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 34.0 in stage 0.0 (TID 34) in 35 ms on bdphdp070025 (executor 2) (33/40) 2021-06-07 16:47:28,121 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 37.0 in stage 0.0 (TID 37, bdphdp070025, executor 2, partition 37, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:28,122 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 35.0 in stage 0.0 (TID 35) in 37 ms on bdphdp070025 (executor 2) (34/40) 2021-06-07 16:47:28,146 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Starting task 38.0 in stage 0.0 (TID 38, bdphdp070025, executor 2, partition 38, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:28,147 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 36.0 in stage 0.0 (TID 36) in 32 ms on bdphdp070025 (executor 2) (35/40) 2021-06-07 16:47:28,153 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Starting task 39.0 in stage 0.0 (TID 39, bdphdp070025, executor 2, partition 39, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:28,154 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 37.0 in stage 0.0 (TID 37) in 33 ms on bdphdp070025 (executor 2) (36/40) 2021-06-07 16:47:28,185 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 0.0 in stage 1.0 (TID 40, bdphdp070025, executor 2, partition 0, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,187 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 38.0 in stage 0.0 (TID 38) in 41 ms on bdphdp070025 (executor 2) (37/40) 2021-06-07 16:47:28,189 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 1.0 in stage 1.0 (TID 41, bdphdp070025, executor 2, partition 1, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,191 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 39.0 in stage 0.0 (TID 39) in 38 ms on bdphdp070025 (executor 2) (38/40) 2021-06-07 16:47:28,218 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Added broadcast_3_piece0 in memory on bdphdp070025:18606 (size: 7.2 KB, free: 1048.7 MB) 2021-06-07 16:47:28,247 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Added broadcast_1_piece0 in memory on bdphdp070025:18606 (size: 70.5 KB, free: 1048.6 MB) 2021-06-07 16:47:28,335 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 2.0 in stage 1.0 (TID 42, bdphdp070025, executor 2, partition 2, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,335 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 1.0 in stage 1.0 (TID 41) in 146 ms on bdphdp070025 (executor 2) (1/20) 2021-06-07 16:47:28,336 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 3.0 in stage 1.0 (TID 43, bdphdp070025, executor 2, partition 3, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,336 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 0.0 in stage 1.0 (TID 40) in 151 ms on bdphdp070025 (executor 2) (2/20) 2021-06-07 16:47:28,364 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 4.0 in stage 1.0 (TID 44, bdphdp070025, executor 2, partition 4, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,365 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 3.0 in stage 1.0 (TID 43) in 29 ms on bdphdp070025 (executor 2) (3/20) 2021-06-07 16:47:28,365 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 5.0 in stage 1.0 (TID 45, bdphdp070025, executor 2, partition 5, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,366 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 2.0 in stage 1.0 (TID 42) in 31 ms on bdphdp070025 (executor 2) (4/20) 2021-06-07 16:47:28,391 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 6.0 in stage 1.0 (TID 46, bdphdp070025, executor 2, partition 6, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,392 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 7.0 in stage 1.0 (TID 47, bdphdp070025, executor 2, partition 7, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,392 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 4.0 in stage 1.0 (TID 44) in 28 ms on bdphdp070025 (executor 2) (5/20) 2021-06-07 16:47:28,393 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 5.0 in stage 1.0 (TID 45) in 28 ms on bdphdp070025 (executor 2) (6/20) 2021-06-07 16:47:28,415 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Starting task 8.0 in stage 1.0 (TID 48, bdphdp070025, executor 2, partition 8, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,416 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 6.0 in stage 1.0 (TID 46) in 25 ms on bdphdp070025 (executor 2) (7/20) 2021-06-07 16:47:28,416 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Starting task 9.0 in stage 1.0 (TID 49, bdphdp070025, executor 2, partition 9, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,417 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 7.0 in stage 1.0 (TID 47) in 25 ms on bdphdp070025 (executor 2) (8/20) 2021-06-07 16:47:28,429 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 32.0 in stage 0.0 (TID 32) in 366 ms on bdphdp070016 (executor 1) (39/40) 2021-06-07 16:47:28,435 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 33.0 in stage 0.0 (TID 33) in 371 ms on bdphdp070016 (executor 1) (40/40) 2021-06-07 16:47:28,436 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Removed TaskSet 0.0, whose tasks have all completed, from pool 2021-06-07 16:47:28,437 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - ShuffleMapStage 0 (show at NebulaSparkEdgeWriter.scala:83) finished in 5.116 s 2021-06-07 16:47:28,438 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - looking for newly runnable stages 2021-06-07 16:47:28,438 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - running: Set(ShuffleMapStage 1) 2021-06-07 16:47:28,438 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - waiting: Set() 2021-06-07 16:47:28,439 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - failed: Set() 2021-06-07 16:47:28,439 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Starting task 10.0 in stage 1.0 (TID 50, bdphdp070025, executor 2, partition 10, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,439 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 8.0 in stage 1.0 (TID 48) in 24 ms on bdphdp070025 (executor 2) (9/20) 2021-06-07 16:47:28,440 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Starting task 11.0 in stage 1.0 (TID 51, bdphdp070025, executor 2, partition 11, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,440 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 9.0 in stage 1.0 (TID 49) in 24 ms on bdphdp070025 (executor 2) (10/20) 2021-06-07 16:47:28,462 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Starting task 12.0 in stage 1.0 (TID 52, bdphdp070025, executor 2, partition 12, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,462 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 10.0 in stage 1.0 (TID 50) in 23 ms on bdphdp070025 (executor 2) (11/20) 2021-06-07 16:47:28,464 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Starting task 13.0 in stage 1.0 (TID 53, bdphdp070025, executor 2, partition 13, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,464 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 11.0 in stage 1.0 (TID 51) in 24 ms on bdphdp070025 (executor 2) (12/20) 2021-06-07 16:47:28,483 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Starting task 14.0 in stage 1.0 (TID 54, bdphdp070025, executor 2, partition 14, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,483 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 12.0 in stage 1.0 (TID 52) in 21 ms on bdphdp070025 (executor 2) (13/20) 2021-06-07 16:47:28,484 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Starting task 15.0 in stage 1.0 (TID 55, bdphdp070025, executor 2, partition 15, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,484 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 13.0 in stage 1.0 (TID 53) in 21 ms on bdphdp070025 (executor 2) (14/20) 2021-06-07 16:47:28,505 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 16.0 in stage 1.0 (TID 56, bdphdp070025, executor 2, partition 16, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,505 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 14.0 in stage 1.0 (TID 54) in 22 ms on bdphdp070025 (executor 2) (15/20) 2021-06-07 16:47:28,506 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 17.0 in stage 1.0 (TID 57, bdphdp070025, executor 2, partition 17, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,506 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 15.0 in stage 1.0 (TID 55) in 22 ms on bdphdp070025 (executor 2) (16/20) 2021-06-07 16:47:28,526 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Starting task 18.0 in stage 1.0 (TID 58, bdphdp070025, executor 2, partition 18, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,527 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 16.0 in stage 1.0 (TID 56) in 22 ms on bdphdp070025 (executor 2) (17/20) 2021-06-07 16:47:28,527 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Starting task 19.0 in stage 1.0 (TID 59, bdphdp070025, executor 2, partition 19, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,528 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 17.0 in stage 1.0 (TID 57) in 22 ms on bdphdp070025 (executor 2) (18/20) 2021-06-07 16:47:28,547 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 18.0 in stage 1.0 (TID 58) in 21 ms on bdphdp070025 (executor 2) (19/20) 2021-06-07 16:47:28,547 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 19.0 in stage 1.0 (TID 59) in 20 ms on bdphdp070025 (executor 2) (20/20) 2021-06-07 16:47:28,547 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Removed TaskSet 1.0, whose tasks have all completed, from pool 2021-06-07 16:47:28,548 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - ShuffleMapStage 1 (show at NebulaSparkEdgeWriter.scala:83) finished in 4.563 s 2021-06-07 16:47:28,548 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - looking for newly runnable stages 2021-06-07 16:47:28,548 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - running: Set() 2021-06-07 16:47:28,548 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - waiting: Set() 2021-06-07 16:47:28,548 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - failed: Set() 2021-06-07 16:47:28,549 INFO (main) [Logging.scala:logInfo(54)] - advisoryTargetPostShuffleInputSize: 67108864, targetPostShuffleInputSize 67108864. 2021-06-07 16:47:28,596 INFO (main) [Logging.scala:logInfo(54)] - Code generated in 10.197756 ms 2021-06-07 16:47:28,614 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Registering RDD 23 (show at NebulaSparkEdgeWriter.scala:83) 2021-06-07 16:47:28,615 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Got map stage job 2 (show at NebulaSparkEdgeWriter.scala:83) with 1 output partitions 2021-06-07 16:47:28,615 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Final stage: ShuffleMapStage 4 (show at NebulaSparkEdgeWriter.scala:83) 2021-06-07 16:47:28,615 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Parents of final stage: List(ShuffleMapStage 2, ShuffleMapStage 3) 2021-06-07 16:47:28,615 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Missing parents: List() 2021-06-07 16:47:28,615 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting ShuffleMapStage 4 (MapPartitionsRDD[23] at show at NebulaSparkEdgeWriter.scala:83), which has no missing parents 2021-06-07 16:47:28,622 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_4 stored as values in memory (estimated size 36.4 KB, free 363.8 MB) 2021-06-07 16:47:28,625 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_4_piece0 stored as bytes in memory (estimated size 15.5 KB, free 363.8 MB) 2021-06-07 16:47:28,625 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Added broadcast_4_piece0 in memory on bdphdp07jobs14:16197 (size: 15.5 KB, free: 366.1 MB) 2021-06-07 16:47:28,626 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Created broadcast 4 from broadcast at DAGScheduler.scala:1161 2021-06-07 16:47:28,626 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting 1 missing tasks from ShuffleMapStage 4 (MapPartitionsRDD[23] at show at NebulaSparkEdgeWriter.scala:83) (first 15 tasks are for partitions Vector(0)) 2021-06-07 16:47:28,626 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Adding task set 4.0 with 1 tasks 2021-06-07 16:47:28,630 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 0.0 in stage 4.0 (TID 60, bdphdp070016, executor 1, partition 0, NODE_LOCAL, 8097 bytes) 2021-06-07 16:47:28,659 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Added broadcast_4_piece0 in memory on bdphdp070016:35542 (size: 15.5 KB, free: 1048.7 MB) 2021-06-07 16:47:28,759 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Asked to send map output locations for shuffle 0 to 10.107.97.188:62930 2021-06-07 16:47:29,340 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Asked to send map output locations for shuffle 1 to 10.107.97.188:62930 2021-06-07 16:47:30,027 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 0.0 in stage 4.0 (TID 60) in 1400 ms on bdphdp070016 (executor 1) (1/1) 2021-06-07 16:47:30,027 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Removed TaskSet 4.0, whose tasks have all completed, from pool 2021-06-07 16:47:30,028 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - ShuffleMapStage 4 (show at NebulaSparkEdgeWriter.scala:83) finished in 1.410 s 2021-06-07 16:47:30,028 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - looking for newly runnable stages 2021-06-07 16:47:30,028 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - running: Set() 2021-06-07 16:47:30,028 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - waiting: Set() 2021-06-07 16:47:30,028 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - failed: Set() 2021-06-07 16:47:30,028 INFO (main) [Logging.scala:logInfo(54)] - advisoryTargetPostShuffleInputSize: 67108864, targetPostShuffleInputSize 67108864. 2021-06-07 16:47:30,068 INFO (main) [Logging.scala:logInfo(54)] - Starting job: show at NebulaSparkEdgeWriter.scala:83 2021-06-07 16:47:30,071 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Got job 3 (show at NebulaSparkEdgeWriter.scala:83) with 1 output partitions 2021-06-07 16:47:30,071 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Final stage: ResultStage 8 (show at NebulaSparkEdgeWriter.scala:83) 2021-06-07 16:47:30,072 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Parents of final stage: List(ShuffleMapStage 7) 2021-06-07 16:47:30,072 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Missing parents: List() 2021-06-07 16:47:30,073 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting ResultStage 8 (MapPartitionsRDD[29] at show at NebulaSparkEdgeWriter.scala:83), which has no missing parents 2021-06-07 16:47:30,077 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_5 stored as values in memory (estimated size 38.2 KB, free 363.8 MB) 2021-06-07 16:47:30,080 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_5_piece0 stored as bytes in memory (estimated size 17.0 KB, free 363.7 MB) 2021-06-07 16:47:30,081 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Added broadcast_5_piece0 in memory on bdphdp07jobs14:16197 (size: 17.0 KB, free: 366.1 MB) 2021-06-07 16:47:30,081 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Created broadcast 5 from broadcast at DAGScheduler.scala:1161 2021-06-07 16:47:30,083 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting 1 missing tasks from ResultStage 8 (MapPartitionsRDD[29] at show at NebulaSparkEdgeWriter.scala:83) (first 15 tasks are for partitions Vector(0)) 2021-06-07 16:47:30,083 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Adding task set 8.0 with 1 tasks 2021-06-07 16:47:30,084 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 0.0 in stage 8.0 (TID 61, bdphdp070016, executor 1, partition 0, PROCESS_LOCAL, 7780 bytes) 2021-06-07 16:47:30,108 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Added broadcast_5_piece0 in memory on bdphdp070016:35542 (size: 17.0 KB, free: 1048.7 MB) 2021-06-07 16:47:30,169 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Asked to send map output locations for shuffle 2 to 10.107.97.188:62930 2021-06-07 16:47:30,327 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 0.0 in stage 8.0 (TID 61) in 244 ms on bdphdp070016 (executor 1) (1/1) 2021-06-07 16:47:30,327 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Removed TaskSet 8.0, whose tasks have all completed, from pool 2021-06-07 16:47:30,328 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - ResultStage 8 (show at NebulaSparkEdgeWriter.scala:83) finished in 0.254 s 2021-06-07 16:47:30,329 INFO (main) [Logging.scala:logInfo(54)] - Job 3 finished: show at NebulaSparkEdgeWriter.scala:83, took 0.260338 s +--------------------+----------------------+------+------------------+------------------+----------------+----------------------+ |id_no_applyer_tag_id|id_no_contactor_tag_id|degree|ID_NO_APPLYER |ID_NO_CONTACTOR |CREATED_DATETIME|LAST_MODIFIED_DATETIME| +--------------------+----------------------+------+------------------+------------------+----------------+----------------------+ |500101198506020014 |500101198506020014 |1 |500101198506020014|500101198506020014|1587455660 |1587455664 | |610112199503110857 |610112199503110857 |1 |610112199503110857|610112199503110857|1593413000 |1593413004 | +--------------------+----------------------+------+------------------+------------------+----------------+----------------------+ 2021-06-07 16:47:30,421 INFO (main) [NebulaConfig.scala:check(417)] - NebulaWriteEdgeConfig={space=dcp_zaps_adm_prod_01,edgeName=APPLYER2CONTACTOR,srcField=id_no_applyer_tag_id,srcPolicy=null，dstField=id_no_contactor_tag_id,dstPolicy=null,rankField=degree} 2021-06-07 16:47:30,457 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 66 2021-06-07 16:47:30,457 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 70 2021-06-07 16:47:30,457 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 131 2021-06-07 16:47:30,457 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 91 2021-06-07 16:47:30,457 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 136 2021-06-07 16:47:30,458 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 90 2021-06-07 16:47:30,458 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 113 2021-06-07 16:47:30,458 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 116 2021-06-07 16:47:30,458 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 109 2021-06-07 16:47:30,458 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 123 2021-06-07 16:47:30,458 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 59 2021-06-07 16:47:30,458 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 95 2021-06-07 16:47:30,458 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 122 2021-06-07 16:47:30,458 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 125 2021-06-07 16:47:30,458 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 76 2021-06-07 16:47:30,469 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Removed broadcast_5_piece0 on bdphdp07jobs14:16197 in memory (size: 17.0 KB, free: 366.1 MB) 2021-06-07 16:47:30,472 INFO (main) [NebulaDataSource.scala:createWriter(69)] - create writer 2021-06-07 16:47:30,472 INFO (main) [NebulaDataSource.scala:createWriter(70)] - options {graphaddress=172.21.2.220:3699, spacename=dcp_zaps_adm_prod_01, rankfiled=degree, srcasprop=false, batch=2000, metaaddress=172.21.2.220:45500, label=APPLYER2CONTACTOR, srcpolicy=, rankasprop=false, type=edge, connectionretry=1, timeout=6000, srcvertexfield=id_no_applyer_tag_id, executionretry=1, dstpolicy=, passwd=Wedg#SAD@admp&123., dstvertexfield=id_no_contactor_tag_id, user=clgd_dcp_test, dstasprop=false} 2021-06-07 16:47:30,480 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Removed broadcast_5_piece0 on bdphdp070016:35542 in memory (size: 17.0 KB, free: 1048.7 MB) 2021-06-07 16:47:30,488 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 129 2021-06-07 16:47:30,488 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 134 2021-06-07 16:47:30,488 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 106 2021-06-07 16:47:30,488 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 86 2021-06-07 16:47:30,488 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 112 2021-06-07 16:47:30,488 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 118 2021-06-07 16:47:30,488 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 82 2021-06-07 16:47:30,488 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 56 2021-06-07 16:47:30,488 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 108 2021-06-07 16:47:30,488 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 67 2021-06-07 16:47:30,489 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 75 2021-06-07 16:47:30,489 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 133 2021-06-07 16:47:30,489 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 93 2021-06-07 16:47:30,489 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 102 2021-06-07 16:47:30,489 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 132 2021-06-07 16:47:30,489 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 100 2021-06-07 16:47:30,489 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 77 2021-06-07 16:47:30,489 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 87 2021-06-07 16:47:30,491 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Removed broadcast_3_piece0 on bdphdp07jobs14:16197 in memory (size: 7.2 KB, free: 366.1 MB) 2021-06-07 16:47:30,500 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Removed broadcast_3_piece0 on bdphdp070025:18606 in memory (size: 7.2 KB, free: 1048.7 MB) 2021-06-07 16:47:30,507 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 103 2021-06-07 16:47:30,507 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 104 2021-06-07 16:47:30,507 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 64 2021-06-07 16:47:30,507 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 68 2021-06-07 16:47:30,507 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 94 2021-06-07 16:47:30,507 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 58 2021-06-07 16:47:30,507 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 57 2021-06-07 16:47:30,507 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 61 2021-06-07 16:47:30,507 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 97 2021-06-07 16:47:30,507 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 117 2021-06-07 16:47:30,507 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 65 2021-06-07 16:47:30,507 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 135 2021-06-07 16:47:30,512 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Removed broadcast_2_piece0 on bdphdp07jobs14:16197 in memory (size: 8.5 KB, free: 366.1 MB) 2021-06-07 16:47:30,514 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Removed broadcast_2_piece0 on bdphdp070025:18606 in memory (size: 8.5 KB, free: 1048.7 MB) 2021-06-07 16:47:30,514 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Removed broadcast_2_piece0 on bdphdp070016:35542 in memory (size: 8.5 KB, free: 1048.7 MB) 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 83 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 73 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 114 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 120 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 96 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 99 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 79 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 60 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 89 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 127 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 121 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 84 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 62 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 98 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 124 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 63 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 72 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 119 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 115 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 101 2021-06-07 16:47:30,521 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 130 2021-06-07 16:47:30,521 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 78 2021-06-07 16:47:30,522 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Removed broadcast_4_piece0 on bdphdp07jobs14:16197 in memory (size: 15.5 KB, free: 366.2 MB) 2021-06-07 16:47:30,526 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Removed broadcast_4_piece0 on bdphdp070016:35542 in memory (size: 15.5 KB, free: 1048.7 MB) 2021-06-07 16:47:30,528 INFO (main) [Logging.scala:logInfo(54)] - Code generated in 11.730457 ms 2021-06-07 16:47:30,531 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 128 2021-06-07 16:47:30,531 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 74 2021-06-07 16:47:30,531 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 80 2021-06-07 16:47:30,531 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 110 2021-06-07 16:47:30,531 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 81 2021-06-07 16:47:30,531 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 85 2021-06-07 16:47:30,531 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 126 2021-06-07 16:47:30,531 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 92 2021-06-07 16:47:30,531 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 111 2021-06-07 16:47:30,531 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 69 2021-06-07 16:47:30,531 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 71 2021-06-07 16:47:30,531 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 105 2021-06-07 16:47:30,531 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 88 2021-06-07 16:47:30,531 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 107 2021-06-07 16:47:30,542 INFO (main) [Logging.scala:logInfo(54)] - Block broadcast_6 stored as values in memory (estimated size 1163.6 KB, free 362.8 MB) 2021-06-07 16:47:30,562 INFO (main) [Logging.scala:logInfo(54)] - Block broadcast_6_piece0 stored as bytes in memory (estimated size 71.2 KB, free 362.7 MB) 2021-06-07 16:47:30,563 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Added broadcast_6_piece0 in memory on bdphdp07jobs14:16197 (size: 71.2 KB, free: 366.1 MB) 2021-06-07 16:47:30,563 INFO (main) [Logging.scala:logInfo(54)] - Created broadcast 6 from 2021-06-07 16:47:32,230 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,230 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,232 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,233 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,233 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,233 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,234 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,234 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,234 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,234 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,235 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,235 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,235 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,236 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,236 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,236 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,236 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,237 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,237 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,237 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,237 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,238 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,238 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,238 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,239 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,239 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,239 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,239 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,240 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,240 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,241 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,241 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,241 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,242 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,242 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: int 2021-06-07 16:47:32,243 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,243 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,243 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,244 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: int 2021-06-07 16:47:32,244 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,244 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: int 2021-06-07 16:47:32,245 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: timestamp 2021-06-07 16:47:32,245 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: timestamp 2021-06-07 16:47:32,245 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,246 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,246 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,247 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,247 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,247 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,248 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,248 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,248 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,249 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,249 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,250 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,250 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,250 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,251 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,251 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,251 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,252 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,252 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,252 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,253 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,253 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,253 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,254 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,254 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,254 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,255 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,255 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,255 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,256 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,256 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,256 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,256 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,257 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,257 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,257 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,258 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,258 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,258 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,259 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,259 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,259 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,260 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: int 2021-06-07 16:47:32,260 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,260 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,261 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,261 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,261 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,262 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,262 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,262 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,263 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,263 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,263 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,868 INFO (main) [FileInputFormat.java:listStatus(249)] - Total input paths to process : 1 2021-06-07 16:47:32,877 INFO (main) [FileInputFormat.java:listStatus(249)] - Total input paths to process : 1 2021-06-07 16:47:32,879 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Registering RDD 39 (save at package.scala:298) 2021-06-07 16:47:32,879 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Got map stage job 4 (save at package.scala:298) with 40 output partitions 2021-06-07 16:47:32,879 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Final stage: ShuffleMapStage 9 (save at package.scala:298) 2021-06-07 16:47:32,879 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Parents of final stage: List() 2021-06-07 16:47:32,880 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Missing parents: List() 2021-06-07 16:47:32,880 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting ShuffleMapStage 9 (MapPartitionsRDD[39] at save at package.scala:298), which has no missing parents 2021-06-07 16:47:32,888 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_8 stored as values in memory (estimated size 21.6 KB, free 362.7 MB) 2021-06-07 16:47:32,889 INFO (main) [Logging.scala:logInfo(54)] - Block broadcast_7 stored as values in memory (estimated size 1158.3 KB, free 361.5 MB) 2021-06-07 16:47:32,893 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_8_piece0 stored as bytes in memory (estimated size 8.5 KB, free 361.5 MB) 2021-06-07 16:47:32,893 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Added broadcast_8_piece0 in memory on bdphdp07jobs14:16197 (size: 8.5 KB, free: 366.1 MB) 2021-06-07 16:47:32,894 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Created broadcast 8 from broadcast at DAGScheduler.scala:1161 2021-06-07 16:47:32,894 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting 40 missing tasks from ShuffleMapStage 9 (MapPartitionsRDD[39] at save at package.scala:298) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14)) 2021-06-07 16:47:32,894 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Adding task set 9.0 with 40 tasks 2021-06-07 16:47:32,897 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 0.0 in stage 9.0 (TID 62, bdphdp070025, executor 2, partition 0, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:32,897 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 1.0 in stage 9.0 (TID 63, bdphdp070016, executor 1, partition 1, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:32,897 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 2.0 in stage 9.0 (TID 64, bdphdp070025, executor 2, partition 2, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:32,897 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 3.0 in stage 9.0 (TID 65, bdphdp070016, executor 1, partition 3, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:32,918 INFO (main) [Logging.scala:logInfo(54)] - Block broadcast_7_piece0 stored as bytes in memory (estimated size 70.5 KB, free 361.5 MB) 2021-06-07 16:47:32,919 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Added broadcast_8_piece0 in memory on bdphdp070016:35542 (size: 8.5 KB, free: 1048.7 MB) 2021-06-07 16:47:32,920 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Added broadcast_7_piece0 in memory on bdphdp07jobs14:16197 (size: 70.5 KB, free: 366.0 MB) 2021-06-07 16:47:32,920 INFO (main) [Logging.scala:logInfo(54)] - Created broadcast 7 from 2021-06-07 16:47:32,925 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Added broadcast_8_piece0 in memory on bdphdp070025:18606 (size: 8.5 KB, free: 1048.7 MB) 2021-06-07 16:47:32,947 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Added broadcast_6_piece0 in memory on bdphdp070016:35542 (size: 71.2 KB, free: 1048.7 MB) 2021-06-07 16:47:32,951 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Added broadcast_6_piece0 in memory on bdphdp070025:18606 (size: 71.2 KB, free: 1048.6 MB) 2021-06-07 16:47:33,008 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 4.0 in stage 9.0 (TID 66, bdphdp070016, executor 1, partition 4, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,009 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 3.0 in stage 9.0 (TID 65) in 112 ms on bdphdp070016 (executor 1) (1/40) 2021-06-07 16:47:33,012 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 5.0 in stage 9.0 (TID 67, bdphdp070016, executor 1, partition 5, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,013 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 1.0 in stage 9.0 (TID 63) in 116 ms on bdphdp070016 (executor 1) (2/40) 2021-06-07 16:47:33,031 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Starting task 6.0 in stage 9.0 (TID 68, bdphdp070025, executor 2, partition 6, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,031 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Starting task 7.0 in stage 9.0 (TID 69, bdphdp070025, executor 2, partition 7, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,032 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 2.0 in stage 9.0 (TID 64) in 135 ms on bdphdp070025 (executor 2) (3/40) 2021-06-07 16:47:33,035 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 0.0 in stage 9.0 (TID 62) in 139 ms on bdphdp070025 (executor 2) (4/40) 2021-06-07 16:47:33,046 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 8.0 in stage 9.0 (TID 70, bdphdp070016, executor 1, partition 8, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,046 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 5.0 in stage 9.0 (TID 67) in 34 ms on bdphdp070016 (executor 1) (5/40) 2021-06-07 16:47:33,051 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 9.0 in stage 9.0 (TID 71, bdphdp070016, executor 1, partition 9, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,051 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 4.0 in stage 9.0 (TID 66) in 43 ms on bdphdp070016 (executor 1) (6/40) 2021-06-07 16:47:33,070 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 10.0 in stage 9.0 (TID 72, bdphdp070025, executor 2, partition 10, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,070 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 6.0 in stage 9.0 (TID 68) in 40 ms on bdphdp070025 (executor 2) (7/40) 2021-06-07 16:47:33,075 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Starting task 11.0 in stage 9.0 (TID 73, bdphdp070025, executor 2, partition 11, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,076 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 7.0 in stage 9.0 (TID 69) in 45 ms on bdphdp070025 (executor 2) (8/40) 2021-06-07 16:47:33,083 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 12.0 in stage 9.0 (TID 74, bdphdp070016, executor 1, partition 12, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,083 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 8.0 in stage 9.0 (TID 70) in 37 ms on bdphdp070016 (executor 1) (9/40) 2021-06-07 16:47:33,095 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 13.0 in stage 9.0 (TID 75, bdphdp070016, executor 1, partition 13, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,095 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 9.0 in stage 9.0 (TID 71) in 44 ms on bdphdp070016 (executor 1) (10/40) 2021-06-07 16:47:33,114 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 14.0 in stage 9.0 (TID 76, bdphdp070025, executor 2, partition 14, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,114 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 11.0 in stage 9.0 (TID 73) in 39 ms on bdphdp070025 (executor 2) (11/40) 2021-06-07 16:47:33,117 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Starting task 15.0 in stage 9.0 (TID 77, bdphdp070025, executor 2, partition 15, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,118 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 10.0 in stage 9.0 (TID 72) in 48 ms on bdphdp070025 (executor 2) (12/40) 2021-06-07 16:47:33,126 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 16.0 in stage 9.0 (TID 78, bdphdp070016, executor 1, partition 16, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,127 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 12.0 in stage 9.0 (TID 74) in 44 ms on bdphdp070016 (executor 1) (13/40) 2021-06-07 16:47:33,135 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 17.0 in stage 9.0 (TID 79, bdphdp070016, executor 1, partition 17, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,136 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 13.0 in stage 9.0 (TID 75) in 41 ms on bdphdp070016 (executor 1) (14/40) 2021-06-07 16:47:33,153 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Starting task 18.0 in stage 9.0 (TID 80, bdphdp070025, executor 2, partition 18, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,153 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 15.0 in stage 9.0 (TID 77) in 36 ms on bdphdp070025 (executor 2) (15/40) 2021-06-07 16:47:33,154 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Starting task 19.0 in stage 9.0 (TID 81, bdphdp070025, executor 2, partition 19, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,155 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 14.0 in stage 9.0 (TID 76) in 41 ms on bdphdp070025 (executor 2) (16/40) 2021-06-07 16:47:33,183 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 20.0 in stage 9.0 (TID 82, bdphdp070016, executor 1, partition 20, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,184 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 16.0 in stage 9.0 (TID 78) in 58 ms on bdphdp070016 (executor 1) (17/40) 2021-06-07 16:47:33,187 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 21.0 in stage 9.0 (TID 83, bdphdp070025, executor 2, partition 21, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,188 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 19.0 in stage 9.0 (TID 81) in 34 ms on bdphdp070025 (executor 2) (18/40) 2021-06-07 16:47:33,192 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 22.0 in stage 9.0 (TID 84, bdphdp070025, executor 2, partition 22, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,192 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 18.0 in stage 9.0 (TID 80) in 39 ms on bdphdp070025 (executor 2) (19/40) 2021-06-07 16:47:33,193 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Starting task 23.0 in stage 9.0 (TID 85, bdphdp070016, executor 1, partition 23, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,193 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 17.0 in stage 9.0 (TID 79) in 58 ms on bdphdp070016 (executor 1) (20/40) 2021-06-07 16:47:33,216 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 24.0 in stage 9.0 (TID 86, bdphdp070025, executor 2, partition 24, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,216 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 21.0 in stage 9.0 (TID 83) in 29 ms on bdphdp070025 (executor 2) (21/40) 2021-06-07 16:47:33,218 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 25.0 in stage 9.0 (TID 87, bdphdp070025, executor 2, partition 25, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,218 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 22.0 in stage 9.0 (TID 84) in 26 ms on bdphdp070025 (executor 2) (22/40) 2021-06-07 16:47:33,223 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Starting task 26.0 in stage 9.0 (TID 88, bdphdp070016, executor 1, partition 26, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,223 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 23.0 in stage 9.0 (TID 85) in 30 ms on bdphdp070016 (executor 1) (23/40) 2021-06-07 16:47:33,225 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Starting task 27.0 in stage 9.0 (TID 89, bdphdp070016, executor 1, partition 27, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,226 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 20.0 in stage 9.0 (TID 82) in 43 ms on bdphdp070016 (executor 1) (24/40) 2021-06-07 16:47:33,241 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 28.0 in stage 9.0 (TID 90, bdphdp070025, executor 2, partition 28, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,242 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 24.0 in stage 9.0 (TID 86) in 26 ms on bdphdp070025 (executor 2) (25/40) 2021-06-07 16:47:33,243 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 29.0 in stage 9.0 (TID 91, bdphdp070025, executor 2, partition 29, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,243 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 25.0 in stage 9.0 (TID 87) in 25 ms on bdphdp070025 (executor 2) (26/40) 2021-06-07 16:47:33,248 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 30.0 in stage 9.0 (TID 92, bdphdp070016, executor 1, partition 30, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,248 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 26.0 in stage 9.0 (TID 88) in 25 ms on bdphdp070016 (executor 1) (27/40) 2021-06-07 16:47:33,255 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Starting task 31.0 in stage 9.0 (TID 93, bdphdp070016, executor 1, partition 31, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,255 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 27.0 in stage 9.0 (TID 89) in 30 ms on bdphdp070016 (executor 1) (28/40) 2021-06-07 16:47:33,269 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 32.0 in stage 9.0 (TID 94, bdphdp070025, executor 2, partition 32, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,269 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 29.0 in stage 9.0 (TID 91) in 26 ms on bdphdp070025 (executor 2) (29/40) 2021-06-07 16:47:33,291 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 33.0 in stage 9.0 (TID 95, bdphdp070016, executor 1, partition 33, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,291 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 30.0 in stage 9.0 (TID 92) in 43 ms on bdphdp070016 (executor 1) (30/40) 2021-06-07 16:47:33,295 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Starting task 34.0 in stage 9.0 (TID 96, bdphdp070016, executor 1, partition 34, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,295 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 31.0 in stage 9.0 (TID 93) in 40 ms on bdphdp070016 (executor 1) (31/40) 2021-06-07 16:47:33,322 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Starting task 35.0 in stage 9.0 (TID 97, bdphdp070025, executor 2, partition 35, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,322 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 28.0 in stage 9.0 (TID 90) in 81 ms on bdphdp070025 (executor 2) (32/40) 2021-06-07 16:47:33,324 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 36.0 in stage 9.0 (TID 98, bdphdp070016, executor 1, partition 36, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,325 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 33.0 in stage 9.0 (TID 95) in 35 ms on bdphdp070016 (executor 1) (33/40) 2021-06-07 16:47:33,327 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 37.0 in stage 9.0 (TID 99, bdphdp070016, executor 1, partition 37, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,327 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 34.0 in stage 9.0 (TID 96) in 33 ms on bdphdp070016 (executor 1) (34/40) 2021-06-07 16:47:33,348 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 38.0 in stage 9.0 (TID 100, bdphdp070016, executor 1, partition 38, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,348 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 36.0 in stage 9.0 (TID 98) in 24 ms on bdphdp070016 (executor 1) (35/40) 2021-06-07 16:47:33,351 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Starting task 39.0 in stage 9.0 (TID 101, bdphdp070025, executor 2, partition 39, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,351 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 35.0 in stage 9.0 (TID 97) in 29 ms on bdphdp070025 (executor 2) (36/40) 2021-06-07 16:47:33,352 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 32.0 in stage 9.0 (TID 94) in 83 ms on bdphdp070025 (executor 2) (37/40) 2021-06-07 16:47:33,352 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 37.0 in stage 9.0 (TID 99) in 25 ms on bdphdp070016 (executor 1) (38/40) 2021-06-07 16:47:33,371 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 38.0 in stage 9.0 (TID 100) in 23 ms on bdphdp070016 (executor 1) (39/40) 2021-06-07 16:47:33,373 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 39.0 in stage 9.0 (TID 101) in 22 ms on bdphdp070025 (executor 2) (40/40) 2021-06-07 16:47:33,373 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Removed TaskSet 9.0, whose tasks have all completed, from pool 2021-06-07 16:47:33,373 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - ShuffleMapStage 9 (save at package.scala:298) finished in 0.488 s 2021-06-07 16:47:33,373 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - looking for newly runnable stages 2021-06-07 16:47:33,373 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - running: Set() 2021-06-07 16:47:33,373 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - waiting: Set() 2021-06-07 16:47:33,373 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - failed: Set() 2021-06-07 16:47:33,959 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:33,959 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:33,960 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:33,960 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:33,960 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:33,960 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:33,961 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: timestamp 2021-06-07 16:47:33,961 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: timestamp 2021-06-07 16:47:33,961 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:33,962 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:33,962 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:33,962 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:33,962 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:33,962 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:33,963 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:34,629 INFO (main) [FileInputFormat.java:listStatus(249)] - Total input paths to process : 1 2021-06-07 16:47:34,631 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Registering RDD 46 (save at package.scala:298) 2021-06-07 16:47:34,631 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Got map stage job 5 (save at package.scala:298) with 20 output partitions 2021-06-07 16:47:34,631 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Final stage: ShuffleMapStage 10 (save at package.scala:298) 2021-06-07 16:47:34,631 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Parents of final stage: List() 2021-06-07 16:47:34,632 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Missing parents: List() 2021-06-07 16:47:34,632 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting ShuffleMapStage 10 (MapPartitionsRDD[46] at save at package.scala:298), which has no missing parents 2021-06-07 16:47:34,636 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_9 stored as values in memory (estimated size 14.9 KB, free 361.4 MB) 2021-06-07 16:47:34,640 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_9_piece0 stored as bytes in memory (estimated size 7.2 KB, free 361.4 MB) 2021-06-07 16:47:34,641 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Added broadcast_9_piece0 in memory on bdphdp07jobs14:16197 (size: 7.2 KB, free: 366.0 MB) 2021-06-07 16:47:34,641 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Created broadcast 9 from broadcast at DAGScheduler.scala:1161 2021-06-07 16:47:34,642 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting 20 missing tasks from ShuffleMapStage 10 (MapPartitionsRDD[46] at save at package.scala:298) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14)) 2021-06-07 16:47:34,642 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Adding task set 10.0 with 20 tasks 2021-06-07 16:47:34,643 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 0.0 in stage 10.0 (TID 102, bdphdp070025, executor 2, partition 0, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,643 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 1.0 in stage 10.0 (TID 103, bdphdp070025, executor 2, partition 1, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,663 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Added broadcast_9_piece0 in memory on bdphdp070025:18606 (size: 7.2 KB, free: 1048.6 MB) 2021-06-07 16:47:34,689 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Added broadcast_7_piece0 in memory on bdphdp070025:18606 (size: 70.5 KB, free: 1048.5 MB) 2021-06-07 16:47:34,719 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Starting task 2.0 in stage 10.0 (TID 104, bdphdp070025, executor 2, partition 2, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,719 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 0.0 in stage 10.0 (TID 102) in 76 ms on bdphdp070025 (executor 2) (1/20) 2021-06-07 16:47:34,720 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Starting task 3.0 in stage 10.0 (TID 105, bdphdp070025, executor 2, partition 3, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,721 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 1.0 in stage 10.0 (TID 103) in 78 ms on bdphdp070025 (executor 2) (2/20) 2021-06-07 16:47:34,735 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 4.0 in stage 10.0 (TID 106, bdphdp070025, executor 2, partition 4, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,735 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 2.0 in stage 10.0 (TID 104) in 16 ms on bdphdp070025 (executor 2) (3/20) 2021-06-07 16:47:34,739 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Starting task 5.0 in stage 10.0 (TID 107, bdphdp070025, executor 2, partition 5, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,739 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 3.0 in stage 10.0 (TID 105) in 19 ms on bdphdp070025 (executor 2) (4/20) 2021-06-07 16:47:34,750 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 6.0 in stage 10.0 (TID 108, bdphdp070025, executor 2, partition 6, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,750 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 4.0 in stage 10.0 (TID 106) in 15 ms on bdphdp070025 (executor 2) (5/20) 2021-06-07 16:47:34,754 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Starting task 7.0 in stage 10.0 (TID 109, bdphdp070025, executor 2, partition 7, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,754 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 5.0 in stage 10.0 (TID 107) in 15 ms on bdphdp070025 (executor 2) (6/20) 2021-06-07 16:47:34,764 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 8.0 in stage 10.0 (TID 110, bdphdp070025, executor 2, partition 8, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,765 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 6.0 in stage 10.0 (TID 108) in 15 ms on bdphdp070025 (executor 2) (7/20) 2021-06-07 16:47:34,767 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Starting task 9.0 in stage 10.0 (TID 111, bdphdp070025, executor 2, partition 9, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,767 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 7.0 in stage 10.0 (TID 109) in 14 ms on bdphdp070025 (executor 2) (8/20) 2021-06-07 16:47:34,779 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 10.0 in stage 10.0 (TID 112, bdphdp070025, executor 2, partition 10, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,779 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 8.0 in stage 10.0 (TID 110) in 15 ms on bdphdp070025 (executor 2) (9/20) 2021-06-07 16:47:34,782 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Starting task 11.0 in stage 10.0 (TID 113, bdphdp070025, executor 2, partition 11, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,783 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 9.0 in stage 10.0 (TID 111) in 16 ms on bdphdp070025 (executor 2) (10/20) 2021-06-07 16:47:34,796 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Starting task 12.0 in stage 10.0 (TID 114, bdphdp070025, executor 2, partition 12, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,797 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 10.0 in stage 10.0 (TID 112) in 19 ms on bdphdp070025 (executor 2) (11/20) 2021-06-07 16:47:34,798 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 13.0 in stage 10.0 (TID 115, bdphdp070025, executor 2, partition 13, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,798 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 11.0 in stage 10.0 (TID 113) in 16 ms on bdphdp070025 (executor 2) (12/20) 2021-06-07 16:47:34,810 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 14.0 in stage 10.0 (TID 116, bdphdp070025, executor 2, partition 14, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,811 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 12.0 in stage 10.0 (TID 114) in 15 ms on bdphdp070025 (executor 2) (13/20) 2021-06-07 16:47:34,812 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Starting task 15.0 in stage 10.0 (TID 117, bdphdp070025, executor 2, partition 15, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,812 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 13.0 in stage 10.0 (TID 115) in 15 ms on bdphdp070025 (executor 2) (14/20) 2021-06-07 16:47:34,824 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Starting task 16.0 in stage 10.0 (TID 118, bdphdp070025, executor 2, partition 16, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,824 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 14.0 in stage 10.0 (TID 116) in 14 ms on bdphdp070025 (executor 2) (15/20) 2021-06-07 16:47:34,826 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 17.0 in stage 10.0 (TID 119, bdphdp070025, executor 2, partition 17, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,826 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 15.0 in stage 10.0 (TID 117) in 14 ms on bdphdp070025 (executor 2) (16/20) 2021-06-07 16:47:34,836 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 18.0 in stage 10.0 (TID 120, bdphdp070025, executor 2, partition 18, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,837 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 16.0 in stage 10.0 (TID 118) in 14 ms on bdphdp070025 (executor 2) (17/20) 2021-06-07 16:47:34,840 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Starting task 19.0 in stage 10.0 (TID 121, bdphdp070025, executor 2, partition 19, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,841 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 17.0 in stage 10.0 (TID 119) in 15 ms on bdphdp070025 (executor 2) (18/20) 2021-06-07 16:47:34,849 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 18.0 in stage 10.0 (TID 120) in 13 ms on bdphdp070025 (executor 2) (19/20) 2021-06-07 16:47:34,853 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 19.0 in stage 10.0 (TID 121) in 13 ms on bdphdp070025 (executor 2) (20/20) 2021-06-07 16:47:34,853 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Removed TaskSet 10.0, whose tasks have all completed, from pool 2021-06-07 16:47:34,854 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - ShuffleMapStage 10 (save at package.scala:298) finished in 0.220 s 2021-06-07 16:47:34,854 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - looking for newly runnable stages 2021-06-07 16:47:34,854 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - running: Set() 2021-06-07 16:47:34,854 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - waiting: Set() 2021-06-07 16:47:34,854 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - failed: Set() 2021-06-07 16:47:34,854 INFO (main) [Logging.scala:logInfo(54)] - advisoryTargetPostShuffleInputSize: 67108864, targetPostShuffleInputSize 67108864. 2021-06-07 16:47:34,867 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Registering RDD 53 (save at package.scala:298) 2021-06-07 16:47:34,867 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Got map stage job 6 (save at package.scala:298) with 1 output partitions 2021-06-07 16:47:34,867 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Final stage: ShuffleMapStage 13 (save at package.scala:298) 2021-06-07 16:47:34,867 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Parents of final stage: List(ShuffleMapStage 12, ShuffleMapStage 11) 2021-06-07 16:47:34,867 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Missing parents: List() 2021-06-07 16:47:34,867 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting ShuffleMapStage 13 (MapPartitionsRDD[53] at save at package.scala:298), which has no missing parents 2021-06-07 16:47:34,870 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_10 stored as values in memory (estimated size 36.4 KB, free 361.4 MB) 2021-06-07 16:47:34,874 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_10_piece0 stored as bytes in memory (estimated size 15.5 KB, free 361.4 MB) 2021-06-07 16:47:34,875 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Added broadcast_10_piece0 in memory on bdphdp07jobs14:16197 (size: 15.5 KB, free: 366.0 MB) 2021-06-07 16:47:34,875 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Created broadcast 10 from broadcast at DAGScheduler.scala:1161 2021-06-07 16:47:34,875 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting 1 missing tasks from ShuffleMapStage 13 (MapPartitionsRDD[53] at save at package.scala:298) (first 15 tasks are for partitions Vector(0)) 2021-06-07 16:47:34,876 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Adding task set 13.0 with 1 tasks 2021-06-07 16:47:34,876 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Starting task 0.0 in stage 13.0 (TID 122, bdphdp070016, executor 1, partition 0, NODE_LOCAL, 8097 bytes) 2021-06-07 16:47:34,896 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Added broadcast_10_piece0 in memory on bdphdp070016:35542 (size: 15.5 KB, free: 1048.6 MB) 2021-06-07 16:47:34,905 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Asked to send map output locations for shuffle 3 to 10.107.97.188:62930 2021-06-07 16:47:34,938 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Asked to send map output locations for shuffle 4 to 10.107.97.188:62930 2021-06-07 16:47:35,213 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 0.0 in stage 13.0 (TID 122) in 337 ms on bdphdp070016 (executor 1) (1/1) 2021-06-07 16:47:35,214 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Removed TaskSet 13.0, whose tasks have all completed, from pool 2021-06-07 16:47:35,214 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - ShuffleMapStage 13 (save at package.scala:298) finished in 0.346 s 2021-06-07 16:47:35,214 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - looking for newly runnable stages 2021-06-07 16:47:35,214 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - running: Set() 2021-06-07 16:47:35,214 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - waiting: Set() 2021-06-07 16:47:35,214 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - failed: Set() 2021-06-07 16:47:35,215 INFO (main) [Logging.scala:logInfo(54)] - advisoryTargetPostShuffleInputSize: 67108864, targetPostShuffleInputSize 67108864. 2021-06-07 16:47:35,241 INFO (main) [Logging.scala:logInfo(54)] - Start processing data source writer: com.vesoft.nebula.connector.writer.NebulaDataSourceEdgeWriter@60666985. The input RDD has 1 partitions. 2021-06-07 16:47:35,244 INFO (main) [Logging.scala:logInfo(54)] - Starting job: save at package.scala:298 2021-06-07 16:47:35,245 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Got job 7 (save at package.scala:298) with 1 output partitions 2021-06-07 16:47:35,245 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Final stage: ResultStage 17 (save at package.scala:298) 2021-06-07 16:47:35,246 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Parents of final stage: List(ShuffleMapStage 16) 2021-06-07 16:47:35,248 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Missing parents: List() 2021-06-07 16:47:35,248 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting ResultStage 17 (MapPartitionsRDD[61] at save at package.scala:298), which has no missing parents 2021-06-07 16:47:35,263 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_11 stored as values in memory (estimated size 46.4 KB, free 361.3 MB) 2021-06-07 16:47:35,268 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_11_piece0 stored as bytes in memory (estimated size 20.8 KB, free 361.3 MB) 2021-06-07 16:47:35,269 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Added broadcast_11_piece0 in memory on bdphdp07jobs14:16197 (size: 20.8 KB, free: 366.0 MB) 2021-06-07 16:47:35,269 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Created broadcast 11 from broadcast at DAGScheduler.scala:1161 2021-06-07 16:47:35,270 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting 1 missing tasks from ResultStage 17 (MapPartitionsRDD[61] at save at package.scala:298) (first 15 tasks are for partitions Vector(0)) 2021-06-07 16:47:35,270 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Adding task set 17.0 with 1 tasks 2021-06-07 16:47:35,270 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 0.0 in stage 17.0 (TID 123, bdphdp070025, executor 2, partition 0, PROCESS_LOCAL, 7780 bytes) 2021-06-07 16:47:35,295 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Added broadcast_11_piece0 in memory on bdphdp070025:18606 (size: 20.8 KB, free: 1048.5 MB) 2021-06-07 16:47:35,441 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Asked to send map output locations for shuffle 5 to 10.107.119.198:36674 2021-06-07 16:47:35,720 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Added rdd_58_0 in memory on bdphdp070025:18606 (size: 360.0 B, free: 1048.5 MB) ============Result:================ job id:job_20210607164631_019186 code:200 extraMessage:null response: Output: 2021-06-07 16:46:33,573 WARN (main) [Logging.scala:logWarning(66)] - The configuration key 'spark.yarn.executor.memoryOverhead' has been deprecated as of Spark 2.3 and may be removed in the future. Please use the new key 'spark.executor.memoryOverhead' instead. 2021-06-07 16:46:33,576 WARN (main) [Logging.scala:logWarning(66)] - The configuration key 'spark.yarn.driver.memoryOverhead' has been deprecated as of Spark 2.3 and may be removed in the future. Please use the new key 'spark.driver.memoryOverhead' instead. ----------neubla的配置信息如下所示：---------- edgeSql:select id_no_applyer_tag_id, id_no_contactor_tag_id, rn as degree, ID_NO_APPLYER, ID_NO_CONTACTOR, CREATED_DATETIME, LAST_MODIFIED_DATETIME from( select applyer.id_no as id_no_applyer_tag_id, contack.id_no as id_no_contactor_tag_id, applyer.id_no ID_NO_APPLYER, contack.id_no ID_NO_CONTACTOR, unix_timestamp(nvl(contack.created_datetime,to_timestamp('9999-12-31 23:59:99','yyyy-MM-dd HH:mm:ss'))) as CREATED_DATETIME, unix_timestamp(nvl(contack.last_modified_datetime,to_timestamp('9999-12-31 23:59:99','yyyy-MM-dd HH:mm:ss'))) as LAST_MODIFIED_DATETIME, row_number() over(partition by applyer.id_no,contack.id_no order by contack.last_modified_datetime desc) as rn from MERGE_TM_APPLY_INFO_ZAPS_DCN as applyer inner join MERGE_TM_CONTACT_ZAPS_ZDCN as contack on applyer.app_no=contack.app_no where applyer.ds='20210427' and contack.ds='20210427' and applyer.id_no is not null and contack.id_no is not null )t where rn=1 edgeName:APPLYER2CONTACTOR srcIdField:id_no_applyer_tag_id dstIdField:id_no_contactor_tag_id batch：2000 ------------------------------------------- 2021-06-07 16:46:35,176 WARN (main) [Logging.scala:logWarning(66)] - The configuration key 'spark.yarn.executor.memoryOverhead' has been deprecated as of Spark 2.3 and may be removed in the future. Please use the new key 'spark.executor.memoryOverhead' instead. 2021-06-07 16:46:35,177 WARN (main) [Logging.scala:logWarning(66)] - The configuration key 'spark.yarn.driver.memoryOverhead' has been deprecated as of Spark 2.3 and may be removed in the future. Please use the new key 'spark.driver.memoryOverhead' instead. 2021-06-07 16:46:35,190 INFO (main) [Logging.scala:logInfo(54)] - Running Spark version 2.4.3 2021-06-07 16:46:35,201 WARN (main) [Logging.scala:logWarning(66)] - Note that spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone/kubernetes and LOCAL_DIRS in YARN). 2021-06-07 16:46:35,205 WARN (main) [Logging.scala:logWarning(66)] - Detected deprecated memory fraction settings: [spark.storage.memoryFraction]. As of Spark 1.6, execution and storage memory management are unified. All memory fractions used in the old model are now deprecated and no longer read. If you wish to use the old memory management, you may explicitly enable `spark.memory.useLegacyMode` (not recommended). 2021-06-07 16:46:35,209 INFO (main) [Logging.scala:logInfo(54)] - Submitted application: hive2nebula 2021-06-07 16:46:35,245 INFO (main) [Logging.scala:logInfo(54)] - Changing view acls to: hduser0334 2021-06-07 16:46:35,245 INFO (main) [Logging.scala:logInfo(54)] - Changing modify acls to: hduser0334 2021-06-07 16:46:35,245 INFO (main) [Logging.scala:logInfo(54)] - Changing view acls groups to: 2021-06-07 16:46:35,246 INFO (main) [Logging.scala:logInfo(54)] - Changing modify acls groups to: 2021-06-07 16:46:35,246 INFO (main) [Logging.scala:logInfo(54)] - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hduser0334); groups with view permissions: Set(); users with modify permissions: Set(hduser0334); groups with modify permissions: Set() 2021-06-07 16:46:35,441 INFO (main) [Logging.scala:logInfo(54)] - Successfully started service 'sparkDriver' on port 32832. 2021-06-07 16:46:35,457 INFO (main) [Logging.scala:logInfo(54)] - Registering MapOutputTracker 2021-06-07 16:46:35,469 INFO (main) [Logging.scala:logInfo(54)] - Registering BlockManagerMaster 2021-06-07 16:46:35,472 INFO (main) [Logging.scala:logInfo(54)] - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 2021-06-07 16:46:35,473 INFO (main) [Logging.scala:logInfo(54)] - BlockManagerMasterEndpoint up 2021-06-07 16:46:35,479 INFO (main) [Logging.scala:logInfo(54)] - Created local directory at /appcom/logs/spark/tmp/blockmgr-66aa398a-9c04-42b2-a584-ba2b8b772237 2021-06-07 16:46:35,489 INFO (main) [Logging.scala:logInfo(54)] - MemoryStore started with capacity 366.3 MB 2021-06-07 16:46:35,533 INFO (main) [Logging.scala:logInfo(54)] - Registering OutputCommitCoordinator 2021-06-07 16:46:35,682 INFO (main) [Logging.scala:logInfo(54)] - Successfully started service 'SparkUI' on port 4040. 2021-06-07 16:46:35,716 INFO (main) [Logging.scala:logInfo(54)] - Bound SparkUI to 0.0.0.0, and started at http://bdphdp07jobs14:4040 2021-06-07 16:46:35,739 INFO (main) [Logging.scala:logInfo(54)] - Added JAR viewfs:///apps-data/hduser0334/nebula-exchange/nebula-sync-2.0.1.jar at viewfs:///apps-data/hduser0334/nebula-exchange/nebula-sync-2.0.1.jar with timestamp 1623055595738 2021-06-07 16:46:35,942 INFO (main) [ConfiguredRMFailoverProxyProvider.java:performFailover(100)] - Failing over to rm2 2021-06-07 16:46:35,953 INFO (main) [Logging.scala:logInfo(54)] - Requesting a new application from cluster with 22 NodeManagers 2021-06-07 16:46:35,988 INFO (main) [Logging.scala:logInfo(54)] - Verifying our application has not requested more than the maximum memory capability of the cluster (20480 MB per container) 2021-06-07 16:46:35,988 INFO (main) [Logging.scala:logInfo(54)] - Will allocate AM container, with 1408 MB memory including 384 MB overhead 2021-06-07 16:46:35,989 INFO (main) [Logging.scala:logInfo(54)] - Setting up container launch context for our AM 2021-06-07 16:46:35,990 INFO (main) [Logging.scala:logInfo(54)] - Setting up the launch environment for our AM container 2021-06-07 16:46:35,995 INFO (main) [Logging.scala:logInfo(54)] - Preparing resources for our AM container 2021-06-07 16:46:36,015 WARN (main) [Logging.scala:logWarning(66)] - Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. 2021-06-07 16:46:37,950 INFO (main) [Logging.scala:logInfo(54)] - Uploading resource file:/appcom/logs/spark/tmp/spark-6f7f1690-5162-48e6-a862-359e0ab0c653/__spark_libs__1272496730700104025.zip -> viewfs://bdphdp07/tmp/hduser0334/spark-stagingdir/.sparkStaging/application_1616590772587_981117/__spark_libs__1272496730700104025.zip 2021-06-07 16:46:45,310 INFO (main) [Logging.scala:logInfo(54)] - Uploading resource file:/appcom/logs/spark/tmp/spark-6f7f1690-5162-48e6-a862-359e0ab0c653/__spark_conf__6038093923542428052.zip -> viewfs://bdphdp07/tmp/hduser0334/spark-stagingdir/.sparkStaging/application_1616590772587_981117/__spark_conf__.zip 2021-06-07 16:46:45,422 INFO (main) [Logging.scala:logInfo(54)] - Changing view acls to: hduser0334 2021-06-07 16:46:45,422 INFO (main) [Logging.scala:logInfo(54)] - Changing modify acls to: hduser0334 2021-06-07 16:46:45,423 INFO (main) [Logging.scala:logInfo(54)] - Changing view acls groups to: 2021-06-07 16:46:45,423 INFO (main) [Logging.scala:logInfo(54)] - Changing modify acls groups to: 2021-06-07 16:46:45,423 INFO (main) [Logging.scala:logInfo(54)] - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hduser0334); groups with view permissions: Set(); users with modify permissions: Set(hduser0334); groups with modify permissions: Set() 2021-06-07 16:46:46,115 INFO (main) [Logging.scala:logInfo(54)] - Submitting application application_1616590772587_981117 to ResourceManager 2021-06-07 16:46:46,142 INFO (main) [YarnClientImpl.java:submitApplication(273)] - Submitted application application_1616590772587_981117 2021-06-07 16:46:46,144 INFO (main) [Logging.scala:logInfo(54)] - Starting Yarn extension services with app application_1616590772587_981117 and attemptId None 2021-06-07 16:46:47,151 INFO (main) [Logging.scala:logInfo(54)] - Application report for application_1616590772587_981117 (state: ACCEPTED) 2021-06-07 16:46:47,154 INFO (main) [Logging.scala:logInfo(54)] - client token: N/A diagnostics: N/A ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: root.queue_3399_01 start time: 1623055606128 final status: UNDEFINED tracking URL: http://bdphdp070002:8088/proxy/application_1616590772587_981117/ user: hduser0334 2021-06-07 16:46:48,161 INFO (main) [Logging.scala:logInfo(54)] - Application report for application_1616590772587_981117 (state: ACCEPTED) 2021-06-07 16:46:49,165 INFO (main) [Logging.scala:logInfo(54)] - Application report for application_1616590772587_981117 (state: ACCEPTED) 2021-06-07 16:46:50,168 INFO (main) [Logging.scala:logInfo(54)] - Application report for application_1616590772587_981117 (state: ACCEPTED) 2021-06-07 16:46:51,172 INFO (main) [Logging.scala:logInfo(54)] - Application report for application_1616590772587_981117 (state: ACCEPTED) 2021-06-07 16:46:51,976 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> bdphdp070003,bdphdp070002, PROXY_URI_BASES -> http://bdphdp070003:8088/proxy/application_1616590772587_981117,http://bdphdp070002:8088/proxy/application_1616590772587_981117, RM_HA_URLS -> bdphdp070003:8088,bdphdp070002:8088), /proxy/application_1616590772587_981117 2021-06-07 16:46:51,977 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /jobs, /jobs/json, /jobs/job, /jobs/job/json, /stages, /stages/json, /stages/stage, /stages/stage/json, /stages/pool, /stages/pool/json, /storage, /storage/json, /storage/rdd, /storage/rdd/json, /environment, /environment/json, /executors, /executors/json, /executors/threadDump, /executors/threadDump/json, /static, /, /api, /jobs/job/kill, /stages/stage/kill. 2021-06-07 16:46:52,120 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - ApplicationMaster registered as NettyRpcEndpointRef(spark-client://YarnAM) 2021-06-07 16:46:52,175 INFO (main) [Logging.scala:logInfo(54)] - Application report for application_1616590772587_981117 (state: RUNNING) 2021-06-07 16:46:52,176 INFO (main) [Logging.scala:logInfo(54)] - client token: N/A diagnostics: N/A ApplicationMaster host: 10.107.108.135 ApplicationMaster RPC port: -1 queue: root.queue_3399_01 start time: 1623055606128 final status: UNDEFINED tracking URL: http://bdphdp070002:8088/proxy/application_1616590772587_981117/ user: hduser0334 2021-06-07 16:46:52,177 INFO (main) [Logging.scala:logInfo(54)] - Application application_1616590772587_981117 has started running. 2021-06-07 16:46:52,182 INFO (main) [Logging.scala:logInfo(54)] - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 16197. 2021-06-07 16:46:52,182 INFO (main) [Logging.scala:logInfo(54)] - Server created on bdphdp07jobs14:16197 2021-06-07 16:46:52,183 INFO (main) [Logging.scala:logInfo(54)] - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 2021-06-07 16:46:52,202 INFO (main) [Logging.scala:logInfo(54)] - Registering BlockManager BlockManagerId(driver, bdphdp07jobs14, 16197, None) 2021-06-07 16:46:52,204 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Registering block manager bdphdp07jobs14:16197 with 366.3 MB RAM, BlockManagerId(driver, bdphdp07jobs14, 16197, None) 2021-06-07 16:46:52,206 INFO (main) [Logging.scala:logInfo(54)] - Registered BlockManager BlockManagerId(driver, bdphdp07jobs14, 16197, None) 2021-06-07 16:46:52,207 INFO (main) [Logging.scala:logInfo(54)] - Initialized BlockManager: BlockManagerId(driver, bdphdp07jobs14, 16197, None) 2021-06-07 16:46:52,296 INFO (main) [Logging.scala:logInfo(54)] - Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /metrics/json. 2021-06-07 16:46:52,417 INFO (main) [Logging.scala:logInfo(54)] - Logging events to viewfs://bdphdp07/sparkeventlogs/application_1616590772587_981117.lz4 2021-06-07 16:46:56,281 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.107.119.198:36674) with ID 2 2021-06-07 16:46:56,393 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Registering block manager bdphdp070025:18606 with 1048.8 MB RAM, BlockManagerId(2, bdphdp070025, 18606, None) 2021-06-07 16:46:56,899 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.107.97.188:62930) with ID 1 2021-06-07 16:46:56,931 INFO (main) [Logging.scala:logInfo(54)] - SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8 2021-06-07 16:46:56,946 INFO (main) [Logging.scala:logInfo(54)] - loading hive config file: file:/appcom/config/spark-config/spark-submit/hive-site.xml 2021-06-07 16:46:56,968 INFO (main) [Logging.scala:logInfo(54)] - spark.sql.warehouse.dir is not set, but hive.metastore.warehouse.dir is set. Setting spark.sql.warehouse.dir to the value of hive.metastore.warehouse.dir ('/user/hive/warehouse/hadoop'). 2021-06-07 16:46:56,968 INFO (main) [Logging.scala:logInfo(54)] - Warehouse path is '/user/hive/warehouse/hadoop'. 2021-06-07 16:46:56,975 INFO (main) [Logging.scala:logInfo(54)] - Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL. 2021-06-07 16:46:56,976 INFO (main) [Logging.scala:logInfo(54)] - Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/json. 2021-06-07 16:46:56,977 INFO (main) [Logging.scala:logInfo(54)] - Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/execution. 2021-06-07 16:46:56,977 INFO (main) [Logging.scala:logInfo(54)] - Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/execution/json. 2021-06-07 16:46:56,978 INFO (main) [Logging.scala:logInfo(54)] - Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /static/sql. 2021-06-07 16:46:57,037 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Registering block manager bdphdp070016:35542 with 1048.8 MB RAM, BlockManagerId(1, bdphdp070016, 35542, None) 2021-06-07 16:46:57,358 INFO (main) [Logging.scala:logInfo(54)] - Registered StateStoreCoordinator endpoint 2021-06-07 16:46:57,363 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: use ccpd_z_ods_safe 2021-06-07 16:46:57,363 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: use ccpd_z_ods_safe 2021-06-07 16:46:57,735 INFO (main) [Logging.scala:logInfo(54)] - Initializing HiveMetastoreConnection version 1.2.1 using Spark classes. 2021-06-07 16:46:58,043 WARN (main) [HiveConf.java:initialize(2762)] - HiveConf of name hive.empty.txt.generate does not exist 2021-06-07 16:46:58,043 WARN (main) [HiveConf.java:initialize(2762)] - HiveConf of name hive.empty.orc.generate does not exist 2021-06-07 16:46:58,044 WARN (main) [HiveConf.java:initialize(2762)] - HiveConf of name hive.strict.checks.type.safety does not exist 2021-06-07 16:46:58,044 WARN (main) [HiveConf.java:initialize(2762)] - HiveConf of name hive.compare.function.adapter does not exist 2021-06-07 16:46:58,044 WARN (main) [HiveConf.java:initialize(2762)] - HiveConf of name hive.exec.schema.evolution does not exist 2021-06-07 16:46:58,044 WARN (main) [HiveConf.java:initialize(2762)] - HiveConf of name hive.decimal.trailing.zero.switch does not exist 2021-06-07 16:46:58,045 INFO (main) [HiveConf.java:setupJDOConnectionPassword(2800)] - set up jdo connection password successful 2021-06-07 16:46:58,135 INFO (main) [HiveMetaStoreClient.java:open(377)] - Trying to connect to metastore with URI thrift://bdphdp070029:9083 2021-06-07 16:46:58,169 INFO (main) [HiveMetaStoreClient.java:open(494)] - Connected to metastore. 2021-06-07 16:47:20,151 INFO (main) [SessionState.java:createPath(641)] - Created local directory: /appcom/tmp/hive/7df4d976-551c-4785-ba3e-770f465f4f06_resources 2021-06-07 16:47:20,162 INFO (main) [SessionState.java:createPath(641)] - Created HDFS directory: /tmp/hduser0334/hive-hduser0334/hduser0334/7df4d976-551c-4785-ba3e-770f465f4f06 2021-06-07 16:47:20,166 INFO (main) [SessionState.java:createPath(641)] - Created local directory: /appcom/tmp/hive/hduser0334/7df4d976-551c-4785-ba3e-770f465f4f06 2021-06-07 16:47:20,177 INFO (main) [SessionState.java:createPath(641)] - Created HDFS directory: /tmp/hduser0334/hive-hduser0334/hduser0334/7df4d976-551c-4785-ba3e-770f465f4f06/_tmp_space.db 2021-06-07 16:47:20,182 INFO (main) [Logging.scala:logInfo(54)] - Warehouse location for Hive client (version 1.2.2) is /user/hive/warehouse/hadoop 2021-06-07 16:47:20,269 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: select id_no_applyer_tag_id, id_no_contactor_tag_id, rn as degree, ID_NO_APPLYER, ID_NO_CONTACTOR, CREATED_DATETIME, LAST_MODIFIED_DATETIME from( select applyer.id_no as id_no_applyer_tag_id, contack.id_no as id_no_contactor_tag_id, applyer.id_no ID_NO_APPLYER, contack.id_no ID_NO_CONTACTOR, unix_timestamp(nvl(contack.created_datetime,to_timestamp('9999-12-31 23:59:99','yyyy-MM-dd HH:mm:ss'))) as CREATED_DATETIME, unix_timestamp(nvl(contack.last_modified_datetime,to_timestamp('9999-12-31 23:59:99','yyyy-MM-dd HH:mm:ss'))) as LAST_MODIFIED_DATETIME, row_number() over(partition by applyer.id_no,contack.id_no order by contack.last_modified_datetime desc) as rn from MERGE_TM_APPLY_INFO_ZAPS_DCN as applyer inner join MERGE_TM_CONTACT_ZAPS_ZDCN as contack on applyer.app_no=contack.app_no where applyer.ds='20210427' and contack.ds='20210427' and applyer.id_no is not null and contack.id_no is not null )t where rn=1 2021-06-07 16:47:20,269 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: select id_no_applyer_tag_id, id_no_contactor_tag_id, rn as degree, ID_NO_APPLYER, ID_NO_CONTACTOR, CREATED_DATETIME, LAST_MODIFIED_DATETIME from( select applyer.id_no as id_no_applyer_tag_id, contack.id_no as id_no_contactor_tag_id, applyer.id_no ID_NO_APPLYER, contack.id_no ID_NO_CONTACTOR, unix_timestamp(nvl(contack.created_datetime,to_timestamp('9999-12-31 23:59:99','yyyy-MM-dd HH:mm:ss'))) as CREATED_DATETIME, unix_timestamp(nvl(contack.last_modified_datetime,to_timestamp('9999-12-31 23:59:99','yyyy-MM-dd HH:mm:ss'))) as LAST_MODIFIED_DATETIME, row_number() over(partition by applyer.id_no,contack.id_no order by contack.last_modified_datetime desc) as rn from MERGE_TM_APPLY_INFO_ZAPS_DCN as applyer inner join MERGE_TM_CONTACT_ZAPS_ZDCN as contack on applyer.app_no=contack.app_no where applyer.ds='20210427' and contack.ds='20210427' and applyer.id_no is not null and contack.id_no is not null )t where rn=1 2021-06-07 16:47:20,577 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,582 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,583 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,583 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,583 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,584 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,584 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,585 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,585 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,586 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,586 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,586 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,587 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,587 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,587 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,587 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,588 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,588 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,589 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,589 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,589 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,590 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,590 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,590 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,591 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,591 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,591 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,592 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,592 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,593 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,593 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,594 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,594 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,594 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,595 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: int 2021-06-07 16:47:20,595 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,595 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,596 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,596 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: int 2021-06-07 16:47:20,596 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,597 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: int 2021-06-07 16:47:20,597 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: timestamp 2021-06-07 16:47:20,597 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: timestamp 2021-06-07 16:47:20,598 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,598 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,598 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,599 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,599 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,599 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,600 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,600 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,600 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,601 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,601 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,601 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,602 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,602 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,602 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,602 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,603 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,603 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,603 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,604 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,604 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,604 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,605 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,605 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,605 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,605 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,606 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,606 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,606 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,607 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,607 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,607 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,607 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,608 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,608 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,608 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,609 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,609 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,609 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,609 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,610 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,610 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,610 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: int 2021-06-07 16:47:20,610 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,611 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,611 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,611 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,612 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,612 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,612 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,612 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,613 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,614 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,614 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,622 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: array 2021-06-07 16:47:20,827 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,827 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,827 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,828 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,828 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,828 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,829 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: timestamp 2021-06-07 16:47:20,829 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: timestamp 2021-06-07 16:47:20,829 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,829 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,830 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,830 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,830 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,830 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:20,831 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:21,263 WARN (main) [Logging.scala:logWarning(66)] - Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.debug.maxToStringFields' in SparkEnv.conf. 2021-06-07 16:47:21,471 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 1 2021-06-07 16:47:21,472 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 4 2021-06-07 16:47:21,472 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 0 2021-06-07 16:47:21,472 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 2 2021-06-07 16:47:21,472 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 3 2021-06-07 16:47:21,472 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 5 2021-06-07 16:47:21,569 INFO (main) [Logging.scala:logInfo(54)] - Code generated in 191.542498 ms 2021-06-07 16:47:21,645 INFO (main) [Logging.scala:logInfo(54)] - Code generated in 21.677265 ms 2021-06-07 16:47:21,669 INFO (main) [Logging.scala:logInfo(54)] - Code generated in 15.050502 ms 2021-06-07 16:47:21,701 INFO (main) [Logging.scala:logInfo(54)] - Code generated in 20.598301 ms 2021-06-07 16:47:21,712 INFO (main) [Logging.scala:logInfo(54)] - Code generated in 8.807863 ms 2021-06-07 16:47:21,720 INFO (main) [Logging.scala:logInfo(54)] - Code generated in 5.98673 ms 2021-06-07 16:47:21,916 INFO (main) [Logging.scala:logInfo(54)] - Block broadcast_0 stored as values in memory (estimated size 1163.6 KB, free 365.2 MB) 2021-06-07 16:47:22,079 INFO (main) [Logging.scala:logInfo(54)] - Block broadcast_0_piece0 stored as bytes in memory (estimated size 71.2 KB, free 365.1 MB) 2021-06-07 16:47:22,081 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Added broadcast_0_piece0 in memory on bdphdp07jobs14:16197 (size: 71.2 KB, free: 366.2 MB) 2021-06-07 16:47:22,083 INFO (main) [Logging.scala:logInfo(54)] - Created broadcast 0 from 2021-06-07 16:47:22,406 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,407 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,407 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,407 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,407 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,408 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,408 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,408 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,409 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,409 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,409 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,409 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,410 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,410 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,410 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,411 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,411 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,411 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,411 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,412 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,412 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,412 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,413 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,413 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,413 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,414 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,414 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,414 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,414 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,415 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,415 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,415 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,416 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,416 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,416 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: int 2021-06-07 16:47:22,417 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,417 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,417 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,417 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: int 2021-06-07 16:47:22,418 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,418 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: int 2021-06-07 16:47:22,418 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: timestamp 2021-06-07 16:47:22,419 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: timestamp 2021-06-07 16:47:22,419 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,419 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,419 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,420 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,420 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,420 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,421 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,421 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,421 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,421 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,422 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,422 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,422 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,423 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,423 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,423 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,423 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,424 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,424 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,424 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,425 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,425 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,425 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,425 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,426 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,426 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,426 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,426 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,427 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,427 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,427 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,427 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,428 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,428 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,428 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,429 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,429 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,429 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,429 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,429 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,430 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,430 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,430 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: int 2021-06-07 16:47:22,431 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,431 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,431 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,431 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,432 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,432 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,432 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,432 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,433 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,433 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,433 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:22,967 WARN (main) [HiveConf.java:initialize(2762)] - HiveConf of name hive.empty.txt.generate does not exist 2021-06-07 16:47:22,968 WARN (main) [HiveConf.java:initialize(2762)] - HiveConf of name hive.empty.orc.generate does not exist 2021-06-07 16:47:22,968 WARN (main) [HiveConf.java:initialize(2762)] - HiveConf of name hive.strict.checks.type.safety does not exist 2021-06-07 16:47:22,968 WARN (main) [HiveConf.java:initialize(2762)] - HiveConf of name hive.compare.function.adapter does not exist 2021-06-07 16:47:22,969 WARN (main) [HiveConf.java:initialize(2762)] - HiveConf of name hive.exec.schema.evolution does not exist 2021-06-07 16:47:22,969 WARN (main) [HiveConf.java:initialize(2762)] - HiveConf of name hive.decimal.trailing.zero.switch does not exist 2021-06-07 16:47:22,970 INFO (main) [HiveConf.java:setupJDOConnectionPassword(2800)] - set up jdo connection password successful 2021-06-07 16:47:23,115 INFO (main) [FileInputFormat.java:listStatus(249)] - Total input paths to process : 1 2021-06-07 16:47:23,131 INFO (main) [FileInputFormat.java:listStatus(249)] - Total input paths to process : 1 2021-06-07 16:47:23,290 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Registering RDD 9 (show at NebulaSparkEdgeWriter.scala:83) 2021-06-07 16:47:23,290 INFO (main) [Logging.scala:logInfo(54)] - Code generated in 149.759248 ms 2021-06-07 16:47:23,296 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Got map stage job 0 (show at NebulaSparkEdgeWriter.scala:83) with 40 output partitions 2021-06-07 16:47:23,297 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Final stage: ShuffleMapStage 0 (show at NebulaSparkEdgeWriter.scala:83) 2021-06-07 16:47:23,298 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Parents of final stage: List() 2021-06-07 16:47:23,299 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Missing parents: List() 2021-06-07 16:47:23,303 INFO (main) [Logging.scala:logInfo(54)] - Block broadcast_1 stored as values in memory (estimated size 1158.3 KB, free 364.0 MB) 2021-06-07 16:47:23,303 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting ShuffleMapStage 0 (MapPartitionsRDD[9] at show at NebulaSparkEdgeWriter.scala:83), which has no missing parents 2021-06-07 16:47:23,337 INFO (main) [Logging.scala:logInfo(54)] - Block broadcast_1_piece0 stored as bytes in memory (estimated size 70.5 KB, free 363.9 MB) 2021-06-07 16:47:23,338 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Added broadcast_1_piece0 in memory on bdphdp07jobs14:16197 (size: 70.5 KB, free: 366.2 MB) 2021-06-07 16:47:23,339 INFO (main) [Logging.scala:logInfo(54)] - Created broadcast 1 from 2021-06-07 16:47:23,350 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_2 stored as values in memory (estimated size 21.6 KB, free 363.9 MB) 2021-06-07 16:47:23,363 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_2_piece0 stored as bytes in memory (estimated size 8.5 KB, free 363.9 MB) 2021-06-07 16:47:23,364 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Added broadcast_2_piece0 in memory on bdphdp07jobs14:16197 (size: 8.5 KB, free: 366.2 MB) 2021-06-07 16:47:23,365 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Created broadcast 2 from broadcast at DAGScheduler.scala:1161 2021-06-07 16:47:23,377 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting 40 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[9] at show at NebulaSparkEdgeWriter.scala:83) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14)) 2021-06-07 16:47:23,378 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Adding task set 0.0 with 40 tasks 2021-06-07 16:47:23,496 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Starting task 0.0 in stage 0.0 (TID 0, bdphdp070016, executor 1, partition 0, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:23,498 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Starting task 1.0 in stage 0.0 (TID 1, bdphdp070025, executor 2, partition 1, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:23,498 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Starting task 2.0 in stage 0.0 (TID 2, bdphdp070016, executor 1, partition 2, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:23,499 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Starting task 3.0 in stage 0.0 (TID 3, bdphdp070025, executor 2, partition 3, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:23,605 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:23,606 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:23,606 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:23,606 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:23,607 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:23,607 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:23,607 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: timestamp 2021-06-07 16:47:23,607 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: timestamp 2021-06-07 16:47:23,608 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:23,608 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:23,608 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:23,609 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:23,609 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:23,609 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:23,610 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:23,978 INFO (main) [FileInputFormat.java:listStatus(249)] - Total input paths to process : 1 2021-06-07 16:47:23,980 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Registering RDD 16 (show at NebulaSparkEdgeWriter.scala:83) 2021-06-07 16:47:23,981 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Got map stage job 1 (show at NebulaSparkEdgeWriter.scala:83) with 20 output partitions 2021-06-07 16:47:23,981 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Final stage: ShuffleMapStage 1 (show at NebulaSparkEdgeWriter.scala:83) 2021-06-07 16:47:23,981 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Parents of final stage: List() 2021-06-07 16:47:23,981 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Missing parents: List() 2021-06-07 16:47:23,981 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting ShuffleMapStage 1 (MapPartitionsRDD[16] at show at NebulaSparkEdgeWriter.scala:83), which has no missing parents 2021-06-07 16:47:23,987 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_3 stored as values in memory (estimated size 14.9 KB, free 363.9 MB) 2021-06-07 16:47:23,993 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_3_piece0 stored as bytes in memory (estimated size 7.2 KB, free 363.8 MB) 2021-06-07 16:47:23,994 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Added broadcast_3_piece0 in memory on bdphdp07jobs14:16197 (size: 7.2 KB, free: 366.1 MB) 2021-06-07 16:47:23,994 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Created broadcast 3 from broadcast at DAGScheduler.scala:1161 2021-06-07 16:47:23,995 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting 20 missing tasks from ShuffleMapStage 1 (MapPartitionsRDD[16] at show at NebulaSparkEdgeWriter.scala:83) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14)) 2021-06-07 16:47:23,996 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Adding task set 1.0 with 20 tasks 2021-06-07 16:47:26,250 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Added broadcast_2_piece0 in memory on bdphdp070016:35542 (size: 8.5 KB, free: 1048.8 MB) 2021-06-07 16:47:26,642 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Added broadcast_0_piece0 in memory on bdphdp070016:35542 (size: 71.2 KB, free: 1048.7 MB) 2021-06-07 16:47:26,659 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Added broadcast_2_piece0 in memory on bdphdp070025:18606 (size: 8.5 KB, free: 1048.8 MB) 2021-06-07 16:47:27,067 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Added broadcast_0_piece0 in memory on bdphdp070025:18606 (size: 71.2 KB, free: 1048.7 MB) 2021-06-07 16:47:27,503 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 4.0 in stage 0.0 (TID 4, bdphdp070016, executor 1, partition 4, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,505 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 5.0 in stage 0.0 (TID 5, bdphdp070016, executor 1, partition 5, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,532 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 2.0 in stage 0.0 (TID 2) in 4033 ms on bdphdp070016 (executor 1) (1/40) 2021-06-07 16:47:27,533 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 0.0 in stage 0.0 (TID 0) in 4048 ms on bdphdp070016 (executor 1) (2/40) 2021-06-07 16:47:27,584 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Starting task 6.0 in stage 0.0 (TID 6, bdphdp070016, executor 1, partition 6, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,587 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 5.0 in stage 0.0 (TID 5) in 83 ms on bdphdp070016 (executor 1) (3/40) 2021-06-07 16:47:27,589 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Starting task 7.0 in stage 0.0 (TID 7, bdphdp070016, executor 1, partition 7, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,592 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 4.0 in stage 0.0 (TID 4) in 90 ms on bdphdp070016 (executor 1) (4/40) 2021-06-07 16:47:27,640 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 8.0 in stage 0.0 (TID 8, bdphdp070016, executor 1, partition 8, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,641 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 6.0 in stage 0.0 (TID 6) in 57 ms on bdphdp070016 (executor 1) (5/40) 2021-06-07 16:47:27,645 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 9.0 in stage 0.0 (TID 9, bdphdp070016, executor 1, partition 9, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,646 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 7.0 in stage 0.0 (TID 7) in 56 ms on bdphdp070016 (executor 1) (6/40) 2021-06-07 16:47:27,699 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Starting task 10.0 in stage 0.0 (TID 10, bdphdp070016, executor 1, partition 10, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,700 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 9.0 in stage 0.0 (TID 9) in 55 ms on bdphdp070016 (executor 1) (7/40) 2021-06-07 16:47:27,700 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Starting task 11.0 in stage 0.0 (TID 11, bdphdp070016, executor 1, partition 11, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,701 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 8.0 in stage 0.0 (TID 8) in 61 ms on bdphdp070016 (executor 1) (8/40) 2021-06-07 16:47:27,748 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 12.0 in stage 0.0 (TID 12, bdphdp070016, executor 1, partition 12, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,749 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 10.0 in stage 0.0 (TID 10) in 50 ms on bdphdp070016 (executor 1) (9/40) 2021-06-07 16:47:27,761 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 13.0 in stage 0.0 (TID 13, bdphdp070016, executor 1, partition 13, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,761 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 11.0 in stage 0.0 (TID 11) in 61 ms on bdphdp070016 (executor 1) (10/40) 2021-06-07 16:47:27,814 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 14.0 in stage 0.0 (TID 14, bdphdp070016, executor 1, partition 14, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,814 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 12.0 in stage 0.0 (TID 12) in 66 ms on bdphdp070016 (executor 1) (11/40) 2021-06-07 16:47:27,819 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Starting task 15.0 in stage 0.0 (TID 15, bdphdp070016, executor 1, partition 15, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,820 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 13.0 in stage 0.0 (TID 13) in 60 ms on bdphdp070016 (executor 1) (12/40) 2021-06-07 16:47:27,856 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 16.0 in stage 0.0 (TID 16, bdphdp070016, executor 1, partition 16, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,857 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 14.0 in stage 0.0 (TID 14) in 43 ms on bdphdp070016 (executor 1) (13/40) 2021-06-07 16:47:27,863 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 17.0 in stage 0.0 (TID 17, bdphdp070016, executor 1, partition 17, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,864 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 15.0 in stage 0.0 (TID 15) in 46 ms on bdphdp070016 (executor 1) (14/40) 2021-06-07 16:47:27,901 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 18.0 in stage 0.0 (TID 18, bdphdp070016, executor 1, partition 18, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,901 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 16.0 in stage 0.0 (TID 16) in 45 ms on bdphdp070016 (executor 1) (15/40) 2021-06-07 16:47:27,910 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Starting task 19.0 in stage 0.0 (TID 19, bdphdp070016, executor 1, partition 19, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,910 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 17.0 in stage 0.0 (TID 17) in 47 ms on bdphdp070016 (executor 1) (16/40) 2021-06-07 16:47:27,949 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 20.0 in stage 0.0 (TID 20, bdphdp070016, executor 1, partition 20, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,950 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 18.0 in stage 0.0 (TID 18) in 49 ms on bdphdp070016 (executor 1) (17/40) 2021-06-07 16:47:27,953 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 21.0 in stage 0.0 (TID 21, bdphdp070025, executor 2, partition 21, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,953 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 1.0 in stage 0.0 (TID 1) in 4456 ms on bdphdp070025 (executor 2) (18/40) 2021-06-07 16:47:27,954 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 22.0 in stage 0.0 (TID 22, bdphdp070025, executor 2, partition 22, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,954 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 3.0 in stage 0.0 (TID 3) in 4456 ms on bdphdp070025 (executor 2) (19/40) 2021-06-07 16:47:27,955 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Starting task 23.0 in stage 0.0 (TID 23, bdphdp070016, executor 1, partition 23, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,956 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 19.0 in stage 0.0 (TID 19) in 47 ms on bdphdp070016 (executor 1) (20/40) 2021-06-07 16:47:27,989 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 24.0 in stage 0.0 (TID 24, bdphdp070016, executor 1, partition 24, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,989 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 20.0 in stage 0.0 (TID 20) in 40 ms on bdphdp070016 (executor 1) (21/40) 2021-06-07 16:47:27,990 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 25.0 in stage 0.0 (TID 25, bdphdp070016, executor 1, partition 25, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:27,990 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 23.0 in stage 0.0 (TID 23) in 35 ms on bdphdp070016 (executor 1) (22/40) 2021-06-07 16:47:28,006 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 26.0 in stage 0.0 (TID 26, bdphdp070025, executor 2, partition 26, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:28,006 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 22.0 in stage 0.0 (TID 22) in 53 ms on bdphdp070025 (executor 2) (23/40) 2021-06-07 16:47:28,007 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 27.0 in stage 0.0 (TID 27, bdphdp070025, executor 2, partition 27, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:28,007 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 21.0 in stage 0.0 (TID 21) in 55 ms on bdphdp070025 (executor 2) (24/40) 2021-06-07 16:47:28,026 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 28.0 in stage 0.0 (TID 28, bdphdp070016, executor 1, partition 28, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:28,026 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 25.0 in stage 0.0 (TID 25) in 36 ms on bdphdp070016 (executor 1) (25/40) 2021-06-07 16:47:28,027 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 29.0 in stage 0.0 (TID 29, bdphdp070016, executor 1, partition 29, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:28,027 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 24.0 in stage 0.0 (TID 24) in 39 ms on bdphdp070016 (executor 1) (26/40) 2021-06-07 16:47:28,045 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Starting task 30.0 in stage 0.0 (TID 30, bdphdp070025, executor 2, partition 30, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:28,045 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 27.0 in stage 0.0 (TID 27) in 38 ms on bdphdp070025 (executor 2) (27/40) 2021-06-07 16:47:28,046 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 31.0 in stage 0.0 (TID 31, bdphdp070025, executor 2, partition 31, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:28,047 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 26.0 in stage 0.0 (TID 26) in 41 ms on bdphdp070025 (executor 2) (28/40) 2021-06-07 16:47:28,064 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 32.0 in stage 0.0 (TID 32, bdphdp070016, executor 1, partition 32, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:28,064 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 29.0 in stage 0.0 (TID 29) in 37 ms on bdphdp070016 (executor 1) (29/40) 2021-06-07 16:47:28,065 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 33.0 in stage 0.0 (TID 33, bdphdp070016, executor 1, partition 33, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:28,065 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 28.0 in stage 0.0 (TID 28) in 40 ms on bdphdp070016 (executor 1) (30/40) 2021-06-07 16:47:28,080 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Starting task 34.0 in stage 0.0 (TID 34, bdphdp070025, executor 2, partition 34, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:28,080 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 31.0 in stage 0.0 (TID 31) in 34 ms on bdphdp070025 (executor 2) (31/40) 2021-06-07 16:47:28,085 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 35.0 in stage 0.0 (TID 35, bdphdp070025, executor 2, partition 35, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:28,085 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 30.0 in stage 0.0 (TID 30) in 40 ms on bdphdp070025 (executor 2) (32/40) 2021-06-07 16:47:28,115 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 36.0 in stage 0.0 (TID 36, bdphdp070025, executor 2, partition 36, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:28,116 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 34.0 in stage 0.0 (TID 34) in 35 ms on bdphdp070025 (executor 2) (33/40) 2021-06-07 16:47:28,121 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 37.0 in stage 0.0 (TID 37, bdphdp070025, executor 2, partition 37, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:28,122 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 35.0 in stage 0.0 (TID 35) in 37 ms on bdphdp070025 (executor 2) (34/40) 2021-06-07 16:47:28,146 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Starting task 38.0 in stage 0.0 (TID 38, bdphdp070025, executor 2, partition 38, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:28,147 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 36.0 in stage 0.0 (TID 36) in 32 ms on bdphdp070025 (executor 2) (35/40) 2021-06-07 16:47:28,153 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Starting task 39.0 in stage 0.0 (TID 39, bdphdp070025, executor 2, partition 39, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:28,154 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 37.0 in stage 0.0 (TID 37) in 33 ms on bdphdp070025 (executor 2) (36/40) 2021-06-07 16:47:28,185 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 0.0 in stage 1.0 (TID 40, bdphdp070025, executor 2, partition 0, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,187 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 38.0 in stage 0.0 (TID 38) in 41 ms on bdphdp070025 (executor 2) (37/40) 2021-06-07 16:47:28,189 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 1.0 in stage 1.0 (TID 41, bdphdp070025, executor 2, partition 1, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,191 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 39.0 in stage 0.0 (TID 39) in 38 ms on bdphdp070025 (executor 2) (38/40) 2021-06-07 16:47:28,218 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Added broadcast_3_piece0 in memory on bdphdp070025:18606 (size: 7.2 KB, free: 1048.7 MB) 2021-06-07 16:47:28,247 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Added broadcast_1_piece0 in memory on bdphdp070025:18606 (size: 70.5 KB, free: 1048.6 MB) 2021-06-07 16:47:28,335 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 2.0 in stage 1.0 (TID 42, bdphdp070025, executor 2, partition 2, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,335 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 1.0 in stage 1.0 (TID 41) in 146 ms on bdphdp070025 (executor 2) (1/20) 2021-06-07 16:47:28,336 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 3.0 in stage 1.0 (TID 43, bdphdp070025, executor 2, partition 3, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,336 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 0.0 in stage 1.0 (TID 40) in 151 ms on bdphdp070025 (executor 2) (2/20) 2021-06-07 16:47:28,364 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 4.0 in stage 1.0 (TID 44, bdphdp070025, executor 2, partition 4, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,365 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 3.0 in stage 1.0 (TID 43) in 29 ms on bdphdp070025 (executor 2) (3/20) 2021-06-07 16:47:28,365 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 5.0 in stage 1.0 (TID 45, bdphdp070025, executor 2, partition 5, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,366 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 2.0 in stage 1.0 (TID 42) in 31 ms on bdphdp070025 (executor 2) (4/20) 2021-06-07 16:47:28,391 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 6.0 in stage 1.0 (TID 46, bdphdp070025, executor 2, partition 6, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,392 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 7.0 in stage 1.0 (TID 47, bdphdp070025, executor 2, partition 7, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,392 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 4.0 in stage 1.0 (TID 44) in 28 ms on bdphdp070025 (executor 2) (5/20) 2021-06-07 16:47:28,393 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 5.0 in stage 1.0 (TID 45) in 28 ms on bdphdp070025 (executor 2) (6/20) 2021-06-07 16:47:28,415 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Starting task 8.0 in stage 1.0 (TID 48, bdphdp070025, executor 2, partition 8, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,416 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 6.0 in stage 1.0 (TID 46) in 25 ms on bdphdp070025 (executor 2) (7/20) 2021-06-07 16:47:28,416 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Starting task 9.0 in stage 1.0 (TID 49, bdphdp070025, executor 2, partition 9, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,417 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 7.0 in stage 1.0 (TID 47) in 25 ms on bdphdp070025 (executor 2) (8/20) 2021-06-07 16:47:28,429 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 32.0 in stage 0.0 (TID 32) in 366 ms on bdphdp070016 (executor 1) (39/40) 2021-06-07 16:47:28,435 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 33.0 in stage 0.0 (TID 33) in 371 ms on bdphdp070016 (executor 1) (40/40) 2021-06-07 16:47:28,436 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Removed TaskSet 0.0, whose tasks have all completed, from pool 2021-06-07 16:47:28,437 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - ShuffleMapStage 0 (show at NebulaSparkEdgeWriter.scala:83) finished in 5.116 s 2021-06-07 16:47:28,438 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - looking for newly runnable stages 2021-06-07 16:47:28,438 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - running: Set(ShuffleMapStage 1) 2021-06-07 16:47:28,438 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - waiting: Set() 2021-06-07 16:47:28,439 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - failed: Set() 2021-06-07 16:47:28,439 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Starting task 10.0 in stage 1.0 (TID 50, bdphdp070025, executor 2, partition 10, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,439 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 8.0 in stage 1.0 (TID 48) in 24 ms on bdphdp070025 (executor 2) (9/20) 2021-06-07 16:47:28,440 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Starting task 11.0 in stage 1.0 (TID 51, bdphdp070025, executor 2, partition 11, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,440 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 9.0 in stage 1.0 (TID 49) in 24 ms on bdphdp070025 (executor 2) (10/20) 2021-06-07 16:47:28,462 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Starting task 12.0 in stage 1.0 (TID 52, bdphdp070025, executor 2, partition 12, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,462 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 10.0 in stage 1.0 (TID 50) in 23 ms on bdphdp070025 (executor 2) (11/20) 2021-06-07 16:47:28,464 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Starting task 13.0 in stage 1.0 (TID 53, bdphdp070025, executor 2, partition 13, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,464 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 11.0 in stage 1.0 (TID 51) in 24 ms on bdphdp070025 (executor 2) (12/20) 2021-06-07 16:47:28,483 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Starting task 14.0 in stage 1.0 (TID 54, bdphdp070025, executor 2, partition 14, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,483 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 12.0 in stage 1.0 (TID 52) in 21 ms on bdphdp070025 (executor 2) (13/20) 2021-06-07 16:47:28,484 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Starting task 15.0 in stage 1.0 (TID 55, bdphdp070025, executor 2, partition 15, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,484 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 13.0 in stage 1.0 (TID 53) in 21 ms on bdphdp070025 (executor 2) (14/20) 2021-06-07 16:47:28,505 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 16.0 in stage 1.0 (TID 56, bdphdp070025, executor 2, partition 16, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,505 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 14.0 in stage 1.0 (TID 54) in 22 ms on bdphdp070025 (executor 2) (15/20) 2021-06-07 16:47:28,506 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 17.0 in stage 1.0 (TID 57, bdphdp070025, executor 2, partition 17, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,506 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 15.0 in stage 1.0 (TID 55) in 22 ms on bdphdp070025 (executor 2) (16/20) 2021-06-07 16:47:28,526 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Starting task 18.0 in stage 1.0 (TID 58, bdphdp070025, executor 2, partition 18, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,527 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 16.0 in stage 1.0 (TID 56) in 22 ms on bdphdp070025 (executor 2) (17/20) 2021-06-07 16:47:28,527 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Starting task 19.0 in stage 1.0 (TID 59, bdphdp070025, executor 2, partition 19, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:28,528 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 17.0 in stage 1.0 (TID 57) in 22 ms on bdphdp070025 (executor 2) (18/20) 2021-06-07 16:47:28,547 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 18.0 in stage 1.0 (TID 58) in 21 ms on bdphdp070025 (executor 2) (19/20) 2021-06-07 16:47:28,547 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 19.0 in stage 1.0 (TID 59) in 20 ms on bdphdp070025 (executor 2) (20/20) 2021-06-07 16:47:28,547 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Removed TaskSet 1.0, whose tasks have all completed, from pool 2021-06-07 16:47:28,548 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - ShuffleMapStage 1 (show at NebulaSparkEdgeWriter.scala:83) finished in 4.563 s 2021-06-07 16:47:28,548 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - looking for newly runnable stages 2021-06-07 16:47:28,548 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - running: Set() 2021-06-07 16:47:28,548 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - waiting: Set() 2021-06-07 16:47:28,548 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - failed: Set() 2021-06-07 16:47:28,549 INFO (main) [Logging.scala:logInfo(54)] - advisoryTargetPostShuffleInputSize: 67108864, targetPostShuffleInputSize 67108864. 2021-06-07 16:47:28,596 INFO (main) [Logging.scala:logInfo(54)] - Code generated in 10.197756 ms 2021-06-07 16:47:28,614 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Registering RDD 23 (show at NebulaSparkEdgeWriter.scala:83) 2021-06-07 16:47:28,615 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Got map stage job 2 (show at NebulaSparkEdgeWriter.scala:83) with 1 output partitions 2021-06-07 16:47:28,615 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Final stage: ShuffleMapStage 4 (show at NebulaSparkEdgeWriter.scala:83) 2021-06-07 16:47:28,615 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Parents of final stage: List(ShuffleMapStage 2, ShuffleMapStage 3) 2021-06-07 16:47:28,615 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Missing parents: List() 2021-06-07 16:47:28,615 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting ShuffleMapStage 4 (MapPartitionsRDD[23] at show at NebulaSparkEdgeWriter.scala:83), which has no missing parents 2021-06-07 16:47:28,622 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_4 stored as values in memory (estimated size 36.4 KB, free 363.8 MB) 2021-06-07 16:47:28,625 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_4_piece0 stored as bytes in memory (estimated size 15.5 KB, free 363.8 MB) 2021-06-07 16:47:28,625 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Added broadcast_4_piece0 in memory on bdphdp07jobs14:16197 (size: 15.5 KB, free: 366.1 MB) 2021-06-07 16:47:28,626 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Created broadcast 4 from broadcast at DAGScheduler.scala:1161 2021-06-07 16:47:28,626 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting 1 missing tasks from ShuffleMapStage 4 (MapPartitionsRDD[23] at show at NebulaSparkEdgeWriter.scala:83) (first 15 tasks are for partitions Vector(0)) 2021-06-07 16:47:28,626 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Adding task set 4.0 with 1 tasks 2021-06-07 16:47:28,630 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 0.0 in stage 4.0 (TID 60, bdphdp070016, executor 1, partition 0, NODE_LOCAL, 8097 bytes) 2021-06-07 16:47:28,659 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Added broadcast_4_piece0 in memory on bdphdp070016:35542 (size: 15.5 KB, free: 1048.7 MB) 2021-06-07 16:47:28,759 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Asked to send map output locations for shuffle 0 to 10.107.97.188:62930 2021-06-07 16:47:29,340 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Asked to send map output locations for shuffle 1 to 10.107.97.188:62930 2021-06-07 16:47:30,027 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 0.0 in stage 4.0 (TID 60) in 1400 ms on bdphdp070016 (executor 1) (1/1) 2021-06-07 16:47:30,027 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Removed TaskSet 4.0, whose tasks have all completed, from pool 2021-06-07 16:47:30,028 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - ShuffleMapStage 4 (show at NebulaSparkEdgeWriter.scala:83) finished in 1.410 s 2021-06-07 16:47:30,028 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - looking for newly runnable stages 2021-06-07 16:47:30,028 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - running: Set() 2021-06-07 16:47:30,028 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - waiting: Set() 2021-06-07 16:47:30,028 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - failed: Set() 2021-06-07 16:47:30,028 INFO (main) [Logging.scala:logInfo(54)] - advisoryTargetPostShuffleInputSize: 67108864, targetPostShuffleInputSize 67108864. 2021-06-07 16:47:30,068 INFO (main) [Logging.scala:logInfo(54)] - Starting job: show at NebulaSparkEdgeWriter.scala:83 2021-06-07 16:47:30,071 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Got job 3 (show at NebulaSparkEdgeWriter.scala:83) with 1 output partitions 2021-06-07 16:47:30,071 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Final stage: ResultStage 8 (show at NebulaSparkEdgeWriter.scala:83) 2021-06-07 16:47:30,072 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Parents of final stage: List(ShuffleMapStage 7) 2021-06-07 16:47:30,072 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Missing parents: List() 2021-06-07 16:47:30,073 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting ResultStage 8 (MapPartitionsRDD[29] at show at NebulaSparkEdgeWriter.scala:83), which has no missing parents 2021-06-07 16:47:30,077 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_5 stored as values in memory (estimated size 38.2 KB, free 363.8 MB) 2021-06-07 16:47:30,080 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_5_piece0 stored as bytes in memory (estimated size 17.0 KB, free 363.7 MB) 2021-06-07 16:47:30,081 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Added broadcast_5_piece0 in memory on bdphdp07jobs14:16197 (size: 17.0 KB, free: 366.1 MB) 2021-06-07 16:47:30,081 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Created broadcast 5 from broadcast at DAGScheduler.scala:1161 2021-06-07 16:47:30,083 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting 1 missing tasks from ResultStage 8 (MapPartitionsRDD[29] at show at NebulaSparkEdgeWriter.scala:83) (first 15 tasks are for partitions Vector(0)) 2021-06-07 16:47:30,083 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Adding task set 8.0 with 1 tasks 2021-06-07 16:47:30,084 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 0.0 in stage 8.0 (TID 61, bdphdp070016, executor 1, partition 0, PROCESS_LOCAL, 7780 bytes) 2021-06-07 16:47:30,108 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Added broadcast_5_piece0 in memory on bdphdp070016:35542 (size: 17.0 KB, free: 1048.7 MB) 2021-06-07 16:47:30,169 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Asked to send map output locations for shuffle 2 to 10.107.97.188:62930 2021-06-07 16:47:30,327 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 0.0 in stage 8.0 (TID 61) in 244 ms on bdphdp070016 (executor 1) (1/1) 2021-06-07 16:47:30,327 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Removed TaskSet 8.0, whose tasks have all completed, from pool 2021-06-07 16:47:30,328 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - ResultStage 8 (show at NebulaSparkEdgeWriter.scala:83) finished in 0.254 s 2021-06-07 16:47:30,329 INFO (main) [Logging.scala:logInfo(54)] - Job 3 finished: show at NebulaSparkEdgeWriter.scala:83, took 0.260338 s +--------------------+----------------------+------+------------------+------------------+----------------+----------------------+ |id_no_applyer_tag_id|id_no_contactor_tag_id|degree|ID_NO_APPLYER |ID_NO_CONTACTOR |CREATED_DATETIME|LAST_MODIFIED_DATETIME| +--------------------+----------------------+------+------------------+------------------+----------------+----------------------+ |500101198506020014 |500101198506020014 |1 |500101198506020014|500101198506020014|1587455660 |1587455664 | |610112199503110857 |610112199503110857 |1 |610112199503110857|610112199503110857|1593413000 |1593413004 | +--------------------+----------------------+------+------------------+------------------+----------------+----------------------+ 2021-06-07 16:47:30,421 INFO (main) [NebulaConfig.scala:check(417)] - NebulaWriteEdgeConfig={space=dcp_zaps_adm_prod_01,edgeName=APPLYER2CONTACTOR,srcField=id_no_applyer_tag_id,srcPolicy=null，dstField=id_no_contactor_tag_id,dstPolicy=null,rankField=degree} 2021-06-07 16:47:30,457 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 66 2021-06-07 16:47:30,457 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 70 2021-06-07 16:47:30,457 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 131 2021-06-07 16:47:30,457 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 91 2021-06-07 16:47:30,457 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 136 2021-06-07 16:47:30,458 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 90 2021-06-07 16:47:30,458 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 113 2021-06-07 16:47:30,458 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 116 2021-06-07 16:47:30,458 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 109 2021-06-07 16:47:30,458 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 123 2021-06-07 16:47:30,458 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 59 2021-06-07 16:47:30,458 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 95 2021-06-07 16:47:30,458 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 122 2021-06-07 16:47:30,458 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 125 2021-06-07 16:47:30,458 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 76 2021-06-07 16:47:30,469 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Removed broadcast_5_piece0 on bdphdp07jobs14:16197 in memory (size: 17.0 KB, free: 366.1 MB) 2021-06-07 16:47:30,472 INFO (main) [NebulaDataSource.scala:createWriter(69)] - create writer 2021-06-07 16:47:30,472 INFO (main) [NebulaDataSource.scala:createWriter(70)] - options {graphaddress=172.21.2.220:3699, spacename=dcp_zaps_adm_prod_01, rankfiled=degree, srcasprop=false, batch=2000, metaaddress=172.21.2.220:45500, label=APPLYER2CONTACTOR, srcpolicy=, rankasprop=false, type=edge, connectionretry=1, timeout=6000, srcvertexfield=id_no_applyer_tag_id, executionretry=1, dstpolicy=, passwd=Wedg#SAD@admp&123., dstvertexfield=id_no_contactor_tag_id, user=clgd_dcp_test, dstasprop=false} 2021-06-07 16:47:30,480 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Removed broadcast_5_piece0 on bdphdp070016:35542 in memory (size: 17.0 KB, free: 1048.7 MB) 2021-06-07 16:47:30,488 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 129 2021-06-07 16:47:30,488 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 134 2021-06-07 16:47:30,488 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 106 2021-06-07 16:47:30,488 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 86 2021-06-07 16:47:30,488 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 112 2021-06-07 16:47:30,488 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 118 2021-06-07 16:47:30,488 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 82 2021-06-07 16:47:30,488 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 56 2021-06-07 16:47:30,488 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 108 2021-06-07 16:47:30,488 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 67 2021-06-07 16:47:30,489 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 75 2021-06-07 16:47:30,489 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 133 2021-06-07 16:47:30,489 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 93 2021-06-07 16:47:30,489 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 102 2021-06-07 16:47:30,489 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 132 2021-06-07 16:47:30,489 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 100 2021-06-07 16:47:30,489 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 77 2021-06-07 16:47:30,489 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 87 2021-06-07 16:47:30,491 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Removed broadcast_3_piece0 on bdphdp07jobs14:16197 in memory (size: 7.2 KB, free: 366.1 MB) 2021-06-07 16:47:30,500 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Removed broadcast_3_piece0 on bdphdp070025:18606 in memory (size: 7.2 KB, free: 1048.7 MB) 2021-06-07 16:47:30,507 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 103 2021-06-07 16:47:30,507 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 104 2021-06-07 16:47:30,507 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 64 2021-06-07 16:47:30,507 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 68 2021-06-07 16:47:30,507 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 94 2021-06-07 16:47:30,507 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 58 2021-06-07 16:47:30,507 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 57 2021-06-07 16:47:30,507 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 61 2021-06-07 16:47:30,507 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 97 2021-06-07 16:47:30,507 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 117 2021-06-07 16:47:30,507 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 65 2021-06-07 16:47:30,507 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 135 2021-06-07 16:47:30,512 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Removed broadcast_2_piece0 on bdphdp07jobs14:16197 in memory (size: 8.5 KB, free: 366.1 MB) 2021-06-07 16:47:30,514 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Removed broadcast_2_piece0 on bdphdp070025:18606 in memory (size: 8.5 KB, free: 1048.7 MB) 2021-06-07 16:47:30,514 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Removed broadcast_2_piece0 on bdphdp070016:35542 in memory (size: 8.5 KB, free: 1048.7 MB) 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 83 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 73 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 114 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 120 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 96 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 99 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 79 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 60 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 89 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 127 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 121 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 84 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 62 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 98 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 124 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 63 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 72 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 119 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 115 2021-06-07 16:47:30,520 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 101 2021-06-07 16:47:30,521 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 130 2021-06-07 16:47:30,521 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 78 2021-06-07 16:47:30,522 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Removed broadcast_4_piece0 on bdphdp07jobs14:16197 in memory (size: 15.5 KB, free: 366.2 MB) 2021-06-07 16:47:30,526 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Removed broadcast_4_piece0 on bdphdp070016:35542 in memory (size: 15.5 KB, free: 1048.7 MB) 2021-06-07 16:47:30,528 INFO (main) [Logging.scala:logInfo(54)] - Code generated in 11.730457 ms 2021-06-07 16:47:30,531 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 128 2021-06-07 16:47:30,531 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 74 2021-06-07 16:47:30,531 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 80 2021-06-07 16:47:30,531 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 110 2021-06-07 16:47:30,531 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 81 2021-06-07 16:47:30,531 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 85 2021-06-07 16:47:30,531 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 126 2021-06-07 16:47:30,531 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 92 2021-06-07 16:47:30,531 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 111 2021-06-07 16:47:30,531 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 69 2021-06-07 16:47:30,531 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 71 2021-06-07 16:47:30,531 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 105 2021-06-07 16:47:30,531 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 88 2021-06-07 16:47:30,531 INFO (Spark Context Cleaner) [Logging.scala:logInfo(54)] - Cleaned accumulator 107 2021-06-07 16:47:30,542 INFO (main) [Logging.scala:logInfo(54)] - Block broadcast_6 stored as values in memory (estimated size 1163.6 KB, free 362.8 MB) 2021-06-07 16:47:30,562 INFO (main) [Logging.scala:logInfo(54)] - Block broadcast_6_piece0 stored as bytes in memory (estimated size 71.2 KB, free 362.7 MB) 2021-06-07 16:47:30,563 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Added broadcast_6_piece0 in memory on bdphdp07jobs14:16197 (size: 71.2 KB, free: 366.1 MB) 2021-06-07 16:47:30,563 INFO (main) [Logging.scala:logInfo(54)] - Created broadcast 6 from 2021-06-07 16:47:32,230 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,230 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,232 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,233 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,233 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,233 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,234 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,234 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,234 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,234 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,235 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,235 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,235 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,236 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,236 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,236 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,236 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,237 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,237 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,237 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,237 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,238 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,238 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,238 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,239 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,239 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,239 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,239 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,240 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,240 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,241 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,241 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,241 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,242 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,242 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: int 2021-06-07 16:47:32,243 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,243 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,243 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,244 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: int 2021-06-07 16:47:32,244 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,244 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: int 2021-06-07 16:47:32,245 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: timestamp 2021-06-07 16:47:32,245 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: timestamp 2021-06-07 16:47:32,245 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,246 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,246 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,247 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,247 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,247 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,248 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,248 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,248 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,249 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,249 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,250 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,250 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,250 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,251 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,251 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,251 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,252 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,252 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,252 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,253 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,253 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,253 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,254 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,254 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,254 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,255 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,255 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,255 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,256 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,256 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,256 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,256 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,257 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,257 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,257 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,258 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,258 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,258 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,259 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,259 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,259 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,260 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: int 2021-06-07 16:47:32,260 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,260 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,261 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,261 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,261 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,262 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,262 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,262 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,263 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,263 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,263 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:32,868 INFO (main) [FileInputFormat.java:listStatus(249)] - Total input paths to process : 1 2021-06-07 16:47:32,877 INFO (main) [FileInputFormat.java:listStatus(249)] - Total input paths to process : 1 2021-06-07 16:47:32,879 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Registering RDD 39 (save at package.scala:298) 2021-06-07 16:47:32,879 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Got map stage job 4 (save at package.scala:298) with 40 output partitions 2021-06-07 16:47:32,879 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Final stage: ShuffleMapStage 9 (save at package.scala:298) 2021-06-07 16:47:32,879 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Parents of final stage: List() 2021-06-07 16:47:32,880 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Missing parents: List() 2021-06-07 16:47:32,880 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting ShuffleMapStage 9 (MapPartitionsRDD[39] at save at package.scala:298), which has no missing parents 2021-06-07 16:47:32,888 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_8 stored as values in memory (estimated size 21.6 KB, free 362.7 MB) 2021-06-07 16:47:32,889 INFO (main) [Logging.scala:logInfo(54)] - Block broadcast_7 stored as values in memory (estimated size 1158.3 KB, free 361.5 MB) 2021-06-07 16:47:32,893 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_8_piece0 stored as bytes in memory (estimated size 8.5 KB, free 361.5 MB) 2021-06-07 16:47:32,893 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Added broadcast_8_piece0 in memory on bdphdp07jobs14:16197 (size: 8.5 KB, free: 366.1 MB) 2021-06-07 16:47:32,894 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Created broadcast 8 from broadcast at DAGScheduler.scala:1161 2021-06-07 16:47:32,894 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting 40 missing tasks from ShuffleMapStage 9 (MapPartitionsRDD[39] at save at package.scala:298) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14)) 2021-06-07 16:47:32,894 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Adding task set 9.0 with 40 tasks 2021-06-07 16:47:32,897 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 0.0 in stage 9.0 (TID 62, bdphdp070025, executor 2, partition 0, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:32,897 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 1.0 in stage 9.0 (TID 63, bdphdp070016, executor 1, partition 1, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:32,897 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 2.0 in stage 9.0 (TID 64, bdphdp070025, executor 2, partition 2, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:32,897 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 3.0 in stage 9.0 (TID 65, bdphdp070016, executor 1, partition 3, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:32,918 INFO (main) [Logging.scala:logInfo(54)] - Block broadcast_7_piece0 stored as bytes in memory (estimated size 70.5 KB, free 361.5 MB) 2021-06-07 16:47:32,919 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Added broadcast_8_piece0 in memory on bdphdp070016:35542 (size: 8.5 KB, free: 1048.7 MB) 2021-06-07 16:47:32,920 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Added broadcast_7_piece0 in memory on bdphdp07jobs14:16197 (size: 70.5 KB, free: 366.0 MB) 2021-06-07 16:47:32,920 INFO (main) [Logging.scala:logInfo(54)] - Created broadcast 7 from 2021-06-07 16:47:32,925 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Added broadcast_8_piece0 in memory on bdphdp070025:18606 (size: 8.5 KB, free: 1048.7 MB) 2021-06-07 16:47:32,947 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Added broadcast_6_piece0 in memory on bdphdp070016:35542 (size: 71.2 KB, free: 1048.7 MB) 2021-06-07 16:47:32,951 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Added broadcast_6_piece0 in memory on bdphdp070025:18606 (size: 71.2 KB, free: 1048.6 MB) 2021-06-07 16:47:33,008 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 4.0 in stage 9.0 (TID 66, bdphdp070016, executor 1, partition 4, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,009 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 3.0 in stage 9.0 (TID 65) in 112 ms on bdphdp070016 (executor 1) (1/40) 2021-06-07 16:47:33,012 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 5.0 in stage 9.0 (TID 67, bdphdp070016, executor 1, partition 5, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,013 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 1.0 in stage 9.0 (TID 63) in 116 ms on bdphdp070016 (executor 1) (2/40) 2021-06-07 16:47:33,031 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Starting task 6.0 in stage 9.0 (TID 68, bdphdp070025, executor 2, partition 6, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,031 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Starting task 7.0 in stage 9.0 (TID 69, bdphdp070025, executor 2, partition 7, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,032 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 2.0 in stage 9.0 (TID 64) in 135 ms on bdphdp070025 (executor 2) (3/40) 2021-06-07 16:47:33,035 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 0.0 in stage 9.0 (TID 62) in 139 ms on bdphdp070025 (executor 2) (4/40) 2021-06-07 16:47:33,046 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 8.0 in stage 9.0 (TID 70, bdphdp070016, executor 1, partition 8, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,046 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 5.0 in stage 9.0 (TID 67) in 34 ms on bdphdp070016 (executor 1) (5/40) 2021-06-07 16:47:33,051 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 9.0 in stage 9.0 (TID 71, bdphdp070016, executor 1, partition 9, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,051 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 4.0 in stage 9.0 (TID 66) in 43 ms on bdphdp070016 (executor 1) (6/40) 2021-06-07 16:47:33,070 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 10.0 in stage 9.0 (TID 72, bdphdp070025, executor 2, partition 10, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,070 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 6.0 in stage 9.0 (TID 68) in 40 ms on bdphdp070025 (executor 2) (7/40) 2021-06-07 16:47:33,075 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Starting task 11.0 in stage 9.0 (TID 73, bdphdp070025, executor 2, partition 11, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,076 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 7.0 in stage 9.0 (TID 69) in 45 ms on bdphdp070025 (executor 2) (8/40) 2021-06-07 16:47:33,083 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 12.0 in stage 9.0 (TID 74, bdphdp070016, executor 1, partition 12, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,083 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 8.0 in stage 9.0 (TID 70) in 37 ms on bdphdp070016 (executor 1) (9/40) 2021-06-07 16:47:33,095 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 13.0 in stage 9.0 (TID 75, bdphdp070016, executor 1, partition 13, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,095 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 9.0 in stage 9.0 (TID 71) in 44 ms on bdphdp070016 (executor 1) (10/40) 2021-06-07 16:47:33,114 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 14.0 in stage 9.0 (TID 76, bdphdp070025, executor 2, partition 14, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,114 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 11.0 in stage 9.0 (TID 73) in 39 ms on bdphdp070025 (executor 2) (11/40) 2021-06-07 16:47:33,117 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Starting task 15.0 in stage 9.0 (TID 77, bdphdp070025, executor 2, partition 15, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,118 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 10.0 in stage 9.0 (TID 72) in 48 ms on bdphdp070025 (executor 2) (12/40) 2021-06-07 16:47:33,126 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 16.0 in stage 9.0 (TID 78, bdphdp070016, executor 1, partition 16, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,127 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 12.0 in stage 9.0 (TID 74) in 44 ms on bdphdp070016 (executor 1) (13/40) 2021-06-07 16:47:33,135 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 17.0 in stage 9.0 (TID 79, bdphdp070016, executor 1, partition 17, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,136 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 13.0 in stage 9.0 (TID 75) in 41 ms on bdphdp070016 (executor 1) (14/40) 2021-06-07 16:47:33,153 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Starting task 18.0 in stage 9.0 (TID 80, bdphdp070025, executor 2, partition 18, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,153 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 15.0 in stage 9.0 (TID 77) in 36 ms on bdphdp070025 (executor 2) (15/40) 2021-06-07 16:47:33,154 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Starting task 19.0 in stage 9.0 (TID 81, bdphdp070025, executor 2, partition 19, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,155 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 14.0 in stage 9.0 (TID 76) in 41 ms on bdphdp070025 (executor 2) (16/40) 2021-06-07 16:47:33,183 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 20.0 in stage 9.0 (TID 82, bdphdp070016, executor 1, partition 20, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,184 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 16.0 in stage 9.0 (TID 78) in 58 ms on bdphdp070016 (executor 1) (17/40) 2021-06-07 16:47:33,187 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 21.0 in stage 9.0 (TID 83, bdphdp070025, executor 2, partition 21, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,188 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 19.0 in stage 9.0 (TID 81) in 34 ms on bdphdp070025 (executor 2) (18/40) 2021-06-07 16:47:33,192 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 22.0 in stage 9.0 (TID 84, bdphdp070025, executor 2, partition 22, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,192 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 18.0 in stage 9.0 (TID 80) in 39 ms on bdphdp070025 (executor 2) (19/40) 2021-06-07 16:47:33,193 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Starting task 23.0 in stage 9.0 (TID 85, bdphdp070016, executor 1, partition 23, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,193 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 17.0 in stage 9.0 (TID 79) in 58 ms on bdphdp070016 (executor 1) (20/40) 2021-06-07 16:47:33,216 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 24.0 in stage 9.0 (TID 86, bdphdp070025, executor 2, partition 24, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,216 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 21.0 in stage 9.0 (TID 83) in 29 ms on bdphdp070025 (executor 2) (21/40) 2021-06-07 16:47:33,218 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 25.0 in stage 9.0 (TID 87, bdphdp070025, executor 2, partition 25, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,218 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 22.0 in stage 9.0 (TID 84) in 26 ms on bdphdp070025 (executor 2) (22/40) 2021-06-07 16:47:33,223 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Starting task 26.0 in stage 9.0 (TID 88, bdphdp070016, executor 1, partition 26, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,223 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 23.0 in stage 9.0 (TID 85) in 30 ms on bdphdp070016 (executor 1) (23/40) 2021-06-07 16:47:33,225 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Starting task 27.0 in stage 9.0 (TID 89, bdphdp070016, executor 1, partition 27, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,226 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 20.0 in stage 9.0 (TID 82) in 43 ms on bdphdp070016 (executor 1) (24/40) 2021-06-07 16:47:33,241 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 28.0 in stage 9.0 (TID 90, bdphdp070025, executor 2, partition 28, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,242 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 24.0 in stage 9.0 (TID 86) in 26 ms on bdphdp070025 (executor 2) (25/40) 2021-06-07 16:47:33,243 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 29.0 in stage 9.0 (TID 91, bdphdp070025, executor 2, partition 29, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,243 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 25.0 in stage 9.0 (TID 87) in 25 ms on bdphdp070025 (executor 2) (26/40) 2021-06-07 16:47:33,248 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 30.0 in stage 9.0 (TID 92, bdphdp070016, executor 1, partition 30, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,248 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 26.0 in stage 9.0 (TID 88) in 25 ms on bdphdp070016 (executor 1) (27/40) 2021-06-07 16:47:33,255 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Starting task 31.0 in stage 9.0 (TID 93, bdphdp070016, executor 1, partition 31, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,255 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 27.0 in stage 9.0 (TID 89) in 30 ms on bdphdp070016 (executor 1) (28/40) 2021-06-07 16:47:33,269 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 32.0 in stage 9.0 (TID 94, bdphdp070025, executor 2, partition 32, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,269 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 29.0 in stage 9.0 (TID 91) in 26 ms on bdphdp070025 (executor 2) (29/40) 2021-06-07 16:47:33,291 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 33.0 in stage 9.0 (TID 95, bdphdp070016, executor 1, partition 33, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,291 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 30.0 in stage 9.0 (TID 92) in 43 ms on bdphdp070016 (executor 1) (30/40) 2021-06-07 16:47:33,295 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Starting task 34.0 in stage 9.0 (TID 96, bdphdp070016, executor 1, partition 34, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,295 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 31.0 in stage 9.0 (TID 93) in 40 ms on bdphdp070016 (executor 1) (31/40) 2021-06-07 16:47:33,322 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Starting task 35.0 in stage 9.0 (TID 97, bdphdp070025, executor 2, partition 35, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,322 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 28.0 in stage 9.0 (TID 90) in 81 ms on bdphdp070025 (executor 2) (32/40) 2021-06-07 16:47:33,324 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 36.0 in stage 9.0 (TID 98, bdphdp070016, executor 1, partition 36, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,325 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 33.0 in stage 9.0 (TID 95) in 35 ms on bdphdp070016 (executor 1) (33/40) 2021-06-07 16:47:33,327 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 37.0 in stage 9.0 (TID 99, bdphdp070016, executor 1, partition 37, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,327 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 34.0 in stage 9.0 (TID 96) in 33 ms on bdphdp070016 (executor 1) (34/40) 2021-06-07 16:47:33,348 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 38.0 in stage 9.0 (TID 100, bdphdp070016, executor 1, partition 38, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,348 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 36.0 in stage 9.0 (TID 98) in 24 ms on bdphdp070016 (executor 1) (35/40) 2021-06-07 16:47:33,351 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Starting task 39.0 in stage 9.0 (TID 101, bdphdp070025, executor 2, partition 39, RACK_LOCAL, 8096 bytes) 2021-06-07 16:47:33,351 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 35.0 in stage 9.0 (TID 97) in 29 ms on bdphdp070025 (executor 2) (36/40) 2021-06-07 16:47:33,352 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 32.0 in stage 9.0 (TID 94) in 83 ms on bdphdp070025 (executor 2) (37/40) 2021-06-07 16:47:33,352 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 37.0 in stage 9.0 (TID 99) in 25 ms on bdphdp070016 (executor 1) (38/40) 2021-06-07 16:47:33,371 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 38.0 in stage 9.0 (TID 100) in 23 ms on bdphdp070016 (executor 1) (39/40) 2021-06-07 16:47:33,373 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 39.0 in stage 9.0 (TID 101) in 22 ms on bdphdp070025 (executor 2) (40/40) 2021-06-07 16:47:33,373 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Removed TaskSet 9.0, whose tasks have all completed, from pool 2021-06-07 16:47:33,373 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - ShuffleMapStage 9 (save at package.scala:298) finished in 0.488 s 2021-06-07 16:47:33,373 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - looking for newly runnable stages 2021-06-07 16:47:33,373 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - running: Set() 2021-06-07 16:47:33,373 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - waiting: Set() 2021-06-07 16:47:33,373 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - failed: Set() 2021-06-07 16:47:33,959 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:33,959 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:33,960 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:33,960 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:33,960 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:33,960 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:33,961 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: timestamp 2021-06-07 16:47:33,961 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: timestamp 2021-06-07 16:47:33,961 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:33,962 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:33,962 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:33,962 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:33,962 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:33,962 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:33,963 INFO (main) [Logging.scala:logInfo(54)] - Parsing command: string 2021-06-07 16:47:34,629 INFO (main) [FileInputFormat.java:listStatus(249)] - Total input paths to process : 1 2021-06-07 16:47:34,631 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Registering RDD 46 (save at package.scala:298) 2021-06-07 16:47:34,631 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Got map stage job 5 (save at package.scala:298) with 20 output partitions 2021-06-07 16:47:34,631 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Final stage: ShuffleMapStage 10 (save at package.scala:298) 2021-06-07 16:47:34,631 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Parents of final stage: List() 2021-06-07 16:47:34,632 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Missing parents: List() 2021-06-07 16:47:34,632 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting ShuffleMapStage 10 (MapPartitionsRDD[46] at save at package.scala:298), which has no missing parents 2021-06-07 16:47:34,636 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_9 stored as values in memory (estimated size 14.9 KB, free 361.4 MB) 2021-06-07 16:47:34,640 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_9_piece0 stored as bytes in memory (estimated size 7.2 KB, free 361.4 MB) 2021-06-07 16:47:34,641 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Added broadcast_9_piece0 in memory on bdphdp07jobs14:16197 (size: 7.2 KB, free: 366.0 MB) 2021-06-07 16:47:34,641 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Created broadcast 9 from broadcast at DAGScheduler.scala:1161 2021-06-07 16:47:34,642 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting 20 missing tasks from ShuffleMapStage 10 (MapPartitionsRDD[46] at save at package.scala:298) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14)) 2021-06-07 16:47:34,642 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Adding task set 10.0 with 20 tasks 2021-06-07 16:47:34,643 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 0.0 in stage 10.0 (TID 102, bdphdp070025, executor 2, partition 0, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,643 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 1.0 in stage 10.0 (TID 103, bdphdp070025, executor 2, partition 1, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,663 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Added broadcast_9_piece0 in memory on bdphdp070025:18606 (size: 7.2 KB, free: 1048.6 MB) 2021-06-07 16:47:34,689 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Added broadcast_7_piece0 in memory on bdphdp070025:18606 (size: 70.5 KB, free: 1048.5 MB) 2021-06-07 16:47:34,719 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Starting task 2.0 in stage 10.0 (TID 104, bdphdp070025, executor 2, partition 2, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,719 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 0.0 in stage 10.0 (TID 102) in 76 ms on bdphdp070025 (executor 2) (1/20) 2021-06-07 16:47:34,720 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Starting task 3.0 in stage 10.0 (TID 105, bdphdp070025, executor 2, partition 3, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,721 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 1.0 in stage 10.0 (TID 103) in 78 ms on bdphdp070025 (executor 2) (2/20) 2021-06-07 16:47:34,735 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 4.0 in stage 10.0 (TID 106, bdphdp070025, executor 2, partition 4, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,735 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 2.0 in stage 10.0 (TID 104) in 16 ms on bdphdp070025 (executor 2) (3/20) 2021-06-07 16:47:34,739 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Starting task 5.0 in stage 10.0 (TID 107, bdphdp070025, executor 2, partition 5, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,739 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 3.0 in stage 10.0 (TID 105) in 19 ms on bdphdp070025 (executor 2) (4/20) 2021-06-07 16:47:34,750 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Starting task 6.0 in stage 10.0 (TID 108, bdphdp070025, executor 2, partition 6, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,750 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 4.0 in stage 10.0 (TID 106) in 15 ms on bdphdp070025 (executor 2) (5/20) 2021-06-07 16:47:34,754 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Starting task 7.0 in stage 10.0 (TID 109, bdphdp070025, executor 2, partition 7, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,754 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 5.0 in stage 10.0 (TID 107) in 15 ms on bdphdp070025 (executor 2) (6/20) 2021-06-07 16:47:34,764 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 8.0 in stage 10.0 (TID 110, bdphdp070025, executor 2, partition 8, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,765 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 6.0 in stage 10.0 (TID 108) in 15 ms on bdphdp070025 (executor 2) (7/20) 2021-06-07 16:47:34,767 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Starting task 9.0 in stage 10.0 (TID 111, bdphdp070025, executor 2, partition 9, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,767 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 7.0 in stage 10.0 (TID 109) in 14 ms on bdphdp070025 (executor 2) (8/20) 2021-06-07 16:47:34,779 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 10.0 in stage 10.0 (TID 112, bdphdp070025, executor 2, partition 10, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,779 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 8.0 in stage 10.0 (TID 110) in 15 ms on bdphdp070025 (executor 2) (9/20) 2021-06-07 16:47:34,782 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Starting task 11.0 in stage 10.0 (TID 113, bdphdp070025, executor 2, partition 11, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,783 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 9.0 in stage 10.0 (TID 111) in 16 ms on bdphdp070025 (executor 2) (10/20) 2021-06-07 16:47:34,796 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Starting task 12.0 in stage 10.0 (TID 114, bdphdp070025, executor 2, partition 12, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,797 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 10.0 in stage 10.0 (TID 112) in 19 ms on bdphdp070025 (executor 2) (11/20) 2021-06-07 16:47:34,798 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 13.0 in stage 10.0 (TID 115, bdphdp070025, executor 2, partition 13, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,798 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 11.0 in stage 10.0 (TID 113) in 16 ms on bdphdp070025 (executor 2) (12/20) 2021-06-07 16:47:34,810 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 14.0 in stage 10.0 (TID 116, bdphdp070025, executor 2, partition 14, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,811 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 12.0 in stage 10.0 (TID 114) in 15 ms on bdphdp070025 (executor 2) (13/20) 2021-06-07 16:47:34,812 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Starting task 15.0 in stage 10.0 (TID 117, bdphdp070025, executor 2, partition 15, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,812 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 13.0 in stage 10.0 (TID 115) in 15 ms on bdphdp070025 (executor 2) (14/20) 2021-06-07 16:47:34,824 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Starting task 16.0 in stage 10.0 (TID 118, bdphdp070025, executor 2, partition 16, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,824 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 14.0 in stage 10.0 (TID 116) in 14 ms on bdphdp070025 (executor 2) (15/20) 2021-06-07 16:47:34,826 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - Starting task 17.0 in stage 10.0 (TID 119, bdphdp070025, executor 2, partition 17, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,826 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 15.0 in stage 10.0 (TID 117) in 14 ms on bdphdp070025 (executor 2) (16/20) 2021-06-07 16:47:34,836 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Starting task 18.0 in stage 10.0 (TID 120, bdphdp070025, executor 2, partition 18, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,837 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 16.0 in stage 10.0 (TID 118) in 14 ms on bdphdp070025 (executor 2) (17/20) 2021-06-07 16:47:34,840 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Starting task 19.0 in stage 10.0 (TID 121, bdphdp070025, executor 2, partition 19, NODE_LOCAL, 8094 bytes) 2021-06-07 16:47:34,841 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 17.0 in stage 10.0 (TID 119) in 15 ms on bdphdp070025 (executor 2) (18/20) 2021-06-07 16:47:34,849 INFO (task-result-getter-0) [Logging.scala:logInfo(54)] - Finished task 18.0 in stage 10.0 (TID 120) in 13 ms on bdphdp070025 (executor 2) (19/20) 2021-06-07 16:47:34,853 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Finished task 19.0 in stage 10.0 (TID 121) in 13 ms on bdphdp070025 (executor 2) (20/20) 2021-06-07 16:47:34,853 INFO (task-result-getter-1) [Logging.scala:logInfo(54)] - Removed TaskSet 10.0, whose tasks have all completed, from pool 2021-06-07 16:47:34,854 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - ShuffleMapStage 10 (save at package.scala:298) finished in 0.220 s 2021-06-07 16:47:34,854 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - looking for newly runnable stages 2021-06-07 16:47:34,854 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - running: Set() 2021-06-07 16:47:34,854 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - waiting: Set() 2021-06-07 16:47:34,854 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - failed: Set() 2021-06-07 16:47:34,854 INFO (main) [Logging.scala:logInfo(54)] - advisoryTargetPostShuffleInputSize: 67108864, targetPostShuffleInputSize 67108864. 2021-06-07 16:47:34,867 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Registering RDD 53 (save at package.scala:298) 2021-06-07 16:47:34,867 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Got map stage job 6 (save at package.scala:298) with 1 output partitions 2021-06-07 16:47:34,867 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Final stage: ShuffleMapStage 13 (save at package.scala:298) 2021-06-07 16:47:34,867 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Parents of final stage: List(ShuffleMapStage 12, ShuffleMapStage 11) 2021-06-07 16:47:34,867 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Missing parents: List() 2021-06-07 16:47:34,867 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting ShuffleMapStage 13 (MapPartitionsRDD[53] at save at package.scala:298), which has no missing parents 2021-06-07 16:47:34,870 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_10 stored as values in memory (estimated size 36.4 KB, free 361.4 MB) 2021-06-07 16:47:34,874 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_10_piece0 stored as bytes in memory (estimated size 15.5 KB, free 361.4 MB) 2021-06-07 16:47:34,875 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Added broadcast_10_piece0 in memory on bdphdp07jobs14:16197 (size: 15.5 KB, free: 366.0 MB) 2021-06-07 16:47:34,875 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Created broadcast 10 from broadcast at DAGScheduler.scala:1161 2021-06-07 16:47:34,875 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting 1 missing tasks from ShuffleMapStage 13 (MapPartitionsRDD[53] at save at package.scala:298) (first 15 tasks are for partitions Vector(0)) 2021-06-07 16:47:34,876 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Adding task set 13.0 with 1 tasks 2021-06-07 16:47:34,876 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Starting task 0.0 in stage 13.0 (TID 122, bdphdp070016, executor 1, partition 0, NODE_LOCAL, 8097 bytes) 2021-06-07 16:47:34,896 INFO (dispatcher-event-loop-5) [Logging.scala:logInfo(54)] - Added broadcast_10_piece0 in memory on bdphdp070016:35542 (size: 15.5 KB, free: 1048.6 MB) 2021-06-07 16:47:34,905 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - Asked to send map output locations for shuffle 3 to 10.107.97.188:62930 2021-06-07 16:47:34,938 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Asked to send map output locations for shuffle 4 to 10.107.97.188:62930 2021-06-07 16:47:35,213 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Finished task 0.0 in stage 13.0 (TID 122) in 337 ms on bdphdp070016 (executor 1) (1/1) 2021-06-07 16:47:35,214 INFO (task-result-getter-2) [Logging.scala:logInfo(54)] - Removed TaskSet 13.0, whose tasks have all completed, from pool 2021-06-07 16:47:35,214 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - ShuffleMapStage 13 (save at package.scala:298) finished in 0.346 s 2021-06-07 16:47:35,214 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - looking for newly runnable stages 2021-06-07 16:47:35,214 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - running: Set() 2021-06-07 16:47:35,214 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - waiting: Set() 2021-06-07 16:47:35,214 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - failed: Set() 2021-06-07 16:47:35,215 INFO (main) [Logging.scala:logInfo(54)] - advisoryTargetPostShuffleInputSize: 67108864, targetPostShuffleInputSize 67108864. 2021-06-07 16:47:35,241 INFO (main) [Logging.scala:logInfo(54)] - Start processing data source writer: com.vesoft.nebula.connector.writer.NebulaDataSourceEdgeWriter@60666985. The input RDD has 1 partitions. 2021-06-07 16:47:35,244 INFO (main) [Logging.scala:logInfo(54)] - Starting job: save at package.scala:298 2021-06-07 16:47:35,245 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Got job 7 (save at package.scala:298) with 1 output partitions 2021-06-07 16:47:35,245 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Final stage: ResultStage 17 (save at package.scala:298) 2021-06-07 16:47:35,246 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Parents of final stage: List(ShuffleMapStage 16) 2021-06-07 16:47:35,248 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Missing parents: List() 2021-06-07 16:47:35,248 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting ResultStage 17 (MapPartitionsRDD[61] at save at package.scala:298), which has no missing parents 2021-06-07 16:47:35,263 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_11 stored as values in memory (estimated size 46.4 KB, free 361.3 MB) 2021-06-07 16:47:35,268 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Block broadcast_11_piece0 stored as bytes in memory (estimated size 20.8 KB, free 361.3 MB) 2021-06-07 16:47:35,269 INFO (dispatcher-event-loop-4) [Logging.scala:logInfo(54)] - Added broadcast_11_piece0 in memory on bdphdp07jobs14:16197 (size: 20.8 KB, free: 366.0 MB) 2021-06-07 16:47:35,269 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Created broadcast 11 from broadcast at DAGScheduler.scala:1161 2021-06-07 16:47:35,270 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Submitting 1 missing tasks from ResultStage 17 (MapPartitionsRDD[61] at save at package.scala:298) (first 15 tasks are for partitions Vector(0)) 2021-06-07 16:47:35,270 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - Adding task set 17.0 with 1 tasks 2021-06-07 16:47:35,270 INFO (dispatcher-event-loop-1) [Logging.scala:logInfo(54)] - Starting task 0.0 in stage 17.0 (TID 123, bdphdp070025, executor 2, partition 0, PROCESS_LOCAL, 7780 bytes) 2021-06-07 16:47:35,295 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Added broadcast_11_piece0 in memory on bdphdp070025:18606 (size: 20.8 KB, free: 1048.5 MB) 2021-06-07 16:47:35,441 INFO (dispatcher-event-loop-7) [Logging.scala:logInfo(54)] - Asked to send map output locations for shuffle 5 to 10.107.119.198:36674 2021-06-07 16:47:35,720 INFO (dispatcher-event-loop-0) [Logging.scala:logInfo(54)] - Added rdd_58_0 in memory on bdphdp070025:18606 (size: 360.0 B, free: 1048.5 MB) 2021-06-07 16:47:35,962 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Finished task 0.0 in stage 17.0 (TID 123) in 692 ms on bdphdp070025 (executor 2) (1/1) 2021-06-07 16:47:35,963 INFO (task-result-getter-3) [Logging.scala:logInfo(54)] - Removed TaskSet 17.0, whose tasks have all completed, from pool 2021-06-07 16:47:35,963 INFO (dag-scheduler-event-loop) [Logging.scala:logInfo(54)] - ResultStage 17 (save at package.scala:298) finished in 0.714 s 2021-06-07 16:47:35,963 INFO (main) [Logging.scala:logInfo(54)] - Job 7 finished: save at package.scala:298, took 0.719260 s 2021-06-07 16:47:35,965 INFO (main) [Logging.scala:logInfo(54)] - Data source writer com.vesoft.nebula.connector.writer.NebulaDataSourceEdgeWriter@60666985 is committing. 2021-06-07 16:47:35,966 INFO (main) [NebulaSourceWriter.scala:apply(93)] - failed execs: List() 2021-06-07 16:47:35,967 INFO (main) [Logging.scala:logInfo(54)] - Data source writer com.vesoft.nebula.connector.writer.NebulaDataSourceEdgeWriter@60666985 committed. 2021-06-07 16:47:35,972 INFO (main) [Logging.scala:logInfo(54)] - Stopped Spark web UI at http://bdphdp07jobs14:4040 2021-06-07 16:47:35,975 INFO (YARN application state monitor) [Logging.scala:logInfo(54)] - Interrupting monitor thread 2021-06-07 16:47:35,995 INFO (main) [Logging.scala:logInfo(54)] - Shutting down all executors 2021-06-07 16:47:35,995 INFO (dispatcher-event-loop-6) [Logging.scala:logInfo(54)] - Asking each executor to shut down 2021-06-07 16:47:35,998 INFO (main) [Logging.scala:logInfo(54)] - Stopping SchedulerExtensionServices (serviceOption=None, services=List(), started=false) 2021-06-07 16:47:35,999 INFO (main) [Logging.scala:logInfo(54)] - Stopped 2021-06-07 16:47:36,041 INFO (dispatcher-event-loop-3) [Logging.scala:logInfo(54)] - MapOutputTrackerMasterEndpoint stopped! 2021-06-07 16:47:36,051 INFO (main) [Logging.scala:logInfo(54)] - MemoryStore cleared 2021-06-07 16:47:36,051 INFO (main) [Logging.scala:logInfo(54)] - BlockManager stopped 2021-06-07 16:47:36,052 INFO (main) [Logging.scala:logInfo(54)] - BlockManagerMaster stopped 2021-06-07 16:47:36,056 INFO (dispatcher-event-loop-2) [Logging.scala:logInfo(54)] - OutputCommitCoordinator stopped! 2021-06-07 16:47:36,063 INFO (main) [Logging.scala:logInfo(54)] - Successfully stopped SparkContext 2021-06-07 16:47:36,064 INFO (Thread-1) [Logging.scala:logInfo(54)] - Shutdown hook called 2021-06-07 16:47:36,065 INFO (Thread-1) [Logging.scala:logInfo(54)] - Deleting directory /tmp/spark-3438c208-94fa-47ad-bd6c-af9cc0673b5a 2021-06-07 16:47:36,067 INFO (Thread-1) [Logging.scala:logInfo(54)] - Deleting directory /appcom/logs/spark/tmp/spark-6f7f1690-5162-48e6-a862-359e0ab0c653 ############Execute Success!!!########